Model validation: key steps to a faster, smoother process
Estimated reading time: 4 minutes
Model validation can be a stressful, bloated process - and may not be as value adding as it could be. Phrases like the following may be familiar to you:
- “This has been more stressful than last year - there’s no time to do all of this.”
- “It is becoming too much of a regulatory tick-box exercise.”
- “The capital team are under the cosh – we have business plans being updated. We can’t do validation tests too – especially those sensitivity tests!”
- “What’s the value in this?”
Is there another way?
Unfortunately, you are not alone if this rings true for you. However, there is another way.
Model validation is at its best when it starts with understanding how the model is used (and this goes much further than for required capital), and emphasises a feedback loop to the first line to add genuine value to the business.
In this blog we discuss some ways to optimise the validation process.
The first step in making a model useful is understanding who uses the model and how. Validation should highlight areas where the business could get more value out of the model. Understanding the model better through validation will help the business fundamentally, as the process highlights the differences between the model and the actual business.
Consider, for example, a line of business which is under consideration for doubling in exposure. Validation should explicitly call out the risk profile of that specific line of business and consider key metrics of performance (such as mean loss ratios) and the effects of doubling the book on capital and diversification.
Furthermore, the validation should check whether the methodology would still be appropriate, or if a rework is needed to cater for new terms of the proposed business.
Once you know the key use cases, consider the testing approach and how to improve it. It’s always optimal to go with a top-down validation approach. However, we know we will need to perform deep-dive testing over and above the core tests performed each year.
Focus deep-dives on the use case areas to add value to the company. For example, instead of asking for the same batches of sensitivities year-on-year, an optimised validation should be a more targeted investigation into areas of the model that materially affect use as well as the capital.
Visualisation lends itself perfectly to understanding models. It’s the cornerstone of a value adding validation process. These processes are so often hampered with extra work spent digging into strange results and trying to check what the model results are doing. Visualisation enables you to spend more of the scarcely available time on why things are behaving the way they are.
Additionally, visualisation is key to the communication of complex and technical work, whether to regulators, the Board or our peers. The Prudential Regulation Authority’s (PRA’s) ‘Dear Chief Actuary’ letter of 5 November 2019 talked about “Transparency over key judgements and assumptions in management information (MI).” It concluded that “MI may be difficult to follow” and that “boards would benefit from clearer presentation of the most material assumptions.”
We believe that visualisation, used appropriately, goes a long way in supporting work to ensure these requirements are fulfilled by insurers.
Validation is almost always considered the last item on the chain of events before finalisation of the capital. Typically, for example, any changes in the business plan would lead to running the model first with an update and then the validation tests are run. This process may take several turns and repeats, taking up valuable time.
The validation process ought to consider the automation steps throughout the journey the data have taken. For example, back-testing mean loss ratios could be integrated into the reserving process to ensure the tests are performed at source. Business plan miss-factors could be included in the planning process and automated.
Validation is resource intensive due to the time needed to first understand the model and then to evaluate it. There are certain crunch points to consider which will vary by firm. These include deadlines for model development, external deadlines and short validation deadlines.
Moving some validation off-cycle can mitigate these crunch points. Deep dives, for instance, are probably the most time-consuming parts of validation and can usually be off-cycle. The deep-dive may be on a specific model use (such as the example above of investigating the line of business) or more general such as dependencies.
As with any project there will always be pinch points. A good way of dealing with these is to get help from other departments, or outsource. This frees your team to work on other important work.
Outsourcing ensures independence of validators, who can spot potential issues with fresh eyes and distance from model development.
In addition, this may lead to better insight into your model, especially as externals will have seen other implementations. Outsourcing could also be a means of optimising costs as you may only run validation for a part of the year, instead of having a resource on board full time.
Model validation should not be a tick box exercise. It has all the right ingredients to be a value adding exercise that can help senior management understand what really drives the business from a risk and capital perspective.
The life of the validation actuary can also be made easier if we work smarter, use more visualisation techniques, automate what we can and work in the off-cycles. Sometimes, outsourcing validation can also provide extra value to your process making for better insights into what really drives your company.
Stay up to date
Follow the latest independent commentary and exclusive insights from a range of experts at the forefront of insurance, risk, pensions and investment – tailored to your preference.
Subscribe today