Model Validation

The ECB’s latest update on TRIM outcomes shows that weaknesses in model validation generated the lion’s share of its findings in the area of counterparty credit risk. Ironing out these shortcomings and following best practices will be crucial to banks’ ability to manage model risks, and to avoiding any adverse impact on their decision-making processes.

The recent Targeted Review of Internal Models (TRIM) findings unveiled by the European Central Bank (ECB) related to model validation, together with the launch of the ECB annual validation reporting requirement, are two events that indicate growing supervisory focus on the model validation framework deployed by banks.

Model validation involves the processes and activities that verify models are performing as intended, and is a core element of model risk management (MRM). For instance, the Basel Committee’s minimum standards for internal ratings-based (IRB) institutions require a regular cycle of model validation “that includes monitoring of model performance and stability; review of model relationships; and testing of model outputs against outcomes.”¹

Over the last few years supervisory expectations for model validation have evolved rapidly. November 2018 and July 2019 saw the ECB revised its Guide to Internal Models, which sets out a range of model validation requirements in both its General Topics and Risk-Specific chapters. Most recently, the ECB’s third update on TRIM outcomes published in November 2019 summarises the findings of on-site inspections (OSIs) focused on counterparty credit risk (CCR). Model validation not only generated more findings than any other CCR topic, but also the largest number of severe findings. In particular, the TRIM review “identified weaknesses in the scope and depth of the validation tasks and also shortcomings relating to back-testing due to inappropriate coverage, missing horizons, missing levels or risk measures (e.g. the exposure metric) or a lack of follow-up action.”²

With these findings in mind it may be worthwhile for banks to remind themselves of the ECB’s revisions to the CCR chapter of its Guide which, along with other topics, focused on model validation. Some of the key enhancements were:

• Clarification that model development and model validation should be conducted by independent functions (paragraph 59 of the CCR chapter);
• Confirmation that back-testing at both the actual and the hypothetical portfolio levels represents good practice (para 62); and
• Permission for model validation functions to calculate their own back-testing coverage ratios, but with the requirement that these should be reported and adequately justified (para 64).

Banks should also remember that these enhancements cover only a handful of the Guide’s full range of requirements. In addition to the above stated enhanced expectations, other important best practices include:

• Creating a comprehensive view of all the findings, problems and weaknesses identified during model validation (para 59);
• Including various types of analysis when validating key modelling assumptions (para 60);
• Performing and reporting on back-testing at least once a year (para 61);
• Including back-testing at single transaction level in the validation framework (para 63);
• Adapting statistical back-testing when samples contain forecasts with overlapping time periods (para 65); and
• Amending actual portfolio back-testing if portfolio composition has changed during the observation period (para 67).

Given this background and the increasing reliance of banks on models for their decision-making, it is becoming critical for banks to establish a robust model validation framework that protects them from the consequences of misleading model outputs. In light of the ECB’s revised Guide to internal models and its TRIM CCR findings, we believe that banks should reinforce the following model validation measures, if they have not done so already:

• Robust validation framework: A strong robust MRM framework should set out key policies, processes, roles and responsibilities covering model validation activities.
• Comprehensive group-wide validation policy: Group-wide policies should reflect the necessary intricacies and exceptions arising from different activities and subsidiaries.
• Robust back testing framework: It is critical to establish robust back testing framework in order to determine model’s ability to capture risk exposure. For instance, as indicated above CCR models characterize by their complexity require extra efforts in defining a sound and comprehensive back testing framework.
• Independent model validation: Those performing model validation must be able to carry out effective, unbiased assessments (i.e. demonstrate independence of model validation activities from model development ones). The ECB Guide also sets out expectations for third party model validation.
• Frequent model validation: As mentioned, back-testing of models’ expected and actual outputs should take place at least once a year. However, the frequency of model validation may vary depending on the nature and complexity of the model.
• Initial and periodic validation: Model validation should not only occur before a model’s deployment, but also periodically during its use and whenever it is altered or amended.
• Reporting of validation outcomes: For model validation to be effective, outcomes and severity assessments must be reported to appropriate internal bodies - and be acted upon.
• Following-up of validation issues: Model weaknesses identified during validation should be appropriately documented and monitored to resolution, ensuring that follow-up actions are executed in a timely manner.

Furthermore, the ECB’s supervisory priorities for 2020 – which include upcoming activities in relation to the European Banking Authority’s IRB repair programme - indicate that supervisory focus on the adequacy of internal models will continue to gain traction this year. It is therefore essential for banks to implement robust MRM programmes, including establishing strong model validation frameworks that can adequately determine models’ fitness for purpose.

To sum up, model validation is not only the subject of growing supervisory scrutiny. It is also vital to mitigating the risks of models generating unwarranted results, which could have significant adverse impacts on banks’ decision-making.