RISK MANAGEMENT—Interagency Supervisory Guidance on Counterparty Credit Risk Management

This guidance discusses critical aspects of effective management of counterparty credit risk (CCR), and sets forth sound practices and supervisory expectations for an effective CCR management framework. CCR is the risk that the counterparty to a transaction could default or deteriorate in creditworthiness before the final settlement of a transaction’s cash flows. Unlike the credit risk for a loan, when only the lending banking organization¹ faces the risk of loss, CCR creates a bilateral risk of loss because the market value of a transaction can be positive or negative to either counterparty. The future market value of the exposure and the counterparty’s credit quality are uncertain and may vary over time as underlying market factors change. The guidance is intended for use by banking organizations, especially those with large derivatives portfolios, in setting their risk management practices, as well as by supervisors as they assess and examine such institutions’ management of CCR. For other banking organizations without large derivatives portfolios, risk managers and supervisors should apply this guidance as appropriate, given the size, nature, and complexity of the CCR risk profile of the banking organization.

CCR is a multidimensional form of risk, affected by both the exposure to a counterparty and the credit quality of the counterparty, both of which are sensitive to market-induced changes. It is also affected by the interaction of these risks, for example the correlation² between an exposure and the credit spread of the counterparty, or the correlation of exposures among the banking organization’s counterparties. Constructing an effective CCR management framework requires a combination of risk management techniques from the credit, market, and operational risk disciplines.

CCR management techniques have evolved rapidly over the last decade, along with increased complexity of derivative instruments under management. Banking organizations substantially improved their risk management practices during this time; however, in some cases, implementation of sound practices has been uneven across business lines and counterparty types. Further, the financial crisis of 2007-2009 revealed weaknesses in CCR management at many banking organizations, such as shortcomings in the timeliness and accuracy of exposure aggregation capabilities and inadequate measurement of correlation risks. The crisis also highlighted deficiencies in the ability of banking organizations to monitor and manage counterparty exposure limits and concentration risks, ranging from poor selection of CCR metrics to inadequate system infrastructure.

To address these weaknesses, this guidance reinforces sound governance of CCR management practices, through prudent board and senior management oversight, management reporting, and risk management functions. The guidance discusses relevant topics in risk measurement, including metrics, exposure aggregation and concentration management, stress testing, and associated characteristics of adequate systems infrastructure. It also covers risk control functions, such as counterparty limits, margin practices, validating and backtesting models and systems, managing close-outs,³ managing central counterparty exposures, and controlling legal and operational risks arising from derivatives activities.

CCR management guidelines and supervisory expectations are delineated in various individual and interagency policy statements and guidance,⁴ which remain relevant and applicable. This guidance offers further explanation and clarification, particularly in light of developments in CCR management. However, this guidance is not all-inclusive and banking organizations should reference sound practices for CCR management, such as those advanced by industry, policymaking, and supervisory forums.⁵

The board of directors or a designated board-level committee (board) should clearly articulate the banking organization’s risk tolerance for CCR, by approving relevant policies, including a framework for establishing limits on individual counterparty exposures and concentrations of exposures. Senior management should establish and implement a comprehensive risk measurement and management framework consistent with this risk tolerance that provides for the ongoing monitoring, reporting, and control of CCR exposures.

Senior management should adhere to the board’s established risk tolerance and establish policies and risk management guidelines appropriately.⁶ At a minimum, policies should outline CCR management standards that are in conformance with this guidance. More specifically, they should address the subjects discussed in this document, such as risk measurement and reporting, risk management tools, and processes to manage legal and operational risk. Policies should be detailed and contain a clear escalation process for review and approval of policy exceptions, especially those pertaining to transaction terms and limits.

Banking organizations should report counter party exposures to the board and senior management at a frequency commensurate with the materiality of exposures and the complexity of transactions. Reporting should include concentration analysis and CCR stress testing results, to allow for an understanding of exposures and potential losses under severe market conditions. Reports should also include an explanation of any measurement weaknesses or limitations that may influence the accuracy and reliability of the CCR risk measures.

Senior management should have access to timely, accurate, and comprehensive CCR reporting metrics, including an assessment of significant issues related to the risk management aspects discussed in this guidance. They should review CCR reports at least monthly, with data that are no more than three weeks old. It is general practice for institutions to report:

A banking organization’s board and senior management should clearly delineate the respective roles of business lines versus risk management, both in terms of initiating transactions that have CCR, and of ongoing CCR management. The board and senior management should ensure that the risk management functions have adequate resources, are fully independent from CCR related trading operations (in both activity and reporting), and have sufficient authority to enforce policies and to escalate issues to senior management and the board (independent of the business line).

The board should direct internal audit to regularly assess the adequacy of the CCR management framework as part of the regular audit plan. Such assessments should include credit line approval processes, credit ratings, and credit monitoring. Such an assessment should opine on the adequacy of the CCR infrastructure and processes, drawing where appropriate from individual business line reviews or other internal and external audit work. Please see the relevant section of this guidance regarding the role of CCR model validation or review. The board should review annual reports from internal audit and model validation or review, assessing the findings and confirming that management has taken appropriate corrective actions.

Given the complexity of CCR exposures (particularly regarding OTC derivatives), banking organizations should employ a range of risk measurement metrics to promote a comprehensive understanding of CCR and how it changes in varying environments. Metrics should be commensurate with the size, complexity, liquidity, and risk profile of the CCR portfolio. Banking organizations typically rely on certain metrics as a primary means of monitoring, with secondary metrics used to create a more robust view of CCR exposures. Banking organizations should apply these metrics to single counterparty exposures, groups of counterparties (for example, by internal rating, industry, geographical region), and the consolidated CCR portfolio. Banking organizations should assess their largest exposures, for instance their top 20 exposures, using each primary metric.

Major dealers and large, sophisticated banking organizations with substantial CCR exposure should measure and assess:

Refer to Appendix A for definitions of basic metrics and descriptions of their purposes.

Banking organizations should have the capacity to measure their exposure at various levels of aggregation (for example, by business line, legal entity, or consolidated by industry). Systems should be sufficiently flexible to allow for timely aggregation of all CCR exposures (that is, OTC derivatives, securities financing transactions (SFTs), and other pre-settlement exposures), as well as aggregation of other forms of credit risk to the same counterparty (for example, loans, bonds, and other credit risks). The following are sound CCR aggregation principles:

Concentrated exposures are a significant concern, as CCR can contribute to sudden increases in credit exposure, which in turn can result in unexpectedly large losses in the event of counterparty default. Accordingly, banking organizations should have enterprise-wide processes to effectively identify, measure, monitor, and control concentrated exposures on both a legal entity and enterprise-wide basis.

Concentrations should be identified using both quantitative and qualitative means. An exposure or group of related exposures (for example, firms in the same industry), should be considered a concentration in the following circumstances: exposures (individually or collectively) exceed risk tolerance levels established to ensure appropriate diversification; deterioration of the exposure could result in material loss; or deterioration could result in circumstances that are detrimental to the banking organization’s reputation. All credit exposures should be considered as part of concentration management, including loans, OTC derivatives, names in bespoke and index CDO credit tranches, securities settlements, and money market transactions such as fed funds sold. Total credit exposures should include the size of settlement and clearing lines, or other committed lines.

CCR concentration management should identify, quantify, and monitor:

Banking organizations with significant CCR exposures should maintain a comprehensive stress testing framework, which is integrated into the banking organization’s CCR management. The framework should inform the banking organization’s day-to-day exposure and concentration management, and it should identify extreme market conditions that could excessively strain the financial resources of the banking organization. Regularly, but no less than quarterly, senior management should evaluate stress test results for evidence of potentially excessive risk, and take risk reduction strategies as appropriate.

The severity of factor shocks should be consistent with the purpose of the stress test. When evaluating solvency under stress, factor shocks should be severe enough to capture historical extreme market environments and/or extreme but plausible stressed market conditions. The impact of such shocks on capital resources and earnings should be evaluated. For day-to-day portfolio monitoring, hedging, and management of concentrations, banking organizations should also consider scenarios of lesser severity and higher probability. When conducting stress testing, risk managers should challenge the strength of assumptions made about the legal enforceability of netting and the ability to collect and liquidate collateral.

A sound stress-testing framework should include:

CVA refers to adjustments to transaction valuation to reflect the counterparty’s credit quality. CVA is the fair value adjustment to reflect CCR in valuation of derivatives. As such, CVA is the market value of CCR and provides a market-based framework for understanding and valuing the counterparty credit risk embedded in derivative contracts. CVA may include only the adjustment to reflect the counterparty’s credit quality (a one-sided CVA or just CVA), or it may include an adjustment to reflect the banking organization’s own credit quality. The latter is a two-sided CVA, or CVA plus a debt valuation adjustment (DVA). For the evaluation of the credit risk due to probability of default of counterparties, a one sided CVA is typically used. For the evaluation of the value of derivatives transactions with a counterparty or the market risk of derivatives transactions, a two-sided CVA should be used.

Although CVA is not a new concept, its importance has grown over the last few years, partly because of a change in accounting rules that requires banking organizations to recognize the earnings impact of changes in CVA.¹² During the 2007-2009 financial crisis, a large portion of CCR losses were because of CVA losses rather than actual counterparty defaults.¹³ As such, CVA has become more important in risk management, as a mechanism to value, manage, and make appropriate hedging decisions, to mitigate banking organizations’ exposure to the mark-to-market (MTM) impact of CCR.¹⁴

The following are general standards for CVA measurement and use of CVA for risk management purposes:

CVA management should be consistent with sound risk management practices for other material mark-to-market risks. These practices should include the following:

Banking organizations with material CVA should measure the risk of associated loss on an ongoing basis. In addition to stress tests of the CVA, banking organizations may develop VaR models that include CVA to measure potential losses. While these models are currently in the early stages of development, they may prove to be effective tools for risk management purposes. An advantage of CVA VaR over more traditional CCR risk measures is that it captures the variability of the CCR exposure, the variability of the counterparty’s credit spread, and the dependency between them.

Developing VaR models for CVA is significantly more complicated than developing VaR models for a banking organization’s market risk positions. In developing a CVA VaR model, a banking organization should match the percentile and time horizon for the VaR model to those appropriate for the management of this risk, and include all significant risks associated with changes in the CVA. For example, banking organizations may use the same percentile for CVA VaR as they use for market risk VaR (for example, the 95th or 99th percentile). However, the time horizon for CVA VaR may need to be longer than for market risk (for example, one quarter or one year) because of the potentially illiquid nature of CVA. The following are important considerations in developing a CVA VaR model:

Wrong-way risk occurs when the exposure to a particular counterparty is positively correlated with the probability of default of the counterparty itself. Specific wrong-way risk arises when the exposure to a particular counterparty is positively correlated with the probability of default of the counterparty itself because of the nature of the transactions with the counterparty. General wrong-way risk arises when the probability of default of counterparties is positively correlated with general market risk factors. Wrong-way risk is an important aspect of CCR that has caused major losses at banking organizations.

Accordingly, a banking organization should have a process to systematically identify, quantify, and control both specific and general wrong-way risk across its OTC derivative and SFT portfolios.¹⁶ To prudently manage wrong-way risk, banking organizations should:

Banking organizations should ensure that systems infrastructure keeps up with changes in the size and complexity of their CCR exposures, and the OTC derivatives market in general. Systems should capture and measure the risk of transactions that may be subject to CCR as a fundamental part of the CCR management framework.

Banking organizations should have strong operational processes across all derivatives markets, consistent with supervisory and industry recommendations.¹⁸ Management should strive for a single comprehensive CCR exposure measurement platform.¹⁹ If not currently possible, banking organizations should minimize the number of system platforms and methodologies, as well as manual adjustments to exposure calculations. When using multiple exposure measurement systems, management should ensure that transactions whose future values are measured by different systems are aggregated conservatively.

To maintain a systems infrastructure that supports adequate CCR management, banking organizations should:

For large derivatives market participants, certain trades may be difficult to capture in exposure measurement systems, and are therefore modeled outside of the main measurement system(s). The resulting exposures, commonly referred to as add-ons, are then added to the portfolio potential exposure measure. In limited cases, the use of conservative add-on methodologies may be suitable, if the central system cannot reflect the risk of complex financial products. However, overreliance on add-on methodologies may distort exposure measures. To mitigate measurement distortions, banking organizations should:

Meaningful limits on exposures are an integral part of a CCR management framework, and these limits should be formalized in CCR policies and procedures. For limits to be effective, a banking organization should incorporate these limits into an exposure monitoring system independent of relevant business lines. It should perform ongoing monitoring of exposures against such limits, to ascertain conformance with these limits, and have adequate risk controls that require action to mitigate limit exceptions. Review of exceptions should include escalation to a managerial level that is commensurate with the size of the excess or nature of mitigation required. A sound limit system should include:

Collateral is a fundamental CCR mitigant. Indeed, significant stress events have highlighted the importance of sound margining practices. With this in mind, banking organizations should ensure that they have adequate margin and collateral “haircut”²² guidelines for all products with CCR.²³

Accordingly, banking organizations should:

A banking organization should validate its CCR models initially and on an ongoing basis. Validation of models should include: an evaluation of the conceptual soundness and developmental evidence supporting a given model; an ongoing monitoring process that includes verification of processes and benchmarking; and an outcomes-analysis process that includes backtesting. Validation should identify key assumptions and potential limitations, and it should assess their possible impact on risk metrics. All components of models should be subject to validation along with their combination in the CCR system.

Evaluating the conceptual soundness involves assessing the quality of the design and construction of the CCR models and systems, including documentation and empirical evidence that supports the theory, data, and methods used.

Ongoing monitoring confirms that CCR systems continue to perform as intended. This generally involves process verification, an assessment of model data integrity and systems operation, and benchmarking to assess the quality of a given model. Benchmarking is a valuable diagnostic tool in identifying potential weaknesses. Specifically, it is the comparison of a banking organization’s CCR model estimates with those derived using alternative data, methods, or techniques. Benchmarking can also be applied to particular CCR model components, such as parameter estimation methods or pricing models. Management should investigate the source of any differences in output, and determine whether benchmarking gaps indicate weakness in the banking organization’s models.

Outcomes analysis compares model outputs to actual results during a sample period not used in model development. This is generally accomplished using backtesting. It should be applied to components of CCR models (for example the risk factor distribution and pricing model), the risk measures, and projected exposures. While there are limitations to backtesting, especially for testing the longer time horizon predictions of a given CCR model, it is an essential component of model validation. Banking organizations should have a process for the resolution of observed model deficiencies detected by backtesting. This should include further investigation to determine the problem, and appropriate course of action, including changing a given CCR model.

If the validation of CCR models and infrastructure systems is not performed by staff that is independent from the developers of the models, then an independent review should be conducted by technically competent personnel to ensure the adequacy and effectiveness of the validation. The scope of the independent review should include: validation procedures for all components, the role of relevant parties, and documentation of the model and validation processes. This review should document its results, what action was taken to resolve findings, and its relative timeliness.

Senior management should be notified of validation and review results and should take appropriate and timely corrective actions to address deficiencies. The board should be apprised of summary results, especially unresolved deficiencies. In support of validation activities, internal audit should review and test models and systems validation, and overall systems infrastructure as part of their regular audit cycle.

For more details on validation, please see Appendix B.

Banking organizations should have the ability to effectively manage counterparties in distress, including execution of a close-out. Policies and procedures outlining sound practices for managing a close-out should include:

A central credit counterparty (CCP) facilitates trades between counterparties in one or more financial markets by either guaranteeing trades or novating contracts, and typically requires all participants to be fully collateralized on a daily basis. The CCP thus effectively bears most of the counterparty credit risk in transactions, becoming the buyer for every seller and the seller to every buyer. Well-regulated and soundly-managed CCPs can be an important means of reducing bilateral counterparty exposure in the OTC derivatives market. However, CCPs also concentrate risk within a single entity. Therefore, it is important that banking organizations centrally clear through regulated CCPs with sound risk management processes, and strong financial resources sufficient to meet their obligations under extreme stress conditions.

To manage CCP exposures, banking organizations should regularly, but no less frequently than annually, review the individual CCPs to which they have exposures. This review should include performing and documenting due diligence on each CCP, applying current supervisory or industry standards²⁵ (and any subsequent standards) as a baseline to assess the CCP’s risk management practices.

Banking organizations should ensure proper control of, and access to, legal documentation and agreements. In addition, it is important that systems used to measure CCR incorporate accurate legal terms and provisions. The accessibility and accuracy of legal terms is particularly critical in close-outs, when there is limited time to review the collateral and netting agreements. Accordingly, banking organizations should:

While a counterparty’s ability to pay should be evaluated when assessing credit risk, credit losses can also occur when a counterparty is unwilling to pay, which most commonly occurs when a counterparty questions the appropriateness of a contract. These types of disputes pose not only risk of a direct credit loss, but also risk of litigation costs and/or reputational damage. Banking organizations should maintain policies and procedures to assess client and deal appropriateness. In addition, banking organizations should:

For relevant banking organizations, CCR management should be an integral component of the risk management framework. When considering the applicability of specific guidelines and best practices set forth in this guidance, a banking organization’s senior management and supervisors should consider the size and complexity of its securities and trading activities. Banking organizations should comprehensively evaluate existing practices against the standards in this guidance and implement remedial action as appropriate. A banking organization’s CCR exposure levels and the effectiveness of its CCR management are important factors for a supervisor to consider when evaluating a banking organization’s overall management, risk management, and credit and market risk profile.

This glossary describes commonly used CCR metrics. As discussed above, banking organizations should employ a suite of metrics commensurate with the size, complexity, liquidity, and risk profile of the organization’s CCR portfolio. Major broker-dealer banking organizations should employ the full range of risk measurement metrics to enable a comprehensive understanding of CCR and how it changes in varying environments. Banking organizations of lesser size and complexity should carefully consider which of these metrics they need to track as part of their exposure risk management processes. At a minimum, all banking organizations should calculate current exposure and stress test their CCR exposures. Definitions marked with an asterisk are from the Bank for International Settlements.

Definition: Current exposure is the larger of zero, or the market value of a transaction or a portfolio of transactions within a netting set with a counterparty that would be lost upon the default of the counterparty, assuming no recovery on the value of those transactions in bankruptcy. Current exposure is often also called replacement cost. Current exposure may be reported gross or net of collateral.

Purpose: Allows banking organizations to assess their CCR exposure at any given time, that is, the amount currently at risk.

Definition: JTD exposure is the change in the value of counterparty transactions upon the default of a reference name in CDS positions.

Purpose: Allows banking organizations to assess the risk of a sudden, unanticipated default before the market can adjust.

Definition: Expected exposure is calculated as average exposure to a counterparty at a date in the future.

Purpose: This is often an intermediate calculation for expected positive exposure or CVA. It can also be used as a measure of exposure at a common time in the future.

Definition: EPE is the weighted average over time of expected exposures when the weights are the proportion that an individual expected exposure represents of the entire time interval.*

Purpose: Expected positive exposure is an appropriate measure of CCR exposure when measured in a portfolio credit risk model.

Definition: Peak exposure is a high percentile (typically 95 percent or 99 percent) of the distribution of exposures at any particular future date before the maturity date of the longest transaction in the netting set. A peak exposure value is typically generated for many future dates up until the longest maturity date of transactions in the netting set.*

Purpose: Allows banking organizations to estimate their maximum potential exposure at a specified future date, or over a given time horizon, with a high level of confidence. For collateralized counterparties, this metric should be based on a realistic close-out period, considering both the size and liquidity of the portfolio. Banking organizations should consider peak potential exposure when setting counterparty credit limits.

Definition: Expected shortfall exposure is similar to peak exposure, but is the expected exposure conditional on the exposure being greater than some specified peak percentile.

Purpose: For transactions with very low probability of high exposure, the expected shortfall accounts for large losses that may be associated with transactions with high tail risk.

Definition: Sensitivity to market risk factors, is the change in exposure because of a given market risk factor change (for example, DV01).

Purpose: Provides information on the key drivers of exposure to specific counterparties and on hedging.

Definition: Stressed exposure is a forward-looking measure of exposure based on pre-defined market factor movements (non-statistically generated). These can include single-factor market shocks, historical scenarios, and hypothetical scenarios.

Purpose: Allows banking organizations to consider their counterparty exposure under a severe or stressed scenario. This serves as a supplemental view of potential exposure, and provides banking organizations with additional information on risk drivers. It is best practice to compare stressed exposure to counterparty credit limits.

Definition: The credit valuation adjustment is an adjustment to the mid-market valuation (average of the bid and asked price) of the portfolio of trades with a counterparty. This adjustment reflects the market value of the credit risk resulting from any failure to perform on contractual agreements with a counterparty. This adjustment may reflect the market value of the credit risk of the counterparty or the market value of the credit risk of both the banking organization and the counterparty.*

Purpose: CVA is a measure of the market value of CCR, incorporating both counterparty creditworthiness and the variability of exposure.

Definition: CVA VaR is a measure of the variability of the CVA mark-to-market value and is based on the projected distributions of both exposures and counterparty creditworthiness.

Purpose: Provides banking organizations with an estimate of the potential CVA mark-to-market loss, at a certain confidence interval and over a given time horizon.

Definition: CVA factor sensitivities is the mark-to-market change in CVA resulting from a given market risk factor change (for example, CR01).

Purpose: Allows banking organizations to assess and hedge the market value of the credit or market risks to single names and portfolios and permits banking organizations to monitor excessive buildups in counterparty concentrations.

Definition: Stressed CVA is a forward-looking measure of CVA mark-to-market value based on predefined credit or market factor movements (non-statistically generated). These can include single market factor shocks, historical scenarios, and hypothetical scenarios.

Purpose: Serves as an informational tool, and allows banking organizations to assess the sensitivity of their CVA to a potential mark-to-market loss under defined scenarios.

A banking organization should validate its CCR models, initially and on an ongoing basis. Validation should include three components: an evaluation of the conceptual soundness of relevant models (including developmental evidence); an ongoing monitoring process that includes verification of processes and benchmarking; and an outcomes-analysis process that includes backtesting. The validation should either be independent, or subject to independent review.

Validation is the set of activities designed to give the greatest possible assurances of CCR models’ accuracy and systems’ integrity. Validation should also identify key assumptions and potential limitations, and assess their possible impact on risk metrics. CCR models have several components:

All components of each model should be subject to validation, along with analysis of their interaction in the CCR system. Validation should be performed initially as a model first goes into production. Ongoing validation is a means of addressing situations where models have known weaknesses and ensuring that changes in markets, products, or counterparties do not create new weaknesses. Senior management should be notified of the validation results and should take corrective actions in a timely manner when appropriate.

A banking organization’s validation process should be independent of the CCR model and systems development, implementation, and operation. Alternately, the validation should be subject to independent review, whereby the individuals who perform the review are not biased in their assessment because of involvement in the development, implementation, or operation of the processes or products. Individuals performing the reviews should possess the requisite technical skills and expertise to provide critical analysis, effective challenge, and appropriate recommendations. The extent of such reviews should be fully documented, sufficiently thorough to cover all significant model elements, and include additional testing of models or systems as appropriate. In addition, reviewers should have the authority to effectively challenge developers and model users, elevate concerns or findings as necessary, and either have issues addressed in a prompt and substantial manner or reject a model for use by the banking organization.

The first component of validation is evaluating conceptual soundness, which involves assessing the quality of the design and construction of CCR models. The evaluation of conceptual soundness includes documentation and empirical evidence supporting the theory, data, and methods used. The documentation should also identify key assumptions and potential limitations and assess their possible impact. A comparison to industry practice should be done to identify areas where substantial and warranted improvements can be made. All model components are subject to evaluation, including simplifying assumptions, parameter calibrations, risk-factor diffusion processes, pricing models, and risk metrics. Developmental evidence should be reviewed whenever the banking organization makes material changes in CCR models. Evaluating conceptual soundness includes independent evaluation of whether a model is appropriate for its purpose, and whether all underlying assumptions, limitations, and shortcomings have been identified and their potential impact assessed.

The second component of model validation is ongoing monitoring to confirm that the models were implemented appropriately and continue to perform as intended. This involves process verification, an assessment of models, and benchmarking to assess the quality of the model. Deficiencies uncovered through these activities should be remediated promptly.

Process verification includes evaluating data integrity and operational performance of the systems supporting CCR measurement and reporting. This should be performed on an ongoing basis and includes:

“Benchmarking” means comparing a banking organization’s CCR measures with those derived using alternative data, methods, or techniques. It can also be applied to particular model components, such as parameter estimation methods or pricing models. It is an important complement to backtesting and is a valuable diagnostic tool in identifying potential weaknesses. Differences between the model and the benchmark do not necessarily indicate that the model is in error because the benchmark itself is an alternative prediction. It is important that a banking organization use appropriate benchmarks, or the exercise will be compromised. As part of the benchmarking exercise, the banking organization should investigate the source of the differences and whether the extent of the differences is appropriate.

The third component of validation is outcomes analysis, which is the comparison of model outputs to actual results during a sample period not used in model development. Backtesting is one form of out-of-sample testing. Backtesting should be applied to components of a CCR model, for example the risk factor distribution and pricing model, as well as the risk measures and projected exposures. Outcomes analysis includes an independent evaluation of the design and results of backtesting to determine whether all material risk factors are captured and to assess the accuracy of the diffusion of risk factors and the projection of exposures. While there are limitations to backtesting, especially for testing the longer horizon predictions of a CCR model, banking organizations should incorporate it as an essential component of model validation.

Typical examples of CCR models that require backtesting are expected exposure, peak exposure, and CVA VaR models. Backtesting of models used for measurement of CCR is substantially different than backtesting VaR models for market risk. Notably, CCR models are applied to each counterparty facing the banking organization, rather than an aggregate portfolio. Furthermore, CCR models should project the distribution over multiple dates and over long time horizons for each counterparty. These complications make the interpretation of CCR backtesting results more difficult than that for market risk. Because backtesting is critical to providing feedback on the accuracy of CCR models, it is particularly important that banking organizations exert considerable effort to ensure that backtesting provides effective feedback on the accuracy of these models.

Key elements of backtesting include the following activities:

It may be appropriate to use back-testing methods that compare forecast distributions of exposures with actual distributions. Some CCR measures depend on the whole distribution of future exposures rather than a single exposure percentile (for example, EE and EPE). For this reason, sole reliance on backtesting methods that count the number of times an exposure exceeds a unique percentile threshold may not be appropriate.

Exception counting remains useful, especially for evaluating peak or percentile measures of CCR, but these measures will not provide sufficient insight for expected exposure measures. Hence, banking organizations should test the entire distribution of future exposure estimates and not just a single percentile prediction.

Banking organizations should have policies and procedures in place that describe when back-testing results will generate an investigation into the source of observed backtesting deficiencies, and when model changes should be initiated as a result of backtesting.

Adequate validation and review are contingent on complete documentation of all material aspects of CCR models and systems. This should include all model components and parameter estimation or calibration processes. Documentation should also include the rationale for all material assumptions underpinning its chosen analytical frameworks, including the choice of inputs; distributional assumptions; and weighting of quantitative and qualitative elements. Any subsequent changes to these assumptions should also be documented and justified.

The validation or independent review should be fully documented. Specifically, this would include results, the scope of work, conclusions and recommendations, and responses to those recommendations. This includes documentation of each of the three components of model validation, discussed above. Complete documentation should be done initially and updated over time to reflect ongoing changes and model performance. Ability of the validation (or review) to provide effective challenge should also be documented.

A banking organization should have an internal audit function, independent of business-line management, which assesses the effectiveness of the model validation process. This assessment should ensure the following: proper validation procedures were followed for all components of the CCR model and infrastructure systems; required independence was maintained by validators or reviewers; documentation was adequate for the model and validation processes; and results of validation procedures are elevated, with timely responses to findings. Internal audit should also evaluate systems and operations that support CCR. While internal audit may not have the same level of expertise as quantitative experts involved in the development and validation of the model, they are particularly well suited to evaluate process verification procedures. If any validation or review work is out-sourced, internal audit should evaluate whether that work meets the standards discussed in this section.