# An erroneous sampling procedure in the Asset Quality Review

24 maart 2014 Plaats een reactie

The sampling procedure that is described in the ECB AQR Manual is not correct.

We illustrate this with the help of an example non-retail portfolio that consists of 1000 performing exposures. In this example the exposures follow an exponential distribution (with mu = 10 (EUR mln)) that is shown below.

** **

The largest exposure is almost EUR 70 mln. Total portfolio size is EUR 10.0 bln.

Each exposure is assigned to one of seven buckets. By definition, the smallest 5% of the exposures are not included in the sample. They are assigned to bucket 1.

The 10 largest exposures are a special case, since these exposures are checked in an integral way. They are assigned to bucket 7.

The range between the tenth debtor and the 5^{th} percentile is split into 5 buckets of the same absolute difference in exposure.

Further explanation of the definition of the five buckets is provided on page 85 of the manual:

## Probability to be drawn for the entire portfolio

With the help of the procedure described above we can establish for each of the 1000 exposures a probability to be included in the sample (‘probability to be drawn’). We compare these probabilities to be drawn in the AQR sample with the probabilities to be drawn in a Dollar Unit Sampling (DUS) procedure. In this DUS procedure the probabilities to be drawn are proportional to the size of the exposures.

The comparison with DUS is justified since the average misstatement is calculated in an unweighted fashion (see p. 171 of the AQR Manual). This is only correct if the exposures have been selected proportional to their size.

We sample 8 exposures from the buckets 2 through 7 (see AQR Manual, p. 90, for non-retail granular performing exposures).

Below we compare both types of probabilities.

In this analysis we have a sample of 50 exposures: 8 exposures for buckets 2 through 6 and 10 exposures for bucket 7. The 50 smallest exposures have a probability equal to zero (remember that the 5^{th} percentile of 1000 exposures is the value of the 50^{th} exposure).

In order to work towards a more direct comparison, the analysis is restricted to buckets 2 through 6, i.e. we select only those buckets for which a meaningful sample is drawn. After this restriction, we exclude bucket 1 (no exposures are selected) and bucket 7 (all top 10 exposures are a special case).

## Probability to be drawn – buckets 2 through 6

Below we show the probabilities to be drawn for a sample of 40 exposures (5 buckets * 8 exposures in the sample for each bucket) after restricting the analysis to buckets 2 through 6.

It is remarkable that the AQR probabilities do not match in a neat way with the DUS probabilities.

Suppose that the relative misstatement for each exposure in the 5 buckets is equal to the following values for buckets 2 through 6: [ 0.050, 0.036, 0.023, 0.047, 0.054 ]. These values have been derived from an example in the AQR Manual (see p. 173).

For these values, the expected outcome of the sample average relative misstatement for the AQR procedure equals 4.20%. This is too high, since the expected outcome of the sample average relative misstatement for the DUS sample is 3.93% (which is the correct outcome).

*We conclude that in this example the AQR sampling procedure overestimates the average relative misstatement. We also conclude that the AQR sampling procedure is biased. *

If larger exposures tend to have larger relative misstatements due to an increasing complexity, the AQR sampling will tend to overestimate the sample average relative misstatement due to the high probabilities to be drawn relative to the DUS sampling procedure in the buckets 5 and 6 (see also the graph above).

## An explanation of the bias

This bias is easily explained. The definition of the buckets is not correct. The correct specification would be:

- Calculate the vector with the cumulative sums of the sorted exposures in the buckets 2 through 6.
- Calculate the range as the total exposure size in the buckets 2 through 6.
- Calculate the step size as the range divided by 5.
- Define the limits with the help of
*k***step*,*k*ranges from 1 to 5. - Assign the exposures to the buckets according to their cumulative sum with the help of the above defined limits.

After the buckets have been defined in this way, the probabilities to be drawn according to DUS and the adjusted AQR procedure look as follows:

We now see that the adjusted-AQR probabilities to be drawn follow more closely the DUS probabilities.

After this adjustment, the expected sample average relative misstatement is correctly calculated as 3.93%.

## Conclusion

The AQR sampling procedure is not correct due to a misspecification of the bucket definition. This leads to a biased outcome for the expected sample average of the relative misstatement. However, there is also good news: the misspecification of the buckets can be repaired afterwards (so after sampling has taken place) by applying a weighting scheme to the calculation of the average relative misstatement in which the bias is repaired.