A New Goodness-of-Fit Test for Event Forecasting and its Application to Credit Default Models

49 Pages Posted: 20 Feb 2007 Last revised: 25 Oct 2010

See all articles by Andreas Bloechlinger

Andreas Bloechlinger

University of Applied Sciences Northwestern Switzerland

Markus Leippold

University of Zurich; Swiss Finance Institute

Date Written: August 9, 2009

Abstract

We develop a new goodness-of-fit test for validating the performance of probability forecasts. Our test statistic is particularly powerful under sparseness and dependence in the observed data. To build our test statistic, we start from a formal definition of calibrated forecasts, which we operationalize by introducing two components. The first component tests the level of the estimated probabilities. The second component validates the shape, measuring the differentiation between high and low robability events. After constructing test statistics for both level and shape, we provide a global goodness-of-fit statistic, which is asymptotically x^2 distributed. In a simulation exercise, we find that our approach is correctly sized and more powerful than alternative statistics. In particular, our shape statistic is significantly more powerful than the Kolmogorov-Smirnov test. Under independence our global test has significantly greater power than the popular Hosmer and Lemeshow's x^2 test. Moreover, even under dependence our global test remains correctly sized and consistent. As a timely and important empirical application of our method, we study the validation of a forecasting model for credit default events.

Keywords: Out-of-Sample Validation, Probability Calibration, Hosmer-Lemeshow Statistic, Bernoulli Mixture Models, Credit Risk

JEL Classification: C12, C52, G21

Suggested Citation

Bloechlinger, Andreas and Leippold, Markus, A New Goodness-of-Fit Test for Event Forecasting and its Application to Credit Default Models (August 9, 2009). Available at SSRN: https://ssrn.com/abstract=821884 or http://dx.doi.org/10.2139/ssrn.821884

Andreas Bloechlinger (Contact Author)

University of Applied Sciences Northwestern Switzerland ( email )

Riggenbachstrasse 16
Olten, Solothurn 4600
Switzerland

Markus Leippold

University of Zurich ( email )

Rämistrasse 71
Zürich, CH-8006
Switzerland

Swiss Finance Institute ( email )

c/o University of Geneva
40, Bd du Pont-d'Arve
CH-1211 Geneva 4
Switzerland