On Policymakers' Loss Function and the Evaluation of Early Warning Systems

25 Pages Posted: 12 Feb 2013

See all articles by Peter Sarlin

Peter Sarlin

Hanken School of Economics; RiskLab Finland

Date Written: January 29, 2013

Abstract

This paper introduces a new loss function and Usefulness measure for evaluating early warning systems (EWSs) that incorporate policymakers' preferences between issuing false alarms and missing crises, as well as individual observations. The novelty derives from three enhancements: i) accounting for unconditional probabilities of the classes, ii) computing the proportion of available Usefulness that the model captures, and iii) weighting observations by their importance for the policymaker. The proposed measures are model free such that they can be used to assess signals issued by any type of EWS, such as logit and probit analysis and the signaling approach, and flexible for any type of crisis EWSs, such as banking, debt and currency crises. Applications to two renowned EWSs, and comparisons to two commonly used evaluation measures, illustrate three key implications of the new measures: i) further highlights the importance of an objective criterion for choosing a final specification and threshold value, and for models to be useful ii) the need to be more concerned about the rare class and iii) the importance of correctly classifying observations of the most relevant entities. Beyond financial stability surveillance, this paper also opens the door for cost-sensitive evaluations of predictive models in other tasks.

Keywords: early warning systems, policymakers' preferences, misclassification costs

JEL Classification: E44, E58, F01, F37, G01

Suggested Citation

Sarlin, Peter, On Policymakers' Loss Function and the Evaluation of Early Warning Systems (January 29, 2013). ECB Working Paper No. 1509, Available at SSRN: https://ssrn.com/abstract=2208385 or http://dx.doi.org/10.2139/ssrn.2208385

Peter Sarlin (Contact Author)

Hanken School of Economics

PO Box 479
FI-00101 Helsinki
Finland

RiskLab Finland ( email )

Turku, 20520
Finland

HOME PAGE: http://risklab.fi/people/peter/

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
54
Abstract Views
520
Rank
675,887
PlumX Metrics