A Better Measure of Relative Prediction Accuracy for Model Selection and Model Estimation

Journal of the Operational Research Society (2015) 66, 1352–1362

25 Pages Posted: 25 Jul 2015

See all articles by Chris Tofallis

Chris Tofallis

University of Hertfordshire Business School

Date Written: July 2014

Abstract

Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of forecast accuracy in businesses and organizations. It is however, biased: When used to select among competing prediction methods it systematically selects those whose predictions are too low. This is not widely discussed and so is not generally known among practitioners. We explain why this happens.

We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio: log (prediction/actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection.

Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood.

Keywords: prediction, forecasting, model selection, loss function, regression, time series

JEL Classification: C22, C32, C44, C51, C52, C53

Suggested Citation

Tofallis, Chris, A Better Measure of Relative Prediction Accuracy for Model Selection and Model Estimation (July 2014). Journal of the Operational Research Society (2015) 66, 1352–1362, Available at SSRN: https://ssrn.com/abstract=2635088

Chris Tofallis (Contact Author)

University of Hertfordshire Business School ( email )

College Lane
Hatfield, Hertfordshire AL10 9AB
United Kingdom

HOME PAGE: http://tinyurl.com/ChrisTofallis

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
5,660
Abstract Views
13,203
Rank
2,695
PlumX Metrics