When Does Simulated Data Match Real Data?

GECCO 2011, Dublin, Ireland, July 12-16, 2011

Robert H. Smith School Research Paper No. RHS 06-135

3 Pages Posted: 22 May 2011 Last revised: 25 Jan 2013

See all articles by Forrest Stonedahl

Forrest Stonedahl

Northwestern University - Northwestern Institute for Complex Systems (NICO)

David Anderson

Villanova School of Business

William Rand

North Carolina State University

Date Written: May 19, 2011

Abstract

Agent-based models can replicate real-world patterns, but finding parameters that achieve the best match can be difficult. To validate a model, a real-world dataset is often divided into a training set (to calibrate the parameters) and a test set (to validate the calibrated model). The difference between the training and test data and the simulated data is determined using an error measure. In the context of evolutionary computation techniques, the error measure also serves as a fitness function, and thus affects evolutionary search dynamics. We survey the effect of five different error measures on both a toy problem and a real world problem of matching a model to empirical online news consumption behavior. We use each error measure separately for calibration on the training dataset, and then examine the results of all five error measures on both the training and testing datasets. We show that certain error measures sometimes serve as better fitness functions than others, and in fact using one error measure may result in better calibration (on a different measure) than using the different measure directly. For the toy problem, the Pearson's correlation measure dominated all other measures, but no single error measure was Pareto dominant for the real world problem.

This is the poster version of the paper.

Keywords: agent-based modeling, calibration, business, genetic algorithms, information search, network analysis

Suggested Citation

Stonedahl, Forrest and Anderson, David and Rand, William, When Does Simulated Data Match Real Data? (May 19, 2011). GECCO 2011, Dublin, Ireland, July 12-16, 2011, Robert H. Smith School Research Paper No. RHS 06-135, Available at SSRN: https://ssrn.com/abstract=1846675

Forrest Stonedahl

Northwestern University - Northwestern Institute for Complex Systems (NICO) ( email )

Chambers Hall
600 Foster Street
Evanston, IL 60208-4057
United States

David Anderson

Villanova School of Business ( email )

800 E Lancaster Ave
Villanova, PA 19085
United States

William Rand (Contact Author)

North Carolina State University ( email )

Raleigh, NC 27695
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
143
Abstract Views
1,116
Rank
366,674
PlumX Metrics