Guilt by Correlation: Data Analytics and Evidence-Based Policy
Posted: 31 Mar 2017
Date Written: March 31, 2017
Abstract
Data analytics and algorithmic decision making are becoming increasingly important in a wide range of applications. As experience shows, their positive and negative contributions are increasingly linked to the performance of communications systems and to the adequacy of scrutiny and (often self-) regulatory mechanisms. For instance, an important factor in the 2010 Flash Crash was network congestion associated with a flood of quotes; analytic programs found themselves trading on stale information suggesting persistent price differentials. This generated more traffic as programs sought to investigate, hedge or speculate on apparently new and significant trends. The resulting positive feedback produced significant (if short-run) market fluctuations, automated (circuit breaker) regulatory devices – with their own anomalies – and longer-term changes in the credibility of automatically-generated information. A key factor seems to be adverse interaction between the technical communication and computation ‘planes’ and analytic and economic (in this case) levels; as volumes increase and the time for interpretation and reaction drop interactions between these levels strengthens; faster, more secure, higher-volume and higher-speed communications cannot be assumed to improve performance or robustness of financial or other systems built on top of them. Moreoever, fast and simplified ‘reactionary’ models will outperform (in the short run) complex and/or farsighted ones. The objective of this research is understanding systemic links among data analytics, algorithmic decision-making and the quality and resilience of evidence-based (esp. regulatory and policy) decisions. The research applies game-theoretic modelling to empirical analysis of high-frequency data flows and decisions to identify anomalies and conditions favouring: ‘normalisation of deviance’ (insensitivity to short-term developments that cannot be explained ‘quickly enough’); ‘post-truth’ assessment (irrelevance to decisions of the accuracy or provenance of underlying information); and distrust of expert judgement or its reduction to the same status as other signals or assessments. The emerging findings are intended to inform ‘structural’ guidelines and regulatory strategies to improve the content if not the conduct of data science, the assessment of automated or algorithmic analysis and decision making; and the role of data science and data scientists in policy processes.
Keywords: Data analytics, big data, algorithms, complexity, evidence-based policy, post-truth
Suggested Citation: Suggested Citation