Experiments in the Study of Public Policy: How to Get from Policy Evaluation to the Policy Process

Forthcoming, Handbook of Methods for Comparative Policy Analysis Edited by B. Guy Peters and Guillaume Fontaine, Edward Elgar Publishing Ltd.

25 Pages Posted: 26 Nov 2018

See all articles by Peter John

Peter John

University College London - School of Public Policy; Department of Political Economy, KCL

Date Written: October 30, 2018

Abstract

If experiments have been frequently used to answer practical questions in the evaluation of public policies, they have hardly appeared at all the studies of the policy process, which is the more political science end of public policy, and represent the core set of issues that many public policy scholars examine today to understand and explain decision-making, especially in comparative context. This might be thought to be a function of the nature of experiments in that they are attune to precise evaluation questions rather than assessing complex decision processes. The policy evaluation question of whether a welfare-to-work policy actually works is done by randomising the new intervention to one group of welfare beneficiaries and not another, based on the findings an agency can decide whether to introduce the policy. But this is a long way from understanding decision-making, such as whether policy punctuations are caused by friction (Jones and Baumgartner 2005), for example, which might be hard to test with an experimental method because it may not be possible to locate the random allocation or create an intervention that might test the hypothesis effectively. The theme of this chapter is that such pessimism is not fully justified and that there remains an opportunity to use experiments more extensively in studies of the policy process and in comparative public policy more generally. As well as field trials, there may be an opportunity to use natural experiments to test for theories of the policy process. To make this argument the chapter shows how political scientists have recently directed their attention to decision-making processes and institutions with great effect, creating a set of second-generation experiments different from the first-generation ones mainly done on political behaviour. Even more importantly, and one for public policy scholars to take notice, there has been a parallel development in the field of public management, which is a cognate field to public policy studies. With methods in public management previously being confined to survey methods, observational aggregate data and case studies, the 2010s has seen a veritable explosion of experimental studies which seek to examine decision-making within and across public bureaucracies, such that this sub-discipline has changed dramatically in terms of the questions that are asked and the manner in which they are evaluated (James, Jilke, and Ryzin 2017). That the public management sub-field has moved so fast is both a challenge and a prompt to scholars of public policy to adopt new methods, to answer old and new research questions with experiments. Fortunately, there are some signs of experimental methods recently appearing, mainly the use of new methods for observational data and the analysis of some natural experiments. But much more needs to be done, especially the use of randomised controlled trials; so this chapter is designed as an introduction to researchers and students to give them necessary knowledge about how it could be done.

Suggested Citation

John, Peter and John, Peter, Experiments in the Study of Public Policy: How to Get from Policy Evaluation to the Policy Process (October 30, 2018). Forthcoming, Handbook of Methods for Comparative Policy Analysis Edited by B. Guy Peters and Guillaume Fontaine, Edward Elgar Publishing Ltd., Available at SSRN: https://ssrn.com/abstract=3275167

Peter John (Contact Author)

Department of Political Economy, KCL ( email )

Strand
London, England WC2R 2LS
United Kingdom

University College London - School of Public Policy ( email )

29/30 Tavistock Square
London, WC1H 9QU
United Kingdom

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
108
Abstract Views
515
Rank
453,810
PlumX Metrics