Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales

43 Pages Posted: 17 Mar 2021 Last revised: 27 Mar 2021

See all articles by Christopher Karvetski

Christopher Karvetski

Good Judgment Inc

Carolyn Meinel

affiliation not provided to SSRN

Daniel Maxwell

Independent

Yunzi Lu

University of Pennsylvania

Barbara Mellers

University of Pennsylvania, Psychology; University of Pennsylvania, Wharton School

Philip Tetlock

University of Pennsylvania

Date Written: February 4, 2021

Abstract

Geopolitical forecasting tournaments have stimulated the development of methods for improving probability judgments of real-world events. But these innovations have focused on easier-to quantify variables, like personnel selection, training, teaming, and crowd aggregation—and bypassed messier constructs, like qualitative properties of forecasters’ rationales. Here we adapt methods from natural language processing (NLP) and computational text analysis to identify distinctive reasoning strategies in the rationales of top forecasters, including: (a) cognitive styles, such as dialectical complexity, that gauge tolerance of clashing perspectives and efforts to blend them into coherent conclusions; (b) the use of comparison classes or base rates to inform forecasts; (c) metrics derived from the Linguistic Inquiry and Word Count (LIWC) program. Applying these tools to multiple forecasting tournaments and to forecasters of widely varying skill (from Mechanical Turkers to carefully culled “superforecasters”) revealed that: (a) top forecasters show higher dialectical complexity in their rationales, use more comparison classes, and offer more past-focused rationales; (b) experimental interventions, like training and teaming, that boost accuracy also influence NLP profiles of rationales, nudging them in a “superforecaster-like” direction.

Keywords: geopolitical forecasting, psycholinguistic vocabularies, integrative complexity, comparison class, natural language processing, LIWC

Suggested Citation

Karvetski, Christopher and Meinel, Carolyn and Maxwell, Daniel and Lu, Yunzi and Mellers, Barbara and Tetlock, Philip, Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales (February 4, 2021). Available at SSRN: https://ssrn.com/abstract=3779404 or http://dx.doi.org/10.2139/ssrn.3779404

Christopher Karvetski (Contact Author)

Good Judgment Inc ( email )

230 Park Ave Rm 1515
New York, NY
United States

Carolyn Meinel

affiliation not provided to SSRN

Daniel Maxwell

Independent

Yunzi Lu

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Barbara Mellers

University of Pennsylvania, Psychology ( email )

3815 Walnut Street
Philadelphia, PA 19104-6196
United States

University of Pennsylvania, Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Philip Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
460
Abstract Views
2,234
Rank
115,186
PlumX Metrics