Will We Trust What We Don’t Understand? Impact of Model Interpretability and Outcome Feedback on Trust in AI

40 Pages Posted: 18 Nov 2021

See all articles by Daehwan Ahn

Daehwan Ahn

University of Pennsylvania - Operations & Information Management Department

Abdullah Almaatouq

Massachusetts Institute of Technology (MIT)

Monisha Gulabani

University of Pennsylvania - The Wharton School

Kartik Hosanagar

University of Pennsylvania - Operations & Information Management Department

Date Written: November 16, 2021

Abstract

Despite AI’s superhuman performance in a variety of domains, humans are often unwilling to adopt AI systems. The lack of interpretability inherent in many modern AI techniques is believed to be hurting their adoption, as users may not trust systems whose decision processes they do not understand. We investigate this proposition with a novel experiment in which we use an interactive prediction task to analyze the impact of interpretability and outcome feedback on trust in AI and on human performance in AI-assisted prediction tasks. We find that interpretability led to no robust improvements in trust, while outcome feedback had a significantly greater and more reliable effect. However, both factors had modest effects on participants’ task performance. Our findings suggest that (1) factors receiving significant attention, such as interpretability, may be less effective at increasing trust than factors like outcome feedback, and (2) augmenting human performance via AI systems may not be a simple matter of increasing trust in AI, as increased trust is not always associated with equally sizable improvements in performance. These findings invite the research community to focus not only on methods for generating interpretations but also on techniques for ensuring that interpretations impact trust and performance in practice.

Keywords: Trust in AI Systems, Human-AI Collaboration, Interpretability, Outcome Feedback, Performance, AI-assisted Systems

JEL Classification: C91

Suggested Citation

Ahn, Daehwan and Almaatouq, Abdullah and Gulabani, Monisha and Hosanagar, Kartik, Will We Trust What We Don’t Understand? Impact of Model Interpretability and Outcome Feedback on Trust in AI (November 16, 2021). Available at SSRN: https://ssrn.com/abstract=3964332 or http://dx.doi.org/10.2139/ssrn.3964332

Daehwan Ahn (Contact Author)

University of Pennsylvania - Operations & Information Management Department ( email )

Philadelphia, PA 19104
United States

Abdullah Almaatouq

Massachusetts Institute of Technology (MIT) ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States

Monisha Gulabani

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Kartik Hosanagar

University of Pennsylvania - Operations & Information Management Department ( email )

Philadelphia, PA 19104
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
153
Abstract Views
825
Rank
347,135
PlumX Metrics