GPT-3.5 Hallucinates Nonexistent Citations: Evidence from Economics

43 Pages Posted: 4 Jun 2023 Last revised: 6 Jun 2023

See all articles by Joy Buchanan

Joy Buchanan

Samford University

Olga Shapoval

Samford University

Date Written: June 3, 2023

Abstract

We create a set of prompts from every Journal of Economic Literature (JEL) topic to test the ability of a GPT-3.5 large language model (LLM) to write about economic concepts. For general summaries, ChatGPT can perform well. However, more than 30% of the citations suggested by ChatGPT do not exist. Furthermore, we demonstrate that the ability of the LLM to deliver accurate information declines as the question becomes more specific. This paper provides evidence that, although GPT has become a useful input to research production, fact-checking the output remains important.

Keywords: artificial intelligence, large language models, ChatGPT, writing, research methods

JEL Classification: B4, O33, I2

Suggested Citation

Buchanan, Joy and Shapoval, Olga, GPT-3.5 Hallucinates Nonexistent Citations: Evidence from Economics (June 3, 2023). Available at SSRN: https://ssrn.com/abstract=4467968 or http://dx.doi.org/10.2139/ssrn.4467968

Joy Buchanan (Contact Author)

Samford University ( email )

800 Lakeshore Drive
Birmingham, AL 35229
United States

Olga Shapoval

Samford University ( email )

800 Lakeshore Drive
Birmingham, AL 35229
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
345
Abstract Views
1,498
Rank
159,344
PlumX Metrics