Evaluating Risk Assessments Using Receiver Operating Characteristic (ROC) Analysis: Rationale, Advantages, Insights, and Limitations

Behavioral Sciences and the Law 2013:31:23–39

22 Pages Posted: 2 Dec 2012 Last revised: 16 Apr 2013

See all articles by Douglas Mossman

Douglas Mossman

University of Cincinnati College of Medicine

Date Written: December 2, 2012

Abstract

The last two decades have witnessed major changes in the way that mental health professionals assess, describe, and think about persons’ risk for future violence. Psychiatrists and psychologists have gone from believing that they could not predict violence to feeling certain they can assess violence risk with well-above-chance accuracy. Receiver operating characteristic (ROC) analysis has played a central role in changing this view. This article reviews the key concepts underlying ROC methods, the meaning of the area under the ROC curve (AUC), the relationship between AUC and effect size d, and what these two indices tell us about evaluations of violence risk.

AUC and d provide succinct but incomplete descriptions of discrimination capacity. These indices do not provide details about sensitivity-specificity trade-offs; they do not tell us how to balance false positive and false negative errors; and they do not determine whether a diagnostic system is accurate enough to make practically useful distinctions between violent and nonviolent subject groups. Justifying choices or clinical practices requires a contextual investigation of outcomes, a process that takes us beyond simply knowing global indices of accuracy.

Keywords: violence, risk assessment, receiver operating characteristic analysis, ROC, accuracy

JEL Classification: D81, C19

Suggested Citation

Mossman, Douglas, Evaluating Risk Assessments Using Receiver Operating Characteristic (ROC) Analysis: Rationale, Advantages, Insights, and Limitations (December 2, 2012). Behavioral Sciences and the Law 2013:31:23–39, Available at SSRN: https://ssrn.com/abstract=2184078

Douglas Mossman (Contact Author)

University of Cincinnati College of Medicine ( email )

260 Stetson Street, Suite 3200
P. O. Box 670559
Cincinnati, OH 45219
United States
513-558-4423 (Phone)

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
112
Abstract Views
801
Rank
441,967
PlumX Metrics