The Crisis of Confidence in Research Findings in Psychology: Is Lack of Replication the Real Problem? Or is it Something Else?

Abstract

There have been frequent expressions of concern over the supposed failure of researchers to conduct replication studies. But the large number of meta-analyses in our literatures shows that replication studies are in fact being conducted in most areas of research. Many who argue for replication as the “gold standard” consider a non-significant replication attempt to be strong evidence against the initial study, an interpretation that ignores statistical power, typically low in behavioral research. Many researchers also hold that there is no need to replicate a non-significant finding, believing it will always replicate perfectly, an erroneous belief. These beliefs lead to a widely accepted sequential model of the research process that is deficient because it assumes that a single study can answer a research question, a belief that meta-analysis has shown to be false. Meta-analysis can provide the solution to these problems if the problems of publication bias and questionable research practices are successfully addressed. The real problem is not a lack of replication; it is the distortion of our research literatures caused by publication bias and questionable research practices.

Keywords: Replication; Meta-Analysis; Publication Bias; Questionable Research Practices