Applying Psychometric Theory and Research to Developing a Continuously Distributed Approach to Making Research Funding Decisions
Review of General Psychology, September 2012
Posted: 4 Aug 2015
Date Written: September 2012
Abstract
Scientists and funding agencies have long struggled with identifying optimal methods for allocating scarce research dollars to pools of research funding proposals. Much of the research has focused on the unreliability of the subjective peer review ratings used to estimate the scholarly merit of funding proposals. Yet, psychologists are uniquely positioned to aid in improving the validity of the funding decisions based on subjective peer review evaluations. Importantly, no federal funding agency currently implements a decision-making method that goes beyond a traditional dichotomous system for allocating support (i.e., “fund” vs. “not fund” a proposal). However, a robust line of psychometric research indicates the superiority of continuous, relative to dichotomous, rating systems for reliably and validly estimating the kinds of constructs of interest in peer review evaluations (i.e., domains relevant to estimating the scholarly merit of research proposals). In this paper, we illustrate how applying psychometric research and theory informs the development of a continuously distributed approach to making research funding decisions. This approach allocates a specific degree of support to proposals, commensurate with the continuously distributed peer review ratings used at major funding agencies to estimate the scholarly merit of proposals. Importantly, we illustrate a hypothetical use of the continuously distributed funding approach that — relative to a dichotomous funding approach and using the same amount of funding resources — allocates funding dollars to 260% more funding proposals. We also discuss the strengths and limitations of a continuously distributed funding approach, as well as its implications for scientists and funding agencies.
Suggested Citation: Suggested Citation