Looking at Individual Subjects in Research on Judgment and Decision Making (or Anything)

Abstract

Many questions in judgment and decision-making research, and, indeed, in experimental psychology generally, concern the existence of effects, and the explanation of effects shown to exist. These questions do not concern the prevalence of effects in any particular population. It is thus appropriate to look for effects in single subjects. If one person shows the effect, then it exists. This argument implies that it is sometimes appropriate to test effects across cases or rounds, without testing across subjects. It also implies that, in some experiments, effects in opposite directions may exist. I recommend looking for such effects by carrying out statistical tests on individual subjects. I describe a few methods, varying in formality, that can be used to deal with the inevitable problem of doing multiple tests of the same hypothesis: probability-probability plots; tests of the distribution of p-values; and correction for multiple testing with step-down resampling. I also present a few examples, some of which show effects in both directions and some of which do not.