Recent scandals in psychology demonstrate how easy it is to massage data, or even twist and invent data, in order to produce a desired result. In this report, some psychologists show how it is done:
What the researchers omitted, as they went on to explain in the rest of the paper, was just how many variables they poked and prodded before sheer chance threw up a headline-making result—a clearly false headline-making result.
The odds of statistical bogosity grow when researchers don't have to report all the ways they manipulated their data in exploratory fashion. For example, the researchers "used father's age to control for baseline age across participants," thereby fudging the subjects' actual ages. They factored in lots of completely irrelevant data. And, rather than establish from the outset how many subjects they would test, they tested until they obtained the false result.
The authors of that provocative paper were Joseph P. Simmons and Uri Simonsohn of the University of Pennsylvania, and Leif D. Nelson of the University of California at Berkeley. "Many of us," they wrote—"and this includes the three authors of this article"—end up "yielding to the pressure to do whatever is justifiable to compile a set of studies that we can publish. This is driven not by a willingness to deceive but by the self-serving interpretation of ambiguity. ... "
