How reliable is psychological research? A 2015 report under the name the Reproducibility Project gathered 100 psychology studies from leading journals to answer this question and found that less than 40 of them held up when retested by a separate team of researchers. The report caused ripples of disagreement through the field and was either taken as a reality check or an unreliable study that did nothing but stir up controversy.
Now, a new team of researchers has released a critique that challenges this report and claims that the conclusions reached are wrong due to the use of flawed statistical procedures. After adjusting the statistical methodology, the new study reveals that almost 100 percent of the originally tested studies held up to repeat testing.
Although the 2015 analysis did not find any evidence of fraud or manipulation of the data, the amount of press that it received led some people that didn't know any better to draw this conclusion.
"That study got so much press, and the wrong conclusions were drawn from it," said Timothy Wilson, a professor of psychology at the University of Virginia and co-author of the new critique. "It's a mistake to make generalizations from something that was done poorly, and this we think was done poorly."
The original team wasted no time responding, offering a rebuttal of Wilson's team's findings.
"They are making assumptions based on selectively interpreting data and ignoring data that's antagonistic to their point of view," said Brian Nosek, who coordinated the original replication project.
Although the Reproducibility Project garnered a great deal of attention, the argument of reliability and accuracy in psychological research is not new.
"On some level, I suppose it is appealing to think everything is fine and there is no reason to change the status quo," said Sanjay Srivastava, a psychologist at the University of Oregon who did not participate in any of the two team's research. "But we know too much, from many other sources, to put too much credence in an analysis that supports that remarkable conclusion."
Others, such as Uri Simonsohn, a researcher at the University of Pennsylvania, believe that both teams used statistical techniques that are not ideal for the research in question.
"State-of-the-art techniques designed to evaluate replications say it is 40 percent full, 30 percent empty, and the remaining 30 percent could be full or empty, we can't tell till we get more data," he said.
