Kristin Lane  Image Credit: Pete Mauney

Bard College Psychology Professor is Coauthor of Comprehensive Long-Term Study Published Today in Science, Testing the Reproducibility of Psychology Studies

ANNANDALE-ON-HUDSON, N.Y. – Bard College associate professor of psychology Kristin Lane, as well as Jin Goh ’12 and Douglas Gazarian ’13, were among 270 researchers who participated in the most comprehensive investigation ever conducted about the rate and predictors of reproducibility in a field of science. The four-year study, published today in Science, found that among replications of 100 published findings in three prominent psychology journals, fewer than half produced the same findings as the original study.

“This project shows the promise and challenge of doing scientific work, and lays the foundation for psychology and other sciences to reward practices that lead to robust and replicable findings,” said Lane.

Science is unique from other ways of gaining knowledge by relying on reproducibility to gain confidence in ideas and evidence. Reproducibility means that the results recur when the same data are analyzed again, or when new data are collected using the same methods. As noted by Angela Attwood, team member from University of Bristol, “Scientific evidence does not rely on trusting the authority of the person that made the discovery. Rather, credibility accumulates through independent replication and elaboration of the ideas and evidence.” 

The team emphasized that a failure to reproduce results does not necessarily mean the original report was incorrect, and noted three basic reasons this might occur: 1) Even though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where, or how the replication was conducted might have influenced the results.  2) The replication might have failed to detect the original result by chance. 3) The original result might have been a false positive. Johanna Cohoon, project coordinator from the Center for Open Science, concluded that “reproducing original results may be more difficult than is presently assumed, and interventions may be needed to improve reproducibility.” 

The study also raised a question about the extent to which incentives for scientists are aligned with reproducibility. Research with new, surprising findings is more likely to be published than research examining when, why, or how existing findings can be reproduced. As a consequence, it is often in many scientists’ career interests to pursue innovative research rather than to focus on reproducibility. Many organizations, funders, journals, and publishers are already working on improving reproducibility. For example, in 2014, the journal Psychological Science—one of the journals included in this study—implemented practices to acknowledge open sharing of materials and data to improve reproducibility, and Science has published Transparency and Openness Promotion Guidelines.  

Similar projects have emerged in other fields such as the Reproducibility Project: Cancer Biology. These and widespread efforts to improve research transparency and reproducibility are indications that, as Lane suggested, “Science can work from within to make our practices and research more transparent and replicable, which is how we will ultimately best serve society.”  

For more information on the study, Reproducibility Project: Psychology, please visit https://osf.io/ezcuj/wiki/home/.

Kristin Lane is an associate professor and director of the Psychology Program at Bard. She earned a B.A. from the University of Virginia, an M.S. from Yale University, and a Ph.D. from Harvard University. She is the recipient of the Cabot Postdoctoral Fellowship for Innovation in Teaching, Derek Bok Center for Teaching and Learning, from Harvard University, and has taught at Harvard, Johns Hopkins University Center for Talented Youth, and Tufts University. Her research has appeared in Social Cognition, Sex Roles, Annual Review of Psychology and Law, and other publications, and is funded by the National Institutes of Health.

For more information, contact Kristin Lane at [email protected]. 

# # #

(8/2715)

This event was last updated on 08-27-2015

Back to Top