Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published December 27, 2022 | Published
Journal Article Open

The apparent prevalence of outcome variation from hidden "dark methods" is a challenge for social science


Every working scientist knows that in the details are both devils and angels. Lots of small design decisions have to be made in collecting and analyzing data, and those decisions affect conclusions. But beginning scientists, from rookies in school science fairs to students in early years of a rigorous Ph.D. program, are often surprised how much small decisions matter. Despite this recognition that details matter, when science is communicated, many small decisions made privately by a science team are hidden from view. It is difficult to disclose every detail (and usually little disclosure is required). Such hidden decisions can be thought of as "dark methods," like dark matter which cannot be directly seen because it does not reflect light, but which is evident from its other effects. The Herculean effort resulting in the new many-analyst study (1) which is the subject of my Commentary should force a painful reckoning about the extent of these dark method choices and their influence on conclusions. Design decisions of each team that were coded (107 of them) explained at most 10 to 20% of the outcome variance. Assuming that the coding itself is not too noisy, it seems that hidden decisions account for the lion's share of what different teams conclude.

Additional Information

© 2022 the Author(s). Published by PNAS. This article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND). Author Contributions. C.F.C. wrote the paper. The author declares no competing interest.

Attached Files

Published - pnas.2216020119.pdf


Files (2.6 MB)
Name Size Download all
2.6 MB Preview Download

Additional details

August 22, 2023
October 18, 2023