The apparent prevalence of outcome variation from hidden "dark methods" is a challenge for social science

Abstract

Every working scientist knows that in the details are both devils and angels. Lots of small design decisions have to be made in collecting and analyzing data, and those decisions affect conclusions. But beginning scientists, from rookies in school science fairs to students in early years of a rigorous Ph.D. program, are often surprised how much small decisions matter. Despite this recognition that details matter, when science is communicated, many small decisions made privately by a science team are hidden from view. It is difficult to disclose every detail (and usually little disclosure is required). Such hidden decisions can be thought of as "dark methods," like dark matter which cannot be directly seen because it does not reflect light, but which is evident from its other effects. The Herculean effort resulting in the new many-analyst study (1) which is the subject of my Commentary should force a painful reckoning about the extent of these dark method choices and their influence on conclusions. Design decisions of each team that were coded (107 of them) explained at most 10 to 20% of the outcome variance. Assuming that the coding itself is not too noisy, it seems that hidden decisions account for the lion's share of what different teams conclude.

Files

Additional details