A Caltech Library Service

Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods

Warby, Simon C. and Wendt, Sabrina L. and Welinder, Peter and Munk, Emil G. S. and Carrillo, Oscar and Sorensen, Helge B. D. and Jennum, Poul and Peppard, Paul E. and Perona, Pietro and Mignot, Emmanuel (2014) Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nature Methods, 11 (4). pp. 385-392. ISSN 1548-7091. PMCID PMC3972193.

[img] PDF - Accepted Version
See Usage Policy.

PDF - Supplemental Material
See Usage Policy.

[img] Archive (ZIP) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

Item Type:Article
Related URLs:
URLURL TypeDescription CentralArticle ReadCube access
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2014 Nature America, Inc. Received 17 July 2013. Accepted 31 January 2014. Published online 23 February 2014. The authors thank the Registered Polysomnographic Technologist (RPSGT) experts who participated in the spindle identification task as well as the participants and organizers of the Wisconsin Sleep Cohort who provided the polysomnography data. We also thank C. Liang, E. Leary, H. Ollila and H. Kraemer for their helpful discussions and E.Þ. Ágústsson and H. Moore for their input in the pilot study for this project. We are grateful to the authors of the previously published algorithms who generously shared their code and knowledge about spindle detectors. S.C.W. is supported by the Brain and Behavior Research Foundation and as a Canadian Institutes of Health Research Banting Fellow. E.M. is supported by US National Institutes of Health (NIH) grant NS23724. P.P. received funding from the Caltech SURF program and the NAVY (ONR-MURI N00014-06-1-0734 and UCLA-MURI N00014-10-1-0933). EEG data collection was supported by grants from the National Heart, Lung, and Blood Institute (grant R01HL62252) and the National Center for Research Resources (grant 1UL1RR025011) at the NIH. The authors declare no competing financial interests. Author Contributions: S.C.W., E.M. and P.P. designed the research. P.W. and P.P. designed and coded the Internet interface. S.C.W. and P.W. collected the spindle scoring data. S.C.W. and S.L.W. performed the data analysis. S.L.W. wrote the code to implement the automated spindle detectors. All authors provided input on data analysis and interpretation. P.E.P. also provided source EEG data. H.B.D.S., P.J., E.M. and P.P. also provided financial support. S.C.W., S.L.W. and E.M. wrote the manuscript, which was discussed and edited by all authors.
Funding AgencyGrant Number
Brain and Behavior Research FoundationUNSPECIFIED
Canadian Institutes of Health Research (CIHR)UNSPECIFIED
Caltech Summer Undergraduate Research Fellowship (SURF)UNSPECIFIED
Office of Naval Research (ONR)N00014-06-1-0734
Office of Naval Research (ONR)N00014-10-1-0933
National Heart, Lung, and Blood InstituteUNSPECIFIED
Subject Keywords:Circadian rhythms and sleep; Biomarker research; Neurophysiology; Electroencephalography - EEG
Issue or Number:4
PubMed Central ID:PMC3972193
Record Number:CaltechAUTHORS:20140303-130810631
Persistent URL:
Official Citation:Warby, S. C., Wendt, S. L., Welinder, P., Munk, E. G. S., Carrillo, O., Sorensen, H. B. D., . . . Mignot, E. (2014). Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. [Analysis]. Nat Meth, 11(4), 385-392. doi: 10.1038/nmeth.2855
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:44090
Deposited By: Ruth Sustaita
Deposited On:03 Mar 2014 21:44
Last Modified:27 Mar 2020 19:29

Repository Staff Only: item control page