CaltechAUTHORS
  A Caltech Library Service

Inter-expert and intra-expert reliability in sleep spindle scoring

Wendt, Sabrina L. and Welinder, Peter and Sorensen, Helge B. D. and Peppard, Paul E. and Jennum, Poul and Perona, Pietro and Mignot, Emmanuel and Warby, Simon C. (2015) Inter-expert and intra-expert reliability in sleep spindle scoring. Clinical Neurophysiology, 126 (8). pp. 1548-1556. ISSN 1388-2457. PMCID PMC4426257. https://resolver.caltech.edu/CaltechAUTHORS:20141208-095057448

[img] PDF - Accepted Version
See Usage Policy.

775Kb
[img] MS Word - Supplemental Material
See Usage Policy.

90Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20141208-095057448

Abstract

Objectives: To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings. Methods: The EEG dataset was comprised of 400 randomly selected 115 s segments of stage 2 sleep from 110 sleeping subjects in the general population (57 ± 8, range: 42–72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F_1-scores, Cohen’s kappa (κ), and intra-class correlation coefficient (ICC). Results: We found an average intra-expert F_1-score agreement of 72 ± 7% (κ: 0.66 ± 0.07). The average inter-expert agreement was 61 ± 6% (κ: 0.52 ± 0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system. Conclusions: We estimate that 2–3 experts are needed to build a spindle scoring dataset with ‘substantial’ reliability (κ: 0.61–0.8), and 4 or more experts are needed to build a dataset with ‘almost perfect’ reliability (κ: 0.81–1). Significance: Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1016/j.clinph.2014.10.158DOIArticle
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4426257PubMed CentralArticle
ORCID:
AuthorORCID
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2014 International Federation of Clinical Neurophysiology. Published by Elsevier Ireland Ltd. Accepted 29 October 2014, Available online 10 November 2014. We thank the RPSGT experts who participated in the spindle identification task, and the participants of the Wisconsin Sleep Cohort who provided the polysomnography data. Also, thanks to Eileen B Leary for valuable comments and edits that helped improve the clarity of the manuscript. SCW is supported by the Canadian Institutes of Health Research and the Brain and Behavior Research Foundation. EM is supported by National Institutes of Health (Grant NS23724). EEG data collection was supported by Grants from the National Heart, Lung, and Blood Institute (Grant R01HL62252) and the National Center for Research Resources (Grant 1UL1RR025011) at the National Institutes of Health. All authors report no conflicts of interest for this work.
Funders:
Funding AgencyGrant Number
Canadian Institutes of Health Research (CIHR)UNSPECIFIED
Brain and Behavior Research FoundationUNSPECIFIED
NIHNS23724
NIHR01HL62252
NIH1UL1RR025011
Subject Keywords:Sleep spindles; Agreement; Reliability; Inter-rater; Inter-expert; Intra-rater; Intra-expert; Electroencephalography; Polysomnography; Event detection; Sleep staging; Sleep scoring
Issue or Number:8
PubMed Central ID:PMC4426257
Record Number:CaltechAUTHORS:20141208-095057448
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20141208-095057448
Official Citation:Sabrina L. Wendt, Peter Welinder, Helge B.D. Sorensen, Paul E. Peppard, Poul Jennum, Pietro Perona, Emmanuel Mignot, Simon C. Warby, Inter-expert and intra-expert reliability in sleep spindle scoring, Clinical Neurophysiology, Volume 126, Issue 8, August 2015, Pages 1548-1556, ISSN 1388-2457, http://dx.doi.org/10.1016/j.clinph.2014.10.158. (http://www.sciencedirect.com/science/article/pii/S1388245714007329)
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:52460
Collection:CaltechAUTHORS
Deposited By: Ruth Sustaita
Deposited On:09 Dec 2014 22:09
Last Modified:27 Mar 2020 21:00

Repository Staff Only: item control page