CaltechAUTHORS
  A Caltech Library Service

Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications

Kugener, Guillaume and Pangal, Dhiraj J. and Cardinal, Tyler and Collet, Casey and Lechtholz-Zey, Elizabeth and Lasky, Sasha and Sundaram, Shivani and Markarian, Nicholas and Zhu, Yichao and Roshannai, Arman and Sinha, Aditya and Han, X. Y. and Papyan, Vardan and Hung, Andrew and Anandkumar, Animashree and Wrobel, Bozena and Zada, Gabriel and Donoho, Daniel A. (2022) Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications. JAMA Network Open, 5 (3). Art. No. e223177. ISSN 2574-3805. PMCID PMC8938712. doi:10.1001/jamanetworkopen.2022.3177. https://resolver.caltech.edu/CaltechAUTHORS:20220329-772928599

This is the latest version of this item.

[img] PDF - Published Version
Creative Commons Attribution.

2MB
[img] PDF - Supplemental Material
Creative Commons Attribution.

167kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220329-772928599

Abstract

Importance. Surgical data scientists lack video data sets that depict adverse events, which may affect model generalizability and introduce bias. Hemorrhage may be particularly challenging for computer vision–based models because blood obscures the scene. Objective. To assess the utility of the Simulated Outcomes Following Carotid Artery Laceration (SOCAL)—a publicly available surgical video data set of hemorrhage complication management with instrument annotations and task outcomes—to provide benchmarks for surgical data science techniques, including computer vision instrument detection, instrument use metrics and outcome associations, and validation of a SOCAL-trained neural network using real operative video. Design, Setting, and Participants. For this quailty improvement study, a total of 75 surgeons with 1 to 30 years’ experience (mean, 7 years) were filmed from January 1, 2017, to December 31, 2020, managing catastrophic surgical hemorrhage in a high-fidelity cadaveric training exercise at nationwide training courses. Videos were annotated from January 1 to June 30, 2021. Interventions. Surgeons received expert coaching between 2 trials. Main Outcomes and Measures. Hemostasis within 5 minutes (task success, dichotomous), time to hemostasis (in seconds), and blood loss (in milliliters) were recorded. Deep neural networks (DNNs) were trained to detect surgical instruments in view. Model performance was measured using mean average precision (mAP), sensitivity, and positive predictive value. Results. SOCAL contains 31 443 frames with 65 071 surgical instrument annotations from 147 trials with associated surgeon demographic characteristics, time to hemostasis, and recorded blood loss for each trial. Computer vision–based instrument detection methods using DNNs trained on SOCAL achieved a mAP of 0.67 overall and 0.91 for the most common surgical instrument (suction). Hemorrhage control challenges standard object detectors: detection of some surgical instruments remained poor (mAP, 0.25). On real intraoperative video, the model achieved a sensitivity of 0.77 and a positive predictive value of 0.96. Instrument use metrics derived from the SOCAL video were significantly associated with performance (blood loss). Conclusions and Relevance. Hemorrhage control is a high-stakes adverse event that poses unique challenges for video analysis, but no data sets of hemorrhage control exist. The use of SOCAL, the first data set to depict hemorrhage control, allows the benchmarking of data science applications, including object detection, performance metric development, and identification of metrics associated with outcomes. In the future, SOCAL may be used to build and validate surgical data science models.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1001/jamanetworkopen.2022.3177DOIArticle
http://www.ncbi.nlm.nih.gov/pmc/articles/pmc8938712/PubMed CentralArticle
ORCID:
AuthorORCID
Kugener, Guillaume0000-0002-4697-2847
Pangal, Dhiraj J.0000-0001-7391-9825
Sundaram, Shivani0000-0003-2863-9204
Zada, Gabriel0000-0001-5821-902X
Additional Information:© 2022 Kugener G et al. JAMA Network Open. This is an open access article distributed under the terms of the CC-BY License. Accepted for Publication: January 31, 2022. Published: March 21, 2022. Author Contributions: Dr Donoho had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: Kugener, Pangal, Roshannai, Papyan, Hung, Anandkumar, Wrobel, Zada, Donoho. Acquisition, analysis, or interpretation of data: Kugener, Pangal, Cardinal, Collet, Lechtholz-Zey, Lasky, Sundaram, Markarian, Zhu, Roshannai, Sinha, Han, Wrobel, Donoho. Drafting of the manuscript: Kugener, Pangal, Lechtholz-Zey, Sundaram, Roshannai, Donoho. Critical revision of the manuscript for important intellectual content: Kugener, Pangal, Cardinal, Collet, Lasky, Markarian, Zhu, Sinha, Han, Papyan, Hung, Anandkumar, Wrobel, Zada, Donoho. Statistical analysis: Kugener, Pangal, Collet, Lechtholz-Zey, Zhu, Roshannai, Anandkumar, Donoho. Obtained funding: Zada. Administrative, technical, or material support: Kugener, Pangal, Lasky, Sundaram, Markarian, Zhu, Sinha, Han, Wrobel, Zada, Donoho. Supervision: Kugener, Pangal, Papyan, Hung, Wrobel, Zada, Donoho. Conflict of Interest Disclosures: None reported.
Issue or Number:3
PubMed Central ID:PMC8938712
DOI:10.1001/jamanetworkopen.2022.3177
Record Number:CaltechAUTHORS:20220329-772928599
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20220329-772928599
Official Citation:Kugener G, Pangal DJ, Cardinal T, et al. Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications. JAMA Netw Open. 2022;5(3):e223177. doi:10.1001/jamanetworkopen.2022.3177
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:114119
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:29 Mar 2022 12:27
Last Modified:27 Jul 2022 21:52

Available Versions of this Item

  • Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications. (deposited 29 Mar 2022 12:27) [Currently Displayed]

Commentary/Response Threads

  • Kugener, Guillaume and Pangal, Dhiraj J. and Cardinal, Tyler and Collet, Casey and Lechtholz-Zey, Elizabeth and Lasky, Sasha and Sundaram, Shivani and Markarian, Nicholas and Zhu, Yichao and Roshannai, Arman and Sinha, Aditya and Han, X. Y. and Papyan, Vardan and Hung, Andrew and Anandkumar, Animashree and Wrobel, Bozena and Zada, Gabriel and Donoho, Daniel A. Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications. (deposited 29 Mar 2022 12:27) [Currently Displayed]

Repository Staff Only: item control page