A Caltech Library Service

Autonomous Hierarchical Surgical State Estimation During Robot-Assisted Surgery Through Deep Neural Networks

Qin, Yidan and Allan, Max and Burdick, Joel W. and Azizian, Mahdi (2021) Autonomous Hierarchical Surgical State Estimation During Robot-Assisted Surgery Through Deep Neural Networks. IEEE Robotics and Automation Letters, 6 (4). pp. 6220-6227. ISSN 2377-3766. doi:10.1109/LRA.2021.3091728.

[img] PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Many operations in robot-assisted surgery (RAS) can be viewed in a hierarchical manner. Each surgical task is represented by a superstate, which can be decomposed into finer-grained states. The estimation of these discrete states at different levels of temporal granularity provides a temporal perception of the current surgical scene during RAS, which is a crucial step towards many automated surgeon-assisting functionalities. We propose Hierarchical Estimation of Surgical States through Deep Neural Networks (HESS-DNN), a deep learning-based system that concurrently estimates the current super- and fine-grained states. HESS-DNN incorporates endoscopic vision, robot kinematics, and system events data from the da Vinci Xi surgical system. HESS-DNN is evaluated on a real-world robotic inguinal hernia repair surgery dataset: HERNIA-20, and achieves accurate state estimates of both surgical superstate and the corresponding fine-grained surgical state. We show that HESS-DNN improves state-of-the-art fine-grained state estimation across the entire HERNIA-20 RAS procedure through its hierarchical design. We also analyze the relative contributions of each input data type and HESS-DNN's design to surgical (super)state estimation accuracy.

Item Type:Article
Related URLs:
URLURL TypeDescription
Qin, Yidan0000-0002-7766-1021
Allan, Max0000-0002-6495-065X
Burdick, Joel W.0000-0002-3091-540X
Additional Information:© 2021 IEEE. Manuscript received February 10, 2021; accepted June 5, 2021. Date of publication June 23, 2021; date of current version July 9, 2021. This work was supported by Intuitive Surgical, Inc. This letter was recommended for publication by Associate Editor J. Guo and Editor P. Valdastri upon evaluation of the reviewers’ comments. We would like to thank Dr. Seyedshams Feyzabadi, Dr. Sandra Park, Dr. Humphrey Chow, and Dr. Wenqing Sun for their support.
Funding AgencyGrant Number
Intuitive Surgical, Inc.UNSPECIFIED
Subject Keywords:Surgical robotics: laparoscopy, deep learning methods, AI-based methods, medical robots and systems
Issue or Number:4
Record Number:CaltechAUTHORS:20210630-201406137
Persistent URL:
Official Citation:Y. Qin, M. Allan, J. W. Burdick and M. Azizian, "Autonomous Hierarchical Surgical State Estimation During Robot-Assisted Surgery Through Deep Neural Networks," in IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 6220-6227, Oct. 2021, doi: 10.1109/LRA.2021.3091728
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:109680
Deposited By: Tony Diaz
Deposited On:30 Jun 2021 22:46
Last Modified:16 Nov 2021 19:37

Repository Staff Only: item control page