Machine learning reveals the control mechanics of an insect wing hinge

The authors thank W. Dickson for extensive expertise in instrumentation, programming, data analysis, formatting all the data and code for public repositories, and creating the animations of free flight data in Supplementary Videos 3–8; T. Lindsay for assistance in the design of the epifluorescence microscope and data acquisition software used for muscle imaging; A. Erickson for helpful comments on the manuscript and Supplementary Information; A. Huda for assistance in the construction of genetic lines; J. Omoto for collecting confocal images of wings to visualize resilin using autofluorescence; J. Tuthill and T. Azevedo for a tomographic dataset of the Drosophila wing hinge that was collected at the European Synchrotron Radiation Facility in Grenoble, France; S. Whitehead for analysis of this tomography data to provide a preliminary reconstruction of the hinge sclerites, and for critical feedback on the manuscript text and data presentation; and B. Fabian and R. G. Beutel for providing μ-CT data from their publication on the morphology of the adult fly body. The research reported in this publication was supported by the National Institute of Neurological Disorders and Stroke of the NIH (U19NS104655). I.S. was supported through the AniBody Project Team at HHMI’s Janelia Research Campus for this work.

The data required to perform the analyses in this paper and reconstruct all the data figure are available in the following files: main_muscle_and_wing_data.h5, flynet_data.zip, robofly_data.zip, which are available from the Caltech Data website: https://doi.org/10.22002/aypcy-ck464. main_muscle_and_wing_data.h5 contains the time series of muscle activity and wing kinematics used to train the muscle-to-wing motion CNN and the encoder–decoder used in the latent variable analysis. flynet_data.zip contains a series of data files for training and running Flynet: (1) camera/calibration/cam_calib.txt (example camera calibration data); (2) movies/session_01_12_2020_10_22 (folder containing example movies); (3) labels.h5 and valid_labels.h5 (data for training); and (4) weights_24_03_2022_09_43_14.h5 (example weights). robofly_data.zip contains the MATLAB data files with force and torque data acquired using the dynamically scaled robotic fly.

Extended Data Fig. 1 Automated setup for simultaneous recording of muscle fluorescence and wing motion.

The code required to perform the analyses in this paper and reconstruct all the data figures are available at https://github.com/FlyRanch/mscode-melis-siwanowicz-dickinson. The software is organized into seven submodules: flynet, flynet-kalman, flynet-optimizer, latent-analysis, mpc-simulations, robofly and wing-hinge-cnn. The installation instructions, system requirements and dependency information are given separately in their respective folders. flynet is a neural network and GUI application that requires the dataset flynet_data.zip, and may be used to create Extended Data Fig. 2. An example demonstrating how to train the network can be found in the examples sub-directory and is called train_flynet.py. flynet-kalman is a Kalman filter Python extension used by Flynet. flynet-optimizer is a particle swarm optimization extension module used by Flynet. latent-analysis is a Python library and Jupyter notebook for performing latent variable analysis that requires the dataset main_muscle_and_wing_data.h5, and may be used to create Fig. 6 and Extended Data Fig. 8. mpc-simulations is a Python library and Jupyter notebook for MPC simulations, and may be used to create Fig. 5 and Extended Data Fig. 7. robofly is a Python library and Jupyter notebook for extracting force and torque data from the robotic fly experiments and plotting forces superimposed on 3D wing kinematics. It requires dataset robofly_data.zip, and may be used to create Extended Data Figs. 5 and  6. wing-hinge-cnn is a Python library and Jupyter notebook for creating the muscle-to-wing motion CNN. It requires main_muscle_and_wing_data.h5, and may be used to create Figs. 3 and 4 and Extended Data Fig. 3. An example demonstrating how to train the network can be found in the examples sub-directory as is called train_wing_hinge_cnn.py. The files containing the raw videos of the muscle Ca²⁺ images and high-speed videos of wing motion are too large to be hosted on a publicly accessible website. Example high-speed videos are provided in the folder movies/session_01_12_2020_10_22 mentioned in Data availability. Additional sequences are available upon request by contacting the corresponding author.

The authors declare no competing interests.