Zvyagin, Maxim and Brace, Alexander and Hippe, Kyle and Deng, Yuntian and Zhang, Bin and Orozco Bohorquez, Cindy and Clyde, Austin and Kale, Bharat and Perez-Rivera, Danilo and Ma, Heng and Mann, Carla M. and Irvin, Michael and Pauloski, J. Gregory and Ward, Logan and Hayot-Sasson, Valerie and Emani, Murali and Foreman, Sam and Xie, Zhen and Lin, Diangen and Shukla, Maulik and Nie, Weili and Romero, Josh and Dallago, Christian and Vahdat, Arash and Xiao, Chaowei and Gibbs, Thomas and Foster, Ian and Davis, James J. and Papka, Michael E. and Brettin, Thomas and Stevens, Rick and Anandkumar, Anima and Vishwanath, Venkatram and Ramanathan, Arvind (2022) GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20230322-101633000.20
![]() |
PDF
- Submitted Version
Creative Commons Attribution Non-commercial No Derivatives. 5MB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20230322-101633000.20
Abstract
We seek to transform how new and emergent variants of pandemiccausing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pretraining on over 110 million prokaryotic gene sequences and finetuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.
Item Type: | Report or Paper (Discussion Paper) | ||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Related URLs: |
| ||||||||||||||||||||||||||||||
ORCID: |
| ||||||||||||||||||||||||||||||
Additional Information: | The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license. We thank the Argonne Leadership Computing Facility (ALCF) supported by the DOE under DE-AC02-06CH11357 and the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory supported by the DOE under Contract No. DE-AC02-05CH11231. We thank Bill Allcock, Silvio Rizzi and ALCF, Wahid Bhimji and NERSC for their timely help in enabling us to run these jobs at scale. We also thank Defne Gorgun, Lorenzo Casalino and Rommie Amaro for stimulating discussions. This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the US DOE Office of Science and the National Nuclear Security Administration, the National Institute of Allergy and Infectious Diseases, National Institutes of Health Award Number P01AI165077 (AR), the National Science Foundation Award Number 2117896 and supported by the DOE through the National Virtual Biotechnology Laboratory, a consortium of DOE national laboratories focused on response to COVID-19, with funding from the Coronavirus CARES Act. The authors have declared no competing interest. | ||||||||||||||||||||||||||||||
Group: | COVID-19 | ||||||||||||||||||||||||||||||
Funders: |
| ||||||||||||||||||||||||||||||
PubMed Central ID: | PMC9709791 | ||||||||||||||||||||||||||||||
DOI: | 10.1101/2022.10.10.511571 | ||||||||||||||||||||||||||||||
Record Number: | CaltechAUTHORS:20230322-101633000.20 | ||||||||||||||||||||||||||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20230322-101633000.20 | ||||||||||||||||||||||||||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||||||||||||||||||||||||||
ID Code: | 120313 | ||||||||||||||||||||||||||||||
Collection: | CaltechAUTHORS | ||||||||||||||||||||||||||||||
Deposited By: | George Porter | ||||||||||||||||||||||||||||||
Deposited On: | 22 Mar 2023 18:10 | ||||||||||||||||||||||||||||||
Last Modified: | 23 May 2023 21:02 |
Repository Staff Only: item control page