CaltechAUTHORS
Published October 11, 2022 | Version Submitted
Discussion Paper Open

GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics

Creators

  • 1. ROR icon Argonne National Laboratory
  • 2. ROR icon University of Chicago
  • 3. ROR icon Harvard University
  • 4. ROR icon Northern Illinois University
  • 5. ROR icon New York University
  • 6. ROR icon Technical University of Munich
  • 7. ROR icon Arizona State University
  • 8. ROR icon University of Illinois at Chicago
  • 9. ROR icon California Institute of Technology

Abstract

We seek to transform how new and emergent variants of pandemiccausing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pretraining on over 110 million prokaryotic gene sequences and finetuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.

Additional Information

The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license. We thank the Argonne Leadership Computing Facility (ALCF) supported by the DOE under DE-AC02-06CH11357 and the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory supported by the DOE under Contract No. DE-AC02-05CH11231. We thank Bill Allcock, Silvio Rizzi and ALCF, Wahid Bhimji and NERSC for their timely help in enabling us to run these jobs at scale. We also thank Defne Gorgun, Lorenzo Casalino and Rommie Amaro for stimulating discussions. This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the US DOE Office of Science and the National Nuclear Security Administration, the National Institute of Allergy and Infectious Diseases, National Institutes of Health Award Number P01AI165077 (AR), the National Science Foundation Award Number 2117896 and supported by the DOE through the National Virtual Biotechnology Laboratory, a consortium of DOE national laboratories focused on response to COVID-19, with funding from the Coronavirus CARES Act. The authors have declared no competing interest.

Attached Files

Submitted - 2022.10.10.511571v2.full.pdf

Files

2022.10.10.511571v2.full.pdf

Files (6.0 MB)

Name Size Download all
md5:22d87c8ed27f6695a8e059d6a07429c7
6.0 MB Preview Download

Additional details

Identifiers

PMCID
PMC9709791
Eprint ID
120313
Resolver ID
CaltechAUTHORS:20230322-101633000.20

Funding

Department of Energy (DOE)
DE-AC02-06CH11357
Department of Energy (DOE)
DE-AC02-05CH11231
Department of Energy (DOE)
17-SC-20-SC
NIH
P01AI165077
NSF
DMR-2117896
Coronavirus CARES Act

Dates

Created
2023-03-22
Created from EPrint's datestamp field
Updated
2023-05-23
Created from EPrint's last_modified field

Caltech Custom Metadata

Caltech groups
COVID-19