A Caltech Library Service

High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors

Carrington, Laura and Komatitsch, Dimitri and Laurenzano, Michael and Tikir, Mustafa M. and Michéa, David and Snavely, Allan and Tromp, Jeroen (2008) High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors. In: 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE , Piscataway, NJ, pp. 1-11. ISBN 978-1-4244-2834-2.

[img] PDF - Published Version
See Usage Policy.


Use this Persistent URL to link to this item:


SPECFEM3D_GLOBE is a spectral element application enabling the simulation of global seismic wave propagation in 3D anelastic, anisotropic, rotating and self-gravitating Earth models at unprecedented resolution. A fundamental challenge in global seismology is to model the propagation of waves with periods between 1 and 2 seconds, the highest frequency signals that can propagate clear across the Earth. These waves help reveal the 3D structure of the Earth's deep interior and can be compared to seismographic recordings. We broke the 2 second barrier using the 62K processor Ranger system at TACC. Indeed we broke the barrier using just half of Ranger, by reaching a period of 1.84 seconds with sustained 28.7 Tflops on 32K processors. We obtained similar results on the XT4 Franklin system at NERSC and the XT4 Kraken system at University of Tennessee Knoxville, while a similar run on the 28K processor Jaguar system at ORNL, which has better memory bandwidth per processor, sustained 35.7 Tflops (a higher flops rate) with a 1.94 shortest period. Thus we have enabled a powerful new tool for seismic wave simulation, one that operates in the same frequency regimes as nature; in seismology there is no need to pursue periods much smaller because higher frequency signals do not propagate across the entire globe. We employed performance modeling methods to identify performance bottlenecks and worked through issues of parallel I/O and scalability. Improved mesh design and numbering results in excellent load balancing and few cache misses. The primary achievements are not just the scalability and high teraflops number, but a historic step towards understanding the physics and chemistry of the Earth's interior at unprecedented resolution.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Tromp, Jeroen0000-0002-2742-8299
Additional Information:© 2008 IEEE. Some improvements in the package were implemented while D. Komatitsch and D. Michéa were visitors at the Barcelona Supercomputing Center (BSC, Catalonia, Spain) in the context of the HPC-Europa program. The help of Jesús Labarta, Sergi Girona, José Cela, David Vincente, and of their ParaVer analysis tool was invaluable. The authors would also like to thank Rogeli Grima from BSC and Sébastien Deldon from the Portland Group Inc. for fruitful discussion regarding Altivec and SSE instructions. This research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This research used resources of the National Center for Computational Sciences at Oak Ridge National Laboratory (ORNL), which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. We especially thank Don Maxwell and Steve Poole of ORNL for their help and time. This research used resources provided by the National Institute for Computational Sciences, operated by the University of Tennessee and funded by the National Science Foundation. We especially thank Phil Andrews, Victor Hazelwood, Bruce Loftis, and John Walsh of NICS and Arthur Funk of Cray for their help and time. This research used resources of Texas Advanced Computing Center (TACC) at The University of Texas at Austin. We especially thank John R. Boisseau, Tommy Minyard, and Karl W. Schulz of TACC for their help and time. This work was supported in part by Performance Evaluation Research Institute (PERI), (DE-FC02-06ER25760), a DoE Office of Science SciDAC2 Institute, The Cyberinfrastructure Evaluation Center, (NSF-OCI-0516162), and Workshop on Petascale Computing and the Geosciences, (NSF-GEO-0621611). This material is based in part upon research supported by French ANR grant NUMASIS ANR-05-CIGC-002 and European FP6 Marie Curie International Reintegration Grant MIRG-CT-2005-017461.
Funding AgencyGrant Number
Department of Energy (DOE)DE-FC02-06ER25760
Agence Nationale pour la Recherche (ANR)NUMASIS ANR-05-CIGC-002
Marie Curie FellowshipMIRG-CT-2005-017461
Department of Energy (DOE)DE-AC02-05CH11231
Department of Energy (DOE)DE-AC05-00OR22725
Record Number:CaltechAUTHORS:20160812-144047302
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:69608
Deposited On:12 Aug 2016 22:07
Last Modified:11 Nov 2021 04:17

Repository Staff Only: item control page