A Caltech Library Service

A scalable elliptic solver with task-based parallelism for the SpECTRE numerical relativity code

Vu, Nils L. and Pfeiffer, Harald P. and Bonilla, Gabriel S. and Deppe, Nils and Hébert, François and Kidder, Lawrence E. and Lovelace, Geoffrey and Moxon, Jordan and Scheel, Mark A. and Teukolsky, Saul A. and Throwe, William and Wittek, Nikolas A. and Włodarczyk, Tom (2022) A scalable elliptic solver with task-based parallelism for the SpECTRE numerical relativity code. Physical Review D, 105 (8). Art. No. 084027. ISSN 2470-0010. doi:10.1103/physrevd.105.084027.

[img] PDF - Published Version
Creative Commons Attribution.

[img] PDF - Accepted Version
See Usage Policy.

[img] Other (Input-file configurations to reproduce the results presented in the article) - Supplemental Material
See Usage Policy.

[img] Other (Input-file configurations to reproduce the results presented in the article) - Supplemental Material
See Usage Policy.

[img] Other (Input-file configurations to reproduce the results presented in the article) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


Elliptic partial differential equations must be solved numerically for many problems in numerical relativity, such as initial data for every simulation of merging black holes and neutron stars. Existing elliptic solvers can take multiple days to solve these problems at high resolution and when matter is involved, because they are either hard to parallelize or require a large amount of computational resources. Here we present a new solver for linear and nonlinear elliptic problems that is designed to scale with resolution and to parallelize on computing clusters. To achieve this we employ a discontinuous Galerkin discretization, an iterative multigrid-Schwarz preconditioned Newton-Krylov algorithm, and a task-based parallelism paradigm. To accelerate convergence of the elliptic solver we have developed novel subdomain-preconditioning techniques. We find that our multigrid-Schwarz preconditioned elliptic solves achieve iteration counts that are independent of resolution, and our task-based parallel programs scale over 200 million degrees of freedom to at least a few thousand cores. Our new code solves a classic initial data problem for binary black holes faster than the spectral code SpEC when distributed to only eight cores, and in a fraction of the time on more cores. It is publicly accessible in the next-generation SpECTRE numerical relativity code. Our results pave the way for highly parallel elliptic solves in numerical relativity and beyond.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper
Vu, Nils L.0000-0002-5767-3949
Pfeiffer, Harald P.0000-0001-9288-519X
Bonilla, Gabriel S.0000-0003-4502-528X
Deppe, Nils0000-0003-4557-4115
Hébert, François0000-0001-9009-6955
Kidder, Lawrence E.0000-0001-5392-7342
Lovelace, Geoffrey0000-0002-7084-1070
Moxon, Jordan0000-0001-9891-8677
Scheel, Mark A.0000-0001-6656-9134
Teukolsky, Saul A.0000-0001-9765-4526
Throwe, William0000-0001-5059-4378
Wittek, Nikolas A.0000-0001-8575-5450
Additional Information:© 2022 Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Open access publication funded by the Max Planck Society. Received 15 November 2021; accepted 8 February 2022; published 18 April 2022. The authors thank Tim Dietrich, Francois Foucart, and Hannes Rüter for helpful discussions. N. V. also thanks the Cornell Center for Astrophysics and Planetary Science and TAPIR at Caltech for the hospitality and financial support during research stays. Computations were performed with the SpECTRE code [29] on the Minerva cluster at the Max Planck Institute for Gravitational Physics. Charm++ [38] was developed by the Parallel Programming Laboratory in the Department of Computer Science at the University of Illinois at Urbana-Champaign. The figures in this article were produced with dgpy [76], matplotlib [77,78], TikZ [79] and ParaView [80]. This work was supported in part by the Sherman Fairchild Foundation and by NSF Grants No. PHY-2011961, No. PHY-2011968, and No. OAC-1931266 at Caltech, and NSF Grants No. PHY-1912081 and No. OAC-1931280 at Cornell. G. L. is pleased to acknowledge support from the NSF through Grants No. PHY-1654359 and No. AST-1559694 and from Nicholas and Lee Begovich and the Dan Black Family Trust.
Group:TAPIR, Walter Burke Institute for Theoretical Physics
Funding AgencyGrant Number
Max Planck SocietyUNSPECIFIED
Cornell UniversityUNSPECIFIED
Sherman Fairchild FoundationUNSPECIFIED
Nicholas and Lee BegovichUNSPECIFIED
Dan Black Family TrustUNSPECIFIED
Issue or Number:8
Record Number:CaltechAUTHORS:20220511-815710400
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:114680
Deposited By: Tony Diaz
Deposited On:11 May 2022 22:13
Last Modified:28 Apr 2023 20:16

Repository Staff Only: item control page