SpECTRE: A task-based discontinuous Galerkin code for relativistic astrophysics
We introduce a new relativistic astrophysics code, SpECTRE, that combines a discontinuous Galerkin method with a task-based parallelism model. SpECTRE's goal is to achieve more accurate solutions for challenging relativistic astrophysics problems such as core-collapse supernovae and binary neutron star mergers. The robustness of the discontinuous Galerkin method allows for the use of high-resolution shock capturing methods in regions where (relativistic) shocks are found, while exploiting high-order accuracy in smooth regions. A task-based parallelism model allows efficient use of the largest supercomputers for problems with a heterogeneous workload over disparate spatial and temporal scales. We argue that the locality and algorithmic structure of discontinuous Galerkin methods will exhibit good scalability within a task-based parallelism framework. We demonstrate the code on a wide variety of challenging benchmark problems in (non)-relativistic (magneto)-hydrodynamics. We demonstrate the code's scalability including its strong scaling on the NCSA Blue Waters supercomputer up to the machine's full capacity of 22,380 nodes using 671,400 threads.
Additional Information© 2017 Elsevier Inc. Received 5 September 2016, Revised 21 December 2016, Accepted 29 December 2016, Available online 5 January 2017. We acknowledge helpful discussions with Sebastiano Bernuzzi, Bernd Brügmann, Marcus Bugner, Mani Chandra, Roland Haas, Daniel Hemberger, Jan Hesthaven, Cameron Hummels, Fraser Hutchison, Jonathan Lifflander, Geoffrey Lovelace, Phil Miller, Harald Pfeiffer, David Radice, and Christian Reisswig. We thank the anonymous referee for detailed comments and suggestions which helped to improve the manuscript. Support for F. Foucart was provided by NASA through Einstein Postdoctoral Fellowship grant numbered PF4-150122 awarded by the Chandra X-ray Center, which is operated by the Smithsonian Astrophysical Observatory for NASA under contract NAS8-03060. Authors at Cornell were partially supported by NSF under award Nos. TCAN AST-1333129 and PHY-1306125, and by the Sherman Fairchild Foundation. Authors at Caltech were partially supported by NSF under award Nos. TCAN AST-1333520, CAREER PHY-1151197, and PHY-1404569, and by the Sherman Fairchild Foundation. ES acknowledges support from NSF award OCI-0905046. ES and JM were supported by an NSERC Discovery grant. Research at the Perimeter Institute is supported by the Government of Canada through Industry Canada and by the Province of Ontario through the Ministry of Research & Innovation. Computations were performed on NSF/NCSA Blue Waters under allocation PRAC ACI-1440083, on the NSF XSEDE network under allocations TG-PHY100033 and TG-PHY990007, and on the Caltech compute cluster Zwicky (NSF MRI-R2 award No. PHY-0960291).
Submitted - 1609.00098.pdf