Contents lists available at
ScienceDirect
Astronomy and Computing
journal homepage:
www.elsevier.com/locate/ascom
Full length article
Processing of GASKAP-H
i
pilot survey data using a commercial
supercomputer
I.P. Kemp
a
,
b
,
∗
,
N.M. Pingel
c
,
R. Worth
d
,
J. Wake
d
,
D.A. Mitchell
e
,
S.D. Midgely
f
,
S.J. Tingay
a
,
J. Dempsey
g
,
h
,
H. Dénes
i
,
J.M. Dickey
j
,
S.J. Gibson
k
,
K.E. Jameson
l
,
C. Lynn
g
,
Y.K. Ma
g
,
A. Marchal
g
,
N.M. McClure-Griffiths
g
,
S. Stanimirović
c
,
J. Th. van Loon
m
a
International Centre for Radio Astronomy Research (ICRAR), Curtin University, Bentley, 6102, WA, Australia
b
CSIRO Space and Astronomy, 26 Dick Perry Avenue, Kensington, 6151, WA, Australia
c
University of Wisconsin-Madison, 4554 Sterling Hall, 475 N. Charter Street, Madison, 53706-1507, WI, USA
d
DUG Technology, 76 Kings Park Road, West Perth, 6005, WA, Australia
e
CSIRO Space and Astronomy, Cnr Vimiera & Pembroke Roads, Marsfield, 2122, NSW, Australia
f
Defence Science and Technology Group, Australian Department of Defence, Australia
g
Research School of Astronomy and Astrophysics, The Australian National University, Canberra, ACT 2611 Australia
h
CSIRO Information Management and Technology, GPO Box 1700 Canberra, ACT 2601, Australia
i
School of Physical Sciences and Nanotechnology, Yachay Tech University, Hacienda San José S/N, 100119, Urcuquí, Ecuador
j
School of Natural Sciences, University of Tasmania, TAS 7005, Australia
k
Physics and Astronomy, Western Kentucky University, Bowling Green, KY, USA
l
Caltech, 1200 E California Blvd, Pasadena, CA 91125, USA
m
Lennard-Jones Laboratories, Keele University, ST5 5BG, UK
A R T I C L E I N F O
Keywords:
Radio astronomy
Computational methods
Supercomputing
Magellanic clouds
Neutral atomic hydrogen HI
A B S T R A C T
Modern radio telescopes generate large amounts of data, with the next generation Very Large Array (ngVLA)
and the Square Kilometre Array (SKA) expected to feed up to 292 GB of visibilities per second to the
science data processor (SDP). However, the continued exponential growth in the power of the world’s largest
supercomputers suggests that for the foreseeable future there will be sufficient capacity available to provide
for astronomers’ needs in processing ‘science ready’ products from the new generation of telescopes, with
commercial platforms becoming an option for overflow capacity. The purpose of the current work is to trial
the use of commercial high performance computing (HPC) for a large scale processing task in astronomy, in
this case processing data from the GASKAP-H
i
pilot surveys. We delineate a four-step process which can be
followed by other researchers wishing to port an existing workflow from a public facility to a commercial
provider. We used the process to provide reference images for an ongoing upgrade to ASKAPSoft (the ASKAP
SDP software), and to provide science images for the GASKAP collaboration, using the joint deconvolution
capability of WSClean. We document the approach to optimising the pipeline to minimise cost and elapsed
time at the commercial provider, and give a resource estimate for processing future full survey data. Finally we
document advantages, disadvantages, and lessons learned from the project, which will aid other researchers
aiming to use commercial supercomputing for radio astronomy imaging. We found the key advantage to be
immediate access and high availability, and the main disadvantage to be the need for improved HPC knowledge
to take best advantage of the facility.
1.
Introduction
Modern
radio
interferometer
arrays
generate
large
amounts
of
data.
The
Science
Data
Processor
(SDP)
for
the
36-dish
ASKAP
telescope
was
designed
to
process
3
GB
per
second
of
visibility
data
(
Guzman
and
Marquarding
,
2019
).
The
ngVLA,
the
planned
next
generation
VLA,
is
∗
Corresponding
author
at:
International
Centre
for
Radio
Astronomy
Research
(ICRAR),
Curtin
University,
Bentley,
6102,
WA,
Australia.
E-mail
address:
ian.kemp@postgrad.curtin.edu.au
(I.P.
Kemp)
.
expected
to
produce
a
peak
rate
of
133
GB
per
second
(
Hiliart
,
2019
),
while
SKA-Mid,
currently
under
construction
(
SKA
,
2022
),
will
produce
292
GB
per
second
(
McMullin
et
al.
,
2022
).
Because
of
these
large
data
volumes,
radio
astronomy
is
critically
dependent
upon
high
performance
computing
(HPC)
to
carry
out
imag-
ing
and
other
analysis
to
extract
useful
science,
such
as
for
example
https://doi.org/10.1016/j.ascom.2024.100901
Received
30
August
2024;
Accepted
24
November
2024
Astronomy
and
Computing
51
(2025)
100901
Available
online
30
November
2024
2213-1337/©
2024
The
Authors.
Published
by
Elsevier
B.V.
This
is
an
open
access
article
under
the
CC
BY
license
(
http://creativecommons.org/licenses/by/4.0/
).
I.P.
Kemp
et
al.
the
calibrated
and
mosaiced
image
cubes
for
the
WALLABY
survey
(
Koribalski
et
al.
,
2020
),
the
atomic
neutral
hydrogen
absorption
mea-
surements
described
in
Dempsey
et
al.
(
2022
),
and
the
de-dispersed
real-time
FRB
search
of
the
CRAFT
survey
(
Shannon
et
al.
,
2024
).
In
2011,
Norris
(
2011
)
outlined
the
SKA
Data
Challenge,
and
es-
timated
that
in
2022,
SKA
data
processing
would
require
computing
infrastructure
beyond
the
capacity
of
the
largest
supercomputer
in
the
world.
In
reality,
Norris’
concerns
have
been
mitigated
by
delays
to
the
start
of
SKA
construction.
Now,
13
years
later,
computing
availability
has
already
fulfilled
the
needs
of
the
SKA.
This
was
confirmed
by
Wang
et
al.
(
2020
)
who
successfully
carried
out
processing
of
simulated
SKA
data
in
better
than
real-time,
using
Summit,
then
identified
in
the
Top500
list
of
supercomputing
systems
(
Top500.org
,
2024
)
as
the
world’s
most
performant
computer,
with
a
maximal
achievable
benchmark
performance
(
푅
푚푎푥
)
of
149
PFlop/s.
In
2024,
just
five
years
after
that
work
was
carried
out,
Summit
has
been
decommissioned,
and
the
current
top
listed
system,
El
Capitan,
is
rated
at
푅
푚푎푥
=
1742
PFlop/s,
11
times
the
power
of
Summit.
Current
plans
for
the
SKA
are
to
carry
out
a
number
of
standardised
workflows
at
the
SDP,
with
this
central
facility
providing
a
range
of
science
ready
products
including
image
cubes
and
spectra
(
Nikolic
and
SDP
Consortium,
SKA
,
2014
).
Specialist
or
experimental
processing
by
other
users
of
the
data
will
be
carried
out
using
‘science
regional
centres’
(SRCs).
The
data
post-processing
system
built
for
the
Australian
SRC
has
been
designed
to
be
hardware
agnostic
(
Lee-Wadell
et
al.
,
2022
),
so
that
in
the
future
it
could
be
implemented
using
dedicated
facilities,
national
computing
infrastructure,
or
commercially
provided
HPC
or
cloud
facilities.
This
approach
has
been
foreshadowed
in
the
processing
of
data
from
the
Australian
SKA
Pathfinder
(ASKAP)
(
Schinckel
et
al.
,
2012
;
Hotan
et
al.
,
2021
).
Each
of
the
36
ASKAP
antennas
are
equipped
with
188-
element
phase
array
feed
receivers,
enabling
a
∼
25
deg
2
field-of-view
through
beamforming
36
simultaneous
and
separate
primary
beams,
which
each
extend
one
degree
on
the
sky
at
21
cm.
The
philosophy
at
ASKAP
is
to
produce
and
provide
‘science-ready’
data
products,
includ-
ing
calibrated
visibilities
and
images,
using
specific
software
named
ASKAPSoft
(
Wieringa
et
al.
,
2020
).
The
SDP
is
implemented
on
Setonix,
which
is
currently
the
highest
performance
public
HPC
system
in
Aus-
tralia
(positioned
28
on
the
Top500
list),
with
an
푅
푚푎푥
of
27
PFlop/s,
located
at
the
Pawsey
Supercomputing
Research
Centre
in
Perth.
1
The
data
rate
from
the
observatory
ranges
up
to
2.5
GB/s
(
ATNF
,
2024
),
depending
on
whether
ASKAP
is
observing
in
the
288
MHz
wide
spectral-band
continuum
mode
or
its
various
zoom-modes
for
spectral
line
science.
These
large
data
volumes
from
ASKAP
present
a
unique
opportunity
to
explore
how
to
efficiently
utilise
the
computing
resources
available
to
the
astronomical
community,
in
preparation
for
future
facilities.
One
of
the
science
survey
projects
approved
for
the
ASKAP
telescope
is
GASKAP-H
i
,
a
survey
to
study
the
distribution
and
thermodynamic
properties
of
atomic
neutral
hydrogen
(H
i
)
in
the
Milky
Way
and
nearby
Magellanic
System,
at
high
angular
and
spectral
resolution
(
Dickey
et
al.
,
2013
).
Specific
science
goals
of
this
survey
include
characterising
interstellar
turbulence
in
this
key
component
of
the
interstellar
medium
and
revealing
the
phase
transitions
between
the
multi-phase
atomic
gas
and
star-forming
molecular
gas.
In
preparation
for
the
main
survey,
shallow
(10
−
20
h)
Phase
I
and
Phase
II
pilot
surveys
–
capturing
the
Small
Magellanic
Cloud
(SMC),
Large
Magellanic
Cloud
(LMC),
Magellanic
Bridge,
and
the
start
of
the
Magellanic
Stream
–
have
been
carried
out
to
develop
the
imaging
and
analysis
tools,
and
data
from
the
pilots
have
already
been
used
to
provide
useful
scientific
outputs
in
line
with
the
survey
aims
(see
Pingel
et
al.
,
2022
;
Dickey
et
al.
,
2022
;
Dempsey
et
al.
,
2022
;
Nguyen
et
al.
,
2024
).
1
https://pawsey.org.au/
.
For
GASKAP-H
i
,
the
novel
phased
array
feed
(PAF)
receivers
on
ASKAP
expand
the
instantaneous
field-of-view
of
the
telescope
from
1
◦
×
1
◦
expected
for
a
12
m
dish
to
∼
5
◦
×
5
◦
through
forming
36
simul-
taneous
primary
beams
on
the
sky.
The
standard
approach
to
imaging
a
single
ASKAP
PAF
footprint
is
a
linear
mosaic,
wherein
distinct
deconvolved
images,
(i.e.,
with
the
inherent
PSF
response
from
the
sky-
brightness
distribution
removed),
produced
from
each
individual
beam,
are
combined
in
a
pointed
mosaic.
However,
in
the
case
of
GASKAP-
H
i
,
diffuse
emission
from
the
Milky
Way
and
nearby
Magellanic
system
extend
far
beyond
the
boundaries
of
individual
beams.
Because
com-
mon
deconvolution
algorithms,
such
as
CLEAN
(
Högbom
,
1974
),
are
non-linear
processes,
we
risk
losing
information
about
the
largest
scales
due
to
variations
in
the
beam-to-beam
sky
models.
To
ensure
the
largest
scale
emission
is
accurately
recovered
across
beam
boundaries,
a
joint
deconvolution
approach
is
necessary
—
in
which
the
images
from
each
beam
are
stitched
together
in
a
single
image
before
moving
on
to
deconvolution.
However,
this
imaging
approach
is
computationally
intensive,
so
that
Pawsey’s
Galaxy
supercomputer,
the
predecessor
to
Setonix,
did
not
possess
sufficient
node
memory,
hampering
efforts
to
develop
this
imaging
mode
in
ASKAPSoft.
Fortunately,
the
increased
capabilities
of
Setonix
has
enabled
development
of
joint
deconvolution
in
ASKAPSoft.
Pending
the
enhancements
to
ASKAPSoft,
Pingel
et
al.
(
2022
)
devel-
oped
an
imaging
pipeline
for
GASKAP-H
i
based
on
the
command-line
imaging
software
WSClean
(
Offringa
et
al.
,
2014
).
Due
to
the
limited
availability
of
systems
with
suitable
memory
to
process
this
data,
it
is
worthwhile
to
identify
alternative
computing
resources.
Of
interest
here
is
the
commercial
sector,
which
has
had
a
steadily
increasing
presence
in
the
afore-mentioned
Top500
list
of
supercomputing
systems.
The
November
2024
Top500
list
showed
that
ten
of
the
top
50
publicly-
disclosed
systems
were
owned
by
commercial
providers.
Five
years
ago
the
corresponding
number
was
zero
—
in
June
2019
only
one
machine
in
the
top
20
was
owned
by
a
private
corporation,
and
this
was
an
in-
house
system
for
an
oil
company
(Total)
and
not
available
to
the
public
research
community.
In
the
future
it
may
be
increasingly
possible
for
commercial
supercomputing
to
provide
additional
on-demand
capacity
for
radio
astronomy
data
processing.
The
aim
of
the
present
work
is
to
trial
the
use
of
commercial
supercomputing,
by
processing
data
from
the
GASKAP-H
i
pilot
surveys
using
a
supercomputer
owned
by
DUG
Technology
in
Perth,
Western
Australia.
2
The
work
was
part
of
a
larger
trial
facilitated
by
CSIRO,
3
in
which
teams
from
multiple
disciplines
were
given
access
to
DUG
facilities,
to
evaluate
the
applicability
of
commercial
technology.
We
used
DUG’s
commercial
offering,
which
includes
a
high
level
of
tech-
nical
support,
in
assisting
with
configuration
and
optimisation
of
the
deployment
of
our
specialist
software.
Our
trial
had
four
sub-objectives:
1.
Identify
and
document
the
feasibility,
advantages
and
disad-
vantages
of
using
a
commercial
facility
for
data
processing;
2.
Provide
a
resource
estimate
for
processing
full
survey
data
using
commercial
supercomputing;
3.
Use
the
joint
deconvolution
technique
with
WSClean
to
produce
reference
images
for
the
updated
joint
deconvolution
algorithm
in
ASKAPSoft;
and
4.
Produce
fully
processed
cubes
from
the
pilot
survey
data,
for
use
by
the
science
team.
We
wish
to
use
our
experience
to
inform
the
community,
so
that
astronomers
may
be
better
placed
to
take
advantage
of
the
large
quantities
of
commercial
HPC
which
we
expect
to
enter
the
market
in
the
next
decade.
Our
experience
may
also
help
inform
the
discussion
2
https://dug.com/about-dug/
.
3
https://www.csiro.au/
.
Astronomy
and
Computing
51
(2025)
100901
2
I.P.
Kemp
et
al.
Fig.
1.
Fields
in
the
GASKAP-H
i
Pilot
Surveys.
Cyan
boxes
denote
the
approximate
ASKAP
PAF
footprint
for
the
pilot
phase
I
fields
(20
h
integration);
yellow
boxes
are
those
for
the
pilot
phase
II
fields
(10h
integration).
about
the
prospects
for
incorporation
of
commercial
supercomputing
in
the
model
for
the
SKA
regional
centres.
We
adapted
the
existing
imaging
workflow
which
has
been
de-
scribed
in
Pingel
et
al.
(
2022
),
and
in
this
paper
we
will
focus
on
the
issues
around
porting
the
workflow
and
use
of
commercial
computing
facilities.
The
structure
of
this
paper
is
as
follows:
in
Section
2
we
record
the
observations
used;
in
Sections
3
and
4
we
describe
the
computational
infrastructure
and
software
respectively;
and
in
Section
5
we
describe
the
overall
processing
steps.
In
Section
6
we
explain
how
the
workflow
was
evolved
to
make
optimum
use
of
the
commercial
facility;
in
Sec-
tion
7
we
summarise
our
results;
and
in
Section
8
we
detail
the
lessons
learned
from
the
process.
2.
Observations
and
data
We
used
calibrated
visibilities
from
the
GASKAP-H
i
Phase
I
and
Phase
II
pilot
surveys
performed
with
the
ASKAP
telescope.
Primary
beams
obtained
using
the
holography
methods
outlined
by
Hotan
et
al.
(
2021
)
at
1.4
GHz
were
used
as
the
‘beam_map’
input
for
the
joint
deconvolution
mode
in
WSClean.
Processing
resulted
in
images
with
30
′′
synthesised
beams
with
rms
noise
of
∼
3
K
per
0.24
km
s
−
1
channel.
Observations
were
obtained
using
3
interleaved
positions,
to
provide
uniform
sensitivity
data
across
108
individual
beam
pointings
for
each
field.
Data
were
collected
in
the
spectral
window
1418.501
MHz–1420.944
MHz,
in
2112
channels
of
width
1.157
kHz.
The
pilot
I
survey
consisted
of
20
h
integrations
spread
across
three
fields
centred
on
the
Small
Magellanic
Cloud
(SMC),
a
portion
of
the
Magellanic
Bridge,
and
the
beginning
of
the
Magellanic
Stream
extending
from
the
SMC.
The
pilot
II
survey
consisted
of
10-hour
observations
across
9
fields
across
the
Magellanic
system
(see
Fig.
1
).
In
total,
each
integration
generated
108
×
61
GB,
∼
6600
GB
of
visibilities.
These
visibilities
had
been
calibrated
with
the
standard
ASKAPSoft
pipeline,
as
described
in
Section
3
of
Pingel
et
al.
(
2022
).
In
the
work
reported
here,
each
field
was
imaged
and
deconvolved
using
the
joint
deconvolution
mode
of
the
WSClean
software
package
(
Offringa
et
al.
,
2014
)
to
produce
image
cubes
of
up
to
150
GB
in
size.
The
primary
repository
for
ASKAP
data
is
the
CSIRO
ASKAP
Science
Data
Archive
(CASDA)
(
Huynh
et
al.
,
2020
).
In
CASDA,
the
data
are
located
and
referred
to
using
the
Scheduling
Block
IDs
(SBID).
In
this
work
we
used
data
from
SBID
10941
from
pilot
phase
I,
and
SBIDs
38509,
38466,
38215
from
pilot
phase
II
(see
Fig.
1
).
3.
Hardware
The
DUG
infrastructure
provided
us
with
essentially
two
tiers
of
compute
capability.
>
2000
nodes
were
provided
based
on
Intel
Xeon
Phi
‘Knights
Landing’
processors
with
64
cores
and
128
GB
RAM,
billed
at
A$0.35
per
node-hour.
Also
available
were
eight
nodes
based
on
dual
socket
Intel
‘Cascade
Lake’
processors,
each
with
24
cores
and
192
GB
RAM,
and
eight
nodes
based
on
dual
socket
Intel
‘Ice
Lake’
processors,
each
with
16
cores
and
1
TB
RAM.
These
higher-performance
nodes
were
billed
at
A$2.45
and
A$2.65
per
node-hour
respectively.
In
this
paper,
these
three
node
types
are
referred
to
as
‘KNL’,
‘CLX’
and
‘ILX’.
We
use
the
term
‘sockets’
to
refer
to
the
individual
processors
within
the
node.
With
the
dual-socket
nodes,
the
default
behaviour
of
the
Linux
kernel
is
to
spread
active
processes
across
both
sockets,
and
it
may
migrate
active
processes
between
sockets
to
balance
load.
Although
half
the
node
memory
is
attached
to
each
socket,
each
has
access
to
the
full
node
memory.
However
there
is
a
lower
latency
if
sockets
are
‘pinned’
so
that
the
socket
uses
only
its
attached
memory.
This
detail
is
included
as
background
to
our
later
discussion
on
performance
improvement.
The
file
system
at
DUG
is
based
on
VAST,
4
a
scalable
SSD-based
storage
accessed
via
NFS.
At
the
time
of
this
work,
DUG’s
Perth
deploy-
ment
was
rated
at
90
GB/s
read
and
15
GB/s
write.
Compute
nodes
are
connected
via
NVIDIA
Mellanox
multi-host
NICs,
with
one
card
connecting
four
nodes
to
the
network.
The
total
bandwidth
of
50
Gb/s
can
be
rate
limiting
if
multiple
nodes
access
the
network
concurrently.
4.
Software
The
key
software
tools
for
producing
image
cubes
from
visibility
data
were
WSClean
version
3.0
(
Offringa
et
al.
,
2014
)
(for
imaging)
and
CASA
version
6.1.2.7
(
CASA
Team
et
al.
,
2022
)
for
most
data
manipulation
and
arithmetic
operations.
CASA
tasks
were
scripted
in
Python,
for
execution
in
CASA’s
IPython
application
environment.
WSClean
and
CASA
had
been
installed
by
DUG’s
HPC
team
prior
to
the
project,
for
use
in
an
earlier
activity.
The
original
implementation
of
the
workflow
at
the
Australian
National
University
(ANU)
used
MIRIAD
(
Sault
et
al.
,
1995
)
for
one
task
(Feathering).
To
save
time
required
to
resolve
difficulties
installing
a
consistent
set
of
dependen-
cies
for
this
package,
the
task
which
relied
on
MIRIAD
was
allocated
to
CASA
instead,
and
MIRIAD
was
not
used.
Non-interactive
jobs
were
controlled
using
RJS,
which
is
DUG’s
proprietary
wrapper
for
sbatch,
which
in
turn
is
part
of
the
underlying
scheduler
SLURM
(we
used
version
23.02.07).
RJS
provides
a
simple
method
for
jobs
to
be
executed
in
parallel,
by
supplying
values
for
variables
in
an
external
schema
file.
The
operating
system
kernel
was
CentOS
Linux
release
7.7.1908,
and
RJS
jobs
were
submitted
manually
via
ssh
access.
An
example
set
of
the
RJS
scripts
and
schema
files,
and
iPython
scripts,
used
to
implement
the
workflow
are
available
online
(
Kemp
et
al.
,
2024
).
During
our
project,
workflow
steps
were
initiated
manually,
gener-
ally
with
the
output
of
one
step
being
checked
before
starting
the
next.
Orchestration
using
a
workflow
manager
was
left
to
be
the
first
stage
of
full
scale
deployment
in
the
event
that
the
DUG
platform
was
to
be
used
after
our
trial
was
complete.
4
https://www.vastdata.com/
.
Astronomy
and
Computing
51
(2025)
100901
3
I.P.
Kemp
et
al.
Table
1
Summary
of
imaging
workflow
(More
detail
is
given
in
Pingel
et
al.
(
2022
)).
Note:
9
and
13
were
not
used
in
the
production
workflow.
Step
Name
Description
Job
Type
01
Download
Transfer
visibility
files
from
CASDA
to
DUG
Single
02
Bin
Combine
(average)
frequency
channels
Parallel
03
Listobs
Record
basic
observation
metadata
Manual
04
Rotate
Ph.
Centre
Move
all
interleaves
to
common
phase
centre
Parallel
05
Cont.
Sub.
Suppress
continuum
sources
Parallel
06
Split
Channel
Split
visibility
files
1
per
channel
Parallel
07
Make
Clean
Mask
Create
mask
where
PB
<
20%
Manual
08
Imaging
Create
image
&
PB
for
each
channel
Parallel
10
Collect
images
Consolidate
images
and
PB
Single
11
Import
to
CASA
Convert
from
FITS
to
CASA
image
format
Single
12
Update
Headers
Correct
header
metadata
in
CASA
images
Single
14
Concatenate
Form
image
cube
&
PB
cube
Single
15
Normalise
PB
Ensure
max
value
in
PB
<
1.0
Single
16
PB
correction
Divide
image
cube
by
PB
cube
Single
17
Get
Parkes
Cube
Download
prior
single
dish
cube
Manual
18
Dropdeg
Ensure
ASKAP
&
Parkes
cubes
compatible
Single
19
Update
Headers
Ensure
ASKAP
&
Parkes
cubes
compatible
Single
20
Smooth
Apply
smoothing
to
the
ASKAP
cube
Single
21
Feather
Combine
ASKAP
&
Parkes
images
Single
5.
Workflow
Key
features
of
the
workflow
for
GASKAP-H
i
imaging
is
described
in
Pingel
et
al.
(
2022
).
The
workflow
had
been
optimised
to
run
on
the
AVATAR
cluster
at
the
Research
School
of
Astronomy
and
Astrophysics
at
the
Australian
National
University
(ANU),
and
a
detailed
description
of
the
code,
along
with
the
logic
of
each
step,
is
given
in
the
imaging
guide
(
Pingel
and
Ma
,
2024
).
Observations
were
processed
one
field
at
a
time.
In
the
current
work,
initial
establishment
of
the
workflow
was
carried
out
using
SBID
38215,
which
had
previously
been
processed
using
ANU
facilities.
This
was
done
to
establish
confidence
in
the
modified
workflow,
by
comparison
of
the
resulting
images
with
those
previously
obtained.
A
summary
of
the
workflow
steps
used
at
DUG
is
listed
in
Table
1
.
As
indicated
by
the
rightward
column,
the
process
uses
some
manual
steps
which
were
run
interactively;
some
single
jobs;
and
some
large
jobs
which
were
parallelised,
and
run
concurrently
over
the
largest
number
of
nodes
which
were
available.
A
graphical
representation
of
the
workflow
is
available
as
Fig.
2
in
Pingel
et
al.
(
2022
).
Some
of
the
notable
features
of
the
workflow,
which
will
be
referred
to
later,
are:
•
‘Download’:
Due
to
cost
of
storage
it
was
not
feasible
to
replicate
all
the
visibility
data
on
the
commercial
platform,
nor
was
this
felt
a
desirable
data
management
practice.
Therefore
a
process
step
was
required
to
download
data
from
the
repository
to
the
working
machine
immediately
prior
to
processing;
•
‘Rotate
Phase
Centre’:
In
this
step
visibility
data
from
the
108
beams
are
phase-rotated
to
a
common
reference
coordinate,
which
defines
the
centre
of
the
image.
This
is
a
requirement
for
the
joint
deconvolution
mode
in
WSClean;
•
‘Split
Channel’:
To
reduce
memory
demands
and
processing
time
during
imaging,
visibility
data
are
pre-split
into
individual
chan-
nels
for
each
of
the
108
pointings.
Thus,
we
image
each
spectral
channel
separately
and
in
parallel.
These
images
are
concatenated
into
a
single
cube
in
a
later
step;
•
‘Feather’:
The
physical
limitations
on
spacing
of
the
individual
elements
of
radio
interferometers
limits
the
sensitivity
to
low
spatial
frequencies.
For
ASKAP,
the
shortest
22
m
baselines
filter
out
structures
that
span
>
30
′
on
the
sky,
eliminating
sensitivity
to
large
angular
scales.
These
missing
baselines
are
commonly
referred
to
as
the
missing
short-spacings.
To
compensate
for
these,
image
cubes
from
ASKAP
are
combined
with
images
from
the
GASS
survey
obtained
with
Murriyang,
the
single-dish
Parkes
telescope
(
McClure-Griffiths
et
al.
,
2009
).
The
elapsed
time
for
the
download
of
the
relevant
cube
from
the
Parkes
data
repository
(workflow
step
17)
was
trivial
and
for
this
reason
we
did
not
attempt
to
automate
it.
6.
Methods
and
evolution
of
imaging
workflow
Our
project
contained
four
distinct
stages:
1.
The
working
code
and
workflow
were
ported
from
a
working
installation
with
minimal
changes
required
to
run
successfully;
2.
Experimentation
with
WSClean
parameters
to
obtain
satisfactory
images;
3.
Optimisation
of
deployment
of
the
code
to
the
compute
nodes
to
improve
performance
—
referred
to
as
‘Machine
optimisation’;
4.
Optimisation
of
WSClean
parameters
to
further
improve
perfor-
mance
—
referred
to
as
‘software
optimisation’.
These
stages
are
now
discussed
in
more
detail.
6.1.
Port
of
the
existing
workflow
We
ported
the
working
code
from
the
ANU
implementation,
with
changes
to
meet
updated
science
goals
and
to
take
best
advantage
of
the
DUG
platform.
Initial
trials
were
conducted
using
data
from
SBID
38215.
For
the
current
work
we
used
the
full
spectral
range
and
resolution,
of
2112
channels
of
1.157
kHz,
and
an
image
size
of
4096x4096
7-arcsec
pixels.
This
was
a
significant
increase
in
computing
load
compared
to
previous
work,
which
had
used
4-channel
binning,
giving
a
total
of
528
channels
of
width
4.628
kHz.
When
implementing
the
workflow
at
DUG,
some
technical
issues
emerged
which
will
be
of
interest
to
researchers
attempting
to
replicate
our
processing
on
other
platforms:
•
The
first
issue
was
that
it
was
convenient
to
use
DUG’s
scheduling
tool
RJS.
This
improvement
required
all
the
job
control
scripts
to
be
modified
to
use
RJS
directives
in
place
of
the
Portable
Batch
System
commands
used
at
the
ANU;
•
In
the
original
workflow,
data
were
binned
into
4-channel
bins.
For
improved
science
output
and
because
of
the
additional
pro-
cessing
capacity
available,
we
opted
to
process
the
data
at
the
full
native
spectral
resolution
of
1.157
kHz,
so
no
binning
was
required.
Omission
of
the
binning
step
led
to
failure
of
the
phase
centre
rotation
step.
This
was
attributed
to
the
‘tiling’
scheme
in
the
measurement
set;
to
solve
the
problem
we
adopted
the
simple
expedient
of
reinstating
the
‘bin’
operation,
but
with
width
1.
This
caused
the
observation
data
to
be
re-tiled
at
a
small
compute
cost
and
further
processing
was
successful.
Astronomy
and
Computing
51
(2025)
100901
4
I.P.
Kemp
et
al.
Fig.
2.
Comparison
of
dirty
images
from
ASKAPSoft
(top
row)
and
WSClean
(bottom
row)
with
different
Briggs
Weighting
values.
Images
are
from
SBID
10941,
single
beam
21
A,
Channel
760,
centred
at
1419.380
MHz.
6.2.
WSClean
parameters
To
provide
reference
images
for
the
development
of
the
joint
decon-
volution
mode
in
ASKAPSoft,
we
produced
some
cleaned
single
beam
images
as
well
as
joint
deconvolved
(multibeam)
images,
in
order
to
separate
the
effects
of
cleaning
and
mosaicing.
Single
beam
images
were
made
using
beam
21
A
of
SBID
10941,
centred
on
the
SMC.
These
used
a
single
channel
(Channel
760
centred
on
1419.380
MHz
in
the
LSRK
spectral
reference
frame),
with
an
image
size
of
1024
×
1024
7-
arcsec
pixels.
These
images
were
fully
contained
within
the
area
of
the
ASKAP
footprint
(i.e.,
the
combined
primary
beam
map).
The
key
parameters
affecting
image
quality
were
found
to
be
the
clean
threshold
and
the
Briggs
robustness
value,
which
weights
the
gridded
visibilities
to
provide
a
compromise
between
noise
and
the
final
angular
resolution.
After
some
experimentation,
it
was
found
that
optimal
images
were
obtained
with
a
Briggs
weighting
around
2.0.
A
comparison
for
3
values
of
the
Briggs
weighting
is
given
for
the
single-
beam
images
in
Fig.
2
,
showing
good
qualitative
agreement
between
the
two
imagers.
In
both
cases,
the
sidelobe
structure
becomes
more
apparent
as
the
weighting
transitions
from
uniform
(
−2
.
0
)
to
natural
(2.0),
wherein
structure
from
the
large
amounts
of
baselines
below
2
km
begin
to
dominant
the
image.
Other
WSClean
parameters
used
are
listed
in
the
first
column
of
Table
2
.
For
joint
deconvolved
images
of
the
full
field,
the
image
size
was
initially
increased
to
5000
×
5000
7-arcsec
pixels
(later
rounded
to
4096
×
4096).
This
included
an
area
outside
that
covered
by
the
joint
primary
beam
map,
and
noise
in
this
outer
zone
led
to
rapidly
diverging
clean
operations.
We
opted
to
deal
with
this
by
providing
a
clean
mask
to
exclude
parts
of
the
image
where
the
joint
beam
response
was
less
than
15%
of
the
peak.
Satisfactory
image
cubes
were
obtained
with
a
clean
threshold
of
21
mJy
and
a
Briggs
weighting
of
1.25.
A
portion
of
the
cleaned
image
cube
at
channel
798
centred
on
1419.424
MHz
is
shown
in
Fig.
3
.
The
image
includes
the
SMC
Bar
(rightward
of
RA
=
15
◦
)
and
Wing
(leftward
of
RA
=
15
◦
),
and
resolves
linear
plumes
at
the
upper
left,
and
fine
filaments
(for
example
centred
on
RA
=
15.5
◦
,
decl
=
−
72.4
◦
).
For
a
further
discussion
of
features
in
the
image,
please
refer
to
the
description
of
Figure
12
of
Pingel
et
al.
(
2022
).
WSClean
parameters
used
to
produce
our
Fig.
3
are
listed
in
the
second
column
of
Table
2
.
Negative
intensities
in
the
image
are
due
to
the
lack
of
a
total
power
measurement,
and
are
mitigated
by
feathering
with
data
from
the
Parkes
single
dish
telescope,
at
a
later
stage
in
the
workflow.
For
this
first
successful
production
imaging,
processing
time
and
cost
were
dominated
by
the
imaging
step.
Using
CLX
nodes,
imaging
consumed
2.9
h
per
channel,
giving
a
total
of
3,061
node-hours
for
the
full
2112-channel
cube
(commercial
cost
∼
A$7,800).
Within
this,
inversion
was
by
far
the
most
expensive
process
(ie.
computation
of
the
image
by
Fourier
Transform
of
gridded
visibility
data),
consuming
typically
2.1
h
per
channel.
The
next
most
significant
cost
was
the
‘Split
Channel’
step
in
which
separate
measurement
sets
were
created
for
each
beam
and
each
channel.
Implemented
on
CLX
nodes,
this
consumed
263
node
hours
(commercial
cost
∼
A$670).
6.3.
Machine
optimisation
Having
established
a
successful
run
of
the
workflow,
with
satisfac-
tory
imaging,
our
next
step
was
to
tune
the
processes
to
the
machine,
to
reduce
processing
time
and
cost.
This
was
done
by
focusing
on
three
areas,
as
follows:
6.3.1.
Data
transfer
An
issue
with
using
the
commercial
platform
was
the
need
to
transfer
data
from
the
system
of
record
for
processing.
To
improve
the
elapsed
time
and
reduce
manual
intervention,
a
process
was
developed
based
on
‘drip
feed’
of
the
required
visibility
files
for
a
given
observa-
tion,
with
automated
restart
and
throttled
to
avoid
excessive
load
on
CASDA.
This
process
achieved
a
download
time
of
6.2
h
for
the
full
data
set
for
a
∼
10
h
observation
(7
TB),
without
manual
intervention.
Astronomy
and
Computing
51
(2025)
100901
5