of 16
36
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 11, 2025
Exoplanet Imaging via Differentiable Rendering
Brandon Y. Feng
, Rodrigo Ferrer-Chávez
, Aviad Levis
, Jason J. Wang
, Katherine L. Bouman
,
and William T. Freeman
, Life Fellow, IEEE
Abstract
—Direct imaging of exoplanets is crucial for advancing
our understanding of planetary systems beyond our solar system,
but it faces significant challenges due to the high contrast be-
tween host stars and their planets. Wavefront aberrations introduce
speckles in the telescope science images, which are patterns of
diffracted starlight that can mimic the appearance of planets,
complicating the detection of faint exoplanet signals. Traditional
post-processing methods, operating primarily in the image inten-
sity domain, do not integrate wavefront sensing data. These data,
measured mainly for adaptive optics corrections, have been over-
looked as a potential resource for post-processing, partly due to the
challenge of the evolving nature of wavefront aberrations. In this
paper, we present a differentiable rendering approach that lever-
ages these wavefront sensing data to improve exoplanet detection.
Our differentiable renderer models wave-based light propagation
through a coronagraphic telescope system, allowing gradient-based
optimization to significantly improve starlight subtraction and in-
crease sensitivity to faint exoplanets. Simulation experiments based
on the James Webb Space Telescope configuration demonstrate
the effectiveness of our approach, achieving substantial improve-
ments in contrast and planet detection limits. Our results show-
case how the computational advancements enabled by differen-
tiable rendering can revitalize previously underexploited wavefront
data, opening new avenues for enhancing exoplanet imaging and
characterization.
Index Terms
—Exoplanet imaging, high-contrast imaging, diffe-
rentiable rendering, wavefront aberration estimation.
I. I
NTRODUCTION
E
XOPLANETS, or extrasolar planets, are planets that orbit
stars outside our solar system. Studying exoplanets is
Received 7 August 2024; revised 24 November 2024; accepted 25 December
2024. Date of publication 3 January 2025; date of current version 15 January
2025. This work was supported in part by NSF under Award 2019786, in part by
the NSF AI Institute for Artificial Intelligence and Fundamental Interactions, in
part by CIF under Award 1955864, and in part by the Occlusion and Directional
Resolution in Computational Imaging. The work of Aviad Levis was supported
in part by the NSERC-Discovery Grant. The work of Jason J. Wang and Rodrigo
Ferrer-Chávez was supported by STScI under Grant JWST-ERS-01386 and
Grant JWST-GO-04050, in part by NASA under Grant NAS5-03127. The
work of Katherine L. Bouman was supported in part by the NSF CAREER
under Award 2048237, and in part by the Amazon AI4Science Partnership
Discovery Grant, and the Carver Mead New Adventures Fund. The associate
editor coordinating the review of this article and approving it for publication
was Prof. Mohit Gupta.
(Corresponding author: Brandon Y. Feng.)
Brandon Y. Feng and William T. Freeman are with the Computer Science
and Artificial Intelligence Laboratory, Massachusetts Institute of Technology,
Cambridge, MA 02139 USA (e-mail: branfeng@mit.edu; billf@mit.edu).
Rodrigo Ferrer-Chávez and Jason J. Wang are with the Department of Physics
and Astronomy, Northwestern University, Evanston, IL 60208 USA (e-mail: ro-
drigoferrer-chavez2029@u.northwestern.edu; jason.wang@northwestern.edu).
Aviad Levis is with the Department of Computer Science, University of
Toronto, Toronto, ON M5S 1A1, Canada (e-mail: aviad.levis@gmail.com).
Katherine L. Bouman is with the Departments of Computing and Mathemat-
ical Sciences, Electrical Engineering, and Astronomy, California Institute of
Technology, Pasadena, CA 91125 USA (e-mail: klbouman@caltech.edu).
Digital Object Identifier 10.1109/TCI.2025.3525971
crucial for understanding planetary system formation and evolu-
tion. Moreover, it provides insights into the conditions necessary
for habitability and the potential for finding life beyond Earth,
addressing the profound question: “Are we alone?”
Sincethefirstconfirmeddiscoveryofanexoplanetin1992
[1]
,
thousands of exoplanets have been found. The majority of them
have been detected using indirect methods, such as the transit
method
[2]
, which measures the dimming of a star as a planet
passes in front of it, or the radial velocity method
[3]
, which
detects wobbles in a star’s motion due to gravitational pulls from
orbiting planets. While indirect techniques have been highly
successful, they provide limited information about the planets
themselves.
Direct imaging of exoplanets is essential for finding planets
like our own
[4]
,
[5]
. With the advent of powerful instruments
such as the James Webb Space Telescope (JWST), the Roman
Space Telescope, the upcoming class of Extremely Large Tele-
scopes, and the proposed Habitable Worlds Observatory, we are
entering a new era of exoplanet exploration, where the discovery
and study of Earth-like planets in habitable zones are within
reach
[6]
. Compared to indirect methods, direct imaging could
allow us to better understand the properties of exoplanets and
characterize their atmospheres, as well as provide more details
aboutthesysteminwhichtheyreside.However,directlyimaging
exoplanets remains a formidable challenge due to the extreme
contrast in brightness between the bright host stars and the faint
planets. Earth-like planets could be ten billion times fainter than
the Sun-like stars they orbit
[7]
, making it difficult to distinguish
the planet from the overwhelming star glare and the instrumental
noise of our telescopes. Therefore, with current technology,
direct imaging is largely constrained to observing planets that
are sufficiently far from the star glare or are particularly mas-
sive. Detecting potentially habitable, Earth-like planets remains
beyond our current capabilities, and significant technological
advancements will be important to push the boundaries of direct
imaging toward the detection and characterization of potentially
habitable worlds.
Coronagraphy is a key instrumentation technique in high-
contrast direct imaging to mitigate the impact of starlight. Coro-
nagraphic telescope instruments employ a series of optical ele-
ments to suppress the overwhelming starlight while preserving
thelightfromthesurroundingplanets
[7]
,
[8]
,
[9]
. However, even
with the advanced coronagraphic systems employed by leading
telescopeslikeJWST,starlightsuppressionisnotperfect,andthe
residual diffracted starlight produces speckle patterns that fur-
ther complicate the problem
[10]
. The speckles can be brighter
than the planets of interest and of very similar spatial scales. The
speckles may also evolve on a variety of timescales depending
on their different physical causes, such as defects or alignment
drifts in different parts of the telescope instrument.
© 2025 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
https://creativecommons.org/licenses/by/4.0/
FENG et al.: EXOPLANET IMAGING VIA DIFFERENTIABLE RENDERING
37
Fig. 1. A 1D toy example illustrating the basic principles and challenges of the problem addressed in this paper, simplified by examining the PSF in 1D. T
he first
panel shows the photon counts resulting from a star and a planet, including the effects of wavefront aberration. The second panel additionally displa
ys an incorrect
star PSF used in practice (measured under a different wavefront aberration), and the level of photon noise in observations. The third panel shows 1) Ou
tdated:
subtraction using the outdated star PSF, which leaves significant residuals that can be mistaken for the planet signal, posing the risk of false positi
ve detection or
too low signal-to-noise; 2) Optimal: perfect starlight subtraction, representing the upper bound of performance, limited only by noise. The core im
aging problem
is separating the planet signals from the star signals. Our goal is to improve on the outdated starlight subtraction result via differentiable optimi
zation, achieving
a more accurate separation of the star and planet signals. This will effectively result in accurately reducing starlight residuals and improving pla
net detection
sensitivity.
In the face of these challenges, post-processing techniques
have become a vital tool, in addition to instrumentation ad-
vancements. Current post-processing techniques for exoplanet
imaging generally involve three steps: taking reference images,
using them to estimate the star’s point spread function (PSF),
and subtracting this estimated PSF from the science images to
reveal the underlying planet signal. Common methods to collect
reference images are Angular Differential Imaging (ADI)
[11]
,
which observes the same scene at various roll angles, Spectral
Differential Imaging (SDI)
[12]
, which observes the same scene
simultaneouslyatdifferentwavelengths,andReferenceStarDif-
ferential Imaging (RDI)
[13]
, which directly observes different
stars as reference. With these reference images, techniques like
principal component analysis (PCA) are used to estimate the star
PSF, which is then subtracted from the science image
[14]
,
[15]
.
In a nutshell, the post-processing problem for exoplanet imag-
ing boils down to
effectively separating signals of the stars
from the planets
in the presence of speckle patterns arising
from wavefront aberrations, which can mimic planet signals
and complicate detection. These complications can lead to
false positives (when speckles are mistaken for planets) and
false negatives due to over-subtraction (when genuine planet
signals are removed along with speckles). Fig.
1
provides an
intuitive toy example illustrating the underlying challenges of
this problem. While useful, current post-processing techniques
have several limitations: they do not leverage wavefront sensing
data, which could provide hints about speckle structure; they
cannot account for dynamic wavefront aberrations occurring as
the telescope moves between observations; they may require
additional reference images, consuming valuable telescope time
that could be used for scientific observations; and importantly,
they do not model the underlying physical phenomena that give
rise to the speckle patterns. The first attempt to address this
last limitation
[16]
proved computationally challenging due to
then-available optimization strategies.
Exoplanet imaging stands to benefit greatly from differen-
tiable rendering techniques, which have recently transformed
computer graphics, computer vision, and computational imag-
ing
[17]
,
[18]
,
[19]
. By applying differentiable rendering, we
may overcome current post-processing limitations, providing
a more comprehensive and adaptable framework for tack-
ling high-contrast exoplanet imaging challenges
[20]
,
[21]
.
Differentiable rendering encourages consideration of the entire
image formation process rather than focusing solely on final
images. This perspective naturally leads us to examine the wave
space, considering light propagation before it reaches the detec-
tor.Inthispaper,weproposeadifferentiablerenderingtechnique
for exoplanet detection, which differs from the conventional
approach that focuses on the image intensity domain and relies
solely on image detector data. We extend exoplanet imaging
post-processing considerations to encompass the entire wave
space, employing wave-optics modeling for image formation
and incorporating wavefront sensing data.
Our case study focuses on the James Webb Space Telescope
(JWST), where wavefront sensing measurements primarily re-
flect minor misalignments of the telescope’s hexagonal mirrors.
1
These wavefront sensing data provide valuable aberration esti-
mates for our observations, offering information for a successful
star-planet signal separation. However, the available aberration
estimates derived from these wavefront sensing data are gener-
ally
outdated
for a given image, as numerous science observa-
tions can be conducted between two wavefront measurements,
which may be separated by several days. Consequently, by the
time a science observation is performed, the underlying wave-
front aberration affecting the observed image may have drifted
by an unknown amount from the most recent available estimate.
Our differentiable rendering method transforms these out-
dated aberration estimates into a valuable resource for post-
processing, effectively overcoming the temporal mismatch
through a gradient-based optimization to refine outdated esti-
mates and improve starlight subtraction. We construct a differ-
entiable model of the image formation process that takes the
estimated wavefront aberration map as input. This model allows
us to compute a reconstruction loss between the rendered image
and the captured image, enabling gradients to backpropagate
and refine the estimated wavefront aberrations. By using the
refined aberration estimates to render a final star PSF, we obtain
a more accurate estimate of the star signals. This improved
accuracy allows us to surpass the detection limits of traditional
post-processing pipelines.
The main contributions of this work are:

We introduce a differentiable renderer with fast and paral-
lelizable GPU implementation for high-contrast imaging,
1
These measurements are typically taken every few days, with mirror realign-
ment occurring only when errors exceed a specified threshold. Due to far less
frequent corrections compared to image measurements, uncorrected aberrations
will still affect detector images between adjustments.
38
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 11, 2025
accurately modeling light wave propagation through a
coronagraphic optical system.

We leverage differentiability to refine the underlying
wavefront aberrations through gradient-based optimiza-
tion, fully utilizing the wave space information.

We show the effectiveness of our wave space approach
through simulation experiments, achieving significant im-
provements in starlight subtraction and planet detection.

We provide insights into the robustness of our method un-
der different observing conditions and its potential impact
on future exoplanet imaging missions.
II. B
ACKGROUND
A. Definitions
To ensure clarity and consistency throughout the paper, we
formally define the following key terms:
Contrast
: We use contrast to refer to the brightness ratio
between the host star and the orbiting exoplanet. It is expressed
in a logarithmic scale, with a contrast level of
10
x
indicating
that the planet is
10
x
times fainter than its host star.
2
Aberration
: We denote wavefront aberrations as
φ
,atwo-
dimensional representation of the phase deviations of a wave-
front from an ideal, unaberrated wavefront (i.e., a plane wave).
This 2D map quantifies the optical path differences across the
pupil of the imaging system, with each point in the map corre-
sponding to a specific location in the pupil. In real-world sys-
tems, wavefront aberration reflects the optical path differences
measured by a wavefront sensor and expressed in units of length.
For JWST, the optical path differences across the primary mirror
on the telescope usually have a root mean square value of tens of
nanometers. In this work, during optimization, we represent the
wavefront aberrations
φ
explicitly as a 2D grid of
1024
×
1024
pixels. Future work may explore parameterizing
φ
with Zernike
polynomials or other learned or analytical basis functions to
potentially reducing the complexity of the optimization.
Drift
: In this paper, drift refers to the gradual change of
the wavefront aberrations over time, which can be caused by
factors such as thermal variations, mechanical instabilities, or
the movement of optical elements. For our purpose, it quantifies
wavefront aberration from an outdated measurement that has
been taken at an earlier timestep.
SNR (Signal-to-Noise Ratio)
: In the context of exoplanet
imaging, SNR is commonly used to measure the quality of the
recovered planet signal, SNR in exoplanet detection is
computed
over the annulus of the planet
, rather than the entire image or
field of view
[22]
. The signal term refers to the planet bright-
ness in the final subtracted image, and the noise term is the
standard deviation of pixel values in the planet’s annulus (width
approximating the spatial size of a planet), excluding the planet
signal. We adopt this annulus SNR to quantitatively assess the
planet signal recovery in accordance with existing exoplanet
2
Planets of a contrast level can have different detector intensities depending
on their positions in the field of view. A planet close to the center of the field of
view will be attenuated by the focal plane mask designed to occult the star. Thus,
the true contrast level of the planet is not necessarily reflected by its brightness
on the detector
. Similarly, the brightness of the star, i.e. the peak brightness of the
astronomical object unocculted by the coronagraph, is not the peak brightness
of the occulted star PSF recorded on the detector.
imaging literature
[22]
, where the standard for a valid planet
detection is to have an annulus SNR above 5.
B. Coronagraphy
Coronagraphy is a key technique employed in high-contrast
imaging to suppress the overwhelming starlight and enhance the
detectability of faint exoplanets. Fig.
2
illustrates a schematic of
a coronagraphic system used in the JWST NIRCam instrument.
Coronagraphicinstruments typicallyconsist of aseries of optical
elements, including apodizers, focal plane masks, and Lyot
stops. These optical elements are designed to selectively block
and filter the light from the central star while preserving the light
from the surrounding planets
[7]
,
[8]
,
[9]
.
As shown in Fig.
2
, after the entrance pupil (a 6.5 m mirror
composed of smaller individual hexagonal mirrors), a focal
plane mask blocks the central starlight in the image plane.
Then, a Lyot stop filters out the remaining diffracted light in
the pupil plane before the final science image is formed on the
detector. The performance of coronagraphic systems is critically
dependent on the wavefront quality of the incoming light. Wave-
front aberrations, caused by factors like atmospheric turbulence,
optical defects, and mechanical disturbances, produce speckles
in science images that can resemble exoplanets
[10]
,
[23]
.
C. Wavefront Aberration Sources
While ground-based instruments primarily contend with
rapidly evolving atmospheric turbulence, space-based tele-
scopes like JWST face a different set of issues. As shown in
Fig.
3
, aberrations in space primarily stem from factors such as
mirror misalignments, thermal distortions, and mechanical vi-
brations, evolving over longer timescales ranging from minutes
to days. A critical source of wavefront aberrations common to
both ground-based and space-based instruments is non-common
path aberrations (NCPAs)
[24]
. These aberrations occur in the
optical path unique to the science camera downstream of the
wavefront sensor. Consequently, NCPAs are neither directly
measured by the wavefront sensor nor perfectly corrected by
adaptive optics systems. The result is the presence of static or
slowly evolving speckles in science images, which can mimic
or obscure potential exoplanet signals.
This paper focuses specifically on space-based scenarios,
where the potential for long, uninterrupted observations with-
out atmospheric turbulence provides unique opportunities for
discovering and characterizing new exoplanets. The wavefront
aberration sources considered in this work are primarily those
associated with telescope optics, which are the dominant factors
limiting the performance of space-based high-contrast imaging
systems. Specifically, three types of aberrations are included:
known but outdated primary mirror aberrations, known static
instrumental aberrations, and unknown NCPAs. We take the
JWST as our case study due to 1) its unprecedented sensitivity
to new exoplanet populations, 2) its extraordinary stability, and
3) the wide availability of wavefront data for the telescope from
both in-lab testing and in-flight measurements.
D. Post-Processing Techniques
Various post-processing techniques have been developed to
estimate and then subtract the PSF from the science images,