41377_2022_820_MOESM1

Supplementary

Information

for

Deep Learning Acceleration of Multiscale Superresolution Localization

Photoacoustic Imaging

Jongbeom Kim, Gyuwon Kim, Lei Li,

Pengfei Zhang

Jin Young Kim, Yeonggeun Kim,

Hyung Ham Kim,

Lihong V. Wang

, Seungchul Lee

and Chulhong Kim

Corresponding author. Email:

chulhong@postech.edu

seunglee

@postech.ac.kr and

LVW

@caltech.edu

This file includes:

Supplementary Text

Sup

plementary M

aterials

and M

ethods

Fig. S1

Label

free localization

based

PAM

imaging

process

Fig. S2.

Labeled localization

based

PACT

imaging

process

Fig. S

Customized 2D and 3D discriminator network architecture.

Fig. S4.

3D Graphs of evaluation metrics

depending on frame/droplet counts used to

reconstruct a sparse localization

based image for the training and test set.

Fig. S

Configuration of an optical

resolution photoacoustic microscopy system

Fig. S

Spatial resolution of an optical

resolution photoacoustic microscopy system.

Fig. S

Configuration of a photoacoustic computed tomography system

Table S1

Summary of generator networks.

Table S2

Details of the discriminator networks

Table S3

Comparison of 3D PSNR and 3D MS

SSIM metrics depending on frame counts

used to reconstruct a sparse localization

based image for the training and test set.

Table S

Comparison of 2D PSNR and 2D MS

SSIM metrics depending on droplet counts

used to

reconstruct a sparse localization

based image for the training and test sets.

Table S

yper parameters for training

Movie files:

Movie

ormation of sparse,

DNN

and dense localization

based

PAM images.

Movie

ormation of sparse,

DNN

and dense localization

based

PACT images.

Supplementary Text:

Label

free l

ocalization optical

resolution photoacoustic microscopy (OR

PAM)

adopte

previously

reported localization process

ing

procedure

reconstruct

label

free

super

resolution

PAM images

(Fig. S1)

The localization process

ing

was applied to OR

PAM

volumetric images obtained by

galvanometer scanner OR

PAM

(OptichoM,

Opticho

, South

Korea).

The system

described in Fig. S

The

PAM system

with a fast temporal resolution

scan speed of 500 Hz)

could

capture

intrinsic red blood cells

(RBC

)

instantaneously

generating p

hotoacoustic (PA) signals

from the

location

(Fig. S1)

Because e

ach frame

captured

the

signals

the flowing blood at different points

, a densely

connected localization image

could

reconstruct

from

multiple

PAM frames obtained

while

continuously imaging

the

same

region of

a mouse ear.

Finally, l

ocalization frame

were

translated

from OR

PAM f

rame

localizing the PA signals

within the frames,

and

then were

superimposed

to create

a dense

localization

based

PAM image

Labeled

localization

photoacoustic

computed

tomography (PACT)

previously

reported localization algorithm

for PACT

was

applied to produce datasets for a 2D

deep neural network

(Fig. S2)

A PACT system (Fig. S

) continuously

imaged

the cortical layer

of a mouse

brain

during injection of

small

dyed

droplets

that had a

higher optical absorption

contrast than RBCs

the optical wavelength of

780 nm

Thanks to the higher contrast, the droplets

could be tracked and localized with high precision.

maging for half an hour

a frame rate of 20

resulted in a total of 36,000 frames.

As in

the localization OR

PAM environm

ent, the flow

within the blood vessel

caused

each droplet

to be

detected at different

subsequent

positions.

In the

localization PACT processing, droplets

were

extracted from the acquired PACT images

and then

all the localized

droplets were

combined to produce a

dense localization

based

PACT image

(Fig.

S2).

Supplementary

aterials and

ethods

Label

free l

ocalization

PAM

regular OR

PAM images

for training

were obtained

using

PAM

system

(Fig.

)

with

an optical fiber to

deliver optical pulses on

the target.

Thanks to the fiber, we

could

obtain

volumetric images

from a fixed target

, reduc

ing

motion artifacts during

the

imaging

experiments

o excite a target

, t

system

use

a fast nano

pulse laser system with

maximum

pulse repetition

rate of 600 kHz (VPFL

10, Spectra

Physics, USA).

The

laser

beams are collimated with a fiber

optic collimator (TC25FC

543, Thorlabs, USA) and then focused by an objective lens (LA1213

A, Thorlabs, USA) (Figs.

a, b). The laser beams pas

s through the center hole of a customized

ring

shaped ultrasound transducer

with a focal length of 21 mm, an outer diameter of 15 mm, an

inner diameter of 2.5 mm, a central frequency of

20 MHz

and a bandwidth of 60%

. In this PAM

system, we use

galvanom

eter scanner (GVS001, Thorlabs, USA)

and

newly designed

the

mirror of the galvanometer scanner to steer optical beams downward (Fig.

The focused beam

is reflected by

the

mirror

steered by

the

scanner

and

irradiate

the target, thereby inducing

waves

The PA waves

are

then reflected by the mirror of the scanner and measured by the

transducer.

A multifunctional data acquisition board (DAQ, NI PCIe

6321, National Instruments,

USA) synchronizes all

the

mechanic

al systems (i

the

laser system,

galvanometer scanner, linear

motorized stages

and

the

digitizer). B

scan images are obtained by

fast angular scanning of the

galvanometer scanner, and volumetric images are acquired

scanning

the linear motorized

stage

slowly

during the fast angular scanning.

The maximum B

mode imaging speed

reached

with lateral

and axial

resolution

9.1

μm

and of

114

μm

(Fig. S

)

respectively,

under a

scanning range of ~

1.5

, 4

00 pixels, and a laser repetition rate of 400 kHz.

The measured

resolutions matched well with the theoretical lateral and

axial

resolutions of 8.5

μm

and 113

μm

respectively

for

optical

numerical aperture of 0.032,

central frequency of 20 MHz

and

6dB bandwidth of 60%

The measured PA signals, pre

amplified by an amplifier (ZX60

3018G

S+, Mini

Circuits, 26

dB gain, USA), are finally transferred into digital signals by the digitizer

(ATS

9350, Alarzatech, USA)

and saved in binary format.

ocalization

based

PAM image

were

reconstructed

through

the following processes:

Fast

acquired volumetric OR

PAM images

were

precisely aligned via

intensity

based image

registration algorithm

. The aligned volume data

was spatially interpolated to a size of 4x

along x

and y

axes by bicubic interpolation. After

being

normalized and convolved with an averaging filter

with a kernel size of

× 3 × 3

to emphasize local

maximum points,

the OR

PAM images were

transferred into volumetric localization frames by determining

the local maximum points in

MATLAB.

A super resolutio

n volumetric localization OR

PAM image was then reconstructed by

superimposi

ng all the localization frames.

The localization OR

PAM imaging improved the spatial

resolution by a factor of 2.5

in vivo

Labeled l

ocalization

PACT

The PACT system used in this study is shown in Fig.

. A Ti: Sapphire laser (LS

2145

150,

Symphotic Tii; 20 Hz pulse repetition rate; 12 ns pulse width) is used to output 780 nm

pulses

for

PA excitation. The laser beam is first homogenized by an optical diffuser (EDC

5, RPC Photonics)

and then

illuminate

the mouse brain from

above

. The PA signals are detected by a full

ring

ultrasonic transducer array (Imasonic) with a 10

cm diameter, a

MHz central frequency, more

than 90% one

way bandwidth, and 512 elements. Each element (20

mm height, 0.61

mm pitch

and 0.1

mm inter

element space) is cylindrically focused to produce an axial focal distance of 45

mm (acoustic NA, 0.2). The combined fo

ci of all 512 elements form an approximately uniform

imaging region with a 20

mm diameter and 1

mm thickness.

In our experiments, a

lab

made 512

channel preamplifier (26 dB gain)

was

directly connected to the ultrasonic transducer array

housing, with minim

ized connection cable length to minimize cable noise. The pre

amplified

photoacoustic signals

were

digitized using a 512

channel data acquisition system (four

SonixDAQs, Ultrasonix Medical ULC; 128 channels each; 40 MHz sampling rate; 12 bits dynamic

range

) with programmable amplification up to 51 dB. The digitized radio frequency data were first

stored in the onboard buffer, then transferred to a computer. The digitized raw data were fed into

a half

time dual

speed

sound universal back

projection algori

thm for image reconstruction

The in

plane resolution of this system was previously quantified as ~150 μ m

for

an imaging size

of 10 mm

12 mm and

a pixel size of 25

μm

In the PACT localization experiments,

780

iodide

hydrophobic dye (425311, Sigma

Aldrich)

was used as an optical contrast agent in the

oplets. A mixture of 67% (v/v) clove oil (C8392, Sigma

Aldrich) and 33% (v/v) peanut oil

(P2144, Sigma

Aldrich) was the solvent. The oil mixture was prepared

that the final solution

had a density close to

that of

water, which guaranteed good stability o

f the droplets in

both

water

and whole blood

. It took 24

–

48 hours to fully dissolve the dye in the oil solvent, obtaining a

maximum concentration of 2 mM. A mixture of 20 μL of the dye solution (2 mM) and 2 μL of

surfactant (span® 80, S6760

250ML, Sigma

Ald

rich) was added to 1 mL of distilled water and

then vibrated for 10 s to form

droplet suspension. The final droplet suspension had a

concentration of ~ 4×10

To reconstruct a localization PACT image, a cortical layer of a mouse brain was imaged

r 30 minutes with a frame rate of 20 Hz during droplet injection

(Fig. S

)

To trace the droplets

in the brain, the time

lapse PACT images were first denoised by applying a 2

D adaptive noise

removal filter

, t

hen

subtraction of adjacent frames highlighted moving droplets. To localize the

single droplets, the differential images were converted into binary images by thresholding the pixel

values at 1/4 of their maxima. The bright spots within a range of 16 to 64 pix

els in the binary

images were the regions containing droplets. Any spots with a roundness

less than 0.7 were

abandoned, which removed droplet clusters and artifacts. The centroids of the bright spots in the

binary images were determined

coarsely loca

single droplets in the differential images. Then,

a ROI of each droplet, centered at its centroid, was isolated from the differential images. The ROI

(11 × 11 pixels) was fitted with a 2

D Gaussian function, yielding a precise localization of the

center

of each droplet. Every droplet was characterized by a 2

D Gaussian

distributed spot with a

radius equal to its localization uncertainty. Adding up all the

droplet

yield

super

resolution

localization PACT image,

which

improv

the spatial resolution b

y a factor of 6

in vivo

Supplementary Figures

Fig. S

Label

free l

ocalization

based

PAM

imag

ing process

A regular OR

PAM frame

is first translated into a localization frame, and then all the localization frames are

superimposed to reconstruct a dense localization

based OR

PAM image. PA, photoacoustic;

PAM, optical

resolution photoacoustic microscopy; RBC

, red blood cell; Dense local.,

dense localization

based PA image.

Fig. S

Labeled l

ocalization

based

PACT

imag

ing process

Droplets containing a

hydrophobic dye are intravenously injected. Each droplet is localized from regular PACT

images. All the extracted droplets are combined to reconstruct a dense localization

based

PACT image. PA, photoacoustic; PACT, photoacoustic comp

uted tomography; Dense

local., dense localization

based PA image.

Fig.

Customized 2D and 3D discriminator network architecture

Fig. S

Graphs of evaluation metrics depending on

frame/droplet counts used to

reconstruct a sparse localization

based image for the training and test set. Graphs for (

)

PSNR and (

)

3D MS

SSIM evaluation metrics of localization OR

PAM for frame counts of

2, 3, 4, 5, 6, 8, 10, 15, and 30. Graphs for (

)

2D PSNR and (

)

2D MS

SSIM evaluation

metrics of localization PACT for droplet counts of

1/32, 1/28, 1/20, 1/24, 1/16, 1/12, 1/8,

1/4, and 1/2 of the dense images' droplet counts

. OR

PAM, optical

resolution photoacoustic

microscopy; PSNR, peak signal

noise ratio; MS

SSIM, multi

scale structural similarity;

PACT, photoacoustic computed tomography.

Fig.

Configuration of

an optical

resolution photoacoustic microscopy system

(

)

model of the system.

(

)

up view

of the scanning part outlined by the red dashed box

(

)

FOC, fiber optic collimator; LS, linear stage; MLS, motorized linear stage; A

amplifier; OL, objective lens; GS, galvanometer scanner; M, mirror; RT, ring transducer

Fig. S

Spatial resolution of an optical

resolution photoacoustic microscopy system.

(

)

MAP image of the edge of a patterned microstructure.

(

)

Cross

sectional PA B

scan image of

a carbon fiber.

(

)

Fitted ESF of PA data

marked by t

he line a

′

(

)

and LSF

defined

the

first derivative of the ESF. The lateral resolution was measured by the FWHM of the LSF.

(

)

Fitted LSF of PA data marked by the line

′

marked in

(

)

. The axial resolution was measured

by the FWHM of the LSF. PA, photoacoustic; ESF, edge spread function; LSF, line spread

function;

MAP, maximum amplitude projection; FWHM, full width at half maximum.

Fig.

Configuration

of a photoacoustic computed tomography system. A mouse brain

photoacoustically imaged during intracarotid injection of a droplet suspension through a

catheter. ED, engineered diffuser; UTA, ultrasonic transducer array; A

, amplifier; DAQ,

data acquisit

ion board

; PC, personal computer

a, b

The spatial dropout and batch normalization operations were omitted in the 2D localization

PACT network

because they

deteriorated the results

Table S

Summary of generator networks

The batch normalization operation was omitted in the 2D localization PACT network

because

the operation deteriorated the results.

Table S

Details

the

discriminator networks.