AI‐guided histopathology predicts brain metastasis in lung cancer patients - The Journal of Pathology - 2024 - Zhou - AI‐guided histopathology predicts brain metastasis in lung cancer patients.pdf

AI-guided histopathology predicts brain metastasis in lung

cancer patients

Haowen Zhou

†

, Mark Watson

†

, Cory T Bernadt

, Steven (Siyu) Lin

, Chieh-yu Lin

, Jon H Ritter

Alexander Wein

, Simon Mahler

, Sid Rawal

, Ramaswamy Govindan

, Changhuei Yang

and Richard J Cote

Department of Electrical Engineering, California Institute of Technology, Pasadena, CA, USA

Department of Pathology and Immunology, Washington University School of Medicine, Saint Louis, MO, USA

Department of Medicine, Washington University School of Medicine, Saint Louis, MO, USA

*Correspondence to: RJ Cote, Department of Pathology and Immunology, Washington University Sc

hool of Medicine, Saint Louis, MO, USA.

E-mail:

rcote@wustl.edu

†

These authors contributed equally to this work.

Abstract

Brain metastases can occur in nearly half of patients with early and locally advanced (stage I

–

III) non-small cell lung

cancer (NSCLC). There are no reliable histopathologic or molecular means to identify those who are likely to develop

brain metastases. We sought to determine if deep learning (DL) could be applied to routine H

E-stained primary

tumor tissue sections from stage I

–

III NSCLC patients to predict the development of brain metastasis. Diagnostic

slides from 158 patients with stage I

–

III NSCLC followed for at least 5 years for the development of brain metastases

(Met

, 65 patients) versus no progression (Met

, 93 patients) were subjected to whole-slide imaging. Three separate

iterations were performed by

fi

rst selecting 118 cases (45 Met

, 73 Met

) to train and validate the DL algorithm,

while 40 separate cases (20 Met

, 20 Met

) were used as the test set. The DL algorithm results were compared to a

blinded review by four expert pathologists. The DL-based algorithm was able to distinguish the eventual development

of brain metastases with an accuracy of 87% (

p

< 0.0001) compared with an average of 57.3% by the four

pathologists and appears to be particularly useful in predicting brain metastases in stage I patients. The DL algorithm

appears to focus on a complex set of histologic features. DL-based algorithms using routine H

E-stained slides may

identify patients who are likely to develop brain metastases from those who will remain disease free over extended

(>5 year) follow-up and may thus be spared systemic therapy.

TheJournalofPathology

published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great

Britain and Ireland.

Keywords:

non-small cell lung cancer; deep learning; brain metastasis; digital pathology; arti

fi

cial intelligence

Received 12 July 2023; Revised 30 November 2023; Accepted 16 January 2024

No con

fl

icts of interest were declared.

Introduction

Non-small cell lung cancer (NSCLC) remains a leading

cause of cancer death globally. Despite potentially

curative surgery, nearly a third of early-stage (stage I

–

III)

cases will recur with distant metastases [

]. Major

advances in the treatment of primary NSCLC with

therapeutic agents targeted to speci

fi

c protein coding

(

‘

actionable

’

) mutations and immune checkpoint blockade

therapy targeting programmed death-1 (PD-1) or PD-1

ligands have dramatically improved primary outcomes

in NSCLC. However, innate and acquired resistance to

therapies and disease progression to distant metastatic

sites remain a signi

fi

cant cause of morbidity. An increased

understanding of tumor biology has suggested that the

tumor microenvironment of primary NSCLC may dictate

future metastatic behavior [

–

]. Brain metastases, in

particular, are a common cause of morbidity and mortality

in NSCLC [

]. The stage of the disease is the most

commonly used predictor of outcome for NSCLC

(and other cancers). However, although stage provides

a general risk assessment for a population of patients

with similar characteristics, staging is unable to predict

which individual patients will or will not progress to metas-

tasis. Histopathologic analy

sis, even when supplemented

by genomic or molecular biomarkers, cannot accurately

predict the metastatic potential of NSCLC, particularly

in early-stage patients, where risk assessment may lead

to impactful treatment decisions [

The growing discipline of arti

fi

cial intelligence (AI),

especially in the form of deep learning (DL) networks

applied to image analysis, has the potential to identify

subtle and complex histopathologic features that may

not be appreciated by even the most experienced pathol-

ogist, and to correlate these patterns with biologic and

Journal of Pathology

JPathol

May 2024;

263:

–

Published online 4 March 2024 in Wiley Online Library

(

wileyonlinelibrary.com

)

DOI:

10.1002/path.6263

ORIGINAL ARTICLE

TheJournalofPathology

published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

This is an open access article under the terms of the

Creative Commons Attribution-NonCommercial

License, which permits use, distribution and

reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

clinical behavior, such as tumor metastatic potential.

DL algorithms have been trained to automatically and

accurately identify known diagnostic histopathologic

features that recapitulate the abilities of pathologists to

identify these features (e.g. for the diagnosis of prostate

cancer) [

–

]. However, the use of weakly/unsupervised

DL to identify features that cannot be recognized by

pathologists, such as progression and survival potential

based on routine histologic preparations, has been less

well explored [

11,12

]. Attention-based learning has also

been utilized to identify subregions of histopathology to

identify patterns of highest diagnostic value [

Here we demonstrate how a DL network can be

effectively trained on digital images from routine

H&E-stained NSCLC tumor tissue slides to predict

brain metastatic progression within 5 years of the initial

diagnosis and, importantly, accurately identify those

cases that do not progress after 5 or more years of

follow-up. Furthermore, based on the regions of interest

(ROI) that most strongly contribute to the DL algorithm

’

ability to predict progression versus no progression, it

appears that the basis of prediction relies on subtle and

complex histologic features of tumor cells, non-tumor

cells, and the tumor microenvironment.

Materials and methods

Ethics statement

All procedures related to this study were conducted under

an Institutional Review Board-approved protocol, which

allowed for the selection of tissue blocks and slides from

pre-existing institutional diagnostic material, linkage to

non-identifying, limited clini

cal datasets (where available),

and de-identi

fi

cation of all images through an

‘

honest

broker

’

mechanism.

Patient cohort and whole-slide imaging

This study was based on a cohort of patients with stage

–

III NSCLC diagnosed and treated at Washington

University School of Medicine with long-term follow-

up (>5 years or until metastasis). Of the patients

included in the study, 113 had stage I and 41 had higher

stage disease (see Table

for patient and tumor charac-

teristics). One representative block of tumor tissue from

a registry cohort of 198 treatment-naïve NSCLC patients

was used to create a fresh H&E slide, which was then

scanned at 40

magni

fi

cation with an Aperio/Leica AT2

slide scanner (Leica Biosystems, Deer Park, IL, USA).

All cases were initially subject to blind review to assess

tumor adequacy and annotated for ROI by circling an

approximate contour of the primary tumor, including the

entirety of the tumor microenvironment. Forty cases

were initially disquali

fi

ed as being non-representative

or insuf

fi

cient for adequate evaluation. The clinical char-

acteristics of the remaining 158 cases that were used for

this study [65 with known CNS progression (Met

) and

93 with no recurrence (Met

)] are summarized in

Table

and represented diagrammatically in Figure

The median time to progression or the follow-up time

of these cohorts was 12.2 and 106 months, respec-

tively. To retrieve an adequate number of cases, some

heterogeneity in stage and histology features was per-

mitted, although these were generally well represented in

both Met

and Met

cases. All cases were coded, and

clinical parameters (stage and histology) were unknown

to the DL team. Case outcomes were correlated with DL

predictions only after training/validation and subsequent

testing processes were complete. To compare the abil-

ity of expert pathologic assessment of the histopathol-

ogy to predict progression directly against the DL

model, pathologist reviewers were also blinded to out-

come and stage data, although they were obviously

privy to histologic subtype.

Data and image preprocessing

Figure

outlines the image processing algorithm employed

for this study. The Otsu thresholding method [

]was

implemented to exclude regions of plain glass from

the annotated regions in each whole-slide image

(WSI); 1,000 image tiles were randomly sampled from

the ROI in each WSI, each with 256

256 pixels or

130

μ

under 20

magni

fi

cation, down-sampled

from a 512

512 pixels 40

image. On average, the

sampled image tiles accounted for about 10% of the total

ROI in each slide scan. Image tile colors in both the

training/validation set and the testing set were normalized

to the color statistics

of one reference image [

]. For

the training sets, data augmentation, including a

random crop to a size of 224

224 pixels, random

fl

ips, and random rotations were performed on the

Table 1. Clinical characteristics of the study population.

Met

(

n

=

93)

(

n

=

65)

Gender

Male

Female

Average age at diagnosis (years)

60 (47

–

78)

57 (25

–

73)

Histology

Adenocarcinoma

Squamous cell

Large cell

Bronchial alveolar carcinoma

Poorly differentiated

Mixed

Grade

I124

III

No data available

Stage

I8532

III

No data available

Median follow-up time (months)

106

12.2

H Zhou, M Watson

etal

TheJournalofPathology

published by John Wiley & Sons Ltd

on behalf of The Pathological Society of Great Britain and Ireland.

www.pathsoc.org

JPathol

2024;

263:

–

www.thejournalofpathology.com

10969896, 2024, 1, Downloaded from https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.6263 by California Inst of Technology, Wiley Online Library on [27/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Figure 1.

Data processing pipeline. (A) A single, representative H

E-stained slide of a surgically resected primary NSCLC tumor block was

obtained from 158 patient cases and scanned at 40

magni

fi

cation. Each scan

fi

le was coded and linked to outcome and pathology data, but

blinded to both the DL team and the review pathologists until predictions were

fi

nalized. From each whole-slide scan, regions of high tumor

cellularity and surrounding tumor microenvironment were annotated by one reviewing pathologist. Regions outside the tumor bed as well as

areas of blank glass were masked. (B) One thousand non-overlapping image tiles from the ROI of each scan

fi

le were selected at random. Tiles

were subjected to color normalization and randomized in cropping and orientation to create a data augmentation step. (C) All tiles in the

training set were shuf

fl

ed and fed to the convolution neural network with the ResNet-18 backbone pretrained on ImageNet, with a linear

layer and sigmoid activation for model optimization. In the testing process, the weights in the model were all frozen. A median-pooling

function was used to compute the

fi

nal risk assessment from the collective image tiles of each patient.

AI-guided histopathology predicts brain metastasis

TheJournalofPathology

published by John Wiley & Sons Ltd

on behalf of The Pathological Society of Great Britain and Ireland.

www.pathsoc.org

JPathol

2024;

263:

–

www.thejournalofpathology.com

color-normalized images before using them as input to

the DL model. For the validation and testing sets, the

color-normalization process was mandatory, but the

data augmentation was not needed.

DL study design

The DL model was based on the ResNet-18 convolutional

neural network pretrained on the ImageNet dataset [

The pretrained weights were taken as the initialization for

our model training and all model weights were unfrozen

during the training process. A linear layer, attached with a

sigmoid activation layer, was used as the classi

fi

er and the

output was a

‘

prediction score

’

for an individual image

tile; that is, tiles that the DL assessed as associated with

Met

versus tiles that the DL assessed as associated with

Met

. The prediction scores of all individual image tiles

from each WSI were subjected to a median-pooling layer

to produce the

fi

nal overall progression risk assessment

for each slide (case).

Figure 2.

DL study design. The cases (slide images) were arbitrarily coded from 1 to 158 (shown in the top grayscale bar). The cases

were randomized (shown in randomized grayscale bars) and split into three different partitions to create experiments 1

–

3. Each experiment

utilized a different training set of 30 Met

(orange) and 58 Met

(purple) tumor images and a validation set of 15 Met

and 15 Met

images.

The training/validation was performed using a three-fold cross-validation. Each subsequent set-aside testing set was composed of

20 Met

and 20 Met

case images. The testing sets for experiments 1

–

3 in total represented



75% of the entire 158-case cohort.

H Zhou, M Watson

etal

TheJournalofPathology

published by John Wiley & Sons Ltd

on behalf of The Pathological Society of Great Britain and Ireland.

www.pathsoc.org

JPathol

2024;

263:

–