Developing a multimodal foundation model integrating NLST CT and H&E images for improved lung cancer diagnosis

Principal Investigator

Name
Nancy Guo

Degrees
PhD

Institution
SUNY Binghamton University

Position Title
Developing a multimodal foundation model integrating CT and H&E images

Email
nguo1@binghamton.edu

About this CDAS Project

Study
NLST (Learn more about this study)

Project ID
NLST-1538

Initial CDAS Request Approval
Oct 12, 2026

Title
Developing a multimodal foundation model integrating NLST CT and H&E images for improved lung cancer diagnosis

Summary
Data Request: We seek access to the NLST H&E histopathology data, the 17th dictionary: CDAS 17 - Pathology Image Dataset, through CDAS to develop a multimodal foundation model that jointly learns from histopathological and computed tomography (CT) imaging for improved lung cancer diagnosis.

Project Summary: While existing multimodal medical AI studies have increasingly explored combining different imaging modalities, many approaches remain centered on late-stage fusion, where models are trained independently on individual modalities and their representations are integrated only after modality-specific learning. In contrast, early joint learning across complementary modalities remains relatively limited, particularly for large-scale foundation models.

H&E histopathology provides cellular- and tissue-level information that is complementary to the anatomical and spatial information captured by CT. By jointly pretraining a large foundation model on H&E and publicly available CT data, we aim to learn shared and complementary representations that capture lung cancer characteristics across microscopic and macroscopic scales. The resulting model will be further evaluated and adapted for multiple clinically relevant tasks, including tumor and tissue segmentation, histopathological and imaging-based classification.

The requested H&E data will therefore serve as a critical component for multimodal pretraining and evaluation. Access to diverse H&E specimens will allow the model to learn robust pathological representations and establish cross-modal associations between tissue morphology and radiological phenotypes. We expect that such joint multimodal learning can provide a more comprehensive representation of lung cancer than models developed from isolated modalities, potentially improving downstream segmentation, classification, and prognosis while enabling more integrated pathological understanding of disease characteristics.

Aims

Aim 1. Develop a multimodal foundation model for lung cancer. Using NLST low-dose CT scans, digitized H&E pathology images and matched clinical data, we will train a model that learns shared and complementary representations across macroscopic (radiologic) and microscopic (histologic) scales.

Aim 2. Evaluate the model on clinically relevant tasks and benchmark it against single-modality models. We will adapt the model for tumor and tissue segmentation, and imaging-based classification (e.g., nodule malignancy). Performance will be compared with CT-only and H&E-only models using standard metrics [e.g., Dice, AUC], and tested on external data from TCGA and CPTAC.

Aim 3. Incorporate multiomics data to improve characterization of the tumor microenvironment and immune infiltration. Using TCGA and CPTAC cases with molecular profiling, we will align image-derived features with multiomics data (e.g., transcriptome, proteome) to improve prediction of immune infiltration and microenvironment features, and test whether molecular grounding improves the image representations learned in Aims 1 and 2.

Collaborators

Taixi Chen SUNY Binghamton University
Niranjan Kumar SUNY Binghamton University
Nancy Guo SUNY Binghamton University