Maxine Tan

Monash University Malaysia

Senior Lecturer

NLST (Learn more about this study)

NLST-1471

Aug 12, 2025

A multi-modal foundation model for lung cancer segmentation and risk prediction from imaging and clinical data

In this project, we aim to create a comprehensive multi-modal foundation model that will learn from NLST's computed tomography (CT) scans alongside structured clinical risk data. This innovative model will not only perform voxel-level tumor segmentation but also predict lung cancer risk at the patient level. By utilizing shared representations and multi-task learning, we seek to enhance diagnostic accuracy and resilience compared to traditional single-modality approaches. Ultimately, this model will lay the groundwork for adapting to other related tasks, including prognosis prediction and treatment planning.

Furthermore, the newly developed model will be able to forecast lung cancer risk occurrence over specific time horizons; for example, the new model will be able to predict the risk of cancer occurring in one to three years' time, based on a patient's current (and past) CT scans. In this way, personalized screening paradigms can be recommended on a patient level, to avoid overscreening and/or unnecessary radiology exposure to a patient. This is in line with precision medicine and personalized medicine initiatives, and also saves screening costs, reduces false positives, and avoids unnecessary anxiety to individuals in the long run.

● Develop a comprehensive multimodal deep learning framework that combines 3D convolutional encoders for CT imaging with tabular encoders for structured clinical data (such as age, smoking history, and family history). This framework aims to achieve voxel-level segmentation of lung tumors as well as to predict cancer risk at the patient level.

● Train and assess the model utilizing the NLST dataset, while rigorously comparing its performance against CT-only and clinical-only models. Employ standard evaluation metrics including Dice coefficient, Intersection-over-Union (IoU), Area Under the Curve (AUC), and F1-score to ensure a thorough analysis.

● Explore and put into practice sophisticated strategies for combining different types of data, focusing on joint embeddings and attention mechanisms. This approach aims to seamlessly integrate diverse datasets, enhancing the robustness and generalizability of our models across various screening populations.