Research on fusion algorithm for CT images and pathological images based on attention mechanism
Principal Investigator
Name
YUYI TAN
Degrees
B.S.
Institution
Sun Yat-sen University
Position Title
Student
Email
About this CDAS Project
Study
NLST
(Learn more about this study)
Project ID
NLST-1204
Initial CDAS Request Approval
Mar 11, 2024
Title
Research on fusion algorithm for CT images and pathological images based on attention mechanism
Summary
CT images can provide macroscopic features of lesions, while pathological images can provide microscopic representations of lesions (including nuclear and microenvironmental features). By combining macroscopic and microscopic feature representations, it helps to improve the accuracy of prognosis prediction. Therefore, my research plan to adopt an attention mechanism that integrates multimodal features for prognosis prediction. I can obtain imaging and pathological images of the dataset, and I also hope to apply for follow-up data (data related to survival time).
Aims
1. Use ViT architecture to extract hierarchical embeddings of pathological images, including features at different levels such as cells and tissues
2. Extract CT image embeddings using pre-trained models
3. Use attention fusion modules at different levels, including three main parts: encoder, cross modal attention, and decoder
4. Align intra modal features with inter modal features for prognostic prediction
Collaborators
none