Multimodal Medical Imaging Fusion for Diagnostic Assistance: Towards a General Model
The diseases targeted by this project include lung cancer, respiratory failure (COPD), atherosclerosis (vascular diseases), and cardiopathies & vasculitis (e.g., Horton's disease). There is, however, a possibility of initial data unavailability regarding vasculitis. Moreover, the diseases in question are not mutually exclusive, which may present additional complexity in modeling.
The project will unfold in several key stages. Initially, a comprehensive state-of-the-art review will be conducted. This is followed by an in-depth analysis of existing models and datasets, with a particular focus on labels and potential diagnoses. The review will guide the direction of future work, including the validation of the scope or its restriction, and the identification of use cases.
A critical aspect of the project is the comparison of existing models and datasets against performance metrics. The initial phase will employ open-source datasets, subsequently validated with the dataset from Hospices Civils de Lyon (HCL).
One of the project's innovative approaches is the modeling of thoracic scans into latent spaces (data to vect), which raises the question of whether to use a global space or organ-specific spaces. Various modeling techniques, such as Transformers, RESnet, stable diffusion/Unet?, the principle of Mistral (limited attention), and radiomics, will be explored.
Segmentation to identify Regions of Interest (heart, lungs, vessels, etc.) will be considered, but with a caveat not to divert focus from the primary objectives. The project plans to initially test direct latent space modeling and subsequently compare the results with those obtained from segmentation combined with four latent spaces.
Finally, the project will involve classification (diagnosis) and comparison with physicians' assessments using a risk score. The multimodal approach will integrate clinical data and histology, with separate models proposed for imaging (scans) and histology.
Overall, this project aspires to bridge the gap between current medical imaging techniques and the potential offered by multimodal models, ultimately aiding clinicians in making more accurate and timely diagnoses.
- Analyze existing models and datasets with a focus on labels indicative of thoracic pathologies.
- Compare performance metrics of current models and datasets, starting with open-source data and progressing to HCL datasets.
- Explore the potential of latent space modeling for thoracic scans and decide between a global space or organ-specific spaces.
- Investigate the efficacy of segmentation in identifying Regions of Interest without detracting from the main objective.
- Develop a diagnostic classification system and validate it through risk assessments in collaboration with medical professionals.
- Integrate multimodal data, including clinical information and histology, into a comprehensive model with separate components for imaging and histology.
Academic Supervisors (LIRIS, Université Lyon 1):
- Alexandre Meyer
- Mathieu Lefort
- Hamid Ladjal
Industry Supervisor:
- Simon Achard (Innovation Manager, ATOS)
Medical Advisors:
- Antoine Richard (AI Expert, CHU de Lyon)
- Dr. Thomas Barba (Internal Medicine, CHU de Lyon)