Validation of a multimodal generative model for lung cancer and tuberculosis detection on chest radiograph in screening population
Principal Investigator
Name
Amy Hong
Degrees
M.D., Ph.D.
Institution
Kakaobrain
Position Title
Vice President
Email
About this CDAS Project
Study
NLST
(Learn more about this study)
Project ID
NLST-1197
Initial CDAS Request Approval
Feb 26, 2024
Title
Validation of a multimodal generative model for lung cancer and tuberculosis detection on chest radiograph in screening population
Summary
This project is a cross-over reader study focused on lung cancer and tuberculosis screening, evaluating the impact of providing preliminary reports (prem. report) on the sensitivity, specificity, and confidence levels of readers in detecting cancer and nodules. It includes:
Lung Cancer Screening:
A comparison of screening with and without preliminary reports.
Analysis of changes in sensitivity and specificity in detecting cancer and nodules, including their presence, location, and the readers' confidence levels.
Tuberculosis Screening:
Similarly, it compares the effectiveness of screening with and without preliminary reports on the sensitivity, specificity, and confidence levels in detecting tuberculosis-related findings.
Subgroup Analysis for Lung Cancer Screening:
Evaluates the stand-alone performance of the AI model.
Examines the change in performance specifically regarding cancerous nodules.
Includes a free-text evaluation of the model's findings.
Subgroup Analysis for Tuberculosis Screening:
Assesses the stand-alone performance of the model.
Conducts a free-text evaluation, acknowledging the possibility that the model may struggle with accurately identifying TB but noting that analyzing how the model describes TB patterns can still provide meaningful insights.
Suggests dividing the definitive description of TB patterns into three levels of ambiguity to analyze performance changes at each level.
Overall, the project aims to understand how the inclusion of preliminary reports affects the diagnostic accuracy and confidence of readers in lung cancer and tuberculosis screenings, alongside evaluating the AI model's performance in these screenings, especially in identifying cancerous nodules and describing TB patterns.
Lung Cancer Screening:
A comparison of screening with and without preliminary reports.
Analysis of changes in sensitivity and specificity in detecting cancer and nodules, including their presence, location, and the readers' confidence levels.
Tuberculosis Screening:
Similarly, it compares the effectiveness of screening with and without preliminary reports on the sensitivity, specificity, and confidence levels in detecting tuberculosis-related findings.
Subgroup Analysis for Lung Cancer Screening:
Evaluates the stand-alone performance of the AI model.
Examines the change in performance specifically regarding cancerous nodules.
Includes a free-text evaluation of the model's findings.
Subgroup Analysis for Tuberculosis Screening:
Assesses the stand-alone performance of the model.
Conducts a free-text evaluation, acknowledging the possibility that the model may struggle with accurately identifying TB but noting that analyzing how the model describes TB patterns can still provide meaningful insights.
Suggests dividing the definitive description of TB patterns into three levels of ambiguity to analyze performance changes at each level.
Overall, the project aims to understand how the inclusion of preliminary reports affects the diagnostic accuracy and confidence of readers in lung cancer and tuberculosis screenings, alongside evaluating the AI model's performance in these screenings, especially in identifying cancerous nodules and describing TB patterns.
Aims
Compare lung cancer screening effectiveness with vs. without preliminary reports, analyzing sensitivity, specificity, and reader confidence.
Compare tuberculosis screening effectiveness with vs. without preliminary reports, focusing on sensitivity, specificity, and confidence.
Evaluate the AI model's performance in detecting cancerous nodules and TB patterns, including a free-text evaluation and analysis of performance changes based on pattern ambiguity levels.
Collaborators
Ok Kyu Song, Kakaobrain
Woong Bae, Kakaobrain