Benchmarking of AI and Radiologists for Indeterminate Lung Nodule Malignancy Risk Estimation on Screening CT: The LUNA25 Challenge.

Authors

Peeters D, Obreja B, Antonissen N, Saghir Z, Pastorino U, Silva M, de Bock GH, Gietema H, Gleeson F, Heuvelmans MA, Lam S, Litjens G, Mohamed Hoesein F, Schaefer-Prokop C, Scholten E, Snoeckx A, van der Heijden EHFM, Vliegenthart R, Prokop M, Jacobs C, ...show more LUNA25 Consortium

Affiliations

• Diagnostic Image Analysis Group, Department of Medical Imaging, Radboudumc, Radboud University Medical Center, Route 767, Room 2.30, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands.
• Department of Medicine, Section of Pulmonary Medicine, Herlev-Gentofte Hospital, Hellerup, Denmark.
• Department of Thoracic Surgery, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy.
• Department of Medicine and Surgery, University of Parma, Parma, Italy.
• Department of Epidemiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands.
• Department of Radiology and Nuclear Medicine, Maastricht University, Maastricht University Medical Center, Maastricht, the Netherlands.
• Department of Oncology, University of Oxford, Oxford, United Kingdom.
• Department of Medicine, University of British Columbia, Vancouver, Canada.
• Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands.
• Department of Radiology, Utrecht University, University Medical Center Utrecht, Utrecht, the Netherlands.

Abstract

Purpose To compare the performance of an artificial intelligence (AI) system with that of radiologists for estimating malignancy risk of indeterminate-size nodules (5-15 mm) on low-dose CT (LDCT) within a standardized and transparent evaluation framework. Materials and Methods Teams participating in the AI study had access to a public dataset of 555 malignant and 5608 benign nodules on 4069 baseline LDCT scans from the National Lung Screening Trial (NLST) to develop AI systems. External testing was performed on 156 malignant and 312 benign size-matched nodules, all of indeterminate size, from 463 baseline scans collected from three large European lung cancer screening trials between 2004 and 2018, and the best-performing AI system (based on area under the receiver operating characteristic curve [AUC]) was selected. An observer study was conducted in which radiologists assessed 300 randomly selected nodules (100 malignant, 200 benign) from the external test set. Radiologists categorized nodules as low, intermediate, or high-risk, and the ≥ intermediate-risk threshold (intermediate or high-risk) was used to define a positive test. The selected AI system was compared with radiologists on this subset using the AUC. Results The selected AI system demonstrated superior performance to the 65 radiologists' average (AUC, 0.78 (95% CI: 0.73, 0.84) vs 0.70 (95% CI: 0.65, 0.74), P = .001). When using the ≥ intermediate-risk threshold, the AI system correctly classified 12% more malignant nodules at matched specificity and yielded 20% fewer false positives at matched sensitivity. Conclusion The selected AI system was superior to radiologists in estimating malignancy risk of indeterminate lung nodules on LDCT. ©RSNA, 2026.