AI Information Design in Radiological Decision Making
Principal Investigator
Name
Pranav Rajpurkar
Degrees
Ph.D.
Institution
Harvard Medical School
Position Title
Assistant Professor of Biomedical Informatics
Email
About this CDAS Project
Study
NLST
(Learn more about this study)
Project ID
NLST-926
Initial CDAS Request Approval
Jun 22, 2022
Title
AI Information Design in Radiological Decision Making
Summary
We plan to investigate how radiologists combine their own information with AI
predictions when making assessments and decisions. The proposed research uses a retrospective
experimental design based on the NLST images and patient data. The experiment asks
radiologists to conducts lung cancer screens based on the CT images that were recorded as part
of the NLST trial. In addition to the data that was provided to the radiologists during the original
trial our subjects will also have access to a state-of-the-art AI for lung cancer detection. We will
generate the ground-truth for each case based on findings from diagnostic procedures, final
outcomes, and subsequent reads from the NLST trial.
Our experiment contributes to a line of research that asks how the collaboration between
radiologists and AI should be organized. Specifically, we want to understand whether the
availability of detailed AI predictions lead to a reduction in effort by radiologists who may
simply follow those and hence reduced accuracy relative to a Bayesian benchmark that combines
the radiologists and the AI’s assessment ex-post. The tendency to follow the prediction may not
be dangerous when the AI has an accurate prediction. However, if the AI says it is unsure, then it
may be worthwhile for the expert to exert more effort. An appropriately designed information
revelation strategy may then be used to induce effort by the expert, much like in the canonical
examples of Bayesian Persuasion (Kamenica and Gentzkow, 2011). By experimentally varying
the coarseness of the AI information we will be able to construct the optimal information
revelation policy.
The decisions in the experiment will be recorded through a browser-based experimental
interface that we developed ourselves, which was designed to resemble those that are used in a
clinical setting.
predictions when making assessments and decisions. The proposed research uses a retrospective
experimental design based on the NLST images and patient data. The experiment asks
radiologists to conducts lung cancer screens based on the CT images that were recorded as part
of the NLST trial. In addition to the data that was provided to the radiologists during the original
trial our subjects will also have access to a state-of-the-art AI for lung cancer detection. We will
generate the ground-truth for each case based on findings from diagnostic procedures, final
outcomes, and subsequent reads from the NLST trial.
Our experiment contributes to a line of research that asks how the collaboration between
radiologists and AI should be organized. Specifically, we want to understand whether the
availability of detailed AI predictions lead to a reduction in effort by radiologists who may
simply follow those and hence reduced accuracy relative to a Bayesian benchmark that combines
the radiologists and the AI’s assessment ex-post. The tendency to follow the prediction may not
be dangerous when the AI has an accurate prediction. However, if the AI says it is unsure, then it
may be worthwhile for the expert to exert more effort. An appropriately designed information
revelation strategy may then be used to induce effort by the expert, much like in the canonical
examples of Bayesian Persuasion (Kamenica and Gentzkow, 2011). By experimentally varying
the coarseness of the AI information we will be able to construct the optimal information
revelation policy.
The decisions in the experiment will be recorded through a browser-based experimental
interface that we developed ourselves, which was designed to resemble those that are used in a
clinical setting.
Aims
- We propose to optimally design a disease-specific and radiologist-specific AI-radiologist collaboration system that maximizes diagnostic performance under different tolerances for AI autonomy.
- Investigate whether the combination of low-cost imaging technologies and AI or low-skill expertise and AI can substitute for high-cost imaging technologies and high-skill labor
- Measure and model the responsiveness of radiologists’ effort choice to AI assistance, and whether radiologists reduce their own effort when provided with AI assistance
Collaborators
Nikhil Agarwal (MIT Economics)
Alexander Moehring (MIT Sloan)
Tobias Salz (MIT Economics)
Alexander Wolitzky (MIT Economics)