Synthesizing training data for clinical visual question answering

Principal Investigator

Name
Dina Demner-Fushman

Degrees
M.D., Ph.D

Institution
NLM

Position Title
Investigator

Email
ddemner@mail.nih.gov

About this CDAS Project

Study
PLCO (Learn more about this study)

Project ID
PLCO-389

Initial CDAS Request Approval
Aug 9, 2018

Title
Synthesizing training data for clinical visual question answering

Summary
Radiology images are an essential part of clinical decision making and population screening, e.g., for cancer. Automated systems could help clinicians cope with large amounts of images by answering questions about the image contents. An emerging area of artificial intelligence, Visual Question Answering (VQA) in the medical domain explores approaches to this form of clinical decision support. Success of such machine learning tools hinges on availability and design of collections composed of medical images augmented with question-answer pairs directed at the content of the image. We propose to generate data to train such systems by synthesizing large amounts of visual questions and answers leveraging the existing imaging collections.

Aims

Previous research has demonstrated feasibility of leveraging synthesized radiology images for tumor segmentation http://arxiv.org/abs/1807.10225 We hypothesize the approach can be adapted to synthesize the large amounts of radiology images and question-answer pairs needed to develop Visual Question Answering Systems capable of assisting medical students in their training, as well as clinicians and patients in decision support. Our specific aims are to use the images to generate new images to enrich the pathology representation and leverage the image descriptions to generate question answer pairs.

Collaborators

Hoo Chang Shin, NVIDIA Corporation
Asma Ben Abacha, NLM