Skip to Main Content

An official website of the United States government

Testing of chest x-ray computer-aided detection

Principal Investigator
Name
steve worrell
Institution
Riverain Medical
Email
About this CDAS Project
Study
PLCO (Learn more about this study)
Project ID
2009-0003
Initial CDAS Request Approval
Apr 16, 2009
Title
Testing of chest x-ray computer-aided detection
Summary
Riverain Medical (Riverain) develops and markets computer-aided detection (CAD) devices and other pattern recognition devices that can be used by clinicians to help detect lung cancer from chest x-ray (CXR) images. CAD programs mark areas consistent with known lung cancer, by placing a circle around the area. These circles mark Regions of Interest (ROI). These pattern recognition algorithms must be shown to be reasonably safe and effective when used according to the products' labeling. The U.S. Food and Drug Administration (FDA) reviews pre-market approval applications and approves the device for marketing when there is sufficient evidence to consider the device reasonably safe and effective when used according to the label. Riverain has the only FDA PMA approved CXR CAD device. Riverain has digitized most of the PLCO images. Under the terms of the Image and Data Transfer Agreement, Riverain provides the digitized images to NCI, which provides a specified subset of the images and associated clinical information to Riverain. The Agreement indicates that the PLCO data is not intended to be used for FDA or any other clinical study or publication. At the time the agreement was signed Riverain tested its algorithms on a dataset collected in the Johns Hopkins Early Lung Cancer Screening Program ("Hopkins") so Riverain intended to use the PLCO images for training the algorithm. However, Riverain did not do so because it did not want to bias the results of a pilot study at the University of Minnesota screening site, which could be expanded to all PLCO sites. Recently, the FDA informed Riverain that FDA will no longer accept the Hopkins dataset for testing. Consequently, this application requests permission for Riverain to test algorithms using digitized images, submit results to FDA, and publish the results.
Aims

FDA has historically considered two types of studies when reviewing applications. The first type of study is a machine test. A set of chest x-ray images containing both cancers and non-cancer images is analyzed by the CAD program. Typically, no more than half of the images have proven lung cancer (ex: 1/3). The remainder of the images is from people that do not have lung cancer at the time the image was taken. Two statistics are computed from the marked images. The first statistic computed is sensitivity, which is the proportion of cancers that are correctly marked by the program. The second is the average number of false positives per image (FP/I), which is determined from non-cancer images. Typically, a free response operating characteristic (FROC) curve is drawn because sensitivity and FP/I must be balanced. 100% sensitivity could be achieved by placing enough circles on an image. Similarly, no FP/I could be obtained by not marking an image. Neither of these solutions would be clinically useful so a balance point (i.e. operating point) is selected that a clinician could find acceptable. As many as three operating points are typically chosen for a CAD device. For machine tests typically the area under the curve, Az, of the new device is compared to the original algorithm (R1.0) as measured by the area under the FROC curves. H0: Az(test) - Az(R1.0) EQ 0 HA: Az(test) - Az(R1.0) NE 0 Typically, a matched pair t-test is used to compare the average number of FP/I. H0: FP/I (test) GE FP/I(R1.0) HA: FP/I(test) LT FP/I(R1.0) Typically, a bootstrap analysis is used to compare the sensitivity: H0: Sensitivity (test) EQ Sensitivity (R1.0) HA: Sensitivity (test) NE Sensitivity (R1.0) The second study that FDA has historically accepted is a reader study. A multiple-reader multiple-case (MRMC) study design is used. MRMC studies are typically analyzed with an ROC or JFROC analysis. The areas under the curves (Az) without and with CAD are compared. The hypothesis is typically: H0: Az(with CAD) - Az(without CAD) EQ 0 HA: Az(with CAD) - Az(without CAD) NE 0 Note: Abbreviations: EQ is equal, NE is not equal, GT is greater than, GE is Greater than or Equal, and LT is less than. It is possible that FDA will require additonal or alternative tests for Riverain Medical's devices.

Collaborators

Freedman, Matthew MD (Georgetown University)
Vetter, Jennifer (Riverain Medical)
Lo, Ben Ph.D. (Georgetown University)