Real-Time Medical Device Performance Monitoring with CUSUM Control Chart
Principal Investigator
Name
Constantine Gatsonis
Degrees
Ph.D.
Institution
Brown University
Position Title
Professor of Biostatistics
Email
About this CDAS Project
Study
NLST
(Learn more about this study)
Project ID
NLST-1331
Initial CDAS Request Approval
Oct 11, 2024
Title
Real-Time Medical Device Performance Monitoring with CUSUM Control Chart
Summary
This project focuses on developing methods for real-time performance monitoring of AI/ML-based medical devices in healthcare. Specifically, it aims to address the challenge of delayed reference standards in post-market surveillance, where immediate outcome confirmation (e.g., cancer diagnosis) is often not available. To achieve this, the project proposes novel doubly robust estimators for key diagnostics accuracy metrics such as predictive values, sensitivity, and specificity. These estimators are integrated into a modified CUSUM control chart, which will be used to detect performance degradation over time. The National Lung Cancer Screening Trial (NLST) data will be used to evaluate the proposed methods, simulating scenarios where delays in obtaining reference standards affect real-time performance monitoring.
Aims
1. Develop doubly robust estimators for diagnostics accuracy metrics (e.g., sensitivity, specificity, and predictive values) when reference standards are delayed or missing in the target population.
2. Integrate these estimators into a modified CUSUM control chart with dynamic control limits for real-time monitoring of the diagnostics performance of medical devices.
3. Apply the proposed framework to the NLST data to evaluate its effectiveness in detecting performance degradation under synthetic distribution shifts.
Collaborators
Dr Constantine Gatsonis, Dr Jon Steingrimsson