Abstract / Summary
Importance: The Medicare Physician Fee Schedule values ambulatory services using survey-based estimates of physician time, a costly process vulnerable to inaccuracy. EHR audit logs are passively and universally collected and likely meet the American Medical Association Relative Value Scale Update Committee (AMA RUC) Extant Database Standards for alternative data to inform service valuation estimates. However, audit logs' validity for estimating face-to-face time, a core component of many of the most used procedural codes, is unknown. Objective: To assess whether EHR audit logs can facilitate accurate estimation of physician-patient face-to-face time during ambulatory visits. Design: Single-site validation study pairing EHR audit logs with direct audio/video recordings of ambulatory primary care visits (reference standard). We assessed agreement between audit-log estimates and recorded visit duration with Bland-Altman analysis and mean absolute error. Setting: University of Pennsylvania Observer data repository, a multi-modal database of ambulatory visits to Penn Medicine primary care clinics including video and audio recordings, transcripts, clinical EHR data, and EHR audit-log data. Participants: 75 visits from 12 physicians with data collected between August 2023 and February 2025. Exposures: A rule-based algorithm estimated face-to-face time via clusters of physician EHR activity relative to patient check-in time. Inactivity thresholds were applied to differentiate intraservice time with no EHR activity (e.g., a physical exam) from the end of a visit, with longer thresholds used for longer visits. Main Outcomes and Measures: Duration of recorded face-to-face visit time. Results: Recorded face-to-face time averaged 31.6 minutes (SD, 13.9). The audit-log estimator was unbiased (mean bias: -0.63 minutes; 95% CI: -6.0 to 2.6; mean absolute error: 9.0 minutes) and estimated face-to-face time within approximately 2% (38 seconds) of the recorded visit time with no proportional bias (slope: 0.13; P = .25). The 95% limits of agreement were -25.7 to 24.4 minutes, and 37% of estimates fell within 10% of recorded duration. Conclusions and Relevance: In this validation study, EHR audit logs recovered unbiased estimates of ambulatory face-to-face time. Larger-scale studies are required to verify generalizability, but EHR audit logs offer a scalable means to improve the rigor of retrospective physician work relative value unit estimation. This study represents a first step in a potential paradigm shift for measuring components of the fee schedule across procedures, settings, and provider types.