Aidoc's AI tools: What independent research shows in real-world settings.

Performance figures in vendor materials are typically generated under controlled validation conditions. The independent studies reviewed here document meaningful sensitivity gaps in real-world settings — for example, 72.2% overall outpatient ICH sensitivity (Emory, 2025). This gap between validation-study performance and real-world deployment is a recognized challenge in clinical AI broadly. Health systems should require independent, site-specific validation before treating vendor benchmarks as predictive of their own environment.

The pediatric ICH study found that the algorithm — trained predominantly on adult imaging data — produced frequent false positives in children due to normal pediatric anatomical features (choroid plexus calcifications, hyperdense venous sinuses) that differ from adult presentation. This raises a broader due-diligence question health systems should ask of any AI vendor: has the tool been validated on the specific patient populations you serve, including pediatric or other subgroups that may differ from the training cohort?

Two prospective UAB studies — one evaluating PE triage, one evaluating ICH triage — found no statistically significant improvement in radiologist diagnostic accuracy when using the AI compared to reading without it. In the ICH study, specificity was lower with the AI than without it. Prospective controlled study designs are generally considered the strongest basis for evaluating clinical impact; the absence of a detectable benefit in two such studies is a meaningful data point for procurement decisions.

Multiple studies found Aidoc's performance varied significantly by scanner manufacturer, imaging protocol, and the presence of artifacts — motion blur, hardware-related noise, and calcifications. Hospitals use diverse scanner fleets; a tool that performs well on one manufacturer's hardware and fails on another's is not reliably deployable at scale.

The Lahey cervical spine study explicitly flagged that many similar AI algorithms — including Aidoc's — received "little or no external validation" before being deployed clinically. FDA clearance requires demonstrating performance, but does not require external validation across diverse hospital populations or independent replication of performance claims.

AI models can degrade over time as patient populations, scanner firmware, or imaging protocols shift — a phenomenon called model drift. Post-deployment performance monitoring in clinical AI is not yet standardized across the industry. Health systems should ask vendors specifically how ongoing performance is tracked, what thresholds trigger re-validation, and what contractual obligations exist if deployed performance diverges materially from validated benchmarks.