A framework for safe, ethical, and trustworthy AI deployment and integration in hospitals: a systematic review and regulatory gap analysis with emphasis on the U.S. regulatory context

BackgroundThe rapid deployment of artificial intelligence (AI) in U.S. hospitals has outpaced regulatory frameworks. By end-2025, the FDA had authorized over 1,451 AI-enabled medical devices, yet governance remains fragmented and reactive. Although the regulatory analysis focuses on the United States, the review included international evidence where directly relevant to U.S. hospital governance, and U.S.-specific conclusions rest primarily on the U.S.-based evidence.ObjectivesThis systematic review and regulatory gap analysis examined the current state of AI governance in U.S. hospitals, identified critical gaps between existing regulations and governance needs, and proposed a comprehensive framework for safe and trustworthy AI deployment.MethodsThe review was conducted and reported in accordance with PRISMA 2020. Four databases (PubMed, Web of Science, Scopus, Google Scholar) were searched for peer-reviewed articles published from January 2017 through January 2026 (final searches 15–18 January 2026). The PICOS framework structured inclusion criteria. Quality was assessed using CASP, NOS, and QUADAS-2 according to study design. A narrative synthesis approach was employed, followed by a structured regulatory gap analysis of federal statutes, regulations, and agency guidance.ResultsOf 1,065 records identified, 22 studies met the eligibility criteria. In the systematic review, FDA, ONC, and CMS standards were inconsistent and poorly tailored to AI's evolving capabilities, with evidence of inconsistency between pre-market approval and real-world performance, underutilization of continuous surveillance, and failure to conduct subgroup bias analyses. The regulatory gap analysis identified four critical gaps: post-market surveillance deficiencies, equity protection failures, accountability clarification shortcomings, and governance capacity disparities. Interpretation of these findings for U.S. regulatory policy is provided separately in the regulatory gap analysis and the Discussion.ConclusionA lifecycle governance framework is needed, incorporating continuous monitoring, transparency assurance, equity safeguards, accountability mechanisms, and scalable governance structures to ensure safe, ethical, and trustworthy AI deployment in healthcare. Key limitations include the retrospective protocol registration, the restriction to English-language publications, and the heterogeneity of the included evidence.Systematic Review RegistrationOpen Science Framework (OSF): https://osf.io/7q5xt.