Beyond the Black Box: Explainability as an Ethical Imperative for Artificial Intelligence in Intensive Care Decision Making

Photo ID 135634014 © ProductionPerig | Dreamstime.com Abstract Artificial intelligence is increasingly integrated into intensive care decision-making through predictive analytics in clinical decision support systems. While these technologies may improve diagnostic accuracy and patient outcomes, their ethical use depends not only on predictive performance but also on explainability. In the intensive care unit, clinicians must understand and communicate the rationale behind AI-generated recommendations to preserve patient autonomy, maintain trust, and uphold professional accountability. Consequently, explainability should be viewed as an ethical obligation rather than simply a technical improvement. Introduction The intensive care unit is where healthcare teams make their most difficult decisions for critically ill patients. These decisions can range from mechanical ventilation, vasoactive support, and antimicrobial therapy to withdrawing life-sustaining treatment. These are unfolding under various constraints, including time pressure, complex clinical cases, and significant uncertainty. Artificial intelligence (AI) has entered this field, offering significant potential. Predictive models can stratify sepsis risk, forecast hospital length of stay, identify early signs of deterioration, and inform therapeutic decisions in cases of severe infections.[1] The appeal is clear: AI can reliably detect a patient's decline before vital signs visibly worsen, with clear clinical value. However, the most effective machine learning models are also the most opaque. Their output cannot create reasoning that a clinician can follow, interrogate, or explain to a patient. This paper argues that transparent models can still achieve strong clinical performance while offering insights that support clinician trust, patient communication, and accountability. A model that assigns a high mortality probability offers a number, not an account. This opacity is not a minor inconvenience to be solved by future engineering. An AI with recommendations that practitioners cannot understand, communicate, or justify presents a first-order ethical problem in a context where decisions will determine the last hours of human life.[2] This paper argues that explainability, the capacity of an AI system to produce human-understandable accounts of its outputs, is an ethical requirement in intensive care, grounded in patient autonomy, professional accountability, and clinical trust. It is not an optional feature. It is a moral condition of deployment. The Black Box in the ICU AI-powered Clinical Decision Support Systems are already embedded in intensive care workflows, addressing critical areas such as sepsis prediction,[3] antimicrobial recommendations,[4] length-of-stay estimation,[5] and deterioration risk identification.[6] Moreover, such tools often demonstrate strong technical performance, with metrics such as Area Under the Receiver Operating Characteristic Curve (AUROC), sensitivity, and specificity used to evaluate their effectiveness.[7] However, performance metrics are not an ethical guarantee. A system capable of making precise predictions without disclosing its reasoning offers the clinician a suggestion yet lacks an explanation. Practitioners need more than just an AI-generated risk score. They require explanations that identify patient-specific factors driving the output, align with established clinical reasoning, and integrate into existing cognitive frameworks.[8] Without this, the clinician faces a binary choice: accept the AI recommendation on faith or on experience. Neither option supports sound clinical practice, and both foreclose the kind of meaningful human-AI collaboration that responsible deployment requires. Explainability and Patient Autonomy The most fundamental ethical ground for explainability is the patient's right to participate in decisions about their own care. In the ICU, this right is already constrained by illness severity and altered consciousness. In many cases, patients cannot engage in real-time decision-making; in those cases, families and surrogates must act in their place. When AI begins shaping clinical recommendations in these circumstances, a basic ethical question emerges: do patients or families even know an algorithm played a role? Even if they know about the AI tool, the clinician may not be able to explain what the system saw and why it suggested a particular course of action in plain language. If a clinician cannot explain the basis for the AI‑generated recommendation, the disclosure may not be meaningful. Informed consent becomes impossible. Explainability is not a technical feature to be added later; it is the foundation that makes consent ethical. Without it, clinicians may unintentionally present algorithmic outputs as their own judgment, blurring the line between human reasoning and machine inference. That blurring is ethically unacceptable, no matter how accurate the model is. These concerns become even more pressing at the end of life. When AI influences decisions about withdrawing life‑sustaining treatment, opacity carries a moral weight that cannot be ignored. Families making surrogate decisions deserve clarity not just about the medical facts, but about whether an algorithm shaped the recommendation and how. In these moments, explainability is not merely a technical safeguard; it is an expression of respect for persons and for the dignity of the dying. Any AI system that obscures its role in such decisions risks undermining the very autonomy that critical care ethics seeks to protect. Professional Accountability and the Abdication Risk The clinician is the accountable agent in intensive care. The physician who prescribes a vasopressor, initiates mechanical ventilation, or documents a decision to withdraw life support bears professional, legal, and moral responsibility for that choice. AI systems bear none, although their developers, manufacturers, vendors, etc. may bear some in the form of product liability. They cannot be sanctioned, appear before an ethics committee, or explain their reasoning to a court or a grieving family. This asymmetry generates what might be called the abdication risk, in which unexplainable AI outputs gradually displace accountable clinical reasoning without a corresponding transfer of moral or professional responsibility. When AI recommendations shape decisions but cannot be interrogated, accountability gaps widen. The emerging literature on AI governance highlights this directly, as post-hoc explainability methods can complicate accountability by producing explanations that do not accurately reflect model behavior.[9] This potentially allows developers and institutions to evade responsibility for harmful outcomes while appearing transparent. A misleading explanation is not a step toward ethical clarity; rather, it is a step away from it. The ICU literature makes this risk concrete. Research on sepsis prediction in ICUs documents this problem concretely.[10] Ethical concerns about bias, opacity, and the absence of clear accountability mechanisms remain unresolved barriers to safe deployment, even for models with strong predictive performance.[11] For professional accountability, clinicians must be able to reconstruct the reasoning behind their decisions. Predictive accuracy does not compensate for the inability to reconstruct how a recommendation was generated or why it should be trusted. Trust as an Ethical Stake Trust is foundational in intensive care because patients and families enter the ICU profoundly vulnerable. They rely on clinicians for clarity, honesty, and explanations of what is happening and why. When opaque, black-box AI systems are introduced here, they disrupt the transparency that enables these relationships. The question becomes whether and how trust can endure when clinicians cannot understand the reasoning behind AI-generated recommendations. Studies evaluating explainable AI tools in the ICU show that specific, clinically meaningful explanations strengthen clinician trust and support appropriate acceptance of AI recommendations.[12] When an AI system highlights recognizable features and aligns with clinical intuition, clinicians engage with its outputs thoughtfully and maintain the calibrated trust responsible judgment requires. Without such explanations, clinicians are uncertain how to interpret or rely on AI recommendations, directly affecting decision-making quality. Trust also operates at the institutional level, where transparency about AI's role is essential for public confidence. When patients or families learn that decisions were influenced by systems that clinicians cannot explain, institutional trust erodes beyond any single encounter.[13] For trust to remain intact, healthcare organizations must ensure that clinicians can understand and communicate how the AI system came to its conclusion, e.g., which factors or metrics mattered and the source of data. Black-box systems’ opacity undermines institutional integrity and weakens the trust patients, families, and communities depend on in digitally supported critical care. The Limits of Current Explainability Methods Asserting explainability as an ethical requirement does not mean that current explainable AI tools satisfy it. Post-hoc explanation methods, such as Shapley Additive explanations (SHAP) values, Local Interpretable Model-agnostic Explanations (LIME), and attention visualization, produce approximations of model behavior that may not accurately represent why a specific output was generated.[14] An incorrect explanation is not merely unhelpful; it can create false confidence, encourage overreliance on AI outputs, or misdirect clinical reasoning in the very scenarios where caution is most warranted. This limitation sharpens rather than undermines the ethical argument presented here. The claim is not that any explanation suffices. Rather, genuine, validated explanations that acc