Explainable AI in healthcare: what an explanation must let clinicians do

The same technology can sort a worklist, flag a signal, propose a measurement, estimate a risk or help draft a report. The consequences of an error, and therefore what needs explaining, change from one use to the next. Before looking at the interface, the organization documents:

• the target population and exclusions;
• the expected input and minimum data quality;
• the output and its unit;
• the responsible user;
• the point in the care pathway where the result is used;
• the possible action after a positive, negative or uncertain result;
• the risks of an error, or of not using the tool.

The HAS–CNIL guide recommends keeping an inventory of the AI systems in use, updated at least once a year: name and version of each tool, vendor, purposes, regulatory status, risk level under the EU AI Act, user departments, local lead and monitoring arrangements. That is where any explainability question starts: you cannot judge the explanation of a tool when you know neither its version nor its purpose. Starting from the decision rather than the predicted variable is the approach set out in our article on patient risk stratification.

An explanation that works for an engineer is not necessarily useful to a radiologist, a nurse, a primary care physician or a patient. Its level of detail should support action without hiding critical information.

Explainability does not make up for poor validation. A heatmap does not compensate for a model tested on a population too different from the one the hospital treats. Conversely, a model that performs well on average can be misunderstood and lead to errors in use. The two issues have to be handled separately.

In a viewpoint published in The Lancet Digital Health in 2021, Ghassemi, Oakden-Rayner and Beam go further: current explainability methods are unlikely to deliver the trust, transparency or bias reduction expected of them for patient-level decisions. They advocate rigorous internal and external validation instead, and caution against making explainability a requirement for clinically deployed models.

The HAS–CNIL guide keeps both requirements in view. It advises against signing with a vendor if the system’s operating, explainability or learning principles remain opaque to the organization, or if there is no evidence of validation in the intended population. Discrimination, calibration, subgroup performance and thresholds are covered in our article on patient risk stratification, and imaging-specific criteria in the one on AI in medical imaging.

1. Is it faithful to the model?

A post-hoc explanation can approximate a model’s behavior without reproducing it exactly, and two methods can assign different importance to the same variables. The vendor should state the method, its assumptions, its stability and the tests performed. A plausible explanation is not causal evidence.

2. Does the intended user understand it?

Article 14 of the EU AI Act requires that the people overseeing a high-risk system be able to understand its capabilities and limitations and correctly interpret its output. That has to be tested with the clinicians concerned: can they restate the result, identify its limits and choose the right action? A cluttered screen or ambiguous wording can reduce safety, even when the technical documentation is complete.

3. Does it show uncertainty?

A score on its own invites a binary reading. Users need to know whether there is an uncertainty zone, missing data, insufficient image quality or an out-of-scope situation, and the system should be able to abstain when its conditions of use are not met. Input data quality depends on the use: that is the subject of our article on health data quality.

4. Can the result be challenged?

The same article 14 provides that the person overseeing the system can decide not to use it, disregard, override or reverse its output, and stop it safely. In practice, oversight is only real if clinicians can do so without disproportionate friction. Decide who arbitrates, how disagreement is documented and how an incident is reported. A reject button is not enough if the organization penalizes the people who use it.

5. Is it useful to the patient?

Patient information explains the role the system plays in their care without handing them the technical interpretation: purpose, data used, the clinician’s role, main limitations and applicable rights. The HAS–CNIL guide suggests a simple practice: add a sentence to the report naming the system, its version and the physician who validated the document. Under article 86 of the EU AI Act, people affected by a decision based on a high-risk system also have the right to a clear explanation of the role AI played.

6. Does it hold after an update?

An update to the model, the software, the sensor or the care pathway can change results. The organization must know which version is in use, monitor performance and reassess the explanations. Drift can come from the data or from practice, not just from the code.

The French guidance is unambiguous. Under current law, the HAS–CNIL guide concludes that fully automated decisions are ruled out in care settings, and that every care decision made with an AI system must be overseen by the clinician responsible for the patient. It also notes that there is, as yet, no specific liability regime for clinicians using AI: they answer for how they use the tool and for the degree of autonomy they give it.

That calls for explicit roles: who receives the result, who decides, who responds to an alert, who tracks incidents and who can suspend the tool. For high-risk systems, the deploying organization checks that the people in charge of oversight have the competence, training, authority and support they need. The guide also lists the risks to anticipate: overreliance, algorithmic anchoring, automation bias and loss of human expertise.

A useful procedure also sets response times. A highly sensitive alert that cannot be acted on in time creates a false sense of safety; an overly intrusive system feeds alert fatigue. The explanation has to be designed with the workflow.

This is the principle behind MyTwin for clinicians. The patient digital twin, fed by real-world data between visits, informs follow-up; the clinician keeps the interpretation and the decision.

A multidisciplinary committee can gather:

• vendor documentation and model version;
• an analysis of the clinical need and the alternatives;
• validation on representative data;
• risk and data protection assessments;
• misuse and workaround scenarios;
• user comprehension tests;
• a training and support plan;
• performance, equity and safety indicators;
• a log of changes and incidents;
• criteria for suspension or withdrawal.

On personal data, the HAS–CNIL guide points out that under the GDPR the deploying organization is usually the data controller: a record of processing, a data protection impact assessment where required, and information for patients on how their data are used. Reusing data to train or evaluate a model is a separate processing operation. Explainability is no substitute for GDPR compliance.

On incidents, an AI system with medical device status falls under device vigilance: in France, serious incidents must be reported to the national medicines agency (ANSM). For tools without that status, the guide recommends a reporting channel to the vendor and an internal “algorithmovigilance” procedure when the clinical risk is significant.

After deployment, the team tracks usage rates, disagreements, delays, known false positives and false negatives, abstentions, incidents and user experience. It checks whether the tool changes decisions as intended or creates new workarounds. The WHO asks designers, developers and users to assess AI applications continuously and transparently during actual use.

A regular review compares performance with the initial conditions, by site and by clinically relevant subgroup. If the organization lacks the data to reach a conclusion, it says so, limits use and collects the missing evidence.