AI model monitoring in healthcare: running clinical AI after go-live

When an AI system is a medical device, monitoring starts with the manufacturer. In the European Union, the MDCG 2025-6 guidance, endorsed in June 2025 by the EU’s medical device and AI coordination groups, explains that the Medical Device Regulation (Article 83) and the AI Act (Article 72) both require manufacturers to run a post-market surveillance system: systematically collecting and analyzing performance and safety data, assessing and reporting adverse events, and taking corrective action where needed. The AI Act adds a post-market monitoring plan that actively collects performance data throughout the system’s lifetime, proportionate to the technology and its risks.

That does not let the hospital off the hook. The same document notes that deployers must monitor the system’s operation and, where relevant, inform the manufacturer. In France, the national health authority (HAS) and the CNIL go further in their joint guide on the proper use of AI systems in care settings (a working document dated February 2026, in French), which we use throughout as a worked example: a CE mark does not exempt the deployer from setting up local, continuous monitoring of the system in real-world use.

The manufacturer sees all of its installations; the hospital sees its own patients, devices, workflows and teams. The two kinds of monitoring complement each other, and responsibility cannot be allowed to fall into the gap between vendor and provider.

Before any dashboard, document the task: what the system does, for which population, at what point in the pathway, with which inputs, and which human decision follows. It is the same intended-use statement used to judge a tool’s explanations, covered in our article on explainable AI in healthcare.

For monitoring, its main job is to set priorities. Two models with the same average accuracy can carry very different risks. A false negative in a triage tool can delay care; repeated false positives can swamp the team and trigger unnecessary tests. How often an error occurs, how serious it is and how easily it is caught determine what you track, how often, and how fast you respond.

Input data

Track the variables that drive the output: missing-data rates, units, formats, devices, protocols, population and provenance. A new sensor, a new site, an electronic health record upgrade or an epidemiological shift can pull production data away from the data the model was tested on. This data drift, and the dataset documentation that makes it detectable, are covered in our article on health data quality.

Keep indicators tied to the model: measuring every available field just creates noise. Identify the critical inputs, and the known changes before they reach production.

Model performance

Once ground truth becomes available, compute the metrics that suit the task, by site, device and relevant subgroup, with adequate sample sizes and uncertainty intervals. Discrimination, calibration and thresholds are covered in our article on patient risk stratification.

The delay before ground truth arrives matters. A final diagnosis may only be known weeks after the prediction, so the dashboard separates early signals, such as the distribution of scores, from outcomes confirmed later. The HAS-CNIL guide recommends regular audits of the cases the system missed: false negatives are what overconfidence in a tool lets slip through.

Human use

A tool can stay technically stable and still become unsafe if clinicians follow it blindly, ignore it or use it off-label. Measure exposure, acceptance, overrides, workarounds, added time and qualitative feedback. The HAS-CNIL guide gives the rate of false positives corrected by a human and the rate of actual use as examples of indicators, and asks that the post-deployment review look for workarounds, meaning the tool being used for something other than its intended purpose.

Look for automation bias and for systematic rejection alike. When the tool fires alerts, their volume and override rate also fall under alert fatigue.

Effect on the care pathway

Algorithmic performance does not prove clinical benefit. Track decisions, delays, extra tests, adverse events, loss to follow-up and workload, and compare with a relevant period or group where you can. An apparent gain may simply be work shifted onto another team.

We keep that distinction at the heart of MyTwin for clinicians. The patient digital twin is analyzed continuously and alerts the physician only when necessary: it produces a signal, the clinician makes the decision, and clinical outcomes are measured separately.

For each indicator, define an expected range, a watch zone and an escalation threshold. Spell out who checks, how quickly, with what data, and the options available: investigation, restricted use, rollback to a previous version or suspension.

Thresholds are not only statistical. A single serious incident can justify immediate action, while a small shift calls for trend analysis. Governance also keeps the channels apart: technical alert, safety incident, user complaint, regulatory vigilance event. For AI that is a medical device, serious incidents go to the regulator: in the EU, the MDCG 2025-6 guidance recalls that manufacturers must run a vigilance system and report adverse events. In France, the HAS-CNIL guide adds that a serious incident involving a device must be reported to the national medicines and devices agency (ANSM), that someone in the organization should be explicitly in charge of those reports, and that tools without device status deserve a reporting route to the vendor and, where clinical risk is significant, an internal “algorithm vigilance” procedure.

Plan for a safe shutdown

The same guide recommends being able to deactivate the system quickly or stop it immediately after a serious incident, with established fallback procedures that can be rehearsed. Clinicians need to know how to raise a concern and how to work without the tool. Heavy organizational dependence can prevent a suspension at the very moment it becomes necessary.

For the HAS-CNIL guide, a local check remains relevant even for a CE-marked system, to catch biases specific to the local population or technical setup. It recommends at least a usage test on a local dataset that includes cases representative of the patient population and atypical cases, and standardizing those tests so they can be rerun, for example after a major model update. For a model that is not locked and keeps learning, these checks should be repeated regularly to spot drift.

On the organizational side, the guide recommends naming a local lead for each system, in charge of follow-up, frontline feedback and the link with governance. Around that person, a small committee brings together a clinical lead, a quality or risk manager, the technical team and, depending on the issue, the data protection officer, pharmacy, biomedical engineering or the laboratory. Every indicator has a named owner.

How often to review depends on risk and volume: more often for a critical signal or during a pilot, less often once use has settled. A major update, a device change or an extension to a new population triggers a reassessment.

Performance cannot be interpreted if you do not know what changed. Keep track of the version of the model, its parameters, the reference data, the interface and the clinical protocol, and state whether the model learns after deployment or stays locked. The HAS-CNIL guide recommends a full update log (date, version, purpose, expected results) open to both users and management, release notes sent ahead of time by the vendor, a test environment where needed, and a performance check after any major model update to catch regressions.

Predetermined change control plans

Some model changes can be planned and cleared at authorization. In the US, the FDA has issued final guidance, in its August 2025 version, on predetermined change control plans (PCCPs) for AI-enabled device software functions: the plan describes the planned modifications, the methods used to develop, validate and implement them, and an assessment of their impact. In the EU, the MDCG 2025-6 guidance states that changes predetermined by the manufacturer and assessed at the initial conformity assessment do not count as a substantial modification under the AI Act.

A PCCP governs change; it does not replace monitoring. The FDA, Health Canada and MHRA principles in fact ask that it include mechanisms to detect, and then revert or stop, a change that fails to meet its performance criteria.

Monitoring often relies on health data. Define the legal basis, the purpose, who has access, how long data are kept and the security measures. Use aggregated data when that is enough, but keep a controlled route for investigating an individual incident.

Subgroup monitoring may be needed to reveal performance gaps. It has to be justified, proportionate and secure. The absence of a sensitive variable does not guarantee the absence of bias, because a proxy can reproduce it. In a study published in Science, an algorithm underestimated the needs of Black patients because it predicted healthcare costs rather than illness: with unequal access to care, less was spent on them for the same level of need.

Do not quietly reuse monitoring data to retrain the model. Evaluating, developing and deploying are separate purposes, each with its own checks. The CNIL draws this line for French hospitals: evaluating the impact of a deployed AI system requires prior formalities, and data kept beyond the study for other projects turns the processing into a health data warehouse, with its own rules.

A minimal dashboard can include:

• volume of use, population and site;
• missing-data rate;
• distribution of outputs;
• confirmed false positives and false negatives;
• performance by relevant subgroup;
• human override rate;
• incidents and delays;
• the version in service;
• a note on changes in context.

Every indicator should answer three questions. What does a change mean? Who checks? What action is possible? If there is no answer, the indicator is probably decorative.