Hi friend,
Generative AI is hard to regulate.
The systems are open-ended, probabilistic, and it’s hard to test them. To address these challenges in mental health, regulators are adapting their current medical device regulations to be suitable for AI systems (see our briefings on AI regulation in the US, Europe and the UK)
But what happens after a device is authorised?
This is perhaps the hardest part of the regulatory process. AI models change. A company might update its system prompt, retrieval corpus, guardrails or routing logic. The foundation-model provider they use might ship a new version of their model. All of these could change how the device behaves.
Even if the device stays the same, the world around it changes: language evolves and users behave differently. Because of the open-ended and generative nature of LLMs, new inputs can generate new outputs, and new failure modes can emerge.
In short, the product a regulator authorised may not be the product a patient encounters after authorisation.
There are two main approaches regulators are using to combat this challenge: Predetermined Change Control Plans (PCCPs) and the direct probing of deployed models.
Neither is perfect.
PCCPs have received the most attention but face many challenges. Direct testing holds potential in ensuring AI devices work well in the real world but has problems of its own. In this report, our third instalment on AI regulation with May Lee, we explain both approaches, their benefits, drawbacks and some ideas about what needs to be done to ensure AI is safe and effective in mental health.
Note: During the writing of this article, the UK’s National Commission into the Regulation of AI in Healthcare published its recommendations for a new regulatory framework. The report reaches many of the same conclusions discussed here: AI regulation needs to accommodate products that change after authorisation, make greater use of continuous real-world monitoring and make it easier to report problems. Notably, it recommends making PCCPs more flexible by defining the boundaries within which an AI system can change, rather than requiring every future modification to be specified in advance.
What are PCCPs
Medical device regulation was not built for devices that change. Historically, if your device changed, you needed to submit a new regulatory submission. Regulators realised that wouldn’t work for software or AI devices, so they created PCCPs.
A PCCP is a predetermined (duh) plan for certain future changes that manufacturers can agree with a regulator. Once agreed, manufacturers can make these changes without returning for a new submission each time.
Congress gave the FDA explicit authority to authorise PCCPs in 2022. The agency finalised AI-specific guidance in December 2024, and the latest update was in August 2025. The FDA, Health Canada, and MHRA also published joint principles describing what a good PCCP looks like. This includes being focused and bounded, risk-based, evidence-based, transparent, and oriented to the total product lifecycle.
A PCCP has three components:
The description of modifications: this lists the changes you intend to make. These changes must be specific, enumerated changes.
The modification protocol: this describes how you will develop, validate, and implement each one. It includes data management, retraining procedures, performance evaluation and update procedures.
The impact assessment: this analyses what each modification does to the device’s benefit-risk profile, including how the modifications interact with each other.
If the FDA authorises the plan, you can implement those changes without a new submission, provided that the changes stay within the authorised scope and you follow the protocol exactly.
The UK is now following. Predetermined change control for software is written into the draft Medical Devices (Amendment) Regulations 2026, which we covered in Part II.
Europe has no direct equivalent, however. Change control under the EU MDR still runs through the significant change notification process to your Notified Body, which is a different logic and a slower process.
In theory, PCCPs are good regulatory engineering. But they have some problems.
The Problems of PCCPs
1. PCCPs assume you decide when the model changes
PCCP’s architecture assumes the manufacturer always initiates the change in the device. Most generative products are built on third-party foundation models accessed under contract. When the provider (e.g., OpenAI, Anthropic, etc.) ships an update, the device’s behaviour changes. The manufacturer did not initiate this change; they may not have been told in advance, and often cannot even characterise what has changed.
The manufacturer maintains liability, of course and the FDA’s position on outsourcing is clear: the manufacturer is accountable for the whole product regardless of what was built in-house or outsourced. But the PCCP framework doesn’t give manufacturers a way to manage and account for the fact that they cannot always control or know how the underlying model will change.
The FDA is alert to this. Its August 2026 discussion paper asked directly how vendor-initiated model changes should be handled, suggests re-benchmarking a modified device against its premarket baseline, and floats a PCCP as one mechanism. That is the right direction. But what is still missing is a place in the PCCP structure where;
Upstream change gets its own treatment (separate from the modifications manufacturers chose to make)
How manufacturers detect that an upstream model has moved
What monitoring runs continuously against safety-critical behaviours
What performance boundaries trigger investigation
And what manufacturers’ rollback or version-pinning strategy is.
2. PCCPs Work Best for the Changes That Matter Least
A PCCP requires you to list your modifications in advance of making them. But the changes companies most need to make are the ones responding to failure modes they did not anticipate. That is the entire premise of postmarket learning — where you deploy, discover something you didn’t predict, and you fix it. By definition, those fixes are not in your plan. If you could have predetermined them, you would have fixed them at the time.
So the PCCP works best for the changes that matter least, like routine retraining on more data of the same kind, and offers nothing for the changes that matter most. If you find a novel crisis-handling failure six months after launch, you’ll need to get back in the submission queue, which is exactly the situation PCCP was invented to prevent.
It’s also worth noting that if you deploy a static model in a changing world, its performance can degrade over time. Patients use new language; there are new cultural idioms for distress, new presentations of crisis. This is particularly relevant for mental health use cases. We can’t predict these changes, so there is no modification to predetermine.
This would be very difficult to fix through better drafting. It is a structural feature of trying to pre-authorise change in a system whose value lies in adapting to things that can’t be foreseen.
3. It’s Unsettled What Counts as a Modification
PCCP guidance is written around model changes: retraining, reweighting, architecture, etc. However, model changes are not what usually shifts the clinical behaviour of an AI device. Prompt engineering, updates to a retrieval corpus, guardrail configuration, routing logic, temperature settings and safety filter thresholds are much more likely to be updated and to change the behaviour of the device. A prompt change can alter device behaviour more than a retrain, and such changes can be shipped by one engineer in an afternoon.
Do these device modifications require PCCP coverage? It’s currently unsettled. In reality, every company in this space is making these changes weekly, and most are making a good-faith judgment call with very little to go on.
If prompt and configuration changes fall outside the PCCP, then the plan governs the least active surface of the product. If they fall inside, most teams are already out of compliance and do not know it.
4. The Incentives Aren’t Aligned (and PCCPs Can be Gamed)
The more specific a manufacturer’s PCCP, the less flexibility they have to make changes later. But if they make it too broad, the regulator may reject it. This creates an awkward incentive to be specific enough to get the plan approved, but broad enough that future changes still fit within it.
A manufacturer can also define multiple performance thresholds and monitoring protocols in their PCCP, and seek approval for a wide range of future updates. In practice, this can create room for them to essentially operate an adaptive system — making multiple changes based on what they want to do, all technically agreed with the regulator.
What the UK’s National Commission says about PCCPs
The UK’s National Commission into the Regulation of AI in Healthcare recognises many of these challenges with PCCPs. In response, the Commission proposes making them more flexible. Rather than requiring manufacturers to specify every future change in advance, adaptive AI could instead be given agreed boundaries within which it is allowed to change, backed by testing and monitoring. Rather than saying, for example, “we will retrain the model on this dataset and expect performance to improve by X”, a manufacturer might agree a broader envelope of acceptable behaviour and performance. The system could then change within that envelope without every individual modification having to be predicted in advance. The Commission also argues that whether a change is significant should depend on its effect on the device performance — its clinical function, user interaction, autonomy and benefit-risk profile — rather than simply on what was technically changed.
This helps solve the problem that you cannot predetermine a change you do not yet know you will need. But it also makes continuous testing more important. The more freedom a system has to change, the more important it is to know whether it remains within the agreed boundaries.
Direct Probing
For a conventional medical device, regulators can physically buy your device, take it apart and see whether it does what the technical file says. For software devices, however, the only window into how the product performs is the one the manufacturer provides.
One idea for getting around this, and addressing some of the other challenges of AI devices and the limitations of PCCPs, is to directly test the endpoints of Gen AI devices in the wild. This is a somewhat simple concept where a regulator gets programmatic access to the deployed system, sends its own inputs, and observes the outputs.
For mental health, this has some specific benefits.
For example, you cannot ethically run a crisis scenario in a clinical trial. The FDA’s advisory committee spent November 2025 worrying about missed crisis cues and inadequate crisis response. Currently, the only way to systematically test these scenarios is through synthetic testing.
Europe legislated a version of this idea in 2024. EU AI Act Article 74(12) provides that market surveillance authorities shall be granted full access to documentation and to the training, validation, and testing datasets used for developing high-risk AI systems, including, where appropriate and subject to security safeguards, through application programming interfaces or other technical means enabling remote access. Article 74(13) goes further, granting access to source code on a reasoned request where necessary to assess conformity, and where documentation has proved insufficient. Authorities can also conduct or commission testing to verify compliance, and exercise certain powers remotely.
Now, this is not an explicit right to probe your production endpoint. It is remote access to development artefacts plus a testing power. But the components are assembled, API access is named in the statute, and the direction of travel is towards being able to directly test endpoints. For many, this would be the holy grail. Regulators may never have meaningful visibility into model weights, training data or vendor-side changes. But if they can continuously test the product patients actually use, at sufficient scale and against meaningful safety and performance criteria, then they will be much closer to understanding the device’s performance in the real world. Companies are unlikely to want this, however. Continuous evaluation is a level of scrutiny many will be uncomfortable with. However, considering the stakes, perhaps that’s a discomfort they’ll need to put up with.
The FDA, for its part, has already put shadow development on that table as one of the five approaches to clinical confirmation in its August 2026 discussion paper. Regulator-run probing is a short step from something the Agency is already considering.
Direct Probing has Problems Too
Evals (evaluations) is the name for tests attempting to grade the performance of AI devices. In mental health, most evals don’t work very well. In The Map is Not The Territory, we discuss many of these limitations. There is rarely one objectively correct response in mental health conversations, and good performance on a finite set of test conversations tells us little about how a model will behave across the messy, open-ended reality of care.
Direct probing is well suited to catching crisis-handling defects and may detect drift between versions. But it is not good at catching slow, longitudinal harms, which are, in our view, the most serious risks in this category. Meaningful testing requires many samples per scenario, a defined failure threshold, and a view on what failure rate is clinically acceptable. Anyone proposing API access as a general safety mechanism should be asked how it catches sycophancy and other longitudinal harms.
Despite its limitations, direct probing is better than no oversight.
The Need for a Credible Incident-Reporting System
Post-market oversight of AI will struggle if companies and clinicians have little incentive to report failures. Here, healthcare regulators could learn something from the aviation industry.
In 1974, TWA Flight 514 crashed into a Virginia mountain, killing all 92 people on board. Six weeks earlier, a United Airlines crew had made a similar mistake to Flight 514, but luckily for them, they had narrowly avoided an accident. The United crew filed a report of their mistake, but it stayed within United and was never communicated to the broader industry — there was simply no system for sharing this kind of safety information between airlines. The FAA understood that it could not learn about near misses and safety issues if reporting one automatically exposed the person reporting it to potential punishment.
In response, in 1976, the FAA launched the Aviation Safety Reporting System, operated by NASA so that reports could be confidential and independent of the regulator’s enforcement function. Pilots and others who voluntarily report incidents receive confidentiality and, under defined conditions, protection from FAA penalties.
Could something like this be created for manufacturers of Generative AI devices in healthcare?
Focus May Shift From Pre-Market to Post-Market
Because AI can drift, change and perform differently in different settings, post-market evidence may become more informative about actual safety and performance than pre-market evidence. Stronger ongoing monitoring could, for some devices, justify reducing the amount of evidence required before market entry. The NHS Commission recognises this, stating that;
“The nature of many AI-enabled medical devices [...] means that it is becoming increasingly difficult to demonstrate and assure real-world performance and safety at the pre-market stage.”
And…
“The MHRA should consider how to recalibrate the evidentiary requirements to be more weighted to the post-market phase.”
Heed the Call
Regulators face a tough challenge. Their existing processes weren’t built for technology that can change this quickly.
Regulation needs to accommodate this change without losing meaningful oversight. Manufacturers need room to update their products and to be incentivised to surface problems. PCCPs and direct probing have challenges, but they provide an interim safeguard and reduce risk incrementally while evaluation science matures.
The task for regulators is to invest in validated, standardised evaluation methods, developed jointly by engineers, clinicians, statisticians and regulators and, ideally, coordinated internationally. The aim should be to move towards these standards over time, while recognising that regulating AI will remain an evolving challenge.
They must do all this whilst ensuring that the process doesn’t become so burdensome that companies avoid the regulated pathway altogether.
In the words of Dylan, “Come senators, congressmen, please heed the call”. This will be a tough challenge for regulators and policy-makers, but the stakes are high. It’s important they get it right.
If you want to understand more about the fundamentals of mental health AI regulation, you can read Parts I and II in our series below:
And if you want more insider insights on mental health innovation, consider becoming a Hemingway Pro member. You’ll be joining more than 500 founders, executives, clinicians and investors who get access to deep industry insights and tactical guides every week. You’ll also get access to our private Hemingway Slack group and invites to our member-only events.
Many thanks to May for collaborating with us on this piece. Also, thanks to Ben Carrington for his contributions.
That’s all for this edition of The Hemingway Report.
Keep fighting the good fight!
Steve
Founder of Hemingway
