Series: The Patient, the Physician, and the Machine

Part 3 of 3

There is something reassuring about putting a physician’s name on something, even if it’s just my illegible signature.

Healthcare has done this for decades. We sign charts, approve protocols, review cases, and supervise other clinicians. The signature creates a visible line of accountability. It says that someone qualified has looked at this, and someone is responsible.

As AI moves deeper into clinical care, it’s easy to reach for the same model: have the AI do the work, and have a physician supervise it. Except I’m increasingly convinced that we don’t actually know what “supervision” means in this context.

In the first article of this series, I wrote about patients already using AI to fill gaps in healthcare. In the second, I argued that physicians need enough humility to recognize that these tools may outperform us in certain tasks, and that putting a doctor above the machine does not automatically make care safer.

That leaves us with the harder part: what should meaningful oversight actually look like?

I wish I had a clean answer, but I don’t. And given how quickly this technology is evolving, I’m skeptical of anyone who claims they do.

An AI that summarizes a clinical chart is doing something very different from one that recommends a diagnosis or triages a patient. It seems unlikely that one supervision model will work for all of them.

But we can start defining what good oversight should accomplish.

More Than a Checkbox

At its best, clinical oversight catches problems, identifies patterns, and improves care. But we should be careful about assuming that putting a physician in the loop automatically improves quality.

I previously cited a randomized JAMA Network Open trial where an LLM working alone outperformed both physicians using conventional resources and physicians using the LLM. A separate 2024 Nature Medicine study of 140 radiologists found that inaccurate AI predictions could actually worsen physician performance, and that experience or familiarity with AI did not reliably predict who would benefit from its assistance.

Human oversight can be valuable, but it doesn’t become valuable just because a human is present. And even when it works in theory, it can become perfunctory in practice.

Anyone who has worked around clinical operations has seen this. A physician is required to review ten charts. They know and trust the clinician they’re reviewing. So the physician clicks through quickly, satisfies the requirement, and moves on.

Technically, the oversight happened. Practically, very little was overseen. AI could magnify that problem in ways traditional oversight never had to contend with.

Medicine has already seen what happens when clinicians are asked to respond to machine-generated signals at high volume. Sepsis alerts are one example: when alerts fire too often, physicians develop alert fatigue and begin ignoring them, including alerts that may really matter.

Now imagine that problem at AI scale. These systems can generate recommendations and decisions at a volume no physician can actually scrutinize one by one. At some point, traditional “physician oversight” risks becoming an expensive formality that creates the appearance of accountability while lacking the substance of meaningful review.

Another mentionable issue is that AI often communicates with too much confidence. It can sound polished and certain even when it’s wrong. If we’re not careful, we could create a system where physicians technically supervise AI but are really just deferring to it.

Oversight has to do more than that. It should tell us how a system is performing, where it’s failing, and when something needs to change, and a signature alone can’t do that.

A Few Models Worth Exploring

I don’t think we should rush to design a single framework, because there are several models that might make sense depending on the clinical risk.

The most familiar is human review at the point of care, in which AI makes a recommendation and a clinician reviews it before it affects the patient. For higher-risk decisions, that will probably remain essential. But no physician can approve thousands of AI decisions, so it creates an obvious ceiling.

Another possibility is exception-based oversight. AI operates within defined boundaries, while uncertainty, unusual findings, or higher-risk cases are escalated to a clinician. Physicians spend their time where human judgment matters most rather than approving what the system already does reliably.

We’re already testing versions of this approach. In a randomized MASAI breast-screening trial, an AI-supported workflow reduced radiologists’ screen-reading workload by about 44% while increasing cancer detection, without significantly increasing false positives. This is attractive in theory but raises some difficult questions.

Who defines the boundaries? How confident does the system need to be before it acts? What happens when the AI doesn’t actually know that it’s uncertain?

A third model is performance-based oversight, where clinicians monitor the system over time instead of concentrating on each individual output. When it’s making mistakes, what physicians are overriding, where performance is drifting, and specific scenarios where errors are clustering.

At that point, the physician is overseeing a clinical system rather than reviewing every decision the system makes, and that may ultimately be where some forms of AI oversight need to go. But that also looks very different from the supervision structures healthcare has historically built around people.

Of course, these approaches can coexist. The right mix will likely depend on what the AI is doing and the potential risks of it getting something wrong.

The boundaries may also need to move over time. Strong performance could justify greater autonomy within a defined scope, and new patterns of error should tighten that scope. To me, that feels much closer to meaningful oversight: dynamic, measurable, and based on what’s actually happening in patient care.

We Should Be Comfortable Saying “I Don’t Know Yet”

Healthcare likes defined responsibility for good reason, but AI is moving faster than our existing frameworks, and false certainty carries its own risk.

We still have fundamental questions to answer. To name just a few of them: How much AI activity can one physician oversee? What should trigger escalation? When should a tool lose autonomy? Who has the authority to stop it?

I have opinions, not answers. I don’t know exactly what physician oversight of AI will look like five years from now, but I do know that putting a doctor’s signature at the end of an algorithmic workflow will not be enough.

Whatever we eventually call this relationship, the substance matters more than the label. We need to test different models, measure what happens, learn from our mistakes, and change course when the evidence demands it.

Most importantly, we should be able to see clearly if these systems are actually improving patient care.

We have an opportunity to build something meaningful and potentially life-saving before the checkbox becomes the standard, and we should use it.

— Rafid Fadul, MD