
Most healthcare organizations now run AI in production, and few can show they control it. In Black Book’s 2026 report, 78% of organizations have at least one AI or ML system in sustained production, yet only 19% maintain complete controls for ownership, validation, monitoring, and rollback.
The results swing both ways. ECRI named misuse of AI chatbots the top health technology hazard of 2026, while a multisite JAMA study found that AI scribes trim documentation time and EHR use.
In our new article, we’ll discuss where AI delivers value in healthcare software, where it falls short, and how we decide when to use it.
Key Takeaways
- AI pays off in narrow, high-volume tasks where a person reviews the output.
- Ambient scribes reduce burnout, and the time savings are modest.
- Imaging support is the most mature clinical use case.
- Administrative automation carries the lowest clinical risk, and it is a good first project.
- Prediction models and general chatbots fail most often when nobody validated them locally.
- Integration, data quality, and monitoring after launch decide whether a pilot survives.
The Numbers at a Glance
Five findings from named sources frame the rest of this article.
- 78% versus 19% is the gap between organizations running AI in production and organizations with complete controls over it.
- 75% to 80% of FDA-authorized AI devices are radiology devices, out of more than 1,400 devices on the FDA list.
- 32% of AI scribe users reached the heavy-use level, above 50% of visits, where the biggest gains appeared.
- 67% of sepsis patients went unidentified by the Epic Sepsis Model in a 2021 external validation, while it alerted on 18% of hospitalized patients.
- 63% of healthcare organizations use AI and automation in the revenue cycle, and 51% name IT infrastructure limits as the biggest obstacle.

These figures describe the sector. Your results depend on the workflow, the data, and the review process around the tool.
How We Decide Where AI Belongs
We ask four questions before we recommend AI for a feature. In our experience, a yes to all four points to a clear path, and a no to the second or third question is a stop sign.
- Is the task high-volume and repetitive? AI earns its cost when thousands of similar items arrive every week.
- Does a person review the output before it matters? A clinician who signs a draft note is a safety net. An alert that nobody reviews offers no such net.
- Can we measure the result against a baseline? Without a number from before the launch, nobody can say whether the tool helped.
- Is the data clean and reachable? A model inherits the quality of the records that feed it.

The sections below apply those questions to real use cases.
Where AI Delivers Value
Three areas show consistent value. Each one involves a narrow task, a clear definition of correct, and a person checking the result.
Ambient documentation
AI scribes listen to a visit and draft the note for the clinician to review. A Mass General Brigham and UCSF study tracked their use across five US hospitals for more than two years and found modest reductions in EHR and documentation time.
The gains concentrated in frequent users. Clinicians who used the scribe in more than half of their visits saw twice the reduction in total EHR time and three times the reduction in documentation time, but only 32% of users reached that level.
Burnout moved more than time did. A quality-improvement study of 263 ambulatory clinicians, summarized from JAMA Network Open, found burnout fell from 51.9% to 38.8% after 30 days of scribe use.

The revenue case needs care. Extra visits produced a statistically significant but small gain, at $167 per month per clinician, so the case for a scribe rests on clinician wellbeing and retention more than on throughput.
Imaging support
Imaging is the most mature clinical use of AI. Radiology accounts for roughly 75% to 80% of the AI-enabled devices the FDA has authorized, and the list now holds more than 1,400 devices.
It works for structural reasons. The task is narrow, the correct answer is usually verifiable, and a radiologist still reads the study. Triage and prioritization tools help that radiologist work faster without taking over the diagnosis.
Authorization doesn’t guarantee a good fit for your patients. Ask each vendor for performance data from populations that resemble yours.
Administrative automation
Prior authorization, coding, eligibility checks, and denial management carry the lowest clinical risk of any AI use case. An error costs time and money, and a staff member can catch it before it reaches a patient.
Adoption is already broad. An HFMA-FinThrive survey found that 63% of healthcare organizations use AI and automation in the revenue cycle, and 51% named IT infrastructure limits as the biggest obstacle.
That obstacle comes up again below. The tool is seldom the hard part, and the systems around it usually are.
Where AI Falls Short
Failures in healthcare AI follow a pattern. The model works in the demo, then meets local data, busy clinicians, or nobody watching.
Prediction models nobody validated locally
The Epic Sepsis Model is the best-known case. In a 2021 external validation study, University of Michigan researchers analyzed 38,455 hospitalizations and found that the model missed 67% of sepsis patients while generating alerts on 18% of all hospitalized patients.
Epic disputed the findings, saying the study skipped the tuning that has to happen before real-world deployment. Whichever side you favor, the lesson holds: validate and tune a model on your own patients before anyone relies on it.
Alert volume matters as much as accuracy. An alert on 18% of hospitalized patients creates the alert fatigue that the researchers described.
General-purpose chatbots giving medical answers
ECRI ranked misuse of AI chatbots first on its 2026 list of health technology hazards. Its experts documented chatbots suggesting incorrect diagnoses, recommending unnecessary tests, and inventing body parts while sounding like a trusted expert.
ECRI doesn’t call chatbots inherently dangerous. It points to how people understand and use them.
In our view, a chatbot inside a healthcare product needs guardrails: a narrow scope, answers drawn from approved sources, and a clear handoff to a human. A general-purpose model answering open medical questions has none of those by default.
Scribe notes and the review problem
Scribes make errors. In a pilot where 31 physicians used an ambient scribe, reviewers found that 94.7% of 356 assessed notes had no significant errors.
That result is good, and it still means about one note in twenty carried an error that mattered. Review only works if clinicians have the time and the habit to do it.
Legacy systems and messy data
Many AI projects stall at integration. In Black Book’s data, 63% of organizations rank EHR, PACS, RIS, LIS, or workflow integration among the leading reasons that promising pilots fail to scale.
Data quality adds to the problem. In a KLAS and UPMC survey, health system leaders named manual workarounds, spreadsheets, and inconsistent definitions across teams among their top data quality pain points. A model built on that data inherits its gaps.
No monitoring after go-live
Governance is forming faster than controls. 72% of organizations report a formal AI governance body, yet only 29% can produce a complete inventory of their production models within 48 hours.
Monitoring lags too. 76% of organizations track technical accuracy, 32% track clinical outcomes, and 28% track workflow burden. A model that drifts, or a workflow that slows down, can go unnoticed until a clinician complains.
What We’ve Seen in Our Own Projects
Our published healthcare projects are modernization work, and our published AI projects come from other industries. Each one still taught us something about where AI fits in healthcare software.
AI document processing for a mortgage servicing platform
For a commercial real estate finance firm, we built an AI-powered document processing system to replace manual handling of tax documents. We used an event-driven workflow on Azure Durable Functions and third-party AI tools to extract information, and employees can track each document’s status in real time.
Healthcare has the same shape of problem in intake forms, claims attachments, and prior authorization packets. The lesson we carry over is visible status and a person in the loop, so staff can see where every document stands.
An AI roadmap trimmed to the essentials
The client behind ARELIA, a platform for independent music artists, arrived with a roadmap that included AI. We tailored the web application to the most necessary functionality first and built the data integration that everything else depends on.
AI-powered analytics are being developed on top of that foundation, using Azure OpenAI. The same order applies in healthcare: consolidate the data first, then add inference.
Want a second opinion on an AI pilot you’re planning? We’re happy to take a look.
Get in touch →Regulation and Privacy Basics
The rules are still catching up. FDA’s revised clinical decision support guidance, issued January 6, 2026, covers clinician-facing software and stays silent on AI and on consumer-facing tools such as health chatbots. Developers still face uncertainty on the most pressing AI questions.
States are filling gaps. Several state laws that took effect on January 1, 2026, including Texas’s TRAIGA, address disclosing AI use to patients.
Privacy deserves its own review. An ambient scribe sends recordings and transcripts, which contain PHI, to a third-party vendor. That vendor needs a business associate agreement, a security review, and clear rules on data retention and model training.
Cost and ROI: What AI Projects Really Cost
AI costs more than the license, and the return depends on what you measure.
Costs beyond the license
- Integration. Connecting the tool to the EHR, PACS, billing system, or data warehouse.
- Local validation. Testing on your own patients and records before go-live.
- Clinician review time. The minutes spent checking and correcting output.
- Monitoring. Tracking accuracy, drift, and workflow impact after launch.
- Governance. Ownership, audit trails, vendor reviews, and incident response.

We don’t quote a multiplier here, because we haven’t found a trustworthy one for healthcare. Budget for these five lines as separate items.
When returns appear
Administrative automation tends to show returns first, because the results are countable: denials, hours, turnaround. Documentation tools take a few months, and the benefit shows up in clinician wellbeing as much as in time.
Clinical impact takes the longest to prove. It needs outcome data, and only 32% of organizations track clinical outcomes for their AI today.
Numbers to set before you start
- Baseline time on the target task. Measure it for two weeks before launch.
- One success metric. Pick a number a finance lead would accept, such as denial rate or note completion time.
- A review-time budget. Decide how many minutes of checking per item is acceptable.
- A shutdown threshold. Agree in advance on the result that means you switch the tool off.
How to Start Without Regret
- Pick one narrow workflow. Prior authorization intake or visit notes for one clinic beats a hospital-wide rollout.
- Validate on your own data. Run the tool on your records and your patients before anyone relies on it.
- Keep a person in the loop. Measure how long review takes, and count it as part of the cost.
- Monitor after go-live. Track accuracy, drift, performance across patient groups, and workflow burden.
- Write the exit plan first. Decide who can switch the tool off and how work continues without it.
Check the numbers again at 60 and 120 days. That window is long enough to see a trend and short enough to adjust.
Wrapping Up
AI has earned a place in healthcare software and in specific jobs. It drafts notes, prioritizes images, and clears paperwork well when a person checks the result.
It falls short where nobody validates it, nobody watches it, or the systems around it can’t feed it clean data. Those failures usually trace back to the surrounding work.
Our rule of thumb is to start with the task, then the data, then the model. If the task is narrow, the data is reachable, and someone reviews the output, AI is worth a pilot. If any of the three is missing, fix that first.

Several of the projects above started with modernization, and AI came later or not at all. We think that is the right order for most healthcare teams.
If you’re weighing an AI feature for your healthcare product, we’ll give you an honest view, including when the answer is no.
Get in touch →FAQs
-
Does AI actually save clinicians time?
Modestly. A multisite JAMA study found modest reductions in EHR and documentation time from AI scribes, with the largest gains among clinicians who used them for more than half of their visits. The researchers noted that the modest time savings are unlikely to fully explain the drop in burnout.
-
Is AI safe for clinical decisions?
It depends on the tool and the oversight. Narrow, validated tools with clinician review, such as imaging triage, have the strongest record. Models that nobody validated locally, and general-purpose chatbots, carry the most documented risk, and ECRI ranked chatbot misuse as its top health technology hazard for 2026.
-
Do healthcare AI tools need FDA approval?
Some do. Software that diagnoses or recommends treatment may count as a medical device, while administrative and billing software generally falls outside the clinical decision support guidance. FDA’s January 2026 guidance covers clinician-facing software and is silent on AI and consumer chatbots, so check each product’s intended use with regulatory counsel.
-
Is ambient AI scribing HIPAA compliant?
It can be, with the right controls. Recordings and transcripts contain PHI, so the vendor needs a business associate agreement, a security review, and clear rules on retention and data use. Some states also have rules on telling patients when AI is used.
-
How much does it cost to add AI to healthcare software?
License fees are the visible part. Integration, local validation, clinician review time, monitoring, and governance add to the total, and the mix varies by project. Price those five items separately before you set a budget.