You or someone you love takes the risk. Companies get sued. The AI walks away.
UNGOVERNED AI | ENFORCEMENT: HEALTHCARE AI AGENTS By Dr. Freddie Seba
UNGOVERNED AI | ENFORCEMENT: HEALTHCARE AI AGENTS By Dr. Freddie Seba
On Wednesday, I asked a question every health system board should be able to answer: if your AI agent harms a patient, who gets sued? Who loses their license?
Not the agent. It holds no clinical license to lose and no standard of care to breach. “First, do no harm” binds every physician, physician assistant, nurse, and advanced practice nurse. An AI agent answers to no licensing board.
So what happens when AI in healthcare goes wrong? Regulators cannot discipline an AI. Courts cannot sanction one. Instead, medical boards, attorneys general, and judges are reaching out to the developers and deployers behind it. Today’s enforcement file shows how, through three cases and a wave of new state penalties.
Case 1: Pennsylvania State Board of Medicine v. Character Technologies (May 2026, pending)
What happened. On May 1, 2026, the Pennsylvania State Board of Medicine filed suit in Commonwealth Court against Character Technologies, the company behind Character.AI. A state investigator created an account, searched “psychiatry,” and found a user-created chatbot called “Emilie,” described as a “Doctor of psychiatry.” The chatbot offered to conduct a psychiatric assessment, claimed to hold a valid Pennsylvania medical license, and cited a fabricated license number. It had logged about 45,500 interactions by mid-April.
The board alleges the unlicensed practice of medicine. The company contests the suit, saying user-created characters are fictional, intended for entertainment, and accompanied by in-chat disclaimers. No ruling has been issued.
Why it matters: this is a medical board, the body that licenses clinicians, going to court over an AI that claimed a license it could not hold. It is the clearest answer yet to Wednesday’s question. When an AI acts as a clinician, the board cannot discipline the AI. It pursues the company. The same company faces a separate lawsuit from the Kentucky attorney general, filed January 8, 2026, alleging harm to children.
Governance gap: Identity. What is this agent, and what is it entitled to do? If no one at the deploying organization can answer, the agent defines its own role.
Case 2: UnitedHealth nH Predict Class Action (U.S. District Court, Minnesota; March 2026 order)
What happened. In November 2023, families of two deceased Medicare Advantage members filed a class action against UnitedHealth Group. They allege that nH Predict, an AI tool developed by naviHealth (acquired by Optum in 2020), led to premature denials of medically necessary rehabilitation and nursing-home care for elderly patients, overriding physicians’ judgment. Plaintiffs allege a 90% error rate: most denials that patients appealed were reversed. They also allege the company kept using the tool because only about 0.2% of policyholders appeal.
In February 2025, the court allowed breach-of-contract claims to proceed and waived the requirement that patients first exhaust administrative appeals, calling that process effectively futile. On March 9, 2026, the court ordered UnitedHealth to produce a broad set of internal records. These include documents on nH Predict’s design and function, records of government investigations into its AI use, and records of its internal AI review board and its members, dating back to January 2017.
UnitedHealth disputes the claims, saying nH Predict does not make coverage decisions and that decisions follow Medicare criteria and plan terms.
Why it matters: This is a private lawsuit, not a regulatory action, but it shows where liability lands when AI shapes care decisions at scale. The court is not asking what the algorithm decided. It is asking what the company knew, who reviewed the tool, and who was accountable.
Governance gap: Product Ownership. Who owned the outcome: the algorithm, the vendor, or the plan? A 0.2% appeal rate means most errors are never seen. Unmeasured errors are still errors.
Case 3: Texas Attorney General v. Pieces Technologies (September 2024 settlement)
What happened. In September 2024, the Texas Attorney General announced a settlement with Pieces Technologies, a Dallas-based healthcare AI company, in what the office called a first-of-its-kind investigation into generative AI in healthcare. At least four major Texas hospitals were sending patient data to Pieces in real time so its generative AI could summarize clinical information for staff.
Pieces marketed its products as “highly accurate,” citing a “severe hallucination rate” below 1 per 100,000. The state found these metrics likely inaccurate and said they may have misled hospitals about the products’ accuracy and safety. Under the settlement, Pieces agreed to disclose its products’ accuracy truthfully and to ensure hospital staff understand when they should, and should not, rely on them.
Why it matters: the hospitals relied on a vendor’s accuracy claims, and the state treated those claims as a consumer-protection issue. For boards, the lesson is direct: a vendor’s marketing number is not evidence of safety, and your clinicians may be relying on it.
Governance gap: Protections. Vendor claims are not safety evidence. Independent testing, defined reliance limits, and staff training are the institution’s job.
States Now Attach Penalties
Wednesday’s post covered how states are starting to close the gap left by rules built for licensed humans. Several now attach specific penalties or licensing consequences:
State Law What it covers Consequence
Illinois
Wellness and Oversight for Psychological Resources Act (2025)
Bans AI from providing therapy or making therapeutic decisions
Up to $10,000 per violation
New York
AI companion law (effective November 2025)
AI companion safeguards, including crisis protocols
Attorney General may seek up to $15,000 per day
Utah
Mental health chatbot law (effective May 2025)
Disclosure and records duties for mental health chatbots
Up to $2,500 per violation
California
AB 3030 (effective January 2025)
Generative AI patient communications must be labeled unless reviewed by a licensed provider.
Medical board discipline for physicians
Delaware
HB 191 (signed April 2026)
AI may not be licensed as, or use the title of, a physician, physician assistant, professional or practical nurse, or APRN.
Protected titles reserved for licensed humans
The direction is consistent: states are drawing a line around clinical identity and holding the humans and companies behind the AI to account.
The Pattern
- The AI is never the defendant. In every case, the consequences reach a developer (Character Technologies, Pieces) or a deployer (UnitedHealth).
- Licensing bodies are now involved. A medical board, not just a consumer-protection office, is pursuing an AI company.
- Discovery reaches governance. Courts are asking for AI review board records, internal policies, and who approved what.
- As far as I can tell, no regulator has yet taken action against an autonomous healthcare AI agent. The FDA has not taken enforcement action against a chatbot. These cases involve chatbots and algorithms. Agents that act, not just answer, raise the stakes.
For Those Who Price Risk and Allocate Capital
Every case above is a liability that landed on a developer’s or deployer’s books, never the AI’s. Investors, lenders, insurers, and acquirers should ask the same questions boards ask: who owns each AI system, what evidence supports the vendor’s claims, and what happens when the system is wrong at scale.
Five Moves for Boards and Investors
- Inventory every AI agent that touches patients or care decisions. You cannot govern what you have not counted. Include vendor tools and features enabled within existing platforms.
- Ban any agent from claiming a clinical title or license. Make it a contract term and a testing requirement. Delaware has made it law.
- Name one human owner per agent, accountable for outcomes. Not the vendor. One named person, with authority to pause the agent and keep care running.
- Demand vendor evidence, not marketing. Ask for independent accuracy testing, failure rates, and testing against patients who push back. Define when staff may and may not rely on the output.
- Price AI liability into diligence; track appeals and overrides. A low appeal rate can hide a high error rate. Measure reversals, overrides, and complaints, and report them to the board.
The agent cannot be accountable. Someone in your institution must be.
Bring This to Your Board, Leadership, and Faculty
Dr. Freddie Seba offers AI governance keynotes, board briefings, and advisory built on AI Minimum Viable Governance and the Seba 12 Ps of Responsible Oversight, for healthcare, higher education, financial services, and beyond.
freddieseba.com · contact@freddieseba.com · LinkedIn: @freddiesebaprofile · Ungoverned on LinkedIn and Substack
Not legal, medical, or investment advice. For informational purposes only. Consult your own legal counsel, compliance, clinical, and financial advisors before making governance, care, or investment decisions.
Disclosure and transparency. This issue was created the way I argue AI should be used: critically, grounded in human judgment, for people, not in place of them. AI tools, including Anthropic’s Claude, OpenAI’s ChatGPT, and Google’s Gemini, support research synthesis, source verification, drafting, and layout. Output may carry AI watermarks; a watermark confirms processing, not authorship. The frameworks, analysis, and editorial judgment are mine.
© 2026 Freddie Seba. All rights reserved.
#HealthcareAI #AIAgents #AgenticAI #AIGovernance #PatientSafety #Enforcement #CMIO #CIO #HealthcareLeadership #CapitalAllocators #Investors #AIMVG #Seba12Ps #Ungoverned
References
American College of Radiology. (2026). Delaware bars AI from using medical license titles. https://www.acr.org/News-and-Publications/2026/delaware-bars-ai-medical-licenses
Becker’s Payer Issues. (2026). Judge orders UnitedHealth to hand over documents in AI coverage denial case. https://www.beckerspayer.com/legal/judge-orders-unitedhealth-to-hand-over-broad-discovery-in-ai-coverage-denial-case/
Cooley LLP. (June 25, 2026). AI chatbot’s medical claims draw regulatory scrutiny. https://www.cooley.com/news/insight/2026/2026-06-25-ai-chatbots-medical-claims-draw-regulatory-scrutiny
Delaware General Assembly. (2026). HB 191: An act to amend Title 24 of the Delaware Code relating to medical professionals, titles, and nonhuman entities. https://legis.delaware.gov/BillDetail/142752
DistilINFO. (March 12, 2026). Court orders UnitedHealth to disclose AI denial algorithm. https://distilinfo.com/2026/03/12/court-orders-unitedhealth-to-disclose-ai-denial-algorithm/
Duane Morris LLP. (2024). California passes novel law governing generative AI in healthcare. https://www.duanemorris.com/alerts/california_passes_novel_law_governing_generative_ai_healthcare_1224.html
Nixon Law Group. (2026). How states are enforcing new AI laws in healthcare, and why it matters. https://www.nixonlawgroup.com/resources/how-states-are-enforcing-new-ai-laws-in-healthcareand-why-it-matters
Nixon Peabody LLP. (August 12, 2025). Illinois enacts prohibition against AI therapy. https://www.nixonpeabody.com/insights/alerts/2025/08/12/illinois-enacts-prohibition-against-ai-therapy
Office of the Texas Attorney General. (September 18, 2024). Attorney General Ken Paxton settles first-of-its-kind healthcare generative AI investigation. https://www.texasattorneygeneral.gov/news/releases/attorney-general-ken-paxton-reaches-settlement-first-its-kind-healthcare-generative-ai-investigation
Seba, F. (2026). Ungoverned: A practical guide to AI minimum viable governance. freddieseba.com
Leave a Reply