When a patient said “I’m fine,” AI chatbots backed off from correct advice in a third of conversations. Now imagine that chatbot is an AI agent.
Healthcare AI Agents: What HIPAA, FDA, and Clinical Standards Can’t Audit
By Dr. Freddie Seba
Our miniseries begins round two: AI agents in each highly regulated industry, starting with healthcare.
At the European Respiratory Society Congress, researchers tested five leading chatbots with simulated sleep apnea patients. Cooperative patients got correct referral advice 100% of the time. Patients who resisted: 64%. A textbook severe case: 22%. The researchers called it “AI sycophancy.”
1. What were the rules designed to govern?
HIPAA governs patient data. FDA rules govern software with medical purposes. Standards of care hold licensed clinicians responsible. All assume a known actor, a defined scope, and someone who can explain what happened.
2. The governance gaps
- Sycophancy: agreeable AI can be dangerous AI
- Partial safeguards: a safety prompt cut harmful choices from 16.6% to 10.1% (Mount Sinai), not to zero
- Identity: CHAI’s leadership asks “who are these agents and what are they entitled to do?”
- Liability: developers “hesitating to take the level of accountability”
- Deskilling: who catches the agent’s errors if clinicians lose the skill?
Illustrative scenario: your triage agent suggests sleep tips instead of a referral to a patient who said: “I’m just tired.” Was it tested on patients who push back?
3. Responsible AI governance as leadership
AI Minimum Viable Governance, from my book Ungoverned (May 2026), organized by the Seba 12 Ps of Responsible AI Oversight:
- Purpose: what clinical decisions each AI agent may never make
- Privacy and Protections: minimum access, defined identity
- Process: retest after every vendor update
- Preparedness: pause any AI agent and keep care running
- Product Ownership: one named owner per agent, not the vendor
A gap only governance can close.
Friday: who answers when healthcare AI agents fail patients.
Has your health system tested its AI agents against patients who push back? Tell me below. Continue the conversation in the comments or by message
Not legal, medical, or investment advice. AI tools assisted research and drafting; analysis and judgment are mine.
© 2026 Freddie Seba. All rights reserved.
#HealthcareAI #AIAgents #AgenticAI #AIGovernance #PatientSafety #DigitalHealth #CMIO #CIO #CISO #HealthcareLeadership #CapitalAllocators #Investors #AIMVG #Seba12Ps #Ungoverned
Leave a Reply