California Just Banned AI From Making Clinical Decisions — Oct 2, 2026

0:00California just made it law. AI can advise your nurse, not replace them. Welcome to MedAI Times podcast, your daily update on medical AI. Don't forget to like and subscribe. California signs a law keeping hospital AI in an advisory role.

0:14The health secretary says AI gives a better second opinion than any doctor. A fake hospital from Carnegie Mellon fools real physicians and stumps 10 AI models. A head CT tool in Sydney gets a real world test and Google ships its first Gemini 4 model.

0:31Last Friday, Minnesota doctors had just won a contract after a strike where AI was one of the fights. This week, California's nurses won something bigger, a state law. On September 30th, Governor Gavin Newsom signed AB 1979 written by assembly member Mia Bonta and sponsored by the California Nurses Association.

0:50It bars hospitals, clinics and physician offices from using AI to independently perform any clinical function the law reserves for a licensed professional. It also bars AI from directing unlicensed staff to do that work.

1:05Sepsis alerts, risk scores, diagnostic suggestions and chart summaries stay legal as long as a licensed clinician keeps the decision. Documentation is exempt, licensing boards enforce it. A second provision puts consumer health chatbots that read medical records under California's confidentiality law.

1:24Now the other half, Newsom vetoed AB 2575 which would have protected workers from retaliation for rejecting an algorithm's recommendation. His reason, the labor commissioner lacks the medical expertise to enforce it.

1:38The union's reply was that a nurse who sees the algorithm is wrong can still be disciplined or fired. So the human keeps the decision and keeps the risk of overruling the machine. Verdict, signal. Binding law in the biggest health market in the country enforced by boards that have never policed software.

1:57One day earlier at an industry-backed Make America Healthy Again Summit, Health Secretary Robert F. Kennedy Jr. told Vice President J.D. Vance that AI can give a second opinion much better informed than any doctor in the country.

2:12He added that Sam Altman had told him it would now be malpractice to diagnose or prescribe without checking with AI. That is a secondhand quote from a company chief executive and no published study supports the any doctor part.

2:26So in the same week, Sacramento wrote AI into an advisory role and the top federal health official described it as the better doctor. Which is why honest benchmarks matter. Christine Park, Valerie Chen, and Tim Detmers at Carnegie Mellon posted Synthetic Hospital, a public test set of 1,268 invented patients across about 5,600 visits,

2:50built from medical education material with no real patient data. In a blinded review, physicians picked the synthetic charts at 53%, barely better than a coin flip. Ten Frontier and Open Models then tried to rebuild each patient's problem list over time.

3:07The best scored 0.73 level with the average of seven physicians, but well short of the best physician at 0.89. It's a pre-print. The useful part is that anyone can now run it.

3:20Last month, the FDA turned down Harrison.ai's bid to skip pre-market clearance for some radiology software. This week, its emergency head CT tool got a hospital test. Sydney local health district's RPA Greenlight Institute ran it on 3,424 consecutive non-contrast head scans from 2024.

3:427.4% had urgent findings. The AI caught 85% of those with a negative predictive value of 98.3%. Specificity was only 69%, so plenty of false alarms.

3:56The authors report that no missed case needed urgent intervention during that emergency visit. The study was retrospective, and the authors say a prospective trial comes next. On September 28th, the Melbourne-founded scribe company, Heidi, unveiled Heidi 2.

4:12It moves from writing the note to doing the work around it. Agents handle referral management, chart preparation, follow-up coordination and patient outreach, and a clinician approves the final step.

4:25Heidi says it supports 2.8 million patient visits a week, which surface about 10 million follow-up tasks. The rollout started September 29th in English and French. This is a vendor announcement with no outcome data yet.

4:39It also sits right on the line California just drew, where administrative work is allowed and clinical judgment is not. A referral could fall on either side of that line. On September 30th, Google DeepMind announced Gemini 4 Argon, the first Gemini 4 model.

4:56It can generate up to a million tokens in a single run, up from 64,000, and it is pitched at long software projects and cyber defense. Vetted cyber defenders get it first. Paid developers and ultra-subscribers come next with no dates given.

5:13The launch price is $2 per million input tokens and $10 per million output. It has no health-specific claims yet. Anthropic released Claude Sonnet 5.5 on September 28th.

5:25The price is unchanged at $2 in and $10 out per million tokens. Anthropic says it runs more than 30% faster and costs up to 30% less per task. Anthropic named healthcare as a target for sensitive work.

5:39Separately, Reuters reported on a confidential draft of Anthropic's IPO prospectus. It shows 2025 revenue of about $4.6 billion, with 47% of sales routed through Amazon and Google.

5:54This week's spotlight is automation bias, the well-documented tendency of people to go along with a machine's suggestion, even when their own evidence says otherwise. It's the reason an advisory-only rule is only as strong as the clinician's willingness to say no.

6:11A law that keeps the human in charge without protecting the human who overrules the algorithm has handled the first problem and skipped the second. Every source, paper, and vote from this week is linked in the description. When you overrule an AI alert at your hospital, does anyone back you up if you turn out to be wrong?

6:30Thanks for listening. Find us on YouTube and your favorite podcast app. See you tomorrow.