Microsoft’s AI Vs. Doctors: A Look At What Was Measured

🔍 Read the full analysis: Microsoft’s AI Vs. Doctors: A Look At What Was Measured on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get audio and creator gear delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 complex medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a Science paper published October 22. The peer-reviewed study measured performance in a text-based simulation, not treatment of real patients, so its results do not establish how the system would perform in hospitals.

The study, titled Sequential diagnosis with language models, tested whether AI systems and doctors could reach the published diagnosis while requesting information from a case file in stages. Each case began with a short summary; participants had to choose follow-up questions or tests, with the benchmark assigning costs to those requests. The researchers assessed both diagnostic accuracy and the estimated cost of reaching a diagnosis.

On the 304 complex records drawn from the New England Journal of Medicine, Microsoft reported 83.2% accuracy for MAI-DxO. Generalist physicians scored 36.1% when allowed textbooks and internet access, and 22.3% without those resources. On a separate set of 325 emergency department cases, Microsoft reported 80.3% for the AI system and 46.8% for physicians. These are results reported by the study’s authors, all of whom are listed with Microsoft affiliations; peer review does not by itself mean independent researchers have replicated them.

The benchmark’s design is a key part of the findings. MAI-DxO uses a language model with an orchestration layer that has virtual agents propose diagnoses, request information and challenge one another. Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex cases, compared with 83.2% for MAI-DxO. The source account says the orchestration layer added about four percentage points and reduced estimated testing costs, though the published headline accuracy alone does not show how that difference would translate into clinical benefit.

Why the Test Design Matters

The study addresses a meaningful problem: medical diagnosis often involves deciding which question or test to pursue next, rather than choosing an answer from a complete set of facts. A benchmark that makes participants request information sequentially is closer to that part of clinical reasoning than a standard multiple-choice exam. Tracking test costs also lets researchers examine whether a system can reach an answer without ordering every available test.

But a higher score on selected, difficult cases is not evidence that an AI can replace doctors or improve outcomes. The physician comparison was against individual primary care doctors and hospital generalists, not specialist clinicians or teams. The cases were historical text records, and the test did not include a physical examination, real-world imaging or the wider circumstances involved in caring for a patient. Hospitals, insurers and patients should read the findings as a research result about a defined benchmark, not a measurement of routine clinical performance.

The cost figures also need careful interpretation. The benchmark assigned estimated prices to tests using billing codes and a 2023 U.S. hospital price list, along with a fixed physician-visit cost. The source account describes these as first-order approximations. That means the study can compare costs under its assumptions, but it does not establish actual savings for hospitals or patients.

How the Benchmark Was Built

The work builds on an earlier version posted as a non-peer-reviewed preprint in June 2025. That version used OpenAI’s o3 model and reported a different, more dramatic comparison: 80% for the AI against 20% for doctors. The paper published in Science is the later peer-reviewed version, and the figures cited in this report are those attributed to that publication. The source account notes that some implementation details come from the earlier preprint because the full Science paper is behind a paywall; those details may not all match the final article.

In the described benchmark, a gatekeeper model holds each case’s full record and provides further information when requested. If a participant asks for a test result that is absent from the record, the gatekeeper generates a synthetic result. A separate model grades answers against the published diagnosis; physician regrading of a sample showed good, but imperfect, agreement. These features allow the researchers to run a standardized test, but they also mean the benchmark depends on simulated information and model-based scoring.

The source account says the physician cohort in the preprint included 21 U.S. and U.K. generalists, most working in primary care, with a median of 12 years’ experience. Doctors were assessed on a subset of held-out cases, while the AI was assessed on all 304. The Science paper’s reported comparison includes an open-book physician arm. The cases were chosen for difficulty and contained no healthy patients or harmless explanations, according to the account, limiting what the test can show about false alarms and overdiagnosis.

“More than twice as accurate as physicians using textbooks and the internet on complex cases”

— Microsoft, in its blog post about MAI-DxO

What the Scores Cannot Establish

The study does not show how MAI-DxO performs with patients in ordinary clinical settings, where symptoms may be incomplete, communication can be difficult and clinicians must weigh treatment risks as well as diagnostic possibilities. It also does not establish whether using the system improves health outcomes, speeds care, reduces costs in practice or changes how often patients receive unnecessary tests.

The comparison’s limits leave open how the system would fare against specialists or multidisciplinary teams, or when the case mix includes common, mild and non-disease explanations. Because the benchmark cases were selected as difficult and did not include healthy patients or benign explanations, the study could not measure false-positive diagnoses or unnecessary alarm rates in the way a broader clinical evaluation could.

Independent replication is also not established by the information provided. The paper was peer reviewed, but the authors are Microsoft-affiliated, and the source account says details of the final paper are not all available to it. The extent to which the final publication retains every method detail described in the preprint is therefore unclear from the supplied material.

Prospective Patient Testing Planned

Microsoft’s next cited step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with the system alongside their normal care. That design could provide evidence about how the tool works in a live care setting, while keeping the patient’s existing clinical care in place.

The available information does not specify the study’s start date, size, measures of success or publication timeline. Until those details and results are reported, the Science paper remains evidence about performance on simulated historical cases, not proof of safety, effectiveness or cost savings in clinical practice.

Key Questions

What did Microsoft’s study report?

It reported 83.2% accuracy for MAI-DxO on 304 complex NEJM case records. Physicians scored 36.1% with textbooks and internet access and 22.3% without them, according to the paper’s reported results.

Were real patients examined by MAI-DxO?

No. The benchmark used historical, text-based case records in a simulation. It did not test MAI-DxO as a clinician treating real patients.

Who were the doctors in the comparison?

The comparison was with generalist physicians working individually, including primary care doctors and hospital generalists. The source account says the study did not compare the system with specialists or physician teams.

Did the study show that MAI-DxO cuts medical costs?

It included estimated test costs within the benchmark, using assigned prices for requested tests. Those estimates do not establish real-world savings for patients, insurers or hospitals.

What is the next planned test?

The source account describes a planned prospective study at Beth Israel Deaconess Medical Center, where consenting patients would use the system alongside their normal care. The study’s schedule and full design have not been provided here.

Source: ThorstenMeyerAI.com