4 min read
In healthcare, GenAI has a Goldilocks problem. On one side of the system, it is recklessly hot, inflating hospital bills by billions. On the other hand, it is timidly cold, sending sniffles to A&E. Both extremes send the invoice to the same address: the payer, which is most of the time the insurer.
Every AI system deployed in healthcare occupies a position of risk, whether its designers chose to do so deliberately or not. Tune a billing model to maximise reimbursement, and it will find diagnoses to justify higher codes. Tune a triage tool to never miss an emergency, and it will escalate almost everything to a doctor. Neither behaviour is a malfunction. Each system is doing exactly what its incentives and liability structure reward.
For health insurers, this matters because both extremes land on the claims ledger. The aggressive posture inflates the cost of care delivered; the conservative posture inflates the volume of care sought.
The clearest recent example of the aggressive posture comes from the Blue Cross Blue Shield Association. In March 2026, BCBSA and its analytics arm, Blue Health Intelligence, published research analysing commercial claims across roughly 62 million members, focusing on maternity admissions. Among hospitals with the fastest-growing case complexity, many of which had publicly disclosed the use of AI-enabled documentation and coding tools, the share of maternity patients coded with acute posthemorrhagic anaemia rose from around 4 per cent in mid-2022 to more than 12 per cent by early 2025. At comparable hospitals, the rate barely moved. Crucially, the surge in serious diagnoses was not matched by the treatments, such as blood transfusions, that the condition would normally require. As BCBSA's Dr Razia Hashmi put it, "Something is disconnected."
The financial exposure is not marginal. The researchers projected roughly $663 million in potential excess inpatient spending nationally tied to AI-enabled coding, with outpatient exposure of at least $1.67 billion, bringing the total to at least $2.3 billion. Readers wanting the detail should look up the coding-trend exhibits in the BCBSA and BHI report on the association's website; the divergence between high-growth and comparison hospitals is striking in chart form.
The conservative posture is older and better studied. A landmark BMJ audit by Harvard researchers found that symptom checkers gave appropriate triage advice in barely six out of ten cases, and in two-thirds of scenarios where medical attention was unnecessary, they encouraged users to seek care anyway. A five-year follow-up published in the Journal of Medical Internet Research found matters had not improved: median triage accuracy sat around 56 per cent, and the earlier generation of tools made nearly three over-triage errors for every under-triage error. A systematic review of digital triage services reported that one study found 85 per cent of users were advised to see a doctor.
Here is the uncomfortable arithmetic for payers. When a digital front door tells a member with a common cold to attend A&E or book a specialist, the tool's designers have transferred liability, not reduced risk. The member could have seen a GP, spoken to a pharmacist or taken sensible over-the-counter action. Instead, the payer funds an emergency visit that costs many multiples of a primary care visit, and the member loses confidence in digital channels altogether.
This defensive design is rational for the vendor. The reputational and legal costs of one missed emergency dwarf the diffuse, invisible costs of ten thousand unnecessary escalations. But those escalations are not invisible to the insurer; they are the loss ratio. A triage tool that "plays it safe" by defaulting to the most expensive care setting is, from the payer's perspective, an expensive way of saying "I don't know". McKinsey's broader finding, that nearly eight in ten companies use gen AI yet just as many report no significant bottom-line impact, has a specific healthcare flavour: the impact often exists, but it lands as a cost on someone else's balance sheet.
The aggressive posture is simply the same miscalibration pointed the other way. Coding tools that hunt for every defensible diagnosis are optimising revenue for the provider while the payer, employer and ultimately the member absorb the difference.
Defenders of conservative triage argue, fairly, that under-triage kills and over-triage merely costs. The Harvard team itself noted that overly risk-averse advice is not unique to software; telephone lines and clinicians hedge too. And a 2024 Swiss prospective trial of a well-validated symptom checker across 2,543 real patients recorded no hazardous under-triage and an over-triage rate under 18 per cent, evidence that careful design and clinical validation can narrow the gap substantially. On the billing side, providers note that AI documentation tools also capture legitimately under-coded acuity and reduce clinician burnout.
Both points stand. But they strengthen, rather than weaken, the case that risk appetite must be explicitly engineered and audited. The Swiss result was achieved through prospective clinical validation, not by shipping a defensively tuned chatbot. Safety and affordability are not opposites; unvalidated caution is just unmeasured cost.
Health insurers cannot outsource risk appetite to their vendors' legal departments, and they should not outsource the decision rules either. The logic that determines how a claim is coded, how a symptom is assessed and when a member is escalated to a clinician is not a technical implementation detail. It is the codified expertise of the institution's own medical officers, underwriters and claims specialists, refined over decades of managing risk. That logic belongs inside the payer or the hospital, as their intellectual property, not buried in an AI vendor's opaque model where it cannot be inspected, versioned or defended in front of a regulator.
This is where deterministic tools built on expert systems earn their place. A rules-based engine configured by insurance and clinical experts will execute that configured process with 100 per cent repeatability: the same input produces the same output, every single time, and its accuracy is precisely as good as the expertise encoded within it. Every decision can be traced to a rule, every rule to an author, and every change to a version. When the regulator asks why a member was escalated, the answer is a documented clinical protocol, not a probability distribution. Generative AI, by contrast, is probabilistic by design; the same question asked twice can yield two different answers, which is a strength in open conversation and a liability in triage and billing decisions.
The practical architecture follows naturally. Whether deploying conversational AI for member engagement, claims guidance or symptom navigation, payers should encode their decision rules in deterministic, expert-configured systems they own and control, use generative AI at the conversational edges where its flexibility adds value, define acceptable over-triage rates, demand prospective validation evidence, monitor escalation patterns as a cost metric, and scrutinise provider-side AI coding trends in their claims data as BCBSA has done.