After the Rush to Build: What Carriers Learned About AI for Medical Record Review

A small team can now build an AI tool that reads medical records in a few weeks. Keeping it accurate on live claims is harder. Many carriers have learned that the work doesn't end when the first version is built.

The rush started in late 2025, when AI models took a big step up and new tools made it easier for small teams to build their own applications. With adjusters spending 2 to 6 hours a day reading records, many carriers started to build.

By the second half of 2026, the ongoing costs became more apparent: tokens, maintenance, and re-testing every time a model changed. Many are once again weighing build or buy, though most carriers already use some mix of an in-house tool, a specialized platform, and a general AI assistant.

Building gives you control, and ongoing responsibility

Building a tool is easier than ever, and for carriers with the technical resources to support it, there are real advantages. You control the roadmap and shape the tool around your own forms and workflows.

"When you build, the first version comes fast and the demos look great," says Amir Tavafi, Chief AI Officer at SiftMed, "You will see 80 or 85 percent accuracy early and it will feel close to done, but the bigger investment is in everything after that, and it’s a recurring cost." 

Four challenges tend to emerge when AI moves from a demo to real medical files:

1. Medical records are messy. General-purpose models read words well but don't always read the structure of a page. A circled pain score of 7 and a checked "modified duties" box can come back as "patient reports ongoing pain," which drops the information the adjuster needed.

2. Summaries fall behind as records arrive. When a model's context fills, it summarizes what it has and carries on from that summary. By month six, when a claim may pass to a new adjuster with 30+ records, that adjuster might be reading a summary of a summary. Keeping every detail intact through the claim cycle takes serious engineering.

3. The models keep changing. Frontier models used to ship every six months, now new models ship within weeks. Someone has to test each one on your files, compare the outputs and decide whether to switch, with regression testing in place so a change can't quietly lower accuracy.

4. Security and governance stay on your desk. Your team owns PHI security in a sector where the average breach costs $7.42 million. It also owns model governance under OSFI Guideline E-23 (for federally regulated insurers, effective May 1, 2027) or the NAIC model bulletin, now adopted by about half of US states. 

Each of these challenges show up in the budget, and medical data adds another layer of complexity. Processing scanned records, handling handwritten notes and extracting structured information requires more than a model API. Carriers also need the infrastructure, document-processing tools, testing and engineering resources to keep those workflows running reliably. Those costs need to be part of the initial business case, not added later.

Buying shares the work, and some of the control

Buying moves most of the day-to-day maintenance to a vendor, including model testing, support for new form types and integrations. A vendor working with many carriers may have encountered edge cases a single carrier hasn't, though that experience only matters if it's backed by testing and validation.

In return, you give up some control. Your roadmap depends on the vendor's, your data is processed outside your own systems, and a change you need joins someone else's queue. 

"Buying doesn't remove the obligation," Tavafi says. "What it changes is who does the work."

Buying doesn’t hand off the regulatory responsibility, but specialized vendors typically validates and monitors its models, re-tests them as they change, and maintains the security controls and certifications that protect PHI. The carrier’s job shifts to overseeing that work: asking for evidence, reviewing results and confirming the vendor meets its standards.

On either path, the work starts with defining the problem and setting clear success metrics. "Summarize the file" can mean a chronology, a file overview, summary notes or a list of facts to flag, and each needs different output.

"Accuracy starts before anything is summarized," says Holly Hill, CEO and co-founder of SiftMed. "With SiftMed, records are sorted, de-duplicated, categorized and dated before data is summarized and before your team’s eyes land on the file.”

Whether you build or buy, claims professionals should ask how accuracy on your output is measured, and how it is re-tested over time.

General assistants have a place, outside the medical file

General AI assistants work well for drafting emails and letters, summarizing a short report and answering policy or process questions. For the core of medical review, they typically need extra work. 

Large scanned files can hit upload limits, and page-level citations to support a reserve or a tribunal aren't built in. Connecting to claims systems such as Guidewire or OnBase takes development. With consumer tools, organizations may have less control over how PHI is accessed, stored and handled. Enterprise versions address some of this, but teams should know what their tool can and can’t do, especially with the sensitive data in a claim file.

Build, buy or use general AI?

	Build	Buy	General AI
Cost	Highest	Moderate	Lowest
Security	Strong, if your team owns it	Strong, if the vendor is certified	Low, but depends on the product
Accuracy	Depends on testing: only learns from your files	High: Tested across many carriers’ files and edge cases	Low: Especially on long or scanned claims
Customization	Builder has full control	High, with a responsive vendor	Limited without additional development

What to ask beyond the demo

Almost any approach looks good on a clean sample file. A few questions can tell you how it will hold up on a real claim.

1. Can every finding be traced to its source page? An adjuster has to stand behind a summary at a reserve review or a tribunal. Every fact should link to its page, including in exported reports.

2. What happens when new records arrive on an open claim? A good answer is that new records are sorted, checked for duplicates and added to the existing chronology without losing the context or page citations of what was already processed.

3. Who re-evaluates the system when new models come out? New models arrive every few weeks, and a change can quietly lower accuracy on your files. Ask who tests each one, how often, and what results you’ll see before a change reaches your claims.

Automation and control

The industry often describes the adjuster’s role as “human in the loop.” Another way to frame it is to ask how much of the work to automate, and where the team keeps control.

The goal is to automate as much as possible without creating a bottleneck, while keeping control at the points that matter. A required review at every step slows down every claim. The ability to step in where it counts doesn’t.

Automating tasks is different from automating decisions. Sorting, indexing, extracting facts and drafting the summary can run without a manual step at each stage. Adjusters still need to verify sources, look into exceptions and challenge outputs, and the claim decision stays with them. 

Adjusters should be involved in selecting and testing new tools, too. They know where the friction exists in the claims process and can help identify where automation will make the biggest difference.

Build or buy, plan for long-term accuracy

Getting AI to produce a good summary on day one is the easy part on any path. The decision that matters is who maintains accuracy a year later, with hundreds of new records, form changes, and many new models.

A carrier that builds needs a team ready to own that work for years. A carrier that buys needs a vendor that can prove it does that work.