AI Receptionist Maintenance: What Decays and Who Fixes It

Your AI receptionist worked perfectly for 4 months. Then a provider left, nobody thought to tell the phone, and it kept booking new patients with a doctor who no longer works there. Nothing broke. No alert fired. The system simply kept answering from the version of your practice it was handed on day one.

This is the part nobody sells you. An AI receptionist is not an appliance you install once. It is a snapshot of your practice's facts, and those facts go stale on their own schedule. When CMS called offices to check Medicare Advantage directory listings, 48.74% of the 10,504 locations it reviewed had at least one inaccuracy: a wrong phone number, a provider who was not there, or a panel listed as open when it had closed. Those directories were maintained by health plans with compliance staff. Yours is maintained by whoever remembers.

So this post is the boring half of the job. What decays and how fast, why a stale answer is harder to catch than a broken one, the cadence that keeps it honest, who owns each fix, and how to build a regression suite from your own transcripts so a vendor model update cannot quietly change what your phone says.

What goes stale, and how fast

Almost everything a caller asks about changes at some point. Providers join and leave, hours shift, plans are dropped, a service line launches, a fax number changes. None of these events announces itself to your phone system, so each one turns a correct stored answer into a confident wrong one.

The list is mundane, which is exactly why it goes unmanaged. Every row below is a change somebody at the practice already knows about, on a day when updating the phone was nobody's job.

Roster churn moves faster than most owners assume. Premier's provider practice benchmarking data found that across the 24 months from October 2021 to September 2023, roughly 1 in 5 providers in US medical groups was new to their practice. Treat that as directional rather than a rate for your group, but if you are anywhere near it, the roster behind your knowledge base has a shelf life measured in months.

The CMS number is worth sitting with, because those were not careless organizations. CMS was reviewing health-plan directories rather than practice knowledge bases, and the plans behind them had compliance teams and a regulatory obligation to keep them right. Roughly half the listings still carried an error. The failure was not effort. It was that no one owned the specific update at the moment the fact changed.

A stale answer sounds exactly like a correct one

A broken integration fails loudly and someone fixes it that afternoon. A stale fact fails silently. The system retrieves the entry it has, delivers it fluently in your practice's voice, and the caller on the phone has no way to tell that the answer stopped being true 6 weeks ago.

Nothing in the pipeline knows an entry expired. There is no freshness check between the knowledge base and the words the caller hears, so a fact written in January is served in September with exactly the same certainty. The patient has no reason to doubt it, and neither does your front desk, because nobody was on the call.

This is the same mechanism that makes a gap in the knowledge base dangerous. A missing answer produces a plausible guess, and an expired answer produces a plausible fact. How entries should be written, and what must never go into them, is covered in what your practice must load into the knowledge base. If you are still deciding what the system should be allowed to answer at all, what an AI receptionist does and where it stops sets the boundary this post assumes you have already drawn.

The upkeep calendar that keeps it true

Upkeep works when it sits on a schedule rather than on someone noticing. 4 cadences cover almost all of it: a weekly transcript sample, a monthly knowledge-base review, a quarterly replay of your top call intents, and one annual refresh of hours and accepted plans keyed to January.

Four cadences catch nearly all of it. Put them on a schedule, not on someone noticing.

Weekly is a 20-minute job. Pull about 10 transcripts at random, read them end to end, and look for 3 things: an answer you would not have given, an escalation that should have happened and did not, and a question the system could not answer. The third one is free knowledge-base material in your patients' own words.

Monthly is the sweep. Walk the knowledge base against the current roster, the posted hours, the plan list, the price ranges, and the service menu. It takes about 45 minutes when nothing has changed and longer when something has, which is the point.

The January refresh is the one with a hard external deadline. Medicare Open Enrollment runs 15 October to 7 December, and the changes patients make take effect 1 January. A plan list you confirmed in November can be wrong by February, and insurance questions are the answers patients act on most directly. A good rollout already builds a weekly transcript habit in month one, as the week-by-week implementation plan sets out. Treat that habit as the floor, not a launch task that expires.

Who owns each change, and within what window

A calendar with no name attached is a wish. The practices that stay accurate assign each change type to a person and a deadline: the office manager updates hours, whoever handles credentialing updates the provider roster, and whoever handles billing updates the plan list. Unowned facts are the ones that rot.

Ownership is the part practices get wrong. A cadence with no name attached is a wish.

The trick that makes this stick is attaching the update to an event that already has an internal announcement. A credentialing packet, a fee change, a holiday memo, and a lease notice all get circulated anyway. Adding a line to each of those handoffs that says to update the phone answers costs nothing, and it catches most of the drift before a patient finds it.

Field note

Worth confirming before your next quarter starts

• One named person owns each change type, and their backup is named too.
• The plan list carries a review date, not just a last-edited stamp.
• The on-call number and escalation rules were checked this month, not at launch.
• Someone read 10 real transcripts in the last 7 days.

Build a regression suite from your own transcripts

The single most useful artifact you can build is a set of 20 to 40 canonical calls with the answer each one should produce. Pull them from real transcripts, not from imagination, then replay the whole set after every change so you find out what a fix broke somewhere else.

Pull every case from a real transcript. Invented scripts test phrasing nobody uses.

Building it is a 2-hour job, once. Take the transcripts you already sample weekly, pick the calls that represent a whole category, and write down the outcome each should reach: the right visit type booked, the right refusal given, the right escalation triggered. Keep the patient's phrasing rather than tidying it, because the messy phrasing is what breaks things.

The tooling to run this exists at the vendor level and is documented. Retell describes saving scenarios as graded test cases and running them one at a time or as a batch to catch regressions automatically, and drafting cases from real calls. Vapi's voice test suite has an AI tester call your agent following a script, then grades the transcript against a rubric you write. If your vendor offers neither, you can still do it manually with a phone and a scorecard, and 40 calls is one afternoon.

The model changes underneath you too

Your vendor upgrades the speech and language models behind the agent, and behavior can shift without a single edit on your side. Researchers comparing the March and June 2023 releases of one widely used model found accuracy on the same set of questions fall from 84% to 51% in 3 months.

That study looked at a general-purpose chat model, not a healthcare voice agent, so read it as documented proof that version drift is real rather than as a prediction about your line. The mechanism transfers: the same instructions, run against a newer release, can produce different behavior.

The stack underneath a voice agent also moves on a schedule you do not control. OpenAI's deprecations page shows the legacy audio and realtime model families announced on 20 July 2026 for removal on 20 January 2027, and the transcription models announced on 26 August 2026 for removal on 26 February 2027. That is roughly 6 months of notice each time, and somebody has to act on it.

So the practice needs its own check after a vendor update rather than trust in a release note. A release note tells you what the vendor changed. Your 40 graded cases tell you whether the answers your patients hear are still the ones you approved.

Budget the hours, then put them on someone's list

Upkeep costs real staff time, and pretending otherwise is how it gets skipped. Model it at about 3 hours a month: 20 minutes of transcript sampling each week, 45 minutes of knowledge-base review, a quarterly regression replay amortized, and the odd change nobody scheduled. Budget it or lose it.

Route every change by what it touches, so nothing reaches a patient untested.

Here is the arithmetic, and it is modeled rather than measured, so replace it with your own numbers. 4 weekly samples at 20 minutes is 80 minutes. The monthly review is 45. A 2-hour quarterly replay spreads to about 40 minutes a month. Unscheduled changes add roughly 20. That totals about 185 minutes, or 3 hours.

Walk the flow once. A change that does not alter what a caller is told needs nothing. A stored fact like hours, plans, or providers gets edited the same day, by the person who owns it. A vendor-side model or voice change gets the regression suite replayed before your next busy day. Anything else earns a new test case and a closer read of tomorrow's transcripts.

That is far less than the front-desk time a working system gives back, but it is not zero. Budget zero and the phone spends next spring confidently quoting last year's plan list. If you want the size of the phone problem before you commit to running one of these, the free Growth Leak Audit works from your own call numbers.