AI Receptionist Pilot Design: Set the Bars Before Go-Live

Thirty days after go-live the vendor shows you a dashboard where every number is green, and your front desk tells you the phones felt worse. Nobody in the room can settle it, because nobody wrote down what winning looked like before the system answered its first call.

That is the normal outcome, not the unlucky one. MIT's NANDA initiative reported in August 2025 that about 5% of enterprise generative AI pilots produced rapid revenue acceleration while the other 95% showed no measurable impact on profit and loss. Those were enterprise projects rather than practice phone lines, so read it as a warning about how pilots get judged. Gartner made a narrower call in July 2024: at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, with unclear business value among the reasons named.

This post covers the part nobody sells you. How to capture a baseline before the AI answers anything, how to commit pass, extend and stop bars in writing, why the vendor's scorecard measures the vendor, and whether 30 days at your call volume can tell you anything at all.

Why most AI receptionist pilots end in an argument

A pilot with no agreed result is not a pilot, it is a shopping window. When the 30 days end, the vendor produces its dashboard, your staff produce their impressions, and the loudest person in the room decides. An experiment settles that in advance, by naming the numbers everyone agreed to accept.

Two omissions cause almost every one of those arguments, and both happen before go-live rather than during the pilot.

The first is a missing baseline. If you never measured the old setup, there is nothing to compare the new one against, so the conversation collapses into whether the demo calls sounded good. The second is a missing bar. Without a written threshold, whatever number arrives gets interpreted by whoever has the most invested in the answer.

Neither omission is about the technology. If you have not evaluated one of these systems before, start with what an AI receptionist actually does and where it stops, then come back and design the test. The rest of this post assumes you already know what the thing is supposed to do.

Capture the baseline before the AI answers a single call

Four numbers decide everything later, and all four already sit in your phone system: how many calls went unanswered, how many arrived after hours, how long a caller waited for a live person, and how many calls turned into a booked appointment. Pull them for 30 days before anything changes.

Measure the old setup first. Once routing changes, the comparison you needed is gone.

The timing matters more than the precision. Once a number is ported or a routing rule changes, the old call log stops accumulating and your "before" is frozen at whatever you captured.

For reference points rather than targets: aggregated industry figures put unanswered calls at medical practices around 23% and after-hours calls at roughly 41% of volume, with about 62% of callers hanging up rather than leaving a voicemail. Those are aggregator numbers with second-hand attribution, so treat them as a reason to measure your own, not as your own.

If you want a published shape for what a defensible bar looks like, CMS holds Medicare Advantage and Part D call centers to an average hold time of 2 minutes or less and an average disconnect rate of 5% or less. That standard does not bind a private practice, but it is a real regulator putting a number on the same two things you are about to measure.

Baseline the call type you are actually automating, not the whole line. An MGMA poll in March 2026 with 294 applicable responses found eligibility and prior authorization took up the most phone time at 45%, with scheduling at 31%. If the pilot only handles scheduling, a practice-wide average will bury the effect. The rollout mechanics that follow this measurement are covered in the week-by-week implementation guide.

Write the pass, extend and stop bars before go-live

Commit three thresholds to paper before the vendor is live: the number that means expand, the number that means run it longer, and the number that means end it. Each needs a named decider and a review date. A bar you invent after the data arrives is not a bar.

Write the thresholds in your own units. "Booked appointments from after-hours calls rises from 18 to 24 a month" is a bar. "Improves patient access" is a wish. If you want the recovered-revenue model behind that dollar comparison, it is worked line by line in the ROI math for an AI receptionist; this post is only about the decision design that sits on top of it.

One more rule that saves arguments: every bar must be readable off a system you control. If the only place a number exists is the vendor's portal, you have outsourced the verdict along with the phones.

The vendor's scorecard measures the vendor

Containment rate, the metric on nearly every voice AI dashboard, counts calls that never reached a human and divides by total calls. A caller who gave up and hung up counts as contained. It is a real operational number, but it answers the vendor's question, not the one your schedule cares about.

Published figures with their sources. Reference points for setting a bar, not your result.

The gap between the two definitions is not theoretical. One published buyer case describes a deployment approved on a quoted 91% containment rate that showed 58% on the buyer's own operating dashboard six months later, because the contract never defined what counted as a contained call. That is a single vendor-reported case, so treat it as a cautionary example rather than a benchmark.

Vendor evaluation guides put mature containment targets around 75% to 85% and appointment-booking automation at 65% to 80%. Those ranges are useful for sanity-checking a claim, but they use the vendor's numerator and denominator.

Your four counters are different and all four live in your practice management system or your own log:

• Appointments that ended up on the schedule and were kept
• Escalations that reached a human, with the time it took
• Escalations that reached nobody, which is the number vendors rarely surface
• Callers who ended the call early, counted separately from callers who got an answer

Will 30 days at your volume tell you anything?

Probably not, if the pilot only covers after-hours calls at one location. Detecting a booking-rate move from 20% to 35% needs roughly 138 calls in each period, before and after. A practice with 30 after-hours calls a month collects 30. That is an anecdote wearing the clothes of a result.

Those sample sizes are modeled, using the standard formula for comparing two proportions at 80% power and 5% two-sided significance. The narrower the improvement, the more calls it takes: about 294 per period to separate 20% from 30%. Swap in your own rates before quoting either figure.

When the volume is too thin, you have three honest options and no fourth:

1. Extend the window. One vendor benchmark guide describes days 30 to 90 as the real measurement period, since week one underperforms while configuration settles. Plan for 60 to 90 days from the start rather than negotiating an extension under pressure.
2. Widen the pilot. Add daytime overflow calls so the volume arrives faster, accepting that you are now testing a bigger change.
3. Change the question. "Did it handle calls correctly?" needs far fewer calls than "did bookings rise?" A pilot that proves correctness and safety is worth running even when it cannot prove lift.

Listen to a stratified sample, not the dashboard summary

A dashboard is a summary written by the party you are evaluating. The evidence is the recordings and transcripts. Pull about 40 calls spread deliberately across outcomes, listen to them yourself, and you will learn more in an afternoon than a month of green tiles will ever tell you.

Random sampling misses the rare calls. Choose by outcome, and always include clinical routings.

Sampling discipline is thin everywhere, not just in healthcare. Manual review programs typically cover only 2% to 5% of call volume, by one vendor's analysis, and the long-standing contact-centre standard is scoring just 4 calls per agent per month, with 41% of centres managing fewer than that. Nobody listens to everything, so what you choose to listen to is the whole method.

Random sampling is the wrong choice here, because the calls that matter are rare by definition. Stratify by outcome instead: booked, escalated, ended early, transferred to a line nobody answered, and every call the system logged as clinical or urgent.

That last group is not optional listening. An AI receptionist routes clinical and urgent calls to a human and never diagnoses or advises, so each of those calls is a direct test of the boundary. If a single one was handled rather than routed, you have found a stop condition, not a tuning note.

The kill bar deserves as much thought as the pass bar

Name the conditions that end the pilot outright, not just the ones that extend it. A missed urgent escalation, a booking written to the wrong provider, a caller who asked for a person and could not get one: any of these should stop the pilot on the day it happens.

Stopping is a legitimate result, and mature programs treat it that way. A peer-reviewed account of one health system's clinical AI governance reports 10 successful deployments alongside 2 successful retirements and 1 successful non-deployment, counting all three as wins for the process.

The pressure runs the other way, of course. By week three you have paid a setup fee, trained the staff, and told patients about the new system, so month two feels like the sensible choice regardless of the numbers. The bar exists to overrule that feeling, which is why it gets signed while nobody is invested yet.

Field note

Before the pilot starts

• Your 30-day baseline is exported and dated, stored somewhere you control.
• The pass, extend and stop thresholds are written down with a named decider for each.
• A signed BAA is in place before any patient detail reaches the system.
• The escalation path is documented, and you have tested it against your ten most common calls.

Run the pilot as an experiment, not a trial period

The whole design fits on one page: four baseline numbers dated before go-live, three bars with a named decider, a sample of real calls you listen to yourself, and a stop condition you will honor. Everything else in a pilot is logistics, and your vendor is happy to run those.

Decide from the bars you wrote down, so the verdict exists on paper before the meeting.

Walk the flow once before you sign anything. No baseline means restart the clock and measure the old setup first. Too few calls means extend the window and say so plainly rather than reporting a number nobody should trust. A cleared bar means expand to the next call type on exactly the same terms. A stop condition means stop, and put the reason in writing so the next vendor conversation starts from evidence.

The point is that the decision exists on paper before the meeting, so the meeting is about what happened rather than who feels more strongly. If you would rather have the baseline sized from your own call data first, the free Growth Leak Audit does that pass for you.