Module 23 · A/B test your AI agent and drift alerts

Leer en español: Módulo 23 · Prueba A/B de tu agente de IA y alertas de deriva.

Goal: compare variants of the same voice-AI agent (its voice, its style, its model), read the verdict without fooling yourself, know what “Not enough data yet” means and understand what drift alerts do and do not do. Time: ~8 minutes · You need: an AI agent in the live state (Module 22) and real or test calls that it answers.

This module is for whoever operates the agent. If you want to call the API, the guide is Voice AI A/B experiments and drift alerts. For campaign A/B tests, which work the same way, see Module 18.

1. What it is and who turns it on

An A/B test of an agent splits its calls between two to five variants and measures which one gives the better result. There is no switch to turn on: it is available for every live agent, and drift monitoring (section 5) runs by itself for every account. An agent with no test running answers exactly as it always did.

Creating, starting and ending tests requires the account administrator role; with the member role the platform answers “role_forbidden”. Seeing the list and the results is open to any user of the account.

2. Create and start a test

1. Open the agent from Voice AI → AI agents and go into A/B experiments. The screen says “Compare variants of the same agent on the same kind of calls. The control is the agent as it is today.” If the agent is not live, the screen warns “The agent is not live, so an experiment cannot be created or started. Make it live in the agent editor first.”
2. Choose New experiment and give it an Experiment name.
3. In Decides by choose the metric that will give the verdict. Once created, it cannot be changed. The options are: Promise to pay rate, Calls closed without a human, Handoff to a human (lower is better), First audio latency, p95 (lower is better) and Cost per call (lower is better).
4. Define 2 to 5 variants with Add a variant and Remove. Each has a Name, a Weight (a whole number from 1 to 1000, its relative share of the split) and, on exactly one, the Control mark: the agent as it is today, which changes nothing.
5. For each variant that is not the control you can change, and only this: the Voice and its Pace (from 0 slow to 1 fast), a Style paragraph (a single line of up to 600 characters, added after everything the agent says by obligation and unable to replace any of it), the Model, the Reasoning effort and the Prompt form. An empty field keeps the agent as it is.
6. Choose Create experiment. It is born as Draft. If the platform refuses a field, the screen lists it under “The platform refused these fields:” and saves nothing.
7. On the draft's row, choose Start. It becomes Running. An agent can have only one test running at a time. If a model left the catalog since you wrote the draft, starting is refused and the screen says why.

While it runs, the variants are frozen: to try another style, voice or model, create another test. End asks for confirmation (“End this experiment? The agent goes back to answering as it is today and an ended experiment cannot be restarted.”).

What never varies

The form screen lists it under What does NOT vary: the opening legal notice with its text and its audio, the spoken opt-out, the handoff rules (including “I want a person”), the negotiation limits, the list of prohibited phrases, the agent's tools and the compliance checks of every call. A call that a gate blocks never becomes a conversation, so it does not count for any variant either.

3. How calls are split

The split follows the weights and is fixed: the same debt always gets the same variant while the test runs. On an inbound call, which has no debt, the split is per call. Each conversation stores its variant when it opens and is never reassigned. If a variant cannot be resolved (for example, a retired model), that call is answered by the base agent and not by half a variant.

4. Read the verdict

Choose Results on the test's row. The console draws the platform's verdict as it is, without calculating anything on its own. It has three possible answers:

• Not enough data yet (insufficient_data). It is not a tie or a bad result: it means there are not yet enough calls to say anything. The platform asks, per variant, for at least 100 valid conversations and 10 successes when the metric is a rate, or at least 100 calls with the measure when it is a mean (latency or cost); the screen shows the minimum it requires. Wait for more calls. The platform does not give a winner ahead of time, and looking often does not bring it forward.
• No significant difference (no_significant_difference). There is enough data and no variant can be told apart from the control. Rates are compared with a two-proportion test, means with Welch's test, at 0.05 significance and with an adjustment for comparing several variants against the control.
• Winner: X (p = …) (winner). A variant beats the control with significance; the detail gives the difference against what it is compared with and its interval.

A valid conversation is a closed one whose outcome is neither failed nor voicemail. The rates of the main metrics are measured over the valid ones; the failed rate and the compliance-violation rate, over all closed ones.

Three warnings that change how you read it:

• The split did not come out like the weights. If the real proportion of calls deviates from the weights, the screen warns “Warning: the calls were not split between the variants in the proportion of the weights, so the comparison may be biased.” and the platform withholds the winner.
• Cost “Without rate”. If some calls' provider has no rate loaded, the Cost column says Without rate or Partly priced, never zero, and a verdict by cost stays at “not enough data”. Nor will you see cost amounts: the test compares; your invoice is per minute (column Minutes per call).
• “With metrics”. This column says how many closed conversations published their metrics. A low number explains a latency or a cost without data.

If the results do not load, the screen says “The results could not be loaded.” with Retry: a network error is never shown as “no data”.

The safety stop

If a variant commits compliance violations significantly more often than the control (with at least 100 closed conversations in both arms), the platform ends the test by itself, the agent goes back to answering as it does today and the screen says “Ended by the platform: …” with the reason. There is no way to switch this stop off.

5. Drift alerts

Drift is a different thing: it does not compare variants, it watches that what you already have does not degrade. Every 15 minutes the platform reviews, per hour, five measures of each agent and variant, and of each model (all the calls that used it, with or without a test, so a worsening at the provider shows even if you changed nothing): first audio latency (p95), handoff-to-a-human rate, provider error rate, failed-call rate and cost per call. It only warns when the measure gets worse, never when it improves.

Two rules decide:

• Moving band. It compares the hour with those of the previous 7 days. It needs at least 24 hours of those 7 days with 30 calls or more each, and 30 calls in the hour it judges; with less, it does not judge.
• Latency budget. If the hour's first audio p95 exceeds 900 ms, it warns even if the degradation was slow and already looks normal.

Each alert brings its evidence: what was observed, the mean and the threshold, how many calls support it and the hours the rule looked at. An alert opens and, when the measure returns to normal, closes in a new row; the open alert is not erased.

What it does and does not do

• Does: leaves the alert in writing with its evidence, available in the API (GET /v1/ai/agents/drift) and as the webhook event ai.drift.detected that your integration receives, once per open alert (Module 21).
• Does not: it does not pause the agent, end a test, change the model or touch any gate. Deciding what to do is yours.
• You also see it in the console. At the foot of the agent's A/B experiments screen, the Drift alerts panel lists the open and cleared alerts with their evidence and, under What is being judged, the coverage of each series.
• “No alerts” is not always “all fine”. The API also returns the coverage: a series with few calls in its last hour appears as insufficient_sample, and one that does not gather 24 hours with 30 calls or more in the previous 7 days as insufficient_baseline. Those series are not being judged yet.

6. Declared limits

• A style paragraph cannot be proven neutral by itself. It is bounded by the guards that run above the model and do not read the style (the notice is played by code before the model speaks and the prohibited phrases are discarded before being said), and by the safety stop.
• A person with two debts may hear two variants, and someone who calls an inbound line again may land on another one.
• The test has a fixed horizon: there is no early winner.
• Provider cost only exists where the platform loaded a rate.
• Drift is detected by its effects (latency, handoffs, errors, cost), not by comparing the meaning of the model's answers.
• None of this touches the legal texts: if you need to vary a notice, it is a decision for your legal adviser and not an experiment.

Summary

1. A test compares 2 to 5 variants of a live agent; only voice, style, model and reasoning change, never the legal side.
2. “Not enough data yet” means wait for more calls, not that there is a tie.
3. A winner only appears with enough data, a split faithful to the weights and significance; a cost “Without rate” is not read as zero.
4. If a variant violates compliance more, the platform ends the test by itself.
5. Drift warns of what gets worse, with its evidence, and changes nothing by itself.

What next

• The API of these tests and of drift: developer guide.
• Campaign A/B tests: Module 18.
• How each minute is billed: Module 7.