Voice AI A/B experiments and drift alerts

Español: Experimentos A/B de la voz IA y alertas de deriva.

An experiment compares two to five variants of one live AI agent: its voice, a style paragraph, its model and its reasoning settings. Each call is assigned to one variant by a fixed rule, its outcome is attributed to that variant, and core publishes a verdict from the ledger. Next to it, a drift evaluator watches the same metrics in hourly buckets and raises an alert when a variant or a model leaves its band. Both are always on: there is no tenant flag that turns them off. An agent with no running experiment behaves exactly as before.

The campaign counterpart is Campaign A/B experiments; the routes below follow the same shape. The routes, bodies, envelopes and status codes are copied from the AiAgentExperiment* and AiDrift* schemas and the listAiAgentExperiments, createAiAgentExperiment, getAiAgentExperiment, updateAiAgentExperiment, deleteAiAgentExperiment, startAiAgentExperiment, stopAiAgentExperiment, getAiAgentExperimentResults and listAiAgentDrift operations of core's openapi/dialer-v1.yaml.

Who can do what

Writes (POST, PATCH, DELETE, start, stop) take a dd_ key or a dashboard login with the account_admin role. A member login gets 403 role_forbidden. Reads (list, get, results, drift) are open to every tenant-scope token. Cross-tenant resources and unknown identifiers return 404.

Create a draft

POST /v1/ai/agents/{id}/experiments accepts name, primary_metric and variants. primary_metric is one of ptp_rate, contained_rate, handoff_rate, first_audio_p95_ms or cost_per_call, and it cannot be changed afterwards. The variants array has 2–5 entries, each with a name, a weight from 1 through 1,000 and exactly one is_control: true entry. The control is the base agent, so it sends no overrides. The agent must be live; any other agent is 409.

curl -sS -X POST "$API/v1/ai/agents/$AGENT_ID/experiments" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Model and tone",
    "primary_metric": "handoff_rate",
    "variants": [
      {"name": "Control", "weight": 50, "is_control": true, "overrides": {}},
      {"name": "Larger model, direct tone", "weight": 50, "overrides": {
        "model": "openai/gpt-oss-120b",
        "prompt_style": "Be direct and brief; confirm the amount before proposing a plan."
      }}
    ]
  }'

The response is 201 with { "experiment": ... }, in state draft. A malformed body is 400. An illegal experiment is 422: every refused field comes back in one response, each with its path in error.details, and nothing is stored (createAiAgentExperiment).

What a variant may change

The overrides object is a closed list. Only these keys are accepted:

• voice: { "voice_id": "...", "pace": 0..1 }, at least one of the two. voice_id must be in the voice catalog (GET /v1/ai/voices). The disclosure is pre-rendered with the voice of each variant before dialing;
• prompt_style: one line (no line break) of 1–600 characters, appended after everything the agent composes. It cannot replace any of it;
• model: a model of the platform's model catalog. In the vendored spec the catalog is openai/gpt-oss-20b, openai/gpt-oss-120b, llama-3.3-70b-versatile and llama-3.1-8b-instant;
• llm: { "reasoning_effort": "low" | "medium" | "high", "prompt_form": "compact" | "full" }, at least one of the two.

Any other key is refused rather than ignored, with a 422:

{
  "error": {
    "code": "unprocessable",
    "message": "variants[1].overrides has keys outside the closed list: disclosure; allowed: voice, prompt_style, model, llm",
    "details": [
      { "path": "variants[1].overrides", "message": "has keys outside the closed list: disclosure; allowed: voice, prompt_style, model, llm" }
    ]
  }
}

What never varies

Nothing legal is in the list. These are the same in every variant and cannot be put in an experiment: the disclosure (text, lock and pre-rendered audio), the spoken opt-out, the handoff_rules including the human-requested rule, the policy and its tools, the negotiation limits (for example max_settlement_pct) and the prohibited phrases. The compliance gates of the call run before the variant is resolved and are never told about the experiment: a call blocked by a gate creates no conversation and does not count as an exposure of any variant.

Inspect, edit and delete

• GET /v1/ai/agents/{id}/experiments lists every state, newest first, with variants in position order.
• GET /v1/ai/agents/{id}/experiments/{experiment_id} returns one experiment.
• PATCH /v1/ai/agents/{id}/experiments/{experiment_id} accepts name and/or variants; replacing variants replaces the whole set. primary_metric may only be repeated, not changed.
• DELETE /v1/ai/agents/{id}/experiments/{experiment_id} deletes only a draft and returns 204.

Once an experiment runs, or ended, its variants are frozen: editing or deleting it is 409. To test another prompt style, voice or model, create another experiment.

Start and stop

curl -sS -X POST \
  "$API/v1/ai/agents/$AGENT_ID/experiments/$EXPERIMENT_ID/start" \
  -H "Authorization: Bearer $TOKEN"

POST .../start changes draft to running and validates the variants again against the voice and model catalogs as they are now: it is 422 with details if a model left the catalog since the draft was written. It is 409 if the agent is not live, if the agent already has another experiment running, or if the experiment is not a draft.

curl -sS -X POST \
  "$API/v1/ai/agents/$AGENT_ID/experiments/$EXPERIMENT_ID/stop" \
  -H "Authorization: Bearer $TOKEN"

POST .../stop changes running to the terminal ended, and the agent goes back to being answered as the base agent; any other state is 409. ended_reason is null for a manual stop; when core ends the experiment on its own (see the compliance stop) it says why.

Assignment and attribution

Core assigns each unit to a variant with the same pure function as the campaign experiments, over the variant weights in position order. The unit is the debt of the call when it has one and the call itself when it has none, so the same unit gets the same variant for as long as the experiment runs. A debt that already has a sealed call attempt is served the variant of that attempt. The conversation records its experiment and variant when it opens and is never reassigned. A variant that cannot be resolved (an override that no longer validates, a model withdrawn from the catalog) is not served: the call uses the base agent.

Results and verdict

GET /v1/ai/agents/{id}/experiments/{experiment_id}/results returns { "results": ... }. It is recomputed from the ledger on every read (no counters) and reading never changes the experiment. Errors are 401, 404 and 429.

curl -sS \
  "$API/v1/ai/agents/$AGENT_ID/experiments/$EXPERIMENT_ID/results" \
  -H "Authorization: Bearer $TOKEN"

• variants[]: per variant assigned_units, conversations, closed, valid, failed, ptp, handoffs, contained, violating and the rates ptp_rate, contained_rate, handoff_rate, failed_rate, compliance_violation_rate and provider_error_rate (each null when its denominator is zero). A valid conversation is a closed one whose outcome is neither failed nor voicemail; the rates of the primary metrics are over valid conversations, failed_rate and compliance_violation_rate over all closed ones.
• metrics_coverage (with_metrics over closed) says how many closed conversations published per-call metrics: a runtime that does not publish yet shows up here and not as a clean zero.
• first_audio_ms_p95 carries n, mean, variance, p50_mean and p95.
• consumption carries minutes_total and minutes_per_call.
• cost.status is priced, partial or unpriced, with n and unpriced_calls. The amount of the provider cost is never published: it is the platform's cost, not a price, and the tenant is billed per minute.
• unattributed_conversations is non-zero when a conversation's stamp and the experiment disagree.

The verdict

verdict.status is insufficient_data, no_significant_difference or winner.

• insufficient_data: some arm is under 100 valid conversations or under 10 successes for a rate, or under 100 calls with the measure for a mean. verdict.sample_floor states min_n, min_successes (null for a mean) and met. A reason of unpriced means the primary metric is cost_per_call and some arm has an unpriced call.
• Rates use the two-proportion z test, means (latency, cost) use Welch's test in the normal approximation; both are two-sided, alpha 0.05, with a Bonferroni adjustment over the challengers. ptp_rate and contained_rate are better when higher; handoff_rate, first_audio_p95_ms and cost_per_call are better when lower.
• winner carries variant_id, versus_variant_id, the adjusted p_value, difference, ci_low and ci_high; comparisons[] has one entry per challenger. For cost_per_call the verdict keeps its significance and its winner but not the magnitudes (value, difference, ci_low, ci_high are absent).
• allocation_warning is true when the observed split of units deviates from the weights (chi-square, alpha 0.001). While it is true no winner is emitted, and verdict.withheld_by is allocation_warning when a would-be winner was withheld.

The compliance stop

safety.breaches[] lists the variants whose compliance_violation_rate is significantly higher than the control's (z test, alpha 0.05, at least 100 closed conversations in both arms). When that happens core ends the experiment on its own, writes the evidence in ended_reason, and the agent goes back to its base configuration. There is no switch for this stop.

Drift alerts

The drift evaluator runs for every tenant every 15 minutes and changes nothing by itself: it does not pause an agent, end an experiment or touch a gate. It reads hourly buckets (UTC, by the hour the call closed) per scope, (agent, variant | none) and (model), for five metrics (AiDriftMetric): first_audio_p95_ms, handoff_rate, provider_error_rate, failed_rate and cost_per_call. The model scope counts every call that used that model, with or without a variant, so a degradation of the provider shows up even when no variant changed. Every metric alerts only upwards.

Two rules judge them (rule):

• zscore.v1: a moving band over the previous 7 days, with at least 24 baseline buckets and 30 calls in the bucket. It alerts when the bucket is above baseline_mean + k·max(baseline_stddev, stddev_floor), with k = 3.0.
• budget.v1: the p95 of the first audio of the bucket above the latency budget, 900 ms. A moving band does not see a slow degradation that has become the new normal; the budget does.

curl -sS "$API/v1/ai/agents/drift?state=open&limit=50" \
  -H "Authorization: Bearer $TOKEN"

GET /v1/ai/agents/drift answers { "drift": [...], "coverage": [...], "next_cursor": ... }, newest first. It filters by agent_id, variant_id (a variant id, or base for the agent without a variant), model, metric and state (open or cleared); limit is 50 by default and at most 200, and cursor takes the previous next_cursor (null on the last page). model cannot be combined with agent_id or variant_id (400). Errors are 400, 401 and 403.

Each drift[] entry (AiDriftEvent) is its opened row with the evidence it was written with: observed, baseline_mean, baseline_stddev, stddev_floor, threshold, k, sample_n, baseline_n and the series the rule looked at, oldest first. For budget.v1 the baseline fields are null. A later cleared row that cites the alert makes state cleared and fills resolution.

coverage[] (AiDriftCoverage) says what is being judged: a series whose last bucket had fewer than min_sample calls is insufficient_sample, and one without min_baseline baseline buckets is insufficient_baseline. An empty drift list for a series that is not ok means "not enough data yet", never "all fine".

The ai.drift.detected event

Core also announces each opened alert through the outbound webhooks, as the event ai.drift.detected. data is the AiDriftEvent above without resolution (AiDriftDetectedData), with state always open. It is sent once per opened alert; clearing is not announced, so read GET /v1/ai/agents/drift?state=cleared for that. As with every event, deduplicate on the event_id of the signed body.

Declared limits

• A style paragraph cannot be proved legally neutral by a lexical rule. What bounds it are the deterministic guards that run above the model and never read the style, and the compliance stop above.
• A person with two debts can hear two variants (the unit is the debt); on an inbound call the unit is the call, so someone who calls back can land on another variant.
• The test has a fixed horizon: no early winner.
• Provider cost exists only where the platform administrator has loaded a rate; without it the cost is unpriced.
• Drift is seen through its effects (latency, handoffs, errors, cost), not by comparing the meaning of the model's answers.