Campaign A/B experiments

Español: Experimentos A/B de campaña.

An experiment compares two to five variants of one campaign. It runs on voice, SMS and email campaigns (see SMS and email campaigns). Create it as a draft, start it when the variants are ready, and stop it to make it terminal. The routes, request bodies, response envelopes and status codes below are copied from the Experiment schemas and the listCampaignExperiments, createCampaignExperiment, getCampaignExperiment, updateCampaignExperiment, deleteCampaignExperiment, startCampaignExperiment and stopCampaignExperiment operations in the vendored openapi/dialer-v1.yaml.

Create a draft

POST /v1/campaigns/{id}/experiments accepts name, primary_metric and variants. primary_metric is either contact_rate or promise_rate. The variants array has 2–5 entries, each with a name, a weight from 1 through 1,000, and exactly one is_control: true entry.

curl -sS -X POST "$API/v1/campaigns/$CAMPAIGN_ID/experiments" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Window test",
    "primary_metric": "contact_rate",
    "variants": [
      {"name": "Control", "weight": 50, "is_control": true, "overrides": {}},
      {"name": "Afternoon window", "weight": 50, "overrides": {
        "window": {"start": "10:00", "end": "16:00"}
      }}
    ]
  }'

The response is 201 with { "experiment": ... }. A malformed body is 400; an invalid experiment is 422 with error.details, one entry per refused field. window and min_gap_seconds only apply to a voice campaign, and an SMS or email campaign cannot choose promise_rate (a promise points at a call, not at a message): all three are refused with 422 (createCampaignExperiment). A window must be a subset of the campaign window, and an SMS script must satisfy the campaign's own validators, including the STOP requirement.

What a variant may change

The overrides object is a closed list. Only these keys are accepted:

• script: { "field": "tts_prompt" | "voicemail_message" | "sms_body", "text": "..." };
• window: { "start": "HH:MM", "end": "HH:MM" }, never wider than the campaign window;
• min_gap_seconds: an integer from 1 through 2,592,000 (voice campaigns only, see below);
• email_template_id: a UUID for an approved template belonging to the tenant.

Keys such as optout_digit, abandonment_threshold and max_dial_ratio are not experiment controls. They are refused rather than ignored. This closed list and the field constraints come from ExperimentVariantOverrides in the vendored spec; the compliance boundary is also recorded in core ADR #242.

Inspect, edit and delete

• GET /v1/campaigns/{id}/experiments lists every state, newest first, with variants in position order.
• GET /v1/campaigns/{id}/experiments/{experiment_id} returns one experiment.
• PATCH /v1/campaigns/{id}/experiments/{experiment_id} accepts name and/or variants; replacing variants replaces the whole set. primary_metric may only be repeated, not changed.
• DELETE /v1/campaigns/{id}/experiments/{experiment_id} deletes only a draft and returns 204.

Once an experiment is running or ended, its variants are frozen. Editing or deleting it returns 409. Cross-tenant resources and unknown identifiers return 404.

Start and stop

Start a draft with:

curl -sS -X POST \
  "$API/v1/campaigns/$CAMPAIGN_ID/experiments/$EXPERIMENT_ID/start" \
  -H "Authorization: Bearer $TOKEN"

POST .../start changes draft to running and validates the variants again against the current campaign. It returns 422 if the constraints no longer hold and 409 if another experiment is already running or the experiment is not a draft. Experiments run on voice, SMS and email campaigns whose messages or calls the platform sends itself. Starting one on a preview or manual campaign is 422 (details[].path = "campaign"), even though creating the draft was accepted (startCampaignExperiment).

Stop a running experiment with:

curl -sS -X POST \
  "$API/v1/campaigns/$CAMPAIGN_ID/experiments/$EXPERIMENT_ID/stop" \
  -H "Authorization: Bearer $TOKEN"

POST .../stop changes running to terminal ended; any other current state returns 409.

SMS and email campaigns

On an SMS or email campaign the experiment reads the campaign's message ledger instead of the call ledger (getCampaignExperimentResults):

• Attempts are the original outbound messages and their status transitions. A message the provider never took is not_placed, and a contact is a delivered message.
• promises and abandoned are always 0, so promise_rate and abandon_rate are null. Use primary_metric: contact_rate; promise_rate is refused at create time.
• For an SMS campaign a variant can override script with field: "sms_body"; the text must still satisfy the campaign's own validators, including STOP. For a variant's email_template_id, the template must be an approved one of the tenant (ExperimentVariantOverrides).
• The closed override list, the assignment and the verdict rules are the same as for voice.

Core's v0.105.0 release carries the SMS and email experiment commits tagged core#2057 (messages ledgers stamp the variant, the runners apply it, results and start accept the channel). The vendored spec of that release is the source for the lines above.

The separate read GET /v1/campaigns/{id}/email-skips (campaignEmailSkips) explains why an e-mail campaign did not send to a lead; it is a diagnostic of the e-mail runner and takes no experiment parameter, so it is not part of the experiment contract.

Stable assignment and compliance invariants

Core assigns a debt deterministically from sha256(experiment_id <> debt_id), using the first 64 bits and the variant weights in position order. The same experiment/debt pair therefore receives the same variant across retries; the experiment identifier is part of the input, so separate experiments do not share an assignment. This behavior is defined by core's Dialer.Campaigns.Experiments.Assignment implementation and its tests.

The experiment does not create a separate compliance lane. Every variant goes through the same compliance gates, and the legal disclosure is unchanged across variants. The closed override list is intentional: opt-out, abandonment and dial-ratio controls cannot be varied by an experiment. Core ADR #242 documents this boundary and the campaign runner keeps the full compliance path at origination.

Results and verdict

GET /v1/campaigns/{id}/experiments/{experiment_id}/results returns { "results": ... }. It is recomputed from the call ledger on every read (no counters), and a draft answers with zeros and insufficient_data. Errors are 401, 404 and 429.

curl -sS \
  "$API/v1/campaigns/$CAMPAIGN_ID/experiments/$EXPERIMENT_ID/results" \
  -H "Authorization: Bearer $TOKEN"

• variants[]: per variant assigned_debts, attempts, not_placed, placed, contacts, promises, abandoned, successes, and the rates contact_rate, promise_rate and abandon_rate (each null when its denominator is zero). Attempts that never left (not_placed) are outside every denominator.
• verdict.status is insufficient_data, no_significant_difference or winner. A winner carries variant_id, versus_variant_id, p_value, difference, ci_low and ci_high; comparisons[] has one entry per challenger and sample_floor states the minimum sample.
• allocation_warning is true when the observed split of assigned debts deviates from the weights (chi-square, alpha 0.001); then no winner is emitted. verdict.withheld_by is allocation_warning only when a would-be winner was withheld (the status is then no_significant_difference); otherwise it is null, even with allocation_warning: true.
• unattributed_attempts is non-zero when an attempt's experiment stamp and the experiment disagree.

The field definitions are ExperimentResults, ExperimentVariantResults and ExperimentVerdict in the vendored spec (core v0.105.0); the recomputation and verdict rules are recorded in core ADR #249.