Experiments

Start by clarifying what you want to learn and how you will measure success.

• Choose a single workflow to test, such as support triage or account onboarding.
• Decide your primary metric, for example conversion rate, successful resolution rate, or handoff rate.
• Identify a control playbook and one or more variant playbooks you want to compare.

You will configure these choices on the Experiments page.

Open the Experiments page in the AI Config section and create a new experiment.

Fill in the key fields:

• Name – A clear, human-readable title like "Support triage prompt test".
• Workspace – The workspace whose conversations you want to route through this experiment.
• Description – A short hypothesis, for example "Shorter prompt should reduce handoff rate while keeping CSAT stable."

Then configure variants in the Variants table:

• Add at least two variants.
• For each row, set:
  • Name – How you want to label the variant.
  • Playbook – The playbook to run for that variant.
  • Weight – A non-negative integer for its share of traffic.
  • Control – Mark exactly one variant as the control.

From the experiment detail view, run Preflight.

Preflight checks:

• That you have 2 or more variants.
• That exactly 1 variant is marked as control.
• That all weights are non-negative integers.
• That variants are eligible to receive traffic in the selected workspace.

Preflight generates a report with errors (which block activation) and warnings (which do not block but may affect quality). Fix all errors before activation.

When preflight passes, start routing traffic:

• Use Activate/Schedule from Draft to either:
  • Activate now, or
  • Schedule a future start time.
• After activation, status moves to Active, and the health snapshot starts collecting data such as exposures and conversion rate over the last 30 days.

Use the audit timeline to confirm that activation or scheduling was applied at the expected time.

As the experiment runs, you can adjust weights in the Variants table to gradually shift traffic.

A common safe ramp pattern:

[
  {
    "name": "Control",
    "playbookId": "pbk_support_triage_v1",
    "weight": 90,
    "control": true
  },
  {
    "name": "Short prompt",
    "playbookId": "pbk_support_triage_v2",
    "weight": 10,
    "control": false
  }
]

After monitoring results, you might move to a more even split:

[
  {
    "name": "Control",
    "playbookId": "pbk_support_triage_v1",
    "weight": 50,
    "control": true
  },
  {
    "name": "Short prompt",
    "playbookId": "pbk_support_triage_v2",
    "weight": 50,
    "control": false
  }
]

Higher weights send more traffic to that variant. Keep weights as integers; the routing logic uses relative values to determine the final split.

While the experiment is Active, use the health snapshot and audit timeline in the detail view to monitor behavior.

• The health snapshot shows the last 30 days of experiment performance, including exposures, conversion rate, and overall health.
• The audit timeline records key actions, such as Preflight runs, weight changes, and lifecycle events like Activate, Pause, Stop, Complete, and Archive.

Watch for:

• Sudden drops in conversion or quality.
• Variants that appear under-exposed relative to their weights.
• Long-running experiments with stable results that may be ready for a decision.

When you have enough evidence to act, use lifecycle actions to close out or pause the experiment:

• Pause – Temporarily stop routing new traffic while keeping configuration. Use this if you need time to investigate results or incidents.
• Stop – End the experiment when conditions changed or the test is no longer valid, without declaring a winner.
• Complete – Mark the experiment as finished with a decided outcome once you have chosen a winning variant.

Each action updates the experiment status and adds an entry to the audit timeline so you can see who changed what and when.

After you complete or stop the experiment and apply your learnings (for example, by updating your default playbook configuration), Archive the experiment.

Archiving:

• Keeps the experiment available as a historical record, including configuration, health snapshot, and timeline.
• Moves the status to Archived and hides it from the default list unless you toggle include archived.

Use archiving to keep the experiment list focused on current and upcoming tests.

In the Variants table:

• Add a row for each playbook you want to test.
• Enter a clear Name for each variant.
• Select the associated Playbook.
• Set an integer Weight for the traffic share.
• Mark one variant as Control with the control checkbox.

You can edit names, playbooks, and weights while the experiment is in Draft or Paused status.

Before activation, variants must satisfy:

• At least two variants.
• Exactly one control variant.
• All weights are non-negative integers.

If any of these rules are violated, Preflight will fail, and you will not be able to activate the experiment.

Click Preflight to generate a validation report.

Preflight evaluates:

• Variant rules: at least two variants, exactly one control, valid weights.
• Eligibility: whether variants can receive traffic in the chosen workspace.
• Configuration issues that would prevent safe routing.

If Preflight shows errors, fix them and rerun until the report indicates the experiment is eligible to activate. You can also review warnings to understand potential quality risks.

When Preflight passes:

• Use Activate/Schedule for new experiments to:
  • Activate immediately, or
  • Schedule a future start time.
• Use Activate on paused experiments to resume traffic now.
• Use Schedule if you want the experiment to begin automatically later.

The audit timeline records activation and scheduling actions with timestamps.

Control an ongoing experiment based on what you observe:

• Pause – Temporarily stop routing new traffic while preserving configuration. Use this during investigations or incident response.
• Stop – End the experiment if it is no longer valid (for example, traffic mix changed). This does not declare a winner.
• Complete – Finish the experiment and mark it as decided when you have chosen a winning variant and no longer want to split traffic.

After you complete or stop an experiment, you can archive it once you have applied the outcome.

Use Archive to move finished experiments out of your active list.

• Archived experiments retain full detail: configuration, audit timeline, and health snapshot.
• To view them later, enable the include archived filter in the experiment list.

Archiving is reversible by updating the status if you need to revisit or duplicate an experiment setup.

If Preflight reports blocking errors:

• Check that you have at least two variants in the Variants table.
• Confirm that exactly one variant has the control checkbox selected.
• Verify that all weights are non-negative integers (no decimals, blanks, or negative values).
• Review any additional Preflight error messages for workspace or eligibility issues.

After making corrections, rerun Preflight until errors clear.

If a variant shows low or zero exposures despite a non-zero weight:

• Confirm the variant's weight is greater than 0 and saved.
• Make sure the variant's playbook is valid and active for the selected workspace.
• Check the health snapshot and audit timeline for recent changes to weights or status.
• Run Preflight again to detect any new configuration issues.

If necessary, temporarily increase the variant's weight to confirm routing behavior, then adjust back once you verify traffic is flowing.

If the experiment does not move to Active:

• Ensure you have run Preflight and resolved all blocking errors.
• Use Activate/Schedule from Draft, or Activate from Paused.
• If you scheduled a start time, check that the time is in the future and in the correct timezone.

Look at the audit timeline to confirm whether an activation or scheduling action succeeded or if it failed due to validation issues.

If the health snapshot shows concerning trends:

• Pause the experiment to stop new traffic while you investigate.
• Review recent changes in the audit timeline, such as weight edits or playbook updates.
• Check conversations linked to the experiment's playbooks to understand what users are experiencing.

Once you understand the impact, decide whether to Stop the experiment, Complete it with a winner, or adjust weights and Activate again.