AI AssistantExperiments

Experiments

Compare two playbooks on real conversations. Create a draft experiment, check it, run it, watch results, and promote the winning playbook when you're sure.

An experiment splits real conversations between two or more playbooks so you can see which one works better, instead of guessing.

Open AI Assistant → Experiments in the sidebar.

Experiments is an advanced tool. It appears only for admins and for team members who have been given Experiments permission. If you don't see it, ask an admin. Experiments that apply to the whole account (shown as Tenant in the Scope column) can only be changed by an admin.

When to run one

Experiments are worth it when a change could really affect results, for example:

  • A different opening message in a lead-qualification playbook.
  • A direct tone versus a conversational one.
  • Offering a demo early versus late in the conversation.

For small fixes such as typos, just edit and publish the playbook.

The Experiments page

  • Workspace: experiments belong to the workspace you've selected in the sidebar.
  • Status filter: All statuses, Draft, Active, Paused, Completed or Archived. Tick Include archived to see old ones. Click Refresh to reload.
  • Experiment List: Name, Status, Scope (a workspace name, or Tenant for account-wide), and Updated.
  • Click any row to open its details on the right.

Set up an experiment

You need two or more playbooks first. Duplicate your current playbook and change the copy (see AI Playbooks).

Fill in Create Experiment

Enter an Experiment name and a Description. Optionally set a start and end date and time.

Add the variants

Under Initial Variants, each row is one playbook being tested. Give it a Name, the playbook's ID in the playbook_id box, and a Weight. Tick Control on the row that is your current playbook.

Create the draft

Click Create Draft Experiment. It appears in the list with status Draft. Nothing is live yet.

Adjust if needed

Select the experiment to edit its name, description and dates and click Save Experiment. Under Variants you can edit a row and click Save, or use Add Variant to add another.

Weight controls how much of the traffic each variant gets. Equal weights give an even split. Give the untested playbook a smaller weight if you want to limit risk.

Check it, then start it

In Preflight & Lifecycle:

  1. Click Preflight Activate (or Preflight Schedule if you've picked dates). A report says Eligible or Blocked, with a list of Errors (must fix) and Warnings (worth reading).
  2. When it's eligible, click Activate to start now, or set Schedule start / Schedule end and click Schedule.
  3. You're asked to confirm.

While it runs you can use:

ButtonWhat it does
PausePauses the experiment (status becomes Paused).
CompleteMarks the experiment as Completed.
StopStops the experiment.
ArchiveMoves it to Archived. It's hidden unless you tick Include archived.

Roll out gradually

The Rollout section shows the rollout state, the Percentage of traffic, and any hold reason. Admins can click Apply default ramp plan, Hold (with a Hold reason), Resume, or Promote cohort (with a Target %) after ticking the checks Preflight OK, Baseline OK and Guardrails OK. Refresh monitoring shows the next check time and current cohort.

Read the results

Health Snapshot (Last 30 Days) lists each experiment with a health status, how many conversations were Exposures, how many Converted, and the Conversion percentage.

Give an experiment plenty of conversations before you decide. A small sample can mislead you.

Pick a winner

In Winner & Sunset:

  1. Choose the best variant in Select winner variant. Add a Reason (optional).
  2. Click Promote winner. Confirm "Promote this variant to default playbook behavior?"
  3. Click Sunset losers to retire the others.
  4. Changed your mind? Paste the promotion's ID into Promotion audit log id (rollback) and click Rollback winner.

Clean up and history

  • Audit Timeline lists every action taken on the experiment, who did it and when. Click Load More for older entries.
  • Retention shows how much old experiment data can be removed. Dry-run cleanup previews it. Run cleanup permanently deletes it and can't be undone.