Experiments
Run A/B tests on your playbooks to find out which version drives better outcomes — and make decisions based on real data, not guesswork.
Test your playbooks before you commit
Experiments (at AI Assistant > Experiments) let you compare two versions of a playbook side by side to see which one performs better in real conversations.
Instead of guessing whether a change to your playbook will help or hurt, you can run a controlled test, measure the results, and make a data-driven decision.
When to run an experiment
Experiments are most useful when you're considering a change that could meaningfully affect outcomes. Good candidates include:
- Changing the opening message in a lead qualification playbook
- Testing a more direct vs. a more conversational approach
- Trying a different sequence of questions to improve completion rates
- Testing whether offering a demo early vs. late in the conversation affects conversion
If you're making a small, low-risk change (like fixing a typo), you don't need an experiment — just publish the update.
Setting up an experiment
- Go to AI Assistant > Experiments and click Create Experiment.
- Give the experiment a name (e.g., "Lead Qual - Direct vs. Conversational").
- Select the Playbook you want to test.
- Define your two variants:
- Variant A — Usually the current version (your control).
- Variant B — The version with your proposed change.
- Set the Traffic Split — how much of the incoming traffic each variant receives. A 50/50 split is standard, but you can weight it differently if you want to limit exposure to an untested version.
- Define your Success Metric — what you're measuring. Options typically include:
- Playbook completion rate
- Handoff rate (lower is usually better)
- Lead qualification rate
- Conversation resolution rate
- Click Start Experiment.
Reading the results
As conversations flow through the experiment, Autoch.at tracks the performance of each variant. You can check the results at any time on the Experiments page.
The results show:
| Metric | What it tells you |
|---|---|
| Conversations | How many conversations each variant has handled |
| Completion rate | What percentage of conversations reached the playbook's goal |
| Handoff rate | How often each variant escalated to a human |
| Win probability | A statistical estimate of which variant is performing better |
Give your experiment enough time to collect statistically meaningful data before drawing conclusions. A sample size of at least 100 conversations per variant is a good starting point for most use cases.
Applying the winning variant
Once you're confident in the results:
- Click Declare Winner on the variant you want to keep.
- Autoch.at will publish that variant as the active version of the playbook and stop the experiment.
- The losing variant is archived but not deleted — you can reference it later.
If the experiment is inconclusive (no clear winner), you can extend it to collect more data, or try a more significant change in a new experiment.
Last updated 3 weeks ago
Built with Documentation.AI