How do I A/B test AI call scripts?
A/B test AI call scripts by changing one thing — the opening line, a question's wording, the order of questions, the offer — and running both versions on randomly split leads from the same source at the same time. Compare the metric the change was meant to move, such as completed conversations or bookings, after a few hundred calls per version, not after a few dozen.
Rules for a fair test
- 01Change one variable only. Two changes at once tell you nothing about either.
- 02Split leads randomly, not by day or by source.
- 03Run both versions in the same calling window and the same week.
- 04Decide the success metric before starting.
- 05Wait for enough calls — a few hundred per version — before calling a winner.
Tests worth running
| What to vary | Version A | Version B | Metric |
|---|---|---|---|
| Opening line | Generic greeting | Mentions the exact enquiry | Calls past 30 seconds |
| Budget question | Open question | Offers three ranges | Budget field filled |
| Booking ask | Asks when they are free | Offers two slots | Bookings |
| Language of opener | Hindi | Hinglish | Completed conversations |
| Voice tier | Standard | Premium | Qualified leads |
How to split the leads
For bulk campaigns, shuffle the list and split it in half, then upload each half against a different version of the agent. For instant lead calling, route leads to two versions by a simple rule in Zapier, Make or your CRM, such as odd and even lead IDs.
Keep a short log of every test: what changed, the dates, how many calls each version received and the result. A written log stops the same idea being retested six months later and shows which kinds of change, such as opening lines or booking asks, move results most.
Reading the result
Small differences on small samples are usually noise. If version B beats A by a large margin over several hundred calls each, adopt it and test the next idea. If the difference is small, keep the simpler version and move on.
Two script versions can run side by side as two campaigns on two channels, each extra channel costing ₹649/month, so both are tested at the same time on similar leads. Outcomes and custom fields for each version export as CSV for comparison.
Want to hear an AI voice agent handle this on a real call?