When an agency combines audience segments, messages, creative executions, and placements, the number of possible campaign cells grows quickly. A live A/B test can compare the variants it receives, but it cannot answer questions about variants that never enter the experiment. Research can help decide which candidates deserve that live test.
The math that beats your traffic
Five segments, six message angles, four creative executions, and three funnel positions produce 5 × 6 × 4 × 3 = 360 combinations. This is an illustrative planning grid, not a requirement to run 360 independent experiments.
If a hypothetical campaign generated 36,000 eligible observations and allocated them equally, each combination would receive 100 observations. At an assumed 2% conversion rate, that is an expected two conversions per cell. These invented inputs explain the capacity problem; they are not a power calculation, a forecast, or evidence that any particular design can detect a useful difference.
Required traffic depends on the baseline rate, difference worth detecting, allocation, and chosen statistical design. Some programs can use factorial or sequential designs rather than isolated pairwise tests. The constraint is specific to the decision and traffic available, not a law that every campaign must exhaust the whole grid.
"Testable" is not "important"
A high-traffic cell may be easy to test while a strategically important niche is not. Record why a candidate enters the shortlist: a customer need, message uncertainty, creative diagnosis, or prior behavioral evidence. Keep a record of what remains untested.
A message test can ask which claim is clear, credible, and relevant to a defined segment. A creative test can diagnose comprehension and response to executions. Neither result is a measured conversion rate.
There's no single message anyway
An overall ranking can hide useful segment differences. Define the segments before fieldwork and plan the comparisons that will change a decision. A post-hoc winner in a tiny subgroup may reflect noise; a subgroup chosen after seeing the answers also needs a fresh test before it becomes a firm claim.
Consider an illustrative message shortlist for two predefined segments, with 150 respondents in each:
| Segment | Message A relevant | Message B relevant | Decision to investigate |
|---|---|---|---|
| Convenience-led | 102/150 (68%) | 75/150 (50%) | Diagnose why A fits the need |
| Price-led | 81/150 (54%) | 105/150 (70%) | Diagnose why B fits the need |
These illustrative counts show how a segment-level read can change the shortlist. They do not establish significant interactions or guarantee that separate ads will perform better. Use sample-size guidance to plan decision-critical bases and inspect the reasons behind the ratings.
How research escapes it
A practical sequence is to reduce uncertainty in stages:
- Define the client decision, target population, and constraints on the live experiment.
- Test a manageable set of comparable messages, with randomized assignment or exposure order and consistent questions.
- Use the reasons, credibility scores, and segment patterns to revise weak claims. Retest a material rewrite.
- Build a small creative shortlist around those messages and check it with creative-testing best practices.
- Run the selected variants under live delivery conditions, with a preselected outcome and analysis plan.
For example, the team might keep two messages per segment and two executions per message: eight candidates across the two segments. That is a smaller design space, not a promise that eight is the right number or that the discarded options could never win.
Aim the testing you can afford
Survey testing observes responses to a controlled stimulus. Media testing observes behavior under delivery, competition, frequency, and landing-page conditions. An option can be well understood in a survey and still lose in market. Treat the research as evidence for selecting and improving candidates, then use the live experiment to evaluate behavior.
For agencies, the deliverable is a documented shortlist: what was tested, with whom, on which bases, why the candidates advanced, and what remains uncertain. The agency testing workflow connects that research to briefs and client decisions. The campaign research overview shows how to use those findings alongside audience and brand-lift studies.