We give every study on our platform a choice of two respondent experiences: a chat-style conversation that asks one question at a time, or traditional web-form pages. We had opinions about which one is better. Opinions are cheap, so in April we ran the test on ourselves: one 40-question study on tariffs, fielded as two parallel cells on matched samples — 2,803 respondents in total, identical questionnaire, identical field window, different experience.
The toplines tied
Abandonment was 14.4% of starters in both cells. Not close — identical to the decimal. Median completion time was about ten minutes in both.
That result contradicted our own documentation, which claimed chat ran about a minute faster and completed two to three points better. The raw averages leaned that way, but the speed gap came entirely from a handful of marathon sessions on forms, and the completion gap didn't exist at all. We've corrected the page. If a research company won't update its claims when its own experiment disagrees with them, why would you trust its research?
The averages hid the real story
On forms, who gives up depends heavily on who they are. Abandonment ran from 9.3% among households earning $50k or more to 19.1% among households under $50k. Black and Hispanic respondents abandoned forms at around 20%, twice the rate of the most-retained groups.
On chat, the gradient disappears. Every group we measured stayed between roughly 13% and 16%. For under-$50k respondents, that means fatigue dropped by about a third, from 19.1% to 13.3% — and the income-by-experience interaction is one of the most reliable effects in the study (p=.0005).
This matters because weighting can rebalance who counts, but it cannot recover who left. When a fifth of your lower-income sample walks out mid-survey, the remainder carries more weight and your estimates get shakier. It shows up in the numbers: chat completes needed visibly less weighting, carrying about 14% more effective sample per complete, and fewer respondents failed our quality checks in the first place (17.1% vs 20.2%). Stalled sessions — the ones that run past an hour because someone wandered off — were five times as common on forms.
Forms won ground too
This wasn't a sweep. Higher-income respondents abandoned chat more than forms, 14.0% against 9.3% (p=.008). That supports considering forms for a higher-income audience in a similar study. It does not independently establish the best mode for B2B decision-makers, who were not a separate tested population here.
Grid batteries are the other exception. Chat asks grid rows one at a time, and while the answers land in the same place directionally, scale use shifts — respondents pick the extreme-negative point less often when statements arrive one by one. Standalone questions were interchangeable (16 of 19 showed no detectable difference between experiences, and open-ended answers were equally detailed in both), but if you run a tracker with grid batteries, keep it on one experience or bridge the change with a parallel wave.
What this changes
Our advice used to be "chat wherever possible, because it's faster and more engaging." The test forced us to be more precise. In this study, chat was not faster at the median. It retained more lower-income respondents and had a lower observed quality-check failure rate. Those are specific findings about this questionnaire and recruited sample, rather than a universal ranking of respondent experiences.
So the decision rule we now use is about audience, not preference. General-population study where representation matters? Chat. Affluent or professional sample, or a grid-heavy instrument? Forms. Tracker? Whatever it started on.
One study on one topic is a starting point, not a law. We'll replicate this on other subjects and lengths, and when the next experiment disagrees with us, we'll update this post too. That's the deal.
Read the denominator before the comparison
The reported 14.4% abandonment figure uses starters, not the whole recruited audience. Completion-time comparisons concern the respondents included in that analysis; quality-check exclusions answer a different question. A report should show each analysis base separately, with raw and weighted subgroup counts where applicable.
For an illustrative group of 1,000 starters, 14.4% abandonment means 144 did not complete and 856 did. That arithmetic does not reveal whether those 144 had different opinions from the completers. Calibration can rebalance observed attributes among the retained respondents; it cannot reconstruct their missing answers. The weighting methodology explains the effective-base adjustment and its limits.
Choose the next test, not a universal winner
For a new study, compare the modes on a small parallel field before changing the whole program. Keep questionnaire wording, recruitment, eligibility, field window, and quality rules comparable; record assignment and any departures from the plan. Define the primary completion or measurement outcome before examining many subgroup differences. A small p-value on one subgroup does not establish that every question is interchangeable, and failure to detect a difference is not an equivalence test.
A grid-heavy tracker particularly needs a bridge: carry the old experience alongside the proposed one, inspect scale use and subgroup retention, and avoid splicing a mode change into a trend without explanation. Our respondent-experience guide gives the implementation choices. The research platform describes the available workflow; the translation guide connects mode, sampling, and precision to campaign decisions.
The figures above are the original internal experiment's reported results. This article does not publish a cell-level dataset or independently reanalyze it. Use the findings as a hypothesis for a comparable pilot, with the limits of one topic, questionnaire length, and recruited sample kept visible.