We give every study on our platform a choice of two respondent experiences: a chat-style conversation that asks one question at a time, or traditional web-form pages. We had opinions about which one is better. Opinions are cheap, so in April we ran the test on ourselves: one 40-question study on tariffs, fielded as two parallel cells on matched samples — 2,803 respondents in total, identical questionnaire, identical field window, different experience.
The toplines tied
Abandonment was 14.4% of starters in both cells. Not close — identical to the decimal. Median completion time was about ten minutes in both.
That result contradicted our own documentation, which claimed chat ran about a minute faster and completed two to three points better. The raw averages leaned that way, but the speed gap came entirely from a handful of marathon sessions on forms, and the completion gap didn't exist at all. We've corrected the page. If a research company won't update its claims when its own experiment disagrees with them, why would you trust its research?
The averages hid the real story
On forms, who gives up depends heavily on who they are. Abandonment ran from 9.3% among households earning 50k. Black and Hispanic respondents abandoned forms at around 20%, twice the rate of the most-retained groups.
On chat, the gradient disappears. Every group we measured stayed between roughly 13% and 16%. For under-$50k respondents, that means fatigue dropped by about a third, from 19.1% to 13.3% — and the income-by-experience interaction is one of the most reliable effects in the study (p=.0005).
This matters because weighting can rebalance who counts, but it cannot recover who left. When a fifth of your lower-income sample walks out mid-survey, the remainder carries more weight and your estimates get shakier. It shows up in the numbers: chat completes needed visibly less weighting, carrying about 14% more effective sample per complete, and fewer respondents failed our quality checks in the first place (17.1% vs 20.2%). Stalled sessions — the ones that run past an hour because someone wandered off — were five times as common on forms.
Forms won ground too
This wasn't a sweep. Higher-income respondents abandoned chat more than forms, 14.0% against 9.3% (p=.008). If your sample is affluent professionals or B2B decision-makers, forms are still the safer default, and we now say so in our guidance.
Grid batteries are the other exception. Chat asks grid rows one at a time, and while the answers land in the same place directionally, scale use shifts — respondents pick the extreme-negative point less often when statements arrive one by one. Standalone questions were interchangeable (16 of 19 showed no detectable difference between experiences, and open-ended answers were equally detailed in both), but if you run a tracker with grid batteries, keep it on one experience or bridge the change with a parallel wave.
What this changes
Our advice used to be "chat wherever possible, because it's faster and more engaging." The test forced us to be more precise. Chat isn't faster; it's fairer. It costs nothing in time, delivers cleaner data, and its real advantage is representation: it keeps the people surveys chronically under-represent — lower-income households in particular — in the study to the end.
So the decision rule we now use is about audience, not preference. General-population study where representation matters? Chat. Affluent or professional sample, or a grid-heavy instrument? Forms. Tracker? Whatever it started on.
One study on one topic is a starting point, not a law. We'll replicate this on other subjects and lengths, and when the next experiment disagrees with us, we'll update this post too. That's the deal.