Or why the demo is never the whole story
The AI research tool category is new enough that buying decisions are still happening in darkness. You get a demo. It looks impressive. The vendor talks about speed and cost. You compare it to your current stack. The math looks good. So you sign the deal.
Then you get into the details. The tool doesn't quite work the way the demo showed. The output needs more interpretation than expected. Integrating it into your workflows is harder than promised. You've already committed the budget. Now you're stuck with something that solves a problem you didn't actually have.
I've seen this pattern five times in the last year. Five times I've talked to insights leaders who went to market too fast and bought tools that looked good in the room but didn't survive contact with reality.
Here are the five questions that would have changed those conversations.
Does it use real respondents or only synthetic?
This question matters because it shapes what the tool can actually do. There's a narrative in the market that says synthetic data can replace real research. It can't. Synthetic data is incredibly useful, but not as a replacement. It's as a complement.
A platform that leans entirely on synthetic respondents will be fast and cheap, which is why vendors love to lead with it. But it can't estimate incidence. It can't capture lived experience. It can't understand emotional nuance the way a real human can. It can't handle surprise—those moments when your respondents tell you something that contradicts your assumption. Synthetic data can't surprise you because it's shaped by whatever data it was trained on.
The right answer to this question isn't "we only use real respondents" or "we only use synthetic." It's "we use both, and we're clear about when to use which." A platform that can do real respondent research and synthetic research and knows the difference has thought about this problem. One that only does one or the other has chosen convenience over capability.
Ask them: Can you run a study with real respondents today? If they hesitate, that's the answer you need.
What does the fraud detection actually do?
Fraud in survey research has evolved. It used to be obvious—duplicate responses, bot patterns, completion times measured in seconds. Now it's sophisticated. AI-generated responses that pass attention checks. Respondents mimicking human timing patterns. Answers tailored to screening logic to pass eligibility gates. The old fraud is easy to catch. The new fraud requires a different kind of thinking.
When a vendor tells you they have "advanced fraud detection," you need specifics. Are they checking 35+ browser fingerprinting attributes? Can they detect incognito, VPNs, proxies, and emulators? Can they identify device tampering? Do they use dynamic risk scoring that flags suspicious patterns without assuming all unusual behaviour is fraud?
Here's the trap: fraud detection can be aggressive or lenient. An aggressive system will flag a lot of responses. You get clean data but lose volume. A lenient system will miss fraud. You keep volume but data quality suffers. The right system is neither—it's precise. It catches fraud without creating false positives.
Ask them to walk you through a specific fraud scenario. Not the easy ones. Show them a response from a real person using a VPN at unusual hours, completing the survey faster than normal, but with genuine engagement. Can they distinguish that from fraud? If they can't explain how they do it, they probably can't do it.
Can your team run it without vendor hand-holding?
This is the operational question that determines whether the tool becomes part of your stack or becomes a service you buy. Some platforms are designed to be self-serve. Your team builds studies, deploys them, analyses them, writes reports. You own the full pipeline. Other platforms are built for white-glove service. The vendor does the heavy lifting. You provide the brief. They come back with answers.
Both models have value. But you need to know which one you're buying before you sign. Because if you want self-serve and you buy white-glove, you'll feel locked in. You'll be waiting on the vendor every time you want to run something. And if you want white-glove and you buy self-serve, your team will struggle through weeks of learning curve and you'll blame the vendor.
The best platforms offer both options. You can run straightforward studies yourself when you have the bandwidth. You can hand off complex projects to the vendor when you need to. But not every platform is built that way.
Ask them: Can I run a study tomorrow without training? Can my team do it, or just the project manager? What happens if I need help at 2am on a Wednesday?
Does it integrate into your existing data workflows?
This is the detail that kills most implementations. A new research tool is useless if it sits in isolation. It needs to talk to your data stack. You need to push data into it—audience segments, brand attributes, product features. You need to pull data out of it—insights, themes, synthesis. You need to connect it to your activation tools so that what you learn actually influences what you do.
Some platforms have APIs and built-in integrations. Some platforms are islands. They're closed systems. You work inside them and export the results when you're done. The closed system approach is simpler to sell, which is why you'll see it in more demos. But it creates friction. Every integration is a manual bridge. Every workflow has a handoff point. Friction slows adoption. It also slows decision-making because insights take longer to reach the people who need them.
Ask them about their Insights API. Can you programmatically access results? Can you push audience data in? Can you connect to your BI tool? Can you stream results into Slack when key thresholds hit? If they get vague about APIs, assume integration will be painful.
What happens when something goes wrong mid-field?
This is the stress-test question. A platform will work fine on a Tuesday morning when everything is normal. The real test is what happens at 3am when a study has been fielding for six hours, you've had two hundred responses, and suddenly you notice a problem with the logic. Maybe a skip pattern is broken. Maybe you see fraud that the system missed. Maybe you realize the question wording is confusing people.
In your old system, you'd pause the study, fix it, and redeploy. That's relatively straightforward with a panel manager. With an AI platform, the workflow might be different. You need to know what it is before you're in crisis mode.
Some platforms allow you to pause and modify on the fly. Some don't. Some can heal a study mid-field. Some can't. Some will give you support at any hour. Some have business hours only. You need to know this before you're fielding at scale.
Ask them: I find a problem two days into the field. What happens? How long does a fix take? Can I pause? Can I modify the survey? What data do I keep? What do I lose? Listen carefully to the answer. Vendors who've thought about this will have practised the response. Vendors who haven't will get vague.
These five questions won't cover every detail. But they'll separate the platforms that are genuinely built for research from the ones that are built for demos. They'll distinguish between tools that integrate into your workflow and tools that demand you work around them. They'll show you which vendors have thought about real constraints and which ones are selling you the future.
The demo will always look good. These questions will tell you whether the reality matches.