Note: Simulation defaults apply to simulated and test respondents only. They never affect live respondents or how live respondents are screened, and simulated data is never included in delivered datasets.
When designing complex surveys, it is crucial to ensure their accuracy and functionality from the outset. To streamline this process, the MX8 Labs Research Platform includes a unique feature: simulated responses for validation every time a survey is saved. This automated check is essential for ensuring that each survey flows smoothly and every question works as expected.
How Survey Validation Works
Each time you save a survey on the MX8 Labs Research Platform, the system simulates responses to your survey by randomly selecting answers or using any defaults specified within the survey code.
The purpose of this simulated validation is to validate question programming: ensure every question is set up correctly and error-free, and test all logic paths: verify that all possible paths, skips, and terminations (e.g., screening respondents out) within the survey are functional.
When the simulation is complete, the platform will highlight in the survey code any of the following:
- Syntax Issues: Any programming errors in the survey code.
- Unasked Questions: Questions that aren't presented due to flawed skip or logic rules.
- Excessive Term Points: Points at which large numbers of respondents are excluded from the survey, signaling potential issues with the survey's termination logic.
This automated error logging helps identify potential issues early, minimizing the need for link testing and emphasizing upfront survey quality assurance.
Fixing logic errors with default values
If you encounter errors about unasked questions or unusually high termination rates in the simulation, you should:
- Review Skip or Termination Logic: Identify which questions contribute to these logic pathways and adjust them as necessary.
- Set Defaults for Relevant Questions: For questions influencing skip or term decisions, apply defaults so that simulated respondents answer the way a real qualifying respondent would, instead of at random.
Consider this example where the survey screens out anyone who doesn't watch streaming services:
video_devices = s.multi_select_question(
question="Which of the following ways do you watch TV/video? Please select all that apply.",
options=[
"Streaming services",
"TV Apps",
"Cable Provider",
"Antenna",
"NextGen TV",
"Satellite Service",
],
randomize=True,
other_options=["None of the above"],
)
s.terminate_if("Streaming services" not in video_devices,
reason="Sorry, this survey is only for people who watch streaming services."
)
In this scenario, because simulated responses are chosen randomly, they often don't select "Streaming services." Consequently, most simulated respondents screen out and the questions after the screener are never reached, so nothing downstream gets validated. Add a default response that reflects real-world behavior. Streaming is near-universal among the people this survey is for, so setting it as the default produces a test respondent who looks like a real one:
video_devices = s.multi_select_question(
question="Which of the following ways do you watch TV/video? Please select all that apply.",
options=[
"Streaming services",
"TV Apps",
"Cable Provider",
"Antenna",
"NextGen TV",
"Satellite Service",
],
default=["Streaming services"],
randomize=True,
other_options=["None of the above"],
)
s.terminate_if("Streaming services" not in video_devices,
reason="Sorry, this survey is only for people who watch streaming services."
)
By applying a default response, you simulate a more realistic distribution of test respondents, so the simulation reaches the rest of the survey and your logic is validated end to end. The default changes the simulated respondents only; live respondents still answer for themselves and are screened on their real answers.
Validation coverage: questions no simulated respondent reaches
Some questions are asked so rarely that no simulated respondent reaches them — a question behind a narrow condition in a low-IR study, for example. Defining such a question inside a function keeps the validation pass from reporting it as an error when nobody reaches it:
def ask_train_question():
s.text_question("Please tell us about your favorite train?")
if age == 94 and "model trains" in personal_interests:
ask_train_question()
Only respondents who are exactly 94 years old and interested in model trains see this question. If nobody in the simulation matches, the survey still passes validation — but that is a gap in coverage, not a clean bill of health: the question inside the condition was never exercised, so an error in it would not be caught.
Be aware of which questions fall into this category. If a rarely-asked question matters, widen the condition temporarily or set defaults that route simulated respondents into it, so validation actually covers it before you go live.
Conclusion
Although setting defaults requires some additional effort, it significantly reduces the need for repetitive link testing, allowing you to focus on building effective surveys that work correctly from the start.