MX8 Labs cross-tab reports can apply statistical tests to table cells and use color to show the result of each comparison:
- Green: Significantly higher result
- Red: Significantly lower result
- Blue: Normal result, no significant difference
- Transparent: Result is below the reporting threshold (default minimum: 20 respondents)
By default, MX8 Labs uses a residual t-test at a 95% confidence level and does not color cells with fewer than 20 respondents. The residual test identifies cells that are significantly higher or lower than expected from the row and column distribution.
Reports also support row-based and column-based comparisons. The appropriate mode depends on which cells the analysis intends to compare.
Analytic percentages and numeric means use respondent-level Taylor linearisation. This includes Response frequency and Uncoded frequency, whose standard errors and tests apply to each category's share of the selected percentage base. Their comparisons use the joint covariance of the cells, so shared respondents, overlapping groups, and mutually exclusive answers are reflected in the uncertainty of a difference. For cells and :
The tests use two-sided t tails with degrees of freedom, where counts distinct respondents in the full filtered report population. Respondents outside a cell's domain have zero contributions. Reporting weights are treated as fixed, with independent respondents in one sampling stratum; sampling clusters, explicit sampling strata, and finite-population corrections are not modeled. No automatic multiple-comparison adjustment is applied. See Weighting methodology for the estimator and its boundaries.
Choosing a different configuration
- To conduct more granular analyses within smaller subsets of data.
- If a different confidence level (e.g., 90% or 99%) is required to align with specific research standards or regulatory guidelines.
- To perform comparative analyses focused explicitly on rows or columns rather than overall data distribution.
Adjusting the confidence level
The default confidence level in MX8 Labs is 95%, corresponding to a two-sided significance level of 0.05 for each analytic comparison under the test's assumptions. The threshold applies to individual comparisons; it does not control the chance of at least one false positive across all the comparisons in a report.
When to change
- When a higher confidence level (e.g., 99%) is necessary for stricter confidence in findings.
- When lower confidence levels (e.g., 90%) can highlight potential trends or insights for exploratory analysis.
Using a residual t-test
This approach identifies cells within a report table that are significantly higher or lower than expected from the distribution between rows and columns.
The benchmark is the mean of included cells for a one-dimensional or single-row table, or the row total multiplied by the column total and divided by the table total for a two-dimensional table. Because the benchmark is estimated from the same data, the residual calculation includes covariance between the cell and the benchmark.
The numeric residual score is the standardized difference divided by the two-sided t critical value. An absolute score of at least one meets the significance threshold. It is a signed score, not a p-value.
When to use
- This is for standard exploratory analysis or to identify unexpected anomalies in the data.
- If a more focused comparative analysis across rows or columns is required, consider using a row-based or column-based t-test.
Using a row-based t-test
This test compares response rows within the same column. Labels identify the response rows that are statistically lower than the marked cell.
When to use
- When comparing answer categories within the same segment or column context.
- When the intended comparison runs vertically across response rows.
Using a column-based t-test
This test compares columns within the same response row. Labels identify the columns that are statistically lower than the marked cell.
When to use
- When comparing segments or groups for the same response.
- When the intended comparison runs horizontally across columns.
Stat testing for median aggregations
Median and Selected count median use respondent-level bootstrap draws to calculate standard errors and comparisons. The platform repeatedly resamples respondents, preserves their reporting weights and shared comparison context, and recalculates each median to estimate its uncertainty.
For a column-based comparison, the platform evaluates how consistently one segment's median exceeds another segment's median across the paired bootstrap samples. The directional threshold uses a two-sided tail convention. At 95% confidence, at least 97.5% of valid paired draws must have one median strictly above the other for that direction to be marked.
For a residual test, the benchmark is the mean of the included medians in the same comparison group. Each bootstrap draw supplies a residual relative to its group mean, and their variation supplies the residual standard error. The reported score divides the observed residual by this standard error and the two-sided normal critical value, approximately 1.96 at 95% confidence. An absolute score of at least one meets the significance threshold. If the residual standard error is zero or unavailable, the score is zero and no significance is marked.
The report's confidence level controls both modes, and the minimum-respondent threshold determines whether a result can be colored.
Row-based t-tests are not available for median rows. Use a column-based or residual test when a report includes Median or Selected count median. Although the report editor retains the familiar statistical-test names, median significance is based on the bootstrap procedure described here rather than an ordinary t-test calculation.
Conjoint and MaxDiff uncertainty
Conjoint and MaxDiff posterior-draw summaries transform values within each respondent and draw before weighted aggregation. This covers utility and part worth scores, attribute importance, simulated share, above-anchor probability, marginal purchase lift, scaled scores, and ranks, using the model selected for the calculation. Their joint covariance combines average within-draw respondent Taylor covariance with between-draw posterior covariance. The same matrix supplies marginal standard errors and comparisons, retaining dependence between items and overlapping subgroups. Draws do not increase the respondent base.
Probability above anchor is the weighted proportion of draws with item utility above zero. It is not a p-value, a purchase rate, or a test that one demographic group differs from another. Marginal Purchase Lift is a modeled change in purchase probability after replacing an attribute level; it is distinct from the matched-group Lift report described below.
See Utility and simulated share methodology for the likelihoods, formulas and uncertainty calculation.
Matched-group Lift reports
Lift reports use one two-sided conditional score test for average lift across matching runs. Each respondent has one binary outcome for the metric, and their contrast contribution is averaged across valid runs. This preserves respondent reuse: repeating identical matches cannot increase precision. Mean P-Value displays the combined score p-value; P-value threshold retains results at or below the configured value.
The test conditions on fixed matches and observed success totals within the matching strata, assuming exchangeability of outcomes within each stratum. Its conditional null moments are exact, but the normal tail is an approximation and can be optimistic for sparse outcomes. It does not correct unmeasured confounding or multiple comparisons. See Lift measurement methodology for the formula and limitations.
A threshold of 0.95 provides a broad exploratory view; it does not mean 95% confidence. A 0.05 cutoff is the conventional 5% significance threshold for this approximate test under its assumptions. Confirmatory interpretation also requires an appropriate study design and planned comparisons. Specify headline outcomes and X+ thresholds before inspecting results, disclose exploratory comparisons, and report negative and inconclusive findings alongside positive ones. See the Lift report guide for configuration and interpretation.
Changing the reporting threshold
MX8 Labs applies a reporting threshold to each response. If the number of respondents for a specific cell falls below this threshold, the results are not colored in the reports.
This is applied at the level of the individual response to the question, not at the question itself, so that we can filter out significantly low responses to an asymmetric question. For example, let's ask people about their ethnicity. We're more likely to get statistically significant results for common ethnicities. Still, we don't want to filter out ethnicity options entirely because of the small number of respondents in a specific minority.
The default threshold for reporting is 20 respondents.
When to change
- Increase the threshold when cells below a larger minimum base should be left unmarked.
- Lower the threshold when smaller cells must remain visible, while accounting for their lower precision.