Weighting methodology

This page is the technical companion to the Weighted results FAQ. The FAQ explains what weighting does and when it runs; this page documents the estimator, stage composition, respondent-universe selection, weighting diagnostics, and respondent-level uncertainty for analytic cross-tab reports.

Scope

MX8 Labs uses iterative proportional fitting (IPF, also called raking) to calibrate respondent weights to marginal targets supplied through quota configuration. Weighted point estimates are computed from the final calibrated weights. Analytic percentages and numeric means use respondent-level Taylor linearisation for their standard errors and joint covariance for significance testing. Median aggregations, matched-group Lift reports, and discrete-choice outputs use the inference procedures summarized under Significance testing.

Response frequency and Uncoded frequency use the proportion estimator. Their standard errors and statistical tests apply to each category's share of the selected percentage base. Raw counts and respondent bases are reported alongside those shares.

Weighting can be run in one stage or two stages:

• Pre-weighting stage (optional)
• Main weighting stage

Each stage can use its own configured respondent universe.

The Taylor calculations treat the final weights as fixed and respondents as independent sampling units in one stratum. They do not estimate uncertainty from fitting the calibration weights, model sampling clusters or explicit sampling strata, or apply finite-population corrections.

Notation

Let respondents be indexed by $i = 1, \dots, n$ with final calibrated weights $w_i \ge 0$.

For a reporting cell $c$ (for example, "women aged 25-34 in the West region"):

• $S_c$ is the set of respondents in cell $c$.
• $N_c = |S_c|$ is the raw base for that cell.
• $W_c = \sum_{i \in S_c} w_i$ is the weighted base.

For a binary outcome $Y_i \in \{0, 1\}$, the weighted proportion is:

$$\hat{p}_c = \frac{\sum_{i \in S_c} w_i Y_i}{\sum_{i \in S_c} w_i}$$

Weighted means for numeric or rating outcomes are computed analogously.

Target marginals

Weighting targets are supplied as category proportions for one or more questions defined on the respondent source (age bands, gender, region, and so on). Targets may be:

• One-way, e.g. the marginal distribution of age alone.
• Nested (joint), e.g. a target distribution over the joint of age and gender. Joint targets are calibrated directly against the joint cell, not against the two marginals separately.

Only categories with a positive target are treated as required for the eligibility checks described below. A category with a zero target is permitted to be absent from the eligible data.

Where targets come from. Targets are supplied by the study, not imposed by the platform, which is deliberate: calibration answers "representative of which population?", and only the study owner can answer that. For common markets the standard screeners carry census-aligned quotas that can be used as supplied targets, and because quota configuration and weighting targets share the same definitions, a study quota'd to those screeners is calibrated back to them by default. Where a study defines its own universe — a customer base, a category buyer group, a market with no reliable public benchmark — the targets are the client's and should be documented alongside the results, since the validity of the point estimates depends on them. See limitation 2 below.

Respondent-universe selection

Each weighting stage defines its own eligible respondent universe. This is a methodological control, not just an operational filter — calibration answers "representative of which population?", and universe selection defines that population.

Common configurations include:

• Completes-only universes for analysis-sample inference.
• Broader universes (for example including selected terminates) when calibrating to entrant-level representativeness.
• Distinct universes across stages when pre-weighting is used to correct source composition before final reporting calibration.

IPF within a stage

Given a seed tensor $S$ built from the stage's eligible respondents and a set of target margins $\{m_j\}$ on the weighting dimensions, IPF repeatedly rescales $S$ along one dimension at a time so that its marginal on that dimension matches the corresponding target. After each pass it moves to the next dimension and rescales again. The loop continues until the fitted tensor $R$ matches every target to within a convergence tolerance.

Once $R$ has converged, each calibration cell $k$ receives a cell multiplier:

$$a_k = \frac{R_k}{S_k} \cdot \frac{\sum_k S_k}{\sum_k R_k}$$

The right-hand factor normalizes the weights so that their sum equals the eligible respondent count, which keeps weighted bases on the same scale as raw bases. Each respondent inherits the multiplier of their calibration cell as their weight.

Two-stage composition

With both stages configured:

1. Pre-weighting runs first on its configured universe and targets. The seed tensor is built from observed respondent counts and IPF produces a first set of respondent weights.
2. Main weighting runs second on its own configured universe and targets. Critically, its seed tensor is built from the pre-weighting weights, not from raw counts. IPF then produces a second cell multiplier on top of those weights.

Because the second stage's seed is the first stage's output, stages compose multiplicatively:

$$w_i^{\text{final}} = w_i^{\text{pre}} \times w_i^{\text{main}}$$

This supports sequential correction (for example source/frame correction first, reporting calibration second) without collapsing both objectives into a single target system.

Eligibility guardrails

Before any weighting runs, the platform checks every configured stage. A stage is eligible only if:

1. every question referenced by the targets is present in the data,
2. the number of respondents complete across the weighting questions exceeds the minimum base for weighting, and
3. every category with a positive target has at least one eligible respondent in that stage's selected universe (including joint categories for nested targets).

Condition (3) is the most commonly hit. It prevents IPF from trying to put mass into a category that is empty in the eligible sample, which would otherwise produce either non-convergence or implausibly large weights.

If any configured stage fails eligibility, no stage is applied and respondent weights remain at 1.0. This all-or-nothing behavior prevents a partial calibration from introducing targets that were never intended to be hit in isolation. The weighting diagnostics will show which check failed so you can fix the underlying issue — typically by collapsing a sparse category, revising the target, or increasing the sample.

Effective sample size

For any set of weights $\{w_i\}_{i \in S}$, the Kish effective sample size is:

$$n_{\text{eff}} = \frac{\left(\sum_{i \in S} w_i\right)^2}{\sum_{i \in S} w_i^2}$$

Two properties to keep in mind:

• If every respondent has the same weight, $n_{\text{eff}} = |S|$. Unequal weights always reduce $n_{\text{eff}}$ below the raw count.
• $n_{\text{eff}}$ is computed separately for each reporting cell, so it can show differences in weight concentration between subgroups.

At the dataset level, we report the weighting efficiency:

$$\text{Efficiency} = \frac{n_{\text{eff}}}{n}$$

An efficiency near 1 indicates little dispersion in the weights. Low efficiency, or a long right tail in the weight histogram, is a signal to check whether the targets are straining the sample. Kish size and efficiency summarize the weights; uncertainty for a particular outcome also depends on the responses and their relationship to those weights.

Variance approximation

Analytic percentages and numeric means are ratios of weighted totals. For cell $c$, let $a_{ic}$ be respondent $i$'s contribution to the weighted numerator and $b_{ic}$ their contribution to the weighted denominator:

$$\hat{\theta}_c = \frac{\sum_i a_{ic}}{\sum_i b_{ic}}, \qquad g_{ic} = \frac{a_{ic} - \hat{\theta}_c b_{ic}}{\sum_i b_{ic}}$$

Repeated response records are summed within respondent before calculating these contributions. The denominator follows the selected percentage base, including question respondents, respondents offered an option, all respondents, or respondents answering a referenced question.

Let $N$ be the number of distinct respondents in the full filtered report population. Respondents outside a cell's domain contribute zero; demographic groups and topic contexts are domains rather than sampling strata. Taylor variance and standard error are:

$$\widehat{\mathrm{Var}}(\hat{\theta}_c) = \frac{N}{N-1}\sum_i (g_{ic}-\bar{g}_c)^2, \qquad \widehat{\mathrm{SE}}(\hat{\theta}_c) = \bigl[\widehat{\mathrm{Var}}(\hat{\theta}_c)\bigr]^{1/2}$$

For a binary percentage within its respondent base, $a_{ic}=w_iY_i$ and $b_{ic}=w_i$. For a numeric mean, $a_{ic}=w_iX_i$ and $b_{ic}=w_i$, excluding non-finite and non-numeric answers from both totals. Means without outliers apply the configured retention rule and treat the retained set as fixed for inference.

A zero denominator or fewer than two respondents in the filtered population has unavailable inference. Constant observed outcomes can have zero empirical variance. A displayed zero error should not be interpreted as certainty about unobserved outcomes.

See the SAS SURVEYMEANS ratio variance reference for Taylor linearisation of weighted ratios.

Significance testing

Analytic cells retain joint respondent-level covariance. For two cells $A$ and $B$:

$$\widehat{\mathrm{Cov}}(A,B) = \frac{N}{N-1}\sum_i(g_{iA}-\bar{g}_A)(g_{iB}-\bar{g}_B)$$ $$\widehat{\mathrm{Var}}(A-B) = \widehat{\mathrm{Var}}(A)+\widehat{\mathrm{Var}}(B)-2\widehat{\mathrm{Cov}}(A,B)$$

• Column t-tests compare columns within the same row using the variance of their difference.
• Row t-tests compare response rows within the same column using the same contrast calculation.
• Residual t-tests (the default in cross-tabs) compare each cell with an estimated table benchmark and include covariance with that benchmark.

These tests use two-sided t tails with $N-1$ degrees of freedom. Joint covariance accounts for shared respondents, mutually exclusive answers, overlapping groups, and topic contexts. Kish sizes remain descriptive bases; they do not set the test degrees of freedom. No automatic multiple-comparison adjustment is applied.

Other outputs use different inference procedures:

• Median and Selected count median use respondent-level bootstrap comparisons that preserve reporting weights, rather than the analytic standard-error calculation above. See Understanding stat testing.
• Matched-group Lift reports use one two-sided conditional score test for average lift across matching runs, preserving respondent reuse. The test conditions on fixed matches and outcome totals within matching strata, with an approximate normal tail. It uses respondent outcome counts rather than weighted cross-tab errors. See Lift measurement methodology.
• Conjoint and MaxDiff posterior-draw summaries combine average within-draw Taylor covariance with between-draw posterior covariance. The resulting joint matrix supplies both marginal errors and comparisons. See Utility and simulated share methodology.

A small worked example

Suppose a 600-respondent genpop dataset is weighted to national age and gender targets. After weighting, the weights for one reporting cell — "women in the Northeast" — are distributed as follows:

Respondents	Weight	Positive outcomes	Negative outcomes
40	0.7	16	24
60	1.3	24	36

For this cell:

$$\sum w_i = 40 \times 0.7 + 60 \times 1.3 = 28 + 78 = 106$$ $$\sum w_i^2 = 40 \times 0.49 + 60 \times 1.69 = 19.6 + 101.4 = 121.0$$ $$n_{\text{eff}} = \frac{106^2}{121.0} = \frac{11{,}236}{121.0} \approx 92.9$$

The cell has 100 raw respondents and a Kish size of approximately 92.9. Its weighted proportion is:

$$\hat{p} = \frac{16\times0.7+24\times1.3}{106} = 0.40$$

With zero contributions from the other 500 respondents in the filtered population, the Taylor standard error is:

$$\begin{aligned} Q &= 0.7^2\left[16(1-0.4)^2+24(0-0.4)^2\right] \\ &\quad + 1.3^2\left[24(1-0.4)^2+36(0-0.4)^2\right] \\ &= 29.04 \end{aligned}$$ $$\widehat{\mathrm{SE}} = \left[\frac{600}{599}\frac{29.04}{106^2}\right]^{1/2} \approx 0.0509$$

This is about 5.1 percentage points of standard error. The calculation uses both the weights and the observed outcomes; a different distribution of positive outcomes across the weight groups can change the error even when Kish size is unchanged.

Diagnostics to monitor

Per dataset, weighting diagnostics report:

• raw respondent count $n$,
• effective sample size $n_{\text{eff}}$,
• efficiency $n_{\text{eff}}/n$,
• min, median, mean, and max weight,
• a histogram of the weight distribution,
• stage eligibility status and failure reason.

Points to watch for:

1. Efficiency well below 1 — weights are concentrated on fewer respondents. Check whether the targets are achievable from the realized sample and inspect uncertainty for the outcomes being reported.
2. A long right tail on the weight histogram — a small number of respondents are carrying a lot of weight. These cells will have the largest effect on point estimates and the smallest effective base.
3. A stage reported as ineligible — read the failure reason in the diagnostics. Usually this points to an empty target category in the eligible sample.

Assumptions and limitations

1. Taylor inference assumes independent respondents in one sampling stratum, with fixed weights and domain definitions. Sampling clusters, explicit sampling strata, and finite-population corrections are not modeled.
2. The quality of point estimates depends on the validity of the supplied marginal targets and on overlap between the sample and the targets.
3. Extreme weights reduce weighting efficiency and can give individual respondents substantial influence on point estimates and uncertainty. Their relationship to the outcome matters.
4. The calculation does not include uncertainty from estimating calibration weights or selecting an outlier-retention rule.
5. In analytic row and column comparisons, zero contrast variance gives p-value one for a zero difference and the limiting Wald p-value zero for a nonzero difference. Analytic residual tests return score zero when residual variance is zero and do not mark significance. These empirical outcomes do not establish certainty about unobserved outcomes.

Reproducibility

Weighting is deterministic given the input dataset, the marginal targets, the stage configuration, the respondent universes selected for each stage, and the IPF convergence tolerance. Re-running the pipeline on the same inputs produces the same weights and the same reported statistics.