Weighting methodology

This page is the technical companion to the Weighted results FAQ. The FAQ explains what weighting does and when it runs; this page documents the estimator, stage composition, respondent-universe selection, and inference adjustments. It is written for analysts, data-science teams, and methodologists reviewing the approach.

Scope

MX8 Labs uses iterative proportional fitting (IPF, also called raking) to calibrate respondent weights to marginal targets supplied through quota configuration. Point estimates are computed from the final calibrated weights. Standard errors and significance tests use an effective-sample-size adjustment so that precision claims degrade gracefully as weights become more heterogeneous.

Weighting can be run in one stage or two stages:

• Pre-weighting stage (optional)
• Main weighting stage

Each stage can use its own configured respondent universe.

This is a calibration-weighting framework with Kish effective-sample-size adjustment. It is not a full design-based variance estimator: it does not model primary sampling units, explicit strata, or finite-population corrections. In practice, for the kinds of online respondent sources MX8 supports, this framework is well matched to the data and the questions people ask of it, but the scope is worth naming explicitly.

Notation

Let respondents be indexed by $i = 1, \dots, n$ with final calibrated weights $w_i \ge 0$.

For a reporting cell $c$ (for example, "women aged 25-34 in the West region"):

• $S_c$ is the set of respondents in cell $c$.
• $N_c = |S_c|$ is the raw base for that cell.
• $W_c = \sum_{i \in S_c} w_i$ is the weighted base.

For a binary outcome $Y_i \in \{0, 1\}$, the weighted proportion is:

$$\hat{p}_c = \frac{\sum_{i \in S_c} w_i Y_i}{\sum_{i \in S_c} w_i}$$

Weighted means for numeric or rating outcomes are computed analogously.

Target marginals

Weighting targets are supplied as category proportions for one or more questions defined on the respondent source (age bands, gender, region, and so on). Targets may be:

• One-way, e.g. the marginal distribution of age alone.
• Nested (joint), e.g. a target distribution over the joint of age and gender. Joint targets are calibrated directly against the joint cell, not against the two marginals separately.

Only categories with a positive target are treated as required for the eligibility checks described below. A category with a zero target is permitted to be absent from the eligible data.

Respondent-universe selection

Each weighting stage defines its own eligible respondent universe. This is a methodological control, not just an operational filter — calibration answers "representative of which population?", and universe selection defines that population.

Common configurations include:

• Completes-only universes for analysis-sample inference.
• Broader universes (for example including selected terminates) when calibrating to entrant-level representativeness.
• Distinct universes across stages when pre-weighting is used to correct source composition before final reporting calibration.

IPF within a stage

Given a seed tensor $S$ built from the stage's eligible respondents and a set of target margins $\{m_j\}$ on the weighting dimensions, IPF repeatedly rescales $S$ along one dimension at a time so that its marginal on that dimension matches the corresponding target. After each pass it moves to the next dimension and rescales again. The loop continues until the fitted tensor $R$ matches every target to within a convergence tolerance.

Once $R$ has converged, each calibration cell $k$ receives a cell multiplier:

$$a_k = \frac{R_k}{S_k} \cdot \frac{\sum_k S_k}{\sum_k R_k}$$

The right-hand factor normalizes the weights so that their sum equals the eligible respondent count, which keeps weighted bases on the same scale as raw bases. Each respondent inherits the multiplier of their calibration cell as their weight.

Two-stage composition

With both stages configured:

1. Pre-weighting runs first on its configured universe and targets. The seed tensor is built from observed respondent counts and IPF produces a first set of respondent weights.
2. Main weighting runs second on its own configured universe and targets. Critically, its seed tensor is built from the pre-weighting weights, not from raw counts. IPF then produces a second cell multiplier on top of those weights.

Because the second stage's seed is the first stage's output, stages compose multiplicatively:

$$w_i^{\text{final}} = w_i^{\text{pre}} \times w_i^{\text{main}}$$

This supports sequential correction (for example source/frame correction first, reporting calibration second) without collapsing both objectives into a single target system.

Eligibility guardrails

Before any weighting runs, the platform checks every configured stage. A stage is eligible only if:

1. every question referenced by the targets is present in the data,
2. the number of respondents complete across the weighting questions exceeds the minimum base for weighting, and
3. every category with a positive target has at least one eligible respondent in that stage's selected universe (including joint categories for nested targets).

Condition (3) is the most commonly hit. It prevents IPF from trying to put mass into a category that is empty in the eligible sample, which would otherwise produce either non-convergence or implausibly large weights.

If any configured stage fails eligibility, no stage is applied and respondent weights remain at 1.0. This all-or-nothing behavior prevents a partial calibration from introducing targets that were never intended to be hit in isolation. The weighting diagnostics will show which check failed so you can fix the underlying issue — typically by collapsing a sparse category, revising the target, or increasing the sample.

Effective sample size

For any set of weights $\{w_i\}_{i \in S}$, the Kish effective sample size is:

$$n_{\text{eff}} = \frac{\left(\sum_{i \in S} w_i\right)^2}{\sum_{i \in S} w_i^2}$$

Two properties to keep in mind:

• If every respondent has the same weight, $n_{\text{eff}} = |S|$. Unequal weights always reduce $n_{\text{eff}}$ below the raw count.
• $n_{\text{eff}}$ is computed separately for each reporting cell, so a subgroup with stable weights can retain most of its precision even if the dataset overall has heavy weighting.

At the dataset level, we report the weighting efficiency:

$$\text{Efficiency} = \frac{n_{\text{eff}}}{n}$$

An efficiency near 1 means weighting has cost very little precision. Low efficiency, or a long right tail in the weight histogram, is a signal that the targets are straining the sample — often a cue to collapse sparse categories or revisit the quota.

Variance approximation

Standard errors for weighted proportions use a binomial-style approximation with $n_{\text{eff}}$ substituted for the raw base:

$$\widehat{\text{SE}}(\hat{p}_c) = \sqrt{\frac{\hat{p}_c (1 - \hat{p}_c)}{n_{\text{eff},c}}}$$

This is the standard calibration-weighting shortcut: it captures the first-order precision cost of unequal weights without requiring a full Taylor-linearization pass for every estimand. Mean-aggregated outputs use the same pattern after scale-normalizing the estimate to a proportion.

Significance testing

All significance tests in MX8 reports consume $n_{\text{eff}}$ rather than raw counts:

• Column t-tests compare each cell to other cells in its column using the reported weighted mean and standard error, with sample size set to $n_{\text{eff}}$.
• Row t-tests are the same, applied across rows.
• Residual t-tests (the default in cross-tabs) use a critical $t$ with degrees of freedom derived from $n_{\text{eff}} - 1$, so the threshold for flagging a cell as significant tightens when the effective base is small.

The practical consequence is that heavily weighted data produces fewer significant cells than raw counts alone would suggest. This is intentional and avoids overclaiming precision from inflated weighted totals.

A small worked example

Suppose a 600-respondent genpop dataset is weighted to national age and gender targets. After weighting, the weights for one reporting cell — "women in the Northeast" — are distributed as follows:

• 40 respondents with $w_i \approx 0.7$
• 60 respondents with $w_i \approx 1.3$

For this cell:

$$\sum w_i = 40 \times 0.7 + 60 \times 1.3 = 28 + 78 = 106$$ $$\sum w_i^2 = 40 \times 0.49 + 60 \times 1.69 = 19.6 + 101.4 = 121.0$$ $$n_{\text{eff}} = \frac{106^2}{121.0} = \frac{11{,}236}{121.0} \approx 92.9$$

So the 100 raw respondents in this cell carry the information of roughly 93 equally-weighted respondents. If the weighted proportion on a binary outcome is $\hat{p} = 0.40$, the standard error is:

$$\widehat{\text{SE}} = \sqrt{\frac{0.40 \times 0.60}{92.9}} \approx 0.051$$

Compared to $\sqrt{0.24 / 100} \approx 0.049$ if the cell were unweighted — a modest widening that reflects the mild weight dispersion.

Diagnostics to monitor

Per dataset, weighting diagnostics report:

• raw respondent count $n$,
• effective sample size $n_{\text{eff}}$,
• efficiency $n_{\text{eff}}/n$,
• min, median, mean, and max weight,
• a histogram of the weight distribution,
• stage eligibility status and failure reason.

Points to watch for:

1. Efficiency well below 1 — weights are doing a lot of work. Inference is valid but precision is reduced; consider whether the targets are achievable from the realized sample.
2. A long right tail on the weight histogram — a small number of respondents are carrying a lot of weight. These cells will have the largest effect on point estimates and the smallest effective base.
3. A stage reported as ineligible — read the failure reason in the diagnostics. Usually this points to an empty target category in the eligible sample.

Assumptions and limitations

1. Inference assumes independent respondents within calibration cells; there is no explicit modeling of clustering, stratification, or finite-population corrections.
2. The quality of point estimates depends on the validity of the supplied marginal targets and on overlap between the sample and the targets.
3. Extreme weights reduce efficiency and widen standard errors. The eligibility guardrails and the diagnostics are the operational controls for this.
4. The variance approximation is a binomial-style shortcut with $n_{\text{eff}}$ substitution, not a full design-based estimator.

Reproducibility

Weighting is deterministic given the input dataset, the marginal targets, the stage configuration, the respondent universes selected for each stage, and the IPF convergence tolerance. Re-running the pipeline on the same inputs produces the same weights and the same reported statistics.