Documentation

Quota management methodology

This page is the methodology companion to the Quotas function reference and the Setting up quotas how-to guide. The reference documents the quota() and set_quota() calls; the how-to walks through configuring a quota group. This page explains the industry concepts behind quota management: how MX8 Labs decides which respondents to accept, how it steers incoming respondents toward under-filled cells, and how those choices interact with weighting.

Scope

A quota is a rule that controls the composition of your achieved sample by limiting how many respondents fall into each defined segment. Two decisions drive everything else:

  • The fielding strategy: how the survey decides when it is finished and whether an individual respondent is accepted. This is the survey-level Fielding Strategy setting.
  • The quota design: how you carve the sample into cells, whether a single interlocked grid, independent marginal targets, or a mix of both.

MX8 Labs enforces quotas live: every respondent is checked against the current fill counts as they enter, and quotas update in real time. This page covers how that enforcement behaves; for how the achieved sample is calibrated back to target after fielding, see Weighting methodology.

Fielding strategies

The Fielding Strategy setting determines two things: when the survey closes, and whether a given respondent is accepted when the cell they qualify for is already full. MX8 Labs supports three strategies. In code these map to the quota_strategy value the survey is created with (strict, quota, or respondent).

Strict

Both the quotas and the total respondent target are strictly enforced. The survey closes only when every quota line has reached its target and the total number of completes has been reached. A respondent is terminated if any cell they qualify for is already full, even if the overall target has not yet been met.

Use strict when the joint distribution of your sample matters and you would rather field longer than accept an over-quota respondent. A tracker that must match census cells exactly wave over wave is the typical case.

The risk is stalling: the easy cells fill early and close, and toward the end of fielding you are left recruiting only the hardest cells (the classic "high-income Hispanic 18–24" problem). See Empty and hard-to-fill cells below.

Quota driven

The survey closes when all quotas are met or exceeded; the total number of respondents may exceed the target. A respondent is accepted as long as they qualify for at least one cell that still has room. When this strategy is active, MX8 Labs replaces the fixed target-respondents line with an estimated respondents line that projects how many more completes are needed to fill the remaining cells.

Use quota driven when hitting every cell is more important than the exact total, for instance when you need a guaranteed minimum base in each segment and are willing to overshoot the headline N to get there.

Respondent driven

The survey closes when the target number of respondents is reached, even if some quotas are not fully met. Cells can overfill: a respondent is accepted as long as the overall target has not been reached and they qualify for any cell.

Use respondent driven when the total N and speed to field matter most, and you are relying on weighting to correct the residual imbalance across cells. This is the most forgiving strategy and the least likely to stall.

Choosing a strategy

A useful way to think about the three: strict protects the shape of the sample at the cost of time; respondent driven protects the timeline and total N at the cost of shape; quota driven protects the per-cell bases at the cost of the headline total. It is common to start strict and switch to quota driven or respondent driven late in fielding once the easy cells have closed. See Adjusting quotas during fielding in the how-to guide.

Interlocked vs. marginal quotas

The same demographic targets can be enforced in two structurally different ways, and the choice has large consequences for feasibility.

Interlocked (cell) quotas define targets on the full crossing of your dimensions. Age (3 bands) × gender (2) × region (4) produces 24 interlocked cells, each with its own target that sums to 100% across the group. Every respondent occupies exactly one cell. In MX8 Labs this is the default: set_quota() is exclusive, so each respondent is assigned to a single cell, the least-filled one they qualify for (see Click balancing).

Marginal (non-interlocked) quotas define independent targets on each dimension separately: an age target, a gender target, and a region target that each sum to 100% on their own. A respondent counts toward one line in each margin at once. In MX8 Labs you express this as separate set_quota() groups, one group per dimension, each summing to 100%. (A related but distinct option, exclusive=False, lets a respondent count toward several overlapping lines within a single group, which suits overlapping categories such as device ownership rather than independent dimensions, since the lines in one group must still sum to 100%.)

The trade-off is control vs. feasibility:

  • Interlocked quotas control the joint distribution: you guarantee the right number of, say, women aged 18–34 in the West. But cells multiply quickly (3 × 2 × 4 = 24), each cell base shrinks, and rare cells become hard to fill.
  • Marginal quotas control only the margins: you guarantee the right totals for age, gender, and region independently, but not their interaction. They are far easier to fill (3 + 2 + 4 = 9 targets), which is why they are the industry default for large or fast studies where the joint distribution is then recovered through weighting.

A common pattern is to interlock the two or three dimensions where the joint distribution genuinely matters and leave the rest marginal.

Click balancing

"Click balancing" is the practice of steering each incoming respondent toward the cell that is currently furthest from its target, so that fielding fills evenly rather than front-loading whichever cells happen to arrive first. MX8 Labs does this automatically.

When a respondent qualifies for more than one open cell, MX8 Labs computes each qualifying cell's proportion filled: its current count divided by its target, where the target is the cell's min_respondents if set and otherwise its quota proportion times the total. It then assigns the respondent to the least-filled cell. Ties among equally-filled cells are broken by a deterministic shuffle seeded on the quota, so the outcome is stable but not biased toward list order.

Two consequences follow from how this interacts with the fielding strategy:

  • A cell that has reached its target drops out of the candidate pool, so respondents are steered to what remains open. Under respondent driven, a cell that has reached an explicit max_respondents cap is likewise skipped.
  • If a respondent qualifies for no open cell, they are terminated. This is the mechanism behind the "closes when full" behavior of strict and quota-driven fielding.

Because assignment is greedy at the per-respondent level, end-of-fielding cell bases are approximately balanced, not exactly balanced: residual imbalance can persist with small N or high dropout, which is again where weighting comes in. Click balancing for quota cells is the enforcement analogue of the exposure-balancing described in Least-fill balancing methodology; the difference is that quotas terminate respondents, whereas least-fill only changes what each respondent is shown.

Counts increment only on completion

MX8 Labs counts a respondent toward a cell only when they complete the survey. Respondents who are in progress but have not yet finished do not count toward the fill, and are not partially counted. This keeps the quota report a true picture of achieved completes, and it means the targets you set are always in units of completes, not starts.

The practical consequence is at the tail of a fast field: because in-progress respondents are not reserved against a cell, a cell can briefly accept more starts than it ultimately needs, and some of those will complete after the cell is technically full. Set targets on the completes you actually need, and rely on the fielding strategy, rather than in-flight reservation, to govern how tightly the final counts land.

Empty and hard-to-fill cells

Some cells recruit slowly or barely at all: low-incidence segments, expensive demographics, or narrow interlocked cells. How this plays out depends on the strategy:

  • Under strict, a hard cell stays open and blocks completion, because the survey is not finished until every cell is full. Meanwhile the easy cells have closed, so incoming easy respondents are terminated. Fielding stalls near the target with only the hardest cells left open.
  • Under quota driven, the estimated-respondents line grows to reflect how many more completes the remaining cells are projected to need; the survey keeps accepting anyone who fits an open cell.
  • Under respondent driven, hard cells simply finish under target and the survey closes on the total N.

Several levers help:

  • Boost with min_respondents. A cell whose proportional target yields too small a base for analysis (5% of 500 is only 25 respondents) can be given a floor. The floor takes precedence over the proportional target when it is higher. A boost line that sets min_respondents without a quota value raises the base without being weighted, which is useful when you need enough respondents to analyze a group but do not want to distort the sample's proportions.
  • Cap easy cells with max_respondents. Preventing fast cells from over-recruiting reserves remaining completes for the cells that need them.
  • Switch strategy late in fielding. Moving from strict to quota driven (accept anyone who fits a still-open cell) or to respondent driven (close on total N and weight the rest) is the standard response to a stalled strict field. Make the change with enough runway: there is a short lag between respondents completing and the dashboard closing the study, so a very late switch may land after the survey has already closed.

Feasibility and incidence

Before a quota design can fill, the respondents it asks for have to exist in the sample source at a reachable rate. Two numbers govern this: the incidence rate (the share of the source that qualifies) and the resulting feasibility (how many completes the source can realistically deliver in the time available).

MX8 Labs estimates the incidence rate automatically from the survey's screening and quota logic, and surfaces a feasibility estimate for the design. Treat these as a starting point, not a guarantee: an automated estimate cannot see everything a sample partner knows about a particular audience, so always have a person sanity-check the estimate against what you know about the population and the source before committing to field. Very low-incidence designs deserve particular care. See Working with low-incidence studies.

Feasibility and quota design are coupled: every dimension you interlock lowers the incidence of the rarest cells, so a design that is easily feasible as marginal groups can become infeasible once interlocked. Check feasibility after you have decided what to interlock, not before.

The limits of quota sampling

Quota sampling is a non-probability method. Within any cell, respondents are whoever the source supplies, not a random draw from that segment of the population, so quotas control the composition of the sample but cannot by themselves remove selection bias within a cell. This is worth stating plainly when you report results, and it is the reason quotas and weighting are complements rather than substitutes.

The practical failure mode is the impossible cell. Interlock enough dimensions and you eventually ask for a respondent who barely exists: a 24-year-old Asian Hispanic woman earning more than $150k a year in Kansas. Under strict fielding, one such cell can hold an entire study open indefinitely while every other cell sits finished and every otherwise-usable respondent is turned away. This is exactly why MX8 Labs does not default to strict quotas: the default protects you from a single infeasible cell stalling the whole field.

Three habits keep you out of this trap:

  • Interlock only what must be interlocked. Each added dimension multiplies the number of cells and shrinks the rarest one. Keep the joint constraints to the handful that genuinely matter and leave the rest marginal.
  • Size the rarest cell, not the average cell. Before fielding, work out the expected base of your smallest interlocked cell. If it is too small to fill or to analyze, loosen the interlock or add a min_respondents floor and accept that it will be weighted down.
  • Choose a strategy that degrades gracefully. If some cells may be infeasible, a driven strategy plus weighting will finish; strict may not. Reserve strict for designs whose every cell you are confident is fillable.

Weighting as the backstop

No live quota system produces a perfectly balanced achieved sample: click balancing is approximate, cells miss their targets, and boosted cells are deliberately over-represented. Post-field weighting is what closes the gap.

MX8 Labs treats the quota proportions as marginal targets and calibrates respondent weights to them using iterative proportional fitting (raking), so the weighted sample reflects the proportions you specified even when the raw counts do not. The mechanics (two-stage raking, effective sample size, and how precision degrades as weights become more heterogeneous) are documented in Weighting methodology, with a plain-language version in the Weighted results FAQ.

Two cautions:

  • Weighting is a correction, not a rescue. The further the achieved sample is from target, the larger and more heterogeneous the weights, and the more effective sample size (and therefore precision) you lose. Weighting a badly-missed quota back to target buys representativeness at the cost of statistical power. Quotas exist so that weighting has little work to do.
  • Boost-only lines are not weighted. A line with min_respondents and no quota value raises the base but is excluded from the weighting targets, so the boosted group remains over-represented in the raw data unless you weight it separately. That is usually the intent: you wanted the base for sub-analysis, not a change to the headline proportions.

Scale and performance

There is no fixed limit on the number of quota lines or quota groups a survey can carry. The design space is effectively unlimited, so you can enforce as many dimensions as your study genuinely needs. The consideration is performance rather than a hard cap: every incoming respondent is evaluated against every quota group, so a large number of separate groups adds proportionally more work per respondent than the same targets expressed as fewer groups.

In practice this only matters at the extremes. For the great majority of studies the number of groups is small and the cost is negligible; if you find yourself defining a very large number of distinct groups, it is usually a sign that some of them could be consolidated, and consolidating also makes the live quota report easier to read.

Related