Skip to contents

Weighted and survey data

Source: vignettes/tabxplor-weights.Rmd
tabxplor-weights.Rmd
library(tabxplor)

# Pin the legend language: it defaults to "auto" = the ambient locale, so building this English
# vignette on a French machine silently renders French legends and captions (the -fr articles pin
# "fr" for the same reason). Output must not depend on where it is built.
options(tabxplor.lang = "en")
Sys.setenv(LANGUAGE = "en")   # the test-summary / model-fit row labels go through gettext, not this option
library(dplyr)

# Tables render as tabxplor's real html tables (the recommended everyday setting); the shared
# stylesheet is emitted once by tab_css() below, and the hover tooltips are kept off here.
options(tabxplor.print = "html")
options(tabxplor.tab_kable_css = FALSE)
options(tabxplor.tab_kable_tooltips = FALSE)

options(cli.num_colors = 256)
set_color_palette(theme = "light")

Une version française de ce document est disponible : Données pondérées et plans de sondage.

Most survey files come with a weight: a number saying how many people in the population each respondent stands for. The argument wt = uses it, and every percentage and mean in the table becomes an estimate of the population rather than of the people you happened to interview.

That is the easy half. The harder half is the margin of error around those percentages — the confidence intervals, the significance stars, the colours and the tests. There, tabxplor has three levels, and a table’s footer always tells you which one it is on.

you pass level what the intervals and tests carry
wt = w 1 the weighted estimate, on the raw n — no design effect at all
wt = w + design_effect = TRUE 2 the unequal weighting, exactly — blind to clustering and calibration
a survey design as data 3 the full design: strata, clusters, fpc, calibration

This vignette is about choosing between them. vignette("tabxplor") covers cross-tables themselves, and vignette("tabxplor-reg") the regression framework.

Level 1 — weighted percentages, plain margins of error (the default) 

tab(my_survey, education, job, wt = weight, pct = "row")
#> Weighted by weight; confidence intervals and tests use the unweighted sample size.

The percentages are weighted; the margins of error simply count your real respondents. This is the convention almost every teaching text and every point-and-click program uses, and it is why it is the default here. But when the weights are unequal it is optimistic: a sample where some people count for four others carries less information than its size suggests, and the intervals come out usually a bit too narrow.

Level 2 — design_effect = TRUE 

tab(my_survey, education, job, wt = weight, pct = "row", design_effect = TRUE)
options(tabxplor.design_effect = TRUE)   # or once, for the whole session
#> Weighted by weight; confidence intervals and tests account for the weighting.

Now the unequal weighting is taken into account in every interval, star, colour threshold and whole-table test. This is not a rule of thumb: a column of weights is a survey design — the simplest one, with no clusters and no strata — and tabxplor computes that design’s variance exactly, reproducing the reference survey package to the last digit.

Here it is on real data. We give gss_simple a made-up weight (never-married people are under-covered in most surveys, so they count for more), and build the same table twice:

gss_w <- dplyr::mutate(gss_simple, w = ifelse(marital %in% "Never married", 2.5, 0.8))

tab(gss_w, race, party3, wt = w, pct = "row", ci = "cell", na = "drop")
party3
race 1-Democrat 2-Independent,
other 		3-Republican Total
<row%-ci> <row% (n)>
White [40;42]% [21;22]% [37;38]% 100% (16 301)
Black [72;75]% [17;19]% [7;9]% 100% ( 3 093)
Other [48;53]% [30;35]% [15;19]% 100% ( 1 934)
Total [47;48]% [22;23]% [30;31]% 100% (21 328)
Weighted by w; confidence intervals and tests use the unweighted sample size. 
tab(gss_w, race, party3, wt = w, pct = "row", ci = "cell", na = "drop",
    design_effect = TRUE)
party3
race 1-Democrat 2-Independent,
other 		3-Republican Total
<row%-ci> <row% (n)>
White [40;42]% [21;23]% [37;38]% 100% (16 301)
Black [72;76]% [16;20]% [7;10]% 100% ( 3 093)
Other [48;53]% [30;35]% [15;19]% 100% ( 1 934)
Total [47;48]% [22;23]% [29;31]% 100% (21 328)
Weighted by w; confidence intervals and tests account for the weighting.

Same percentages, wider brackets, and a footer that says what changed. Behind them, the 16 292 White respondents are now worth about 11 800 — the price of the unequal weights.

If you weight at all, level 2 is the honest reading of your own numbers, and it costs nothing. It is not the default only because switching it on moves every figure in every table anyone has already produced, so it should be a deliberate, visible act rather than a silent one. One more reason to turn it on: a regression table’s observed columns are always at level 2, so this is what makes a tab() percentage and a tab_reg() observed percentage on the same data directly comparable.

Level 3 — a survey design

Real surveys are not drawn by picking names at random from the whole country. They are usually stratified (drawn separately by region, city size…), often clustered (a few neighbourhoods are picked first, then several households inside each), and their weights are usually calibrated so the sample matches known population totals for age, sex, region. Those three facts are not in the weight column; they live in extra variables. If your file has them, build a design with the survey package and pass it as data:

library(survey)
d <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w, data = my_survey, nest = TRUE)
tab(d, race, marital, pct = "row", color = TRUE, test = TRUE)
#> Design-based (survey): weighted estimates, intervals and tests account for the sample design.

Everything then follows the design: the estimates, the p-values, and every confidence interval, star and colour threshold. Replicate-weight designs (svrepdesign()) and two-phase designs are not supported — tabxplor refuses them rather than approximating.

What level 2 can and cannot do for you

Level 2 sees the weights, and nothing else. That matters, because the three things it cannot see do not pull in the same direction:

seen at level 2 needs a design (level 3)
unequal weights yes, exactly
strata, calibration, finite population would narrow your intervals — by a few percent at most
clusters can make them much wider

So the two corrections you are missing at level 2 are wildly unequal. Ignoring the strata and the calibration costs a few percent, and in the safe direction — your intervals are slightly too cautious. Ignoring clusters goes the other way and can be enormous: on a real clustered school survey, an average that level 2 reports with a margin of error of ±3 points really deserves ±9. On the American NHANES health survey, the distribution of ethnicity comes out nine times too precise, because that survey is built around exactly that variable. These do not cancel out: when clustering is present, it wins.

Which surveys are clustered? Not the ones drawn from a register, or run by web, telephone or mail — there level 2 is simply the right answer, however unequal the weights. But face-to-face household surveys are clustered by construction, because sending interviewers to scattered addresses is unaffordable. The French Enquête Emploi, for instance, is drawn as groups of about twenty neighbouring dwellings; neighbours resemble each other, so twenty interviews in one street tell you less than twenty interviews spread across the country.

Does it matter for your table? Usually much less than those numbers suggest, and there is a simple test. The cluster effect largely cancels out in a comparison, as long as both compared groups are found inside the same neighbourhoods. So ask one question about your row variable:

Does it vary within a neighbourhood, or does it define one?

Sex, age, diploma, occupation, income, opinions — these vary within any neighbourhood, so a comparison between their categories is close to right at level 2. Region, urban/rural, city size, type of neighbourhood, local immigrant density — these are the neighbourhood, and there level 2 is as wrong as it is for a single percentage. The measurements back this up: on the same clustered file where an average is three times too precise, the difference between two groups is only 13 % too precise, and a regression coefficient 8 % — and differences are exactly what tabxplor’s colours and stars test.

Check what your file actually contains before reaching for level 3. Look for a stratum, cluster or calibration variable (strate, grappe, psu, nomen…). Many released research files ship one calibrated weight and nothing else — the cluster is a small geographic area, so it is withheld for confidentiality — and on such a file a survey design would carry only the unequal weighting, which is exactly what level 2 already gives you. The European Social Survey is a useful middle case: it does publish its clusters and strata, but in a separate file you have to download and merge.

Regression on weighted data

There are only two ways to hand tabxplor your weights, and they are the same two: either you give wt = a weight column, or you build the design once and pass the design itself as data.

tab_reg(data, "outcome", c("pred1", "pred2"), wt = "weight")

library(survey)
d <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w, data = my_survey, nest = TRUE)
tab_reg(d, "outcome", c("pred1", "pred2"), empirical = TRUE)

Everything then follows that design, in one regime: the Model_* coefficients and their intervals (survey::svyglm()), the average marginal effects, the per-standard-deviation scaling of numeric predictors, the model-versus-observed gap test — and the observed Obs_* companion columns, whose intervals are built on the design variance of each cell. That last point is what makes the comparison work: the model column and the observed column beside it are measured the same way, so a difference between them is about adjustment, not about two different notions of uncertainty.

Two honest limits. A comparison between two cells (an observed odds ratio, a risk difference) ignores the design covariance between them, so it lands a few percent either side of the exact answer — against the 15–25 % it was out by when those columns were computed as if the sample were simple random. And a nominal outcome has no observed column: its crude value is folded into the model cell as a point, with no interval to make design-based.

Why a weighted tab_reg() is not a weighted tab()

tab_reg() cannot choose: its observed columns have to be measured like the model column beside them, and that one is design-based by construction (survey::svyglm()). tab() can, and keeps the descriptive convention — a weighted estimate on the raw sample size — as its default. So tab_reg() is always at level 2 (or level 3 under a design), while tab() starts at level 1, and tab(design_effect = TRUE) is what brings the two into line. Each table’s footer says which it used.

The fine print

  • Sometimes level 2 makes an interval narrower. That is correct, not a bug. If the weights happen to line up with what you are measuring, unequal selection can carry more information than equal selection, and the effective sample size comes out above your number of respondents.
  • Degrees of freedom. A clustered survey is usually built on few clusters — 15 to 60 is common — and level 3 refers each interval to that number, which widens it again by roughly 8 %. Level 2 has no clusters to count, so it cannot do this: a second, separate reason a face-to-face survey’s true margin of error is wider than what level 2 prints.
  • When level 3 cannot be computed. If a design’s variance fails for a table, it falls back to the level-2 correction and says so in its footer, rather than claiming a design its numbers do not carry.
  • Exact for a cell, cautious for a difference. Level 3 is exact for a single percentage; for a cell-vs-reference difference it cannot carry the covariance between the two cells, so the difference comes out slightly too wide rather than too narrow. It will never hand you a star the design does not support.
  • Pre-aggregated counts and cost. Level 2 needs the individual weights, so a table built with tab_counts() from published counts cannot use it — such a table says so in its footer. And a level-3 table costs roughly three times a weighted one.

Where to go next