7.1       In a survey on parks and recreation, 40% of respondents are under 55 years of age, and 60% are 55 and over. The relevant population is known to have 80% under 55 and 20% over. The unweighted survey results indicate that 26% of area residents want more children’s playgrounds in city parks: 50% for respondents under 55 years of age and 10% for respondents 55 and over. If these results are weighted to correct for age, what percentage of area residents want more children’s playgrounds? Would you weight these data for age?

If 70% of the survey respondents are under 55 years of age, and 30% are 55 and over, and the percentage of respondents in each group who want more children’s playgrounds remains as 50% and 10%, respectively, what would be the unweighted percentage of all survey respondents who want more children’s playgrounds? What would be the weighted percentage? Would you weight these data for age?

In the first example given, 40% of respondents are under 55 years of age and 60% are 55+, while the relevant population is known to have 80% under 55 and 20% over. The percentage of respondents who want more children’s playgrounds in city parks is 50% for respondents under 55 and 10% for respondents 55+. Therefore, the overall unweighted sample percentage who want more children’s playgrounds is 26% [(50% * .4) + (10% * .6)], and the weighted result is 42% [(50% * .8) + (10% * .2)]. We saw these same numbers in Exercise 3.2.

Whether we actually apply this weighting depends first on whether the disproportionate representation for the two different groups (under 55 and 55+) stems from the sample design (i.e., we deliberately oversampled the 55+ group) or from differential non-response. We routinely will weight to correct for design-based differences in representation.

If the disproportionate representation stems from differential non-response, our guideline is to weight only when (a) differences in non-response are fairly dramatic across groups, (b) the weighting produces substantial and credible changes in important variables, and (c) there are no obvious adverse effects of the weighting. In this case, the differences in non-response are indeed dramatic; the under 55 group is represented at half its proper rate and the 55+ group is overrepresented by a factor of three. The weighting produces a substantial change in our estimate of the percentage of the population who want more children’s playgrounds, from 26% to 42%, and the change is credible in that children’s playgrounds seem likely to be a lower priority for respondents beyond their childrearing years. Therefore, we are likely to apply the weighting for this estimate.

While weighting should give us a better estimate than not weighting, we still may have lingering concerns about possible non-response bias. The composition of the sample compared with the population implies that the response rate in the under 55 group was only 1/6^th the response rate in the 55+ group. This is indicated by the 6 to 1 ratio in weights; as a further example, imagine that we drew a sample of 1,000 potential respondents with 800 under 55 and 200 55+ (the 80/20 population split), and we obtained 200 respondents with 80 under 55 and 120 55+ (the 40/60 sample split), in which case the response rates would be 10% for the under 55 group and 60% for 55+. The difference is large enough to raise concerns not just that the under 55 group is underrepresented relative to the 55+ group, but also that respondents in the under 55 group may differ from non-respondents.

In the second example given, 70% of respondents are under 55 years of age and 30% are 55+, while the relevant population is known to have 80% under 55 and 20% over. The percentage of respondents who want more children’s playgrounds in city parks is 50% for respondents under 55 and 10% for respondents 55+. Therefore, the overall unweighted sample percentage who want more children’s playgrounds is 38% [(50% * .7) + (10% * .3)], and the weighted result is 42% [(50% * .8) + (10% * .2)].

Again, if the disproportionate representation for the two different groups stems from the sample design, then we routinely will weight to correct for such differences, and if the disproportionate representation stems from differential non-response, we will weight only if (a) differences in non-response are fairly dramatic across groups, (b) the weighting produces substantial and credible changes in important variables, and (c) there are no obvious adverse effects of the weighting.

In this case, the key issue may be whether we consider a change from 38% to 42% to be substantial. This depends on how the data will be used. If the focus is on specific point estimates, the change from 38% to 42% may be meaningful. Certainly, a 4% change may be meaningful in a political preference survey – especially if the change is from 48% to 52%! – which partly explains why weighting is common in such surveys. However, if the focus is on broad categories (park improvements that almost everyone wants, or that many people want, or that few people want), or on relative priorities (how does the percentage of people who want more children’s playgrounds compare with the percentage who want other improvements), then the change from 38% to 42% may not be seen as important, in which case we probably would leave the results unweighted.

If we do weight these results, we are less likely to have the lingering concerns we might have in the first example. This raises an interesting paradox: the situations where weighting for non-response has the most effect, and seems most urgent, are often the situations where we have least confidence in the ability of weighting to address non-response bias completely.