Skip to content

Statistics · Bayesian and computational

Multilevel Regression and Post-Stratification (MRP)

How MRP turns a national or unrepresentative survey into reliable estimates for provinces, regions and small segments, with a worked example, practical guidance and the mistakes to avoid.

ModQu research article · 1,830 words · 19 references (APA 7th edition)

Download PDF

Clients often want answers for groups that a survey was never designed to measure. A national study of 1,500 adults may be asked to report purchase intent in each province, in each age band within each region, or among owners of a particular car brand. Splitting the sample leaves some cells with a handful of respondents, and the resulting percentages jump from wave to wave for no real reason. Many online samples also over-represent young, educated or highly engaged people.

Multilevel regression and post-stratification (MRP, sometimes nicknamed "Mister P") tackles both problems at once. It fits a regression model that predicts the outcome for every combination of demographic and geographic characteristics, then reweights those predictions to the known make-up of the population. The method was introduced for estimating opinion in American states from national polls (Gelman & Little, 1997; Park et al., 2004) and is now used in political science, public health and commercial research.

The problem MRP solves

There are two traditional ways to estimate a quantity for a small area. Disaggregation simply takes the respondents who live there and computes their average. It is unbiased only if the subsample is representative, and it is very noisy when the subsample is small. Weighting adjusts the sample so that its totals match known population totals, for example by raking to age, sex and region (Deville & Särndal, 1992; Valliant et al., 2018). Weighting corrects some imbalance, but it can only use a few variables before the weights become extreme, and it does nothing to stabilise estimates for small areas.

MRP borrows ideas from both small area estimation and model-based survey inference. Instead of treating each area separately, it lets areas and demographic groups share information through a model, and it uses population data to say how many people of each type live in each area (Rao & Molina, 2015; Little, 1993).

The two steps

Step 1: multilevel regression

The analyst divides the population into poststratification cells, defined by every combination of the variables in the model, such as province × age band × sex × education. A multilevel model, usually a logistic regression for a yes/no outcome, then predicts the outcome in each cell. For respondent i:

logit Pr(yi = 1) = β0 + αagea[i] + αsexs[i] + αedue[i] + αprovp[i] + γ zp[i]

Each set of α terms is a random effect, given a normal distribution with its own standard deviation that the data estimate. A province with few respondents has its effect pulled towards zero, so its prediction leans on the national pattern and on the province-level predictor z. Province-level predictors, such as past market share, average income or category penetration, matter a great deal: they tell the model why provinces differ, and they let it make good predictions for provinces with little data (Gelman & Hill, 2007; Buttice & Highton, 2013).

Step 2: post-stratification

The model gives a predicted probability θ̂c for every cell c, including cells with no respondents at all. These predictions are then averaged with the population count Nc of each cell as the weight. For province p:

θ̂p = Σc ∈ p Nc θ̂c ÷ Σc ∈ p Nc

The cell counts come from a census, a large official survey or a high-quality register. Because the final estimate depends on the population mix rather than the sample mix, an over-representation of young graduates in the sample no longer distorts the result, provided the model captures how the outcome varies with age and education.

A worked example

To show how the method behaves when the truth is known, we built a hypothetical population in which the share of adults intending to switch their home internet provider in the next year varies by province, age, sex and education. Provinces were given their approximate relative sizes. We then drew an online-style sample of 1,500 people in which young adults and degree holders were over-represented: 54% of the sample was aged 18 to 34, against 27% of the population, and 57% held a degree, against about 39%.

In the simulated population, 36.8% of adults intended to switch. The raw sample said 48.7%, because the over-represented groups were more likely to switch. The province estimates were worse. Prince Edward Island had only 7 respondents, and New Brunswick 34; their raw percentages were 43% and 56%, against true values of 29% and 28%.

We fitted the model above, with a province-level predictor for category penetration, and post-stratified to the population cells. The MRP national estimate was 37.9%. Across the ten provinces, the average absolute error fell from 12.6 percentage points for the raw subsample percentages to 2.4 points for MRP.

Dot chart of intention to switch internet provider in ten provinces, ordered by sample size from Ontario with 596 respondents to Prince Edward Island with 7. For each province a dark tick shows the true population value, an open orange circle the raw sample percentage and a filled green dot the MRP estimate. Raw percentages are too high in nine provinces, by as much as 28 points in New Brunswick (56% against a true 28%), while MRP estimates lie within 5 points of the truth in nine provinces and 7 points in New Brunswick. The average absolute error is 12.6 points for raw percentages and 2.4 points for MRP.
Figure 1. Raw subsample percentages and MRP estimates compared with the true values in a simulated population. MRP corrects the demographic skew of the sample and stabilises the provinces with few respondents. Illustrative data; chart by ModQu.

The result has two lessons. First, MRP was not perfect. It put New Brunswick at 35.0% against a true 27.8%, and Nova Scotia at 25.6% against 30.4%. With few respondents, these estimates lean heavily on the model and on the province-level predictor, and whatever makes a province distinctive beyond those inputs is partly lost. Models make assumptions, and when those assumptions miss, the estimates inherit the error. Second, most of the improvement in the large provinces came from post-stratification, which corrected the age and education skew, while most of the improvement in the small provinces came from partial pooling, which stopped seven interviews from being treated as a reliable measurement.

Why MRP works, and when it does not

The best-known demonstration is the study by Wang et al. (2015), who collected opinion polls through an online gaming platform whose users were overwhelmingly young and male. Raw results were badly wrong, but after MRP with detailed demographic cells and previous vote, the forecasts were close to those of established polling averages. Lax and Phillips (2009) found that MRP estimates of state opinion from a single national survey compared well with disaggregation of many pooled surveys. In public health, Zhang et al. (2014) used MRP with a national health survey to estimate disease prevalence for counties, and Downes et al. (2018) argue that it can reduce selection bias in highly selected study samples.

These successes depend on conditions that are worth stating plainly:

MRP and weighting compared

Weighting and MRP rest on the same idea: the sample must be made to look like the population on the variables that matter. Weighting does this by changing how much each respondent counts, MRP by modelling the outcome and projecting it onto the population. Gelman (2007) argues that regression modelling and weighting are best seen as two routes to the same goal, and that the modelling route handles many adjustment variables and small domains more gracefully.

Weighting (raking or calibration) MRP
Adjustment variables A few, before weights become extreme Many, including interactions
Small areas Noisy, often unusable Stabilised by partial pooling
Estimates for several outcomes One set of weights for all A model for each outcome
Transparency Easy to explain and audit Requires model checking and documentation
Population data needed Marginal totals Joint cell counts

In practice the two are complementary. Many organisations publish weighted national figures and use MRP for regional or segment breakdowns. Model-based approaches have also been central to the debate about polling errors: an evaluation of the 2016 United States election polls found that failure to adjust for education was one reason many state polls were wrong (Kennedy et al., 2018), a problem that cell-based adjustment such as MRP is designed to address when education is in the model.

Practical guidance for survey and business research

Common mistakes

Key takeaways

MRP combines a multilevel regression model with population counts to produce estimates for areas and segments that a survey alone cannot support. It corrects demographic imbalance through post-stratification and stabilises small groups through partial pooling. It works best when the model includes the variables that drive both selection and the outcome, when strong area-level predictors are available and when the population frame is accurate. Used with honest uncertainty and external validation, it turns national and even unrepresentative samples into credible regional evidence.

References

Buttice, M. K., & Highton, B. (2013). How does multilevel regression and poststratification perform with conventional national surveys? Political Analysis, 21(4), 449–467. https://doi.org/10.1093/pan/mpt017

Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1), 1–32. https://doi.org/10.18637/jss.v076.i01

Deville, J.-C., & Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87(418), 376–382. https://doi.org/10.1080/01621459.1992.10475217

Downes, M., Gurrin, L. C., English, D. R., Pirkis, J., Currier, D., Spittal, M. J., & Carlin, J. B. (2018). Multilevel regression and poststratification: A modeling approach to estimating population quantities from highly selected survey samples. American Journal of Epidemiology, 187(8), 1780–1790. https://doi.org/10.1093/aje/kwy070

Elliott, M. R., & Valliant, R. (2017). Inference for nonprobability samples. Statistical Science, 32(2), 249–264. https://doi.org/10.1214/16-STS598

Gao, Y., Kennedy, L., Simpson, D., & Gelman, A. (2021). Improving multilevel regression and poststratification with structured priors. Bayesian Analysis, 16(3), 719–744. https://doi.org/10.1214/20-BA1223

Gelman, A. (2007). Struggles with survey weighting and regression modeling. Statistical Science, 22(2), 153–164. https://doi.org/10.1214/088342306000000691

Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. https://doi.org/10.1017/CBO9780511790942

Gelman, A., & Little, T. C. (1997). Poststratification into many categories using hierarchical logistic regression. Survey Methodology, 23(2), 127–135.

Ghitza, Y., & Gelman, A. (2013). Deep interactions with MRP: Election turnout and voting patterns among small electoral subgroups. American Journal of Political Science, 57(3), 762–776. https://doi.org/10.1111/ajps.12004

Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., Saad, L., Witt, G. E., & Wlezien, C. (2018). An evaluation of the 2016 election polls in the United States. Public Opinion Quarterly, 82(1), 1–33. https://doi.org/10.1093/poq/nfx047

Lax, J. R., & Phillips, J. H. (2009). How should we estimate public opinion in the states? American Journal of Political Science, 53(1), 107–121. https://doi.org/10.1111/j.1540-5907.2008.00360.x

Little, R. J. A. (1993). Post-stratification: A modeler's perspective. Journal of the American Statistical Association, 88(423), 1001–1012. https://doi.org/10.1080/01621459.1993.10476368

Lohr, S. L. (2021). Sampling: Design and analysis (3rd ed.). Chapman and Hall/CRC. https://doi.org/10.1201/9780429298899

Park, D. K., Gelman, A., & Bafumi, J. (2004). Bayesian multilevel estimation with poststratification: State-level estimates from national polls. Political Analysis, 12(4), 375–385. https://doi.org/10.1093/pan/mph024

Rao, J. N. K., & Molina, I. (2015). Small area estimation (2nd ed.). Wiley. https://doi.org/10.1002/9781118735855

Valliant, R., Dever, J. A., & Kreuter, F. (2018). Practical tools for designing and weighting survey samples (2nd ed.). Springer. https://doi.org/10.1007/978-3-319-93632-1

Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980–991. https://doi.org/10.1016/j.ijforecast.2014.06.001

Zhang, X., Holt, J. B., Lu, H., Wheaton, A. G., Ford, E. S., Greenlund, K. J., & Croft, J. B. (2014). Multilevel regression and poststratification for small-area estimation of population health outcomes: A case study of chronic obstructive pulmonary disease prevalence using the Behavioral Risk Factor Surveillance System. American Journal of Epidemiology, 179(8), 1025–1033. https://doi.org/10.1093/aje/kwu018