Skip to content

Statistics · Tests and models

Principal Component Analysis and Dimension Reduction

How principal component analysis condenses many correlated survey questions into a few summary dimensions, how to decide how many to keep, and where the method helps or misleads.

ModQu research article · 1,596 words · 17 references (APA 7th edition)

Download PDF

Customer surveys often ask a dozen or more rating questions that overlap: people who rate a store's prices highly also tend to rate its promotions and value for money highly. Analysing each question separately produces long, repetitive tables and unstable models. Principal component analysis (PCA) replaces a set of correlated variables with a smaller set of uncorrelated summary variables, called components, that keep as much of the original information as possible. The idea goes back to Pearson (1901), who sought the lines and planes that best fit a cloud of points, and Hotelling (1933), who developed it as a method for analysing many related test scores.

How PCA works

Each principal component is a weighted sum of the original variables. The first component is the weighted sum with the largest possible variance; the second has the largest variance among weighted sums uncorrelated with the first; and so on, until there are as many components as variables:

PC1 = a11z1 + a12z2 + … + a1pzp

Here z1 to zp are the original variables, usually standardised, and the weights a come from the eigenvectors of their correlation (or covariance) matrix. The variance of each component is the corresponding eigenvalue. Because standardised variables each have variance 1, the eigenvalues of a correlation matrix add up to the number of variables, so an eigenvalue divided by that number gives the share of total variance a component explains (Jolliffe, 2002).

Geometrically, PCA rotates the axes of the data so that the first axis points along the direction of greatest spread. Keeping only the first few components gives the best possible low-dimensional approximation of the data in the least-squares sense, a result formalised by Eckart and Young (1936). Jolliffe and Cadima (2016) and Abdi and Williams (2010) give accessible modern reviews.

Correlation or covariance?

PCA on the covariance matrix lets variables with larger variances dominate. If items are measured in different units, such as spending in dollars and satisfaction on a 0 to 10 scale, the analysis must use the correlation matrix, which is the same as standardising every variable first. When all items share one rating scale, either choice is defensible, but the correlation matrix is the usual default in survey work (Jolliffe, 2002).

A worked example

Suppose a hypothetical grocery chain asks 600 shoppers to rate ten aspects of its stores. The numbers below come from simulated, illustrative data. PCA on the correlation matrix gives the eigenvalues in Figure 1.

Scree plot of eigenvalues for ten hypothetical store-rating items. The first three components have eigenvalues 2.71, 2.17 and 1.75; the remaining seven fall between 0.65 and 0.32. A horizontal line at 1 marks the Kaiser rule, and a parallel analysis line from random data falls from 1.26 to 0.85. Only the first three eigenvalues exceed both lines. Together the three components explain 66 percent of the total variance.
Figure 1. Scree plot for ten store-rating items. Three components stand clearly above the eigenvalues expected from random data and together explain 66% of the variance. Illustrative data; chart by ModQu.

Three components have eigenvalues well above the rest and together explain 66% of the total variance; the fourth adds only 6.5 points. To make the components easier to interpret, the three are rotated with the varimax method (Kaiser, 1958), which redistributes the explained variance so that each item loads strongly on as few components as possible. The loadings, the correlations between items and components, are:

Item Value Products Service and convenience
Low prices 0.86 −0.02 −0.02
Frequent promotions 0.83 −0.05 0.03
Good value for money 0.86 0.00 −0.03
Fresh produce −0.04 0.86 0.10
Product quality −0.05 0.86 0.14
Wide range 0.03 0.82 −0.02
Friendly staff 0.03 0.15 0.75
Short checkout queues −0.04 0.03 0.75
Easy parking 0.01 0.03 0.72
Clean, tidy store −0.04 0.15 0.74

Ten ratings can now be summarised by three scores (value, products, and service and convenience) that can be tracked over time, compared across stores, or used as predictors of loyalty without the instability caused by highly correlated inputs. The names are the analyst's interpretation of the loading pattern; PCA itself supplies only numbers. The final column of a full output would also show each item's communality, the share of its variance captured by the retained components. Here communalities range from about 0.52 for easy parking to 0.76 for fresh produce and product quality, so every item is reasonably well represented.

Using component scores

Once the components are chosen, each respondent receives a score on each one, calculated from their standardised answers and the component weights. These scores have a mean of zero, so a store whose shoppers average +0.4 on the value component is rated better on value than the typical store in the sample. Scores can be fed into later analyses, such as a regression of loyalty on the three components, or compared between customer groups and survey waves.

Two practical cautions apply. First, exact component scores use every item, including those with small loadings, so many analysts prefer simple summated scales: the average of the items that load strongly on each component. These are easier to explain and more stable across samples, although they are no longer perfectly uncorrelated. Second, if the weights are re-estimated in every wave of a tracking study, the meaning of a score can drift. Fixing the weights from a reference wave keeps the scores comparable over time.

How many components to keep

Choosing the number of components is the most important decision in PCA. Several rules are in common use.

In an influential simulation, Zwick and Velicer (1986) found that parallel analysis was among the most accurate methods, while the Kaiser rule often overestimated the number of components. Parallel analysis is a sound default, ideally checked against the scree plot and the interpretability of the resulting components.

PCA is not factor analysis

PCA is often confused with factor analysis, and many software packages present them in the same menu. PCA is a data reduction technique: it summarises the total variance of the observed variables, including measurement error. Factor analysis is a measurement model: it assumes that a small number of unobserved factors cause the correlations among items, and it models only the shared variance. Velicer and Jackson (1990) argue that the two often give similar results in practice, especially with many variables and strong loadings. However, Fabrigar et al. (1999) recommend factor analysis when the goal is to identify latent constructs, such as attitudes or brand perceptions, and to build measurement scales. PCA is the better choice when the goal is simply to summarise or compress data.

Practical uses in business and survey data

Limitations and common mistakes

Key takeaways

PCA turns many correlated variables into a few uncorrelated components that keep most of the information. It is a fast, transparent way to summarise rating batteries, reduce collinearity and build indices. Use the correlation matrix when units differ, decide how many components to keep with parallel analysis rather than the eigenvalue-above-1 rule alone, rotate when interpretation matters, and remember that PCA summarises data rather than measuring latent constructs. Check outliers, report the variance explained, and treat component names as interpretations, not facts.

References

Abdi, H., & Williams, L. J. (2010). Principal component analysis. WIREs Computational Statistics, 2(4), 433–459. https://doi.org/10.1002/wics.101

Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245–276. https://doi.org/10.1207/s15327906mbr0102_10

Dolnicar, S., & Grün, B. (2008). Challenging "factor–cluster segmentation". Journal of Travel Research, 47(1), 63–71. https://doi.org/10.1177/0047287508318910

Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211–218. https://doi.org/10.1007/BF02288367

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. https://doi.org/10.1037/1082-989X.4.3.272

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. https://doi.org/10.1007/BF02289447

Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), 417–441. https://doi.org/10.1037/h0071325

Hubert, M., Rousseeuw, P. J., & Vanden Branden, K. (2005). ROBPCA: A new approach to robust principal component analysis. Technometrics, 47(1), 64–79. https://doi.org/10.1198/004017004000000563

Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). Springer. https://doi.org/10.1007/b98835

Jolliffe, I. T., & Cadima, J. (2016). Principal component analysis: A review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), 20150202. https://doi.org/10.1098/rsta.2015.0202

Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23(3), 187–200. https://doi.org/10.1007/BF02289233

Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1), 141–151. https://doi.org/10.1177/001316446002000116

Organisation for Economic Co-operation and Development, European Union, & Joint Research Centre. (2008). Handbook on constructing composite indicators: Methodology and user guide. OECD Publishing. https://doi.org/10.1787/9789264043466-en

Pearson, K. (1901). LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559–572. https://doi.org/10.1080/14786440109462720

Velicer, W. F., & Jackson, D. N. (1990). Component analysis versus common factor analysis: Some issues in selecting an appropriate procedure. Multivariate Behavioral Research, 25(1), 1–28. https://doi.org/10.1207/s15327906mbr2501_1

Zou, H., Hastie, T., & Tibshirani, R. (2006). Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15(2), 265–286. https://doi.org/10.1198/106186006X113430

Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442. https://doi.org/10.1037/0033-2909.99.3.432