Skip to content

Statistics · Tests and models

Statistical Modeling of Complex Systems: Abstracting Knowledge from Noise

How statisticians model markets, economies and other complex systems, separating stable structure from noise without overfitting, and what the limits of such models mean for business decisions.

ModQu research article · 1,640 words · 16 references (APA 7th edition)

Download PDF

Markets, supply chains, economies, ecosystems and online platforms share a family resemblance. They consist of many interacting parts, they respond to their own past behaviour, and small changes sometimes have large effects. Such complex systems generate data that look noisy, unstable and hard to predict. Statistical modelling cannot remove that complexity, but it can abstract from it: it can identify the parts of a pattern that are stable enough to be called knowledge, and separate them from the variation that is not.

What makes a system complex

Anderson (1972) argued in "More is different" that each level of organisation in nature has properties that cannot simply be deduced from the level below. A market is more than a collection of buyers; its prices, fashions and crashes emerge from the interactions. Several features follow from this view and matter for statistical work:

Signal, noise and abstraction

Every statistical model splits observations into two parts: a structured component that the model explains and a residual that it treats as noise. The word "noise" is a modelling choice, not a fact about the world. It covers measurement error, sampling variation and the influence of everything the model leaves out. In a complex system the last of these is large.

Box (1976) warned that because all models are approximations, the scientist should seek an economical description and worry about the errors that matter, rather than add detail for its own sake. The danger on one side is underfitting: a model too simple to capture real structure. The danger on the other side is overfitting: a model so flexible that it reproduces the noise in the data it was fitted to and fails on new data.

Two-panel chart of simulated data. Panel A shows 24 noisy points generated from a smooth wave-shaped true process, with a straight-line fit that misses the wave, a degree-4 polynomial that follows the true process closely, and a degree-12 polynomial that chases individual points and swings sharply at both ends. Panel B shows average error over 300 simulated samples: error on the fitting data falls steadily from 1.25 at degree 1 to 0.46 at degree 12, below the true noise level of 0.7, while error on new data falls to a minimum of 0.79 at degree 4 and then rises to 1.43 at degree 12.
Figure 1. Underfitting and overfitting. Error on the data used for fitting keeps falling as complexity rises, but error on new data is lowest for a moderately simple model. Illustrative data; chart by ModQu.

Figure 1 shows the trade-off with simulated data, where the true process is known. A straight line misses the wave in the data, with an average error on new observations of 1.29. A degree-4 polynomial captures the wave and achieves the lowest error on new data, 0.79, close to the irreducible noise level of 0.7. A degree-12 polynomial fits the original sample better still, with an error of only 0.46, but that apparent precision is false: on new data its error rises to 1.43, worse than the straight line. The model has learned the noise.

Statisticians guard against this in three main ways. Model selection criteria such as the Akaike information criterion reward fit but charge a penalty for each additional parameter (Akaike, 1974). Regularisation methods such as the lasso shrink coefficients towards zero and set many of them exactly to zero, which keeps models sparse when there are many candidate predictors (Tibshirani, 1996). Cross-validation estimates performance on data not used for fitting, the quantity shown in panel B of Figure 1.

Two cultures of modelling

Breiman (2001) distinguished two cultures in statistical modelling. The data-modelling culture assumes a stochastic model that might have generated the data and focuses on estimating its parameters. The algorithmic culture treats the mechanism as unknown and judges methods by how well they predict. Complex systems test both. Mechanistic models may be badly misspecified, while purely predictive models can fail when the system changes.

The story of Google Flu Trends illustrates the second risk. The service estimated influenza activity from search queries and was at first highly accurate. Lazer et al. (2014) showed that it later overestimated influenza substantially, at one point predicting more than double the proportion of doctor visits for influenza-like illness reported by the United States Centers for Disease Control and Prevention. They attributed the failure partly to "big data hubris" and partly to changes in the search engine itself, which altered the relationship between searches and illness. A complex system had shifted beneath a model that only tracked correlations. Businesses face the same risk whenever a demand model trained before a price change, a new competitor or a disruption to supply is used after it.

Modelling strategies for complex systems

State-space models and filtering

Many systems have a hidden state, such as true demand, brand health or economic momentum, that is observed only through noisy measurements. State-space models describe how the hidden state evolves and how observations relate to it. The Kalman (1960) filter updates the estimate of the state each time a new observation arrives, weighting it by how noisy the measurements are relative to the system. Variants of this approach underlie modern forecasting, tracking of survey indicators over time and the smoothing of noisy monthly series.

Inference by simulation

For many complex models the likelihood, the probability of the observed data given the parameters, cannot be written down or computed. Simulation offers a way around this. In approximate Bayesian computation, parameter values are drawn, the model is simulated, and values are kept when the simulated data resemble the observed data on chosen summary statistics (Beaumont et al., 2002). Wood (2010) showed that for chaotic ecological models, in which a direct likelihood behaves erratically, a synthetic likelihood based on summary statistics of simulated data allows reliable inference. The same ideas apply to agent-based models of consumers or markets.

Calibrating computer models

Large simulation models, from climate to supply-chain digital twins, must be tuned to data. Kennedy and O'Hagan (2001) set out a Bayesian framework for this calibration that explicitly includes a term for model discrepancy, the systematic gap between the simulator and reality. Treating that gap as unknown, rather than zero, prevents the calibration from forcing parameters into implausible values to compensate for missing mechanisms.

Learning structure and causation from data

Newer methods try to recover the structure of a system directly. Brunton et al. (2016) used sparse regression to identify governing equations of dynamic systems from time-series data, keeping only the few terms needed to describe the dynamics. In ecology, Sugihara et al. (2012) argued that in nonlinear coupled systems correlations can appear, vanish and even reverse over time, so correlation is an unreliable guide to causation, and they proposed convergent cross mapping as an alternative test based on how well one variable's history can reconstruct another's.

Checking models against reality

A model of a complex system should be treated as a hypothesis, not a description. Gelman and Shalizi (2013) argue that good statistical practice consists of building models, deriving their implications and checking those implications against data, revising or rejecting models that fail. Predictive checks, in which data simulated from the fitted model are compared with the observed data, are a practical way to find where a model misrepresents the system, for example by producing tails that are too thin or swings that are too regular.

Practical guidance for business and survey data

Common mistakes

Key takeaways

Statistical modelling of complex systems is the disciplined search for stable structure in noisy, interacting and changing data. Good models are deliberately simplified, chosen for their performance on new data rather than their fit to the past, and checked against reality. Tools such as regularisation, state-space filtering, simulation-based inference and calibrated simulators extend what can be learned, but none removes the need for humility: in complex systems, the most valuable knowledge often includes a clear statement of what cannot be predicted.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705

Anderson, P. W. (1972). More is different. Science, 177(4047), 393–396. https://doi.org/10.1126/science.177.4047.393

Beaumont, M. A., Zhang, W., & Balding, D. J. (2002). Approximate Bayesian computation in population genetics. Genetics, 162(4), 2025–2035. https://doi.org/10.1093/genetics/162.4.2025

Box, G. E. P. (1976). Science and statistics. Journal of the American Statistical Association, 71(356), 791–799. https://doi.org/10.1080/01621459.1976.10480949

Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199–231. https://doi.org/10.1214/ss/1009213726

Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016). Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15), 3932–3937. https://doi.org/10.1073/pnas.1517384113

Cont, R. (2001). Empirical properties of asset returns: Stylized facts and statistical issues. Quantitative Finance, 1(2), 223–236. https://doi.org/10.1080/713665670

Gelman, A., & Shalizi, C. R. (2013). Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology, 66(1), 8–38. https://doi.org/10.1111/j.2044-8317.2011.02037.x

Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35–45. https://doi.org/10.1115/1.3662552

Kennedy, M. C., & O'Hagan, A. (2001). Bayesian calibration of computer models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 63(3), 425–464. https://doi.org/10.1111/1467-9868.00294

Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The parable of Google Flu: Traps in big data analysis. Science, 343(6176), 1203–1205. https://doi.org/10.1126/science.1248506

Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 20(2), 130–141. https://doi.org/10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2

Scheffer, M., Bascompte, J., Brock, W. A., Brovkin, V., Carpenter, S. R., Dakos, V., Held, H., van Nes, E. H., Rietkerk, M., & Sugihara, G. (2009). Early-warning signals for critical transitions. Nature, 461(7260), 53–59. https://doi.org/10.1038/nature08227

Sugihara, G., May, R., Ye, H., Hsieh, C.-h., Deyle, E., Fogarty, M., & Munch, S. (2012). Detecting causality in complex ecosystems. Science, 338(6106), 496–500. https://doi.org/10.1126/science.1227079

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

Wood, S. N. (2010). Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310), 1102–1104. https://doi.org/10.1038/nature09319