Skip to content

Statistics · Causal inference

Causal Discovery Algorithms from Observational Data

How algorithms such as PC, GES, FCI and LiNGAM try to learn cause-and-effect structure from observational data, what assumptions they need, and how to use their output responsibly.

ModQu research article · 1,637 words · 16 references (APA 7th edition)

Download PDF

Most causal analysis starts with a diagram that the analyst draws from subject knowledge: price affects satisfaction, satisfaction affects renewal, and so on. The diagram then tells the analyst which variables to adjust for. Causal discovery turns the question around. Instead of assuming the diagram, it asks how much of the causal structure can be learned from the data themselves. A family of algorithms developed since the early 1990s answers that question under clearly stated assumptions (Spirtes et al., 2000; Pearl, 2009).

For business and survey analysts, causal discovery is attractive because many data sets contain dozens of attitudes and behaviours with no agreed theory linking them. Used well, the algorithms narrow the set of plausible causal stories. Used carelessly, they produce confident-looking arrows that the data cannot support.

From correlations to graphs

Causal discovery represents causal structure as a directed acyclic graph (DAG): each variable is a node, and an arrow from A to B means that A is a direct cause of B relative to the variables measured. Two assumptions connect such a graph to data (Spirtes et al., 2000).

Together these assumptions mean that patterns of conditional independence in the data reveal features of the graph. The key pattern is the collider, a variable with two causes that are not otherwise connected. If price fairness and service quality both affect satisfaction, but are unrelated to each other, they become associated once we condition on satisfaction. That signature differs from a chain (A → B → C) or a fork (A ← B → C), where conditioning on the middle variable removes the association. Colliders therefore allow some arrows to be oriented without any experiment (Greenland et al., 1999).

Markov equivalence: what data can and cannot tell

Different graphs can imply exactly the same conditional independencies. The chain A → B → C, the reverse chain A ← B ← C and the fork A ← B → C are indistinguishable from observational data alone. Graphs that share the same independencies form a Markov equivalence class. Most algorithms therefore return not one DAG but a summary of the whole class, often called a completed partially directed acyclic graph (CPDAG). Arrows appear where every graph in the class agrees, and undirected edges where the direction cannot be decided (Chickering, 2002).

This is the most important point for anyone reading discovery output. An undirected edge is not a failure of the software. It is an honest statement that the data are compatible with either direction.

The main families of algorithms

Constraint-based methods

The PC algorithm, named after its authors Peter Spirtes and Clark Glymour, is the best-known constraint-based method (Spirtes & Glymour, 1991). It starts from a complete graph, in which every pair of variables is connected, and removes the edge between two variables whenever a statistical test finds them independent given some set of other variables. It begins with unconditional tests and moves to larger conditioning sets, using only the neighbours that remain. It then orients colliders and applies a small set of rules that propagate those orientations without creating new colliders or cycles.

PC is fast for sparse graphs, and Kalisch and Bühlmann (2007) showed that it can consistently recover the equivalence class even when the number of variables is large relative to the sample, provided the true graph is sparse. Its results depend on the choice of independence test and significance level, and errors in early tests can spread to later steps.

Score-based methods

Score-based methods search over graphs for the one that best balances fit and complexity, using a criterion such as the Bayesian information criterion or a Bayesian score that can include prior knowledge (Heckerman et al., 1995). Greedy equivalence search (GES) searches directly over equivalence classes. It adds edges one at a time while the score improves, then removes edges while the score improves, and in large samples it identifies the correct equivalence class (Chickering, 2002).

Methods that allow hidden variables

PC and GES assume causal sufficiency: no unmeasured variable causes two or more measured ones. In survey data this assumption is rarely credible, because unmeasured traits such as personality or income often drive many answers at once. The Fast Causal Inference (FCI) algorithm relaxes it. FCI returns a partial ancestral graph whose edge marks distinguish "definitely a cause", "definitely not a cause" and "cannot tell", allowing for hidden confounders and selection bias (Spirtes et al., 2000). Zhang (2008) completed its orientation rules, so FCI extracts all the orientation information the independencies contain. The price is that FCI output usually contains fewer firm conclusions.

Methods that use the shape of the distribution

A different idea is to exploit features beyond independence. The LiNGAM model assumes linear relationships and non-Gaussian errors. Under those assumptions the full DAG, not just its equivalence class, can be identified, because regressing in the wrong causal direction leaves residuals that are not independent of the predictor (Shimizu et al., 2006). Related methods use non-linear additive noise models or the stability of relationships across settings (Peters et al., 2017). They can orient edges that PC leaves undirected, but they depend heavily on their distributional assumptions.

A worked example

Consider a hypothetical customer survey of 2,000 respondents with five measures: service quality rating, perceived price fairness, overall satisfaction, renewal intention and number of complaints in the past year. Suppose the true process, unknown to the analyst, is that service quality and price fairness both affect satisfaction, satisfaction affects renewal, and service quality affects complaints.

Three-panel diagram of the PC algorithm on five hypothetical survey variables. Panel 1 shows all ten pairs of variables connected. Panel 2 shows the skeleton left after conditional independence tests: service to satisfaction, price to satisfaction, satisfaction to renewal and service to complaints. Panel 3 shows the output: arrows from service and from price into satisfaction, an arrow from satisfaction to renewal, and a dashed undirected edge between service and complaints.
Figure 1. The PC algorithm on five hypothetical survey variables. Independence tests remove six of the ten possible edges; the collider at satisfaction and one propagation rule orient three edges, while the service–complaints edge stays undirected. Illustrative data; chart by ModQu.

PC proceeds as follows (Figure 1):

  1. Unconditional tests. Service and price are unrelated, so that edge is removed. Price and complaints are also unrelated.
  2. Conditional tests. Service and renewal are associated, but independent given satisfaction, so that edge goes. The same applies to price and renewal, to satisfaction and complaints given service, and to renewal and complaints. Four edges remain.
  3. Colliders. Service and price are not adjacent, and the set that separated them did not include satisfaction. Satisfaction is therefore a collider: service → satisfaction ← price.
  4. Propagation. Because service points into satisfaction and service is not adjacent to renewal, the edge satisfaction–renewal must point towards renewal. Any other direction would create a collider the data did not show.

The edge between service and complaints stays undirected. Poor service could generate complaints, or complaints could colour later ratings of service, and these data cannot separate the two. That finding tells the research team that a design feature, such as the timing of measurement, is needed.

Practical guidance for survey and business data

Limitations and common mistakes

Key takeaways

Causal discovery algorithms learn what they can about causal structure from patterns of conditional independence and, in some methods, from the shape of the data distribution. Constraint-based methods such as PC, score-based methods such as GES, latent-variable methods such as FCI and non-Gaussian methods such as LiNGAM each rest on explicit assumptions (Glymour et al., 2019). Their honest output is usually a class of graphs rather than a single answer. In survey and business research they are most valuable for generating and prioritising causal hypotheses, especially when combined with background knowledge, stability checks and follow-up experiments.

References

Chickering, D. M. (2002). Optimal structure identification with greedy search. Journal of Machine Learning Research, 3, 507–554.

Glymour, C., Zhang, K., & Spirtes, P. (2019). Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10, Article 524. https://doi.org/10.3389/fgene.2019.00524

Greenland, S., Pearl, J., & Robins, J. M. (1999). Causal diagrams for epidemiologic research. Epidemiology, 10(1), 37–48. https://doi.org/10.1097/00001648-199901000-00008

Heckerman, D., Geiger, D., & Chickering, D. M. (1995). Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20(3), 197–243. https://doi.org/10.1007/BF00994016

Heinze-Deml, C., Maathuis, M. H., & Meinshausen, N. (2018). Causal structure learning. Annual Review of Statistics and Its Application, 5(1), 371–391. https://doi.org/10.1146/annurev-statistics-031017-100630

Kalisch, M., & Bühlmann, P. (2007). Estimating high-dimensional directed acyclic graphs with the PC-algorithm. Journal of Machine Learning Research, 8, 613–636.

Maathuis, M. H., Kalisch, M., & Bühlmann, P. (2009). Estimating high-dimensional intervention effects from observational data. The Annals of Statistics, 37(6A), 3133–3164. https://doi.org/10.1214/09-AOS685

Malinsky, D., & Danks, D. (2018). Causal discovery algorithms: A practical guide. Philosophy Compass, 13(1), Article e12470. https://doi.org/10.1111/phc3.12470

Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511803161

Peters, J., Janzing, D., & Schölkopf, B. (2017). Elements of causal inference: Foundations and learning algorithms. MIT Press.

Sachs, K., Perez, O., Pe'er, D., Lauffenburger, D. A., & Nolan, G. P. (2005). Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721), 523–529. https://doi.org/10.1126/science.1105809

Shimizu, S., Hoyer, P. O., Hyvärinen, A., & Kerminen, A. (2006). A linear non-Gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7, 2003–2030.

Spirtes, P., & Glymour, C. (1991). An algorithm for fast recovery of sparse causal graphs. Social Science Computer Review, 9(1), 62–72. https://doi.org/10.1177/089443939100900106

Spirtes, P., Glymour, C., & Scheines, R. (2000). Causation, prediction, and search (2nd ed.). MIT Press. https://doi.org/10.7551/mitpress/1754.001.0001

Uhler, C., Raskutti, G., Bühlmann, P., & Yu, B. (2013). Geometry of the faithfulness assumption in causal inference. The Annals of Statistics, 41(2), 436–463. https://doi.org/10.1214/12-AOS1080

Zhang, J. (2008). On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(16–17), 1873–1896. https://doi.org/10.1016/j.artint.2008.08.001