Skip to content

Statistics · Tests and models

Nonparametric Regression and Smoothing Techniques for Flexible Knowledge Representation

How smoothers such as LOESS, splines and generalised additive models let the data show the shape of a relationship, and how to choose the amount of smoothing without chasing noise.

ModQu research article · 761 words · 6 references (APA 7th edition)

Download PDF

Ordinary regression asks the analyst to decide the shape of a relationship before looking at the data: a straight line, perhaps a quadratic. Many business relationships do not follow such a neat formula. Satisfaction may hold steady while delivery is quick, then fall sharply once waiting passes a threshold. Nonparametric regression estimates the curve directly from the data, assuming only that it is reasonably smooth (Wasserman, 2006).

The idea of a smoother

A smoother estimates the average outcome at each value of a predictor by borrowing information from nearby observations. Three families dominate applied work.

The smoothing parameter

Every smoother has a dial that controls flexibility: the span in LOESS, the penalty weight in a spline. A small span lets the curve follow every local fluctuation, giving low bias but high variance. A large span averages over wide stretches of data, giving a stable curve that may flatten real features. This is the classic bias-variance trade-off (Hastie et al., 2009).

The dial should not be set by eye alone. Cross-validation chooses the value that best predicts observations left out of the fit, and generalised cross-validation offers a fast approximation for splines (Craven & Wahba, 1978). Modern GAM software selects penalties automatically by similar criteria (Wood, 2017).

A worked example

Suppose a hypothetical retailer surveys 200 customers about their satisfaction, on a 0 to 10 scale, and links each answer to the number of days the order took to arrive. Figure 1 shows three fits to the same data.

Scatter plot of 200 hypothetical customers showing satisfaction from 0 to 10 against delivery time from 0 to 10 days. Satisfaction stays near 8 for deliveries of up to about 3 days, falls steeply between 3 and 6 days and levels off near 5.5. A dashed straight line declines steadily and misses the shape. A thin LOESS curve with span 0.06 zigzags with the noise. A thick LOESS curve with span 0.35 follows the underlying pattern smoothly.
Figure 1. Satisfaction by delivery time, fitted with a straight line and with LOESS smoothers of two spans. Illustrative data; chart by ModQu.

The straight line reports that each extra day costs about 0.4 points. That average hides the real story. The LOESS curve with a span of 0.35 shows satisfaction near 8.3 for next-day delivery and still about 7.8 at three days, then a drop to about 5.8 by day six, after which further delay changes little. The operational lesson is quite different: the critical window is days three to six, not every day equally. The very small span, by contrast, produces bumps that would vanish in a new sample.

Practical guidance for survey and business data

Limitations and common mistakes

Key takeaways

Nonparametric regression lets the data reveal shapes that a fixed formula would hide, such as thresholds and plateaus. LOESS, penalised splines and GAMs are the main tools. Their usefulness depends on choosing the smoothing parameter by cross-validation rather than by eye, showing uncertainty, and interpreting only features that are large enough to matter.

References

Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368), 829–836. https://doi.org/10.1080/01621459.1979.10481038

Craven, P., & Wahba, G. (1978). Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation. Numerische Mathematik, 31(4), 377–403. https://doi.org/10.1007/BF01404567

Eilers, P. H. C., & Marx, B. D. (1996). Flexible smoothing with B-splines and penalties. Statistical Science, 11(2), 89–121. https://doi.org/10.1214/ss/1038425655

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

Wasserman, L. (2006). All of nonparametric statistics. Springer. https://doi.org/10.1007/0-387-30623-4

Wood, S. N. (2017). Generalized additive models: An introduction with R (2nd ed.). Chapman and Hall/CRC. https://doi.org/10.1201/9781315370279