Sensitivity Analysis
Sensitivity analysis is the study of how the outputs of a model change when its inputs change. It answers a question that prediction alone cannot: which inputs matter, by how much, and over what range. In a model with thirty parameters, sensitivity analysis is often how we discover that three of them account for most of the variation in the output and the other twenty-seven have little effect.
It is used in three main ways. In Simulation Modelling, to test whether the headline result of a simulation is stable across parameter choices. In calibration, to decide which parameters are worth fitting carefully and which can be left at default. In communication, to report uncertainty around a result rather than only a point estimate.
Local versus global
Local sensitivity analysis perturbs one input at a time around a chosen point. The output is a partial derivative. It is cheap to compute, but it only describes the model’s behaviour in a small neighbourhood and assumes the inputs do not interact. Useful as a first pass; not enough on its own.
Global sensitivity analysis varies inputs across their full plausible range, usually together, and measures how much of the output variance each input explains. It is more expensive to compute but does not depend on a single reference point. For any model that will inform a real decision, global analysis is the appropriate choice.
Methods worth knowing
- One-at-a-time (OAT). The default. Cheap and intuitive. Hides interactions between inputs, so it is best used as a screening step rather than as a conclusion.
- Morris elementary effects. Cheap enough to run on a large parameter space, and produces both a mean effect and a variance-of-effect per input, so it flags non-linearity and interaction without requiring a full Sobol decomposition.
- Sobol indices. The standard method for global sensitivity. Decomposes the output variance into contributions from each input and each interaction. Computationally expensive. First-order Sobol gives the main effect; total-order Sobol captures all interactions involving that input.
- Latin hypercube sampling. A sampling strategy, not a sensitivity method, but the standard way to lay out the design points for any global analysis. Lower variance than plain Monte Carlo at the same sample size.
- Polynomial chaos expansion. Fits a polynomial surrogate to the model and reads off Sobol indices analytically. Fast once the surrogate is fitted, but only works when the input distributions are well-behaved.
- Variance-based methods more generally. A family that includes Sobol. The shared idea is that “important” means “explains output variance.”
Connection to validation
For LLM-driven Agent-Based Modelling and Synthetic Audiences, sensitivity analysis is absent from most current practice. Larooij and Törnberg’s review (see [[Validating Generative Social Simulations — Larooij and Törnberg]]) flags the absence of sensitivity testing as one of three structural weaknesses across thirty-five surveyed studies. A generative ABM whose conclusions flip under a small change to a prompt template or a model version is unreliable, and sensitivity analysis is what surfaces this.
The open research problem is that classical sensitivity analysis assumes continuous, real-valued inputs. LLM prompts live in a discrete, high-dimensional token space. The standard methods need adaptation to apply here, with sampling over prompt edits as the design strategy.
References
- Sensitivity analysis (Wikipedia) — readable overview with the taxonomy
- SALib — Python library for global sensitivity analysis (Sobol, Morris, FAST, delta moment-independent, others)
- Saltelli et al., Global Sensitivity Analysis: The Primer — standard textbook
- Sobol’ method — original paper — variance decomposition
Related notes
- Simulation Modelling — primary application area
- Bayesian Modeling and Probabilistic Programming — propagates input uncertainty through the model; complementary to sensitivity analysis
- Causal Inference — distinguishes “sensitive to” from “caused by”
- LLM-driven Agent-Based Modelling — the open frontier for prompt-space sensitivity
- Validating Generative Social Simulations — Larooij and Törnberg — why sensitivity testing is the missing piece in current generative-ABM practice