Benjamin Scholtz

LLM-driven Agent-Based Modelling

4 min read

LLM-driven Agent-Based Modelling

LLM-driven agent-based modelling replaces the hand-coded behavioural rule inside an agent with a foundation-model call. The agent still has a state, a position in some topology, and a place in a population. What changes is the inner step. Instead of if hungry and food_visible then eat, the agent asks an LLM, “given who you are and what you just observed, what do you do next?”

This sits inside the broader field of Simulation Modelling, specifically in the Agent-Based Modelling family. Classical ABM has spent decades developing calibration, sensitivity analysis, and validation techniques for populations of mechanistic agents. LLM-driven ABM inherits that toolkit, then trades inspectable rules for the open-ended behaviour the foundation model brings.

Why it matters

Two motivations, depending on whose problem you are solving.

For sim modellers, the appeal is expressive behaviour. Many real systems involve agents who interpret ambiguous situations, weigh competing goals, and react to language. Encoding that as discrete transition rules is brittle, and the brittleness shows up as a long tail of edge cases that the model handles badly. An LLM step covers that tail without requiring the modeller to enumerate every case in advance.

For people coming from the Synthetic Audiences direction, LLM-driven ABM is what they should be doing but mostly are not. Today’s synthetic-audience products (Auxos, Electric Twin) ask the LLM a single question per persona and treat the answer as a survey response. There are no dynamics, no agent-to-agent interaction, no topology, no time. LLM-driven ABM keeps those structural ingredients and lets the LLM handle only the reasoning step. That is closer to how a population actually behaves, and it inherits the validation regime that synthetic-audience vendors currently skip.

Reference work

The clearest demonstration is the Stanford Generative Agents paper (Park et al., 2023), commonly known as the Smallville paper. Twenty-five LLM agents in a small simulated town remember, plan, and react to each other over simulated days. Emergent behaviour appears, including coordinating a Valentine’s Day party that no one was told to throw. The architecture has three pieces: a memory stream, a reflection step that builds higher-order memories from raw ones, and a plan step that converts current state plus memories into next actions. Each piece is an LLM call. The agents inhabit a classical ABM topology around them.

Follow-up work has pushed in two directions. The first is scaling, with simulations of hundreds or thousands of agents using cached personas and batched inference. The second is calibration, attempting to match the simulated population’s aggregate behaviour to real survey or panel data so the model has a defensible accuracy claim rather than an anecdotal one.

Open problems

The interesting research questions sit at the boundary with what sim modellers already know.

How do you calibrate an LLM-driven population to real data without overfitting the prompts? Classical ABM has parameter-fitting machinery for this, but the parameter space here is “prompt space”, which is high-dimensional and unstable across model versions.

How do you propagate uncertainty when each agent’s reasoning step is itself stochastic? Sim modelling has Monte Carlo, Latin hypercube, polynomial chaos. None of them directly handle the noise structure of an LLM, which is correlated across prompts in ways the existing toolkit was not designed for.

How do you do sensitivity analysis when the “model” is a 200-token prompt plus a foundation model you do not control? Small prompt edits can produce large behaviour changes, and the modeller has no derivative to work with.

Each of these is a chance for the sim community to bring decades of validation craft to a category that currently markets on speed and a single accuracy number.

References

Graph View

Backlinks