small_Blacktesty.png

thumbnail_Blacktesty.png

Blacktesty.png

large_Whitetesty.png

medium_Whitetesty.png

small_Whitetesty.png

thumbnail_Whitetesty.png

Whitetesty.png

Like most websites, we store cookies for analytics, and well, that's it. You can read all about this in our privacy policy.

The questions leaders bring us, and what we find when we answer them properly.

Bring us one live decision. We'll tell you what it takes to answer it.


## questions-hero
label: What we work on
headline: The questions leaders bring us, and what we find when we answer them properly.
standfirst: Most businesses are complex, not complicated. The parts react to each other and change over time. These are the questions where that matters most, by the part of the business that asks them.

## questions-index

## questions-cta
heading: Got a question like these?
text: Bring us one live decision. We'll tell you what it takes to answer it.
buttonLabel: Talk to us
buttonHref: mailto:hello@zebra.associates


What we work on


*Most models are built to answer "what's the best way to act on this one lever to change this one outcome in our favour?". This might be the effect of any one of ad budget, capex, price, or any other single lever. But good business decisions don't mean looking at each lever in isolation, one at a time. High-value decisions mean pulling several of those levers together in concert. They don't act in isolation: change the ad budget (lever 1) and the right discount (lever 2) changes too. Deploy capex (lever 1) on a hotel site, and the price (lever 2) has more headroom to change.*

Decision-making with data requires causal inference. But the standard approach to causal machine learning (ML) falls apart the moment you're weighing more than one lever at once. Here I'll discuss why, and what our solution is.

Quick recap, because everything below rests on it. There are two fundamentally different jobs we give machine learning, and they only look similar on the surface.

## dual-card-section
card1Label: Predictive ML
card1LabelColor: #A5502A
card1Heading: "What will happen if we change nothing?"
card1Body: Predicting the future of a business left on its current course. It can lean on any pattern in the data: it just has to get the number right. Judged on accuracy.
card2Label: Causal ML
card2LabelColor: #1A4D5C
card2Heading: "What will happen because we act?"
card2Body: Deciding what to change. You pull a lever and the business responds. Here the model has to tell real cause and effect apart from patterns that merely travel together. Judged on what actually happens after you act.

A model can be superb at the first job and horribly wrong for the second: accuracy tells you nothing about lever quality. For more on that, read [Why a highly accurate model can still make terrible decisions](/articles/accurate-model-terrible-decisions).

This post is about what happens next: the way most teams *try* to make predictive models causal, and the trap waiting inside it the moment your business has more than one lever.

Which, of course, it does.

## The standard approach: how causality usually gets shoehorned into a predictive model

Most modelling projects make no attempt at causality at all, and simply look at accuracy or predictive power. That answers the wrong question and is not about the impact of levers. So any attempt at causal inference should be lauded.

But when causal inference *is* attempted, here is the standard recipe. (To be clear, it can be a legitimate one for a single lever, and it is how most applied causal inference is actually done):

**1.** Draw a causal diagram of your business: what drives what.

**2.** Pick the lever you care about and the outcome you care about.

**3.** Use the diagram to sort every variable into **good controls** (include them; they block confounding) and **bad controls** (exclude them; they distort the very effect you're measuring).

**4.** Fit a predictive model (xgboost, doubleML, whatever) with the lever, the outcome, and *only* the good controls.

The crucial mechanic, and the one that is easiest to miss: **in a predictive model, including a variable in the model *is* controlling for that variable.** There is no separate dial marked "control for this one but not that one". Inclusion is controlling for the variable (through conditioning on it).

That's why I call these **conditioning models**: every variable included is conditioned on (controlled for), automatically and unavoidably. The only causal instrument you have is the model's guest list of good and bad controls.

Notice what this approach really is: the model itself does not structure itself causally. **All the causal reasoning lives outside the model**, in your head and your causal diagram. It gets expressed through a single crude choice: what to include and what to leave out. We are shoehorning causality into a machine built for a different job. Double machine learning, for all its sophistication and common use for causal inference, works in exactly the same way causally (if with better statistical hygiene). It's a conditioning model.

The causal structure every conditioning model **implicitly** assumes is that every input independently causes the outcome, but that no input causes another input. The structure of conditioning models looks like the causal diagram on the left, while your business problem might look more like the diagram on the right. The structure in the diagram on the right is smuggled into the model on the left by variable inclusion/exclusion alone.

![Two causal diagrams side by side. On the left, the diagram a conditioning model assumes: ads, discount and traffic each point straight at sales with no arrows between the inputs. On the right, the diagram the business actually has: inputs cause each other and unmeasured demand touches several of them.](/images/science-series/many-levers-2.png)

And this works. Until it doesn't.

Used carefully, with **one** lever, **one** outcome and **one** thoughtfully chosen control set, this recipe works. It can sometimes work if multiple lever-outcome pairs share exactly the same conditioning variables. But that's rare and restrictive.

Usually a trap springs when there is more than one lever, which is to say, in pretty much every business case we've worked on.

## Businesses don't have one lever. They have a control panel.

Almost no real decision is "take action on one lever/dial, and hold everything else sacred". A retailer sets advertising budgets *and* discount depth *and* price *and* stock levels. A subscription business tunes acquisition spend, onboarding, plan pricing and retention offers. A lender moves credit limits, interest rates and collections intensity. And the dials interact: the right discount depends on how much traffic your ads are driving; the right ad budget depends on what margin your discounts have left you. What the business needs is the **best combination of settings across the whole control panel**, and a stack of isolated effects can't give it that.

So let's take a simplified version of that: a retailer with just two levers. And because arguments about causality on real data always dissolve into "well, you can't know the truth", we'll simulate the world, so we *do* know the truth, exactly.

![The simulated ground truth as a causal diagram. Ads, lever one, drive traffic, and traffic drives sales. Discount, lever two, lifts sales directly. Unmeasured demand pushes traffic up and also pushes the trading team to discount deeper when times are lean.](/images/science-series/many-levers-3.png)

Every arrow here is business-mundane. Ads bring people to the site; people on the site buy things. Discounts convert. And when demand goes soft, any trading team reaches for deeper discounts. That is exactly how weak demand ends up statistically glued to big discounts and low sales. Ads, meanwhile, run on a pre-agreed test calendar, so they move independently of all that.

In this world the truth is: every £1 of advertising returns £3 of sales, and every extra point of discount adds £12k of weekly sales. Both levers work. Now let's see what the standard recipe reports.

## One model, two levers, one problem

The data science team fits a conditioning model for weekly sales. The only real choice they get to make is whether **traffic** goes in. Watch what each choice does, over 20,000 simulated weeks and an ordinary regression:

![Results table. Model A, which includes traffic, reports no effect of ads and the right discount effect. Model B, which excludes traffic, reports the right ad effect and a strongly negative discount effect. The simulated truth is that both levers work. Model A has the higher forecast R², and no accuracy metric flags either failure.](/images/science-series/many-levers-4.png)

The model that includes traffic (conditions on it) gets the discount decision right but the ads wrong. The model that excludes it gets the discount decision wrong, but the ads right.

## The mutual adjustment fallacy: the problem with conditioning models

Why does this happen?

Because **traffic plays opposite roles for each of the two levers**. For the *discount* lever, traffic is a **good control**: unmeasured demand reaches sales only through traffic, so traffic carries its fingerprint, and conditioning on it blocks the "weak demand → deep discounts → low sales" confounding that otherwise slanders discounts. For the *ads* lever, traffic is a **bad control**: it is the very channel through which ads work, and conditioning on it amounts to asking "how much do ads lift sales, among weeks with identical traffic?", to which the answer is "none". Not because ads do nothing, but because holding traffic fixed erases the only route by which they work.

![Pull quote: one variable, a good control for one lever and a bad control for the other. A conditioning model has a single guest list, so it cannot be right about both.](/images/science-series/many-levers-5.png)

Model A is the more accurate model on every fold, every backtest, every holdout, so a standard validation pipeline actively *prefers* the model that gets the ad lever backwards. Reaching for a more flexible learner (a boosted tree, a neural net) makes it worse, not better: the more freely a model can use traffic to predict sales, the harder it leans on the very variable that corrupts the ad effect. More data doesn't help either (see [The big data fallacy](/articles/big-data-fallacy)).

This reading error has a name: the **mutual adjustment fallacy**. The seductive line "all effects shown are adjusted for all other variables" describes precisely the thing that makes most of the numbers meaningless: each coefficient is adjusted for the others, but the adjustment that is right for one lever is wrong for the next.

Epidemiologists know the same problem as the **Table 2 fallacy** (Westreich & Greenland, 2013), after the habit of listing a study's full regression output in Table 2, whereupon readers interpret *every* row as a causal effect, when the model was at best designed to be causal for one variable.

In plain business terms: **a conditioning model can tell you the causal effect of one lever: the one it was built around. The numbers it shows for everything else are not usually causal.** They might land near the truth now and then, but only by luck.

## Scientists make this mistake too

Lest you think this only bites careless business analysis, this is common in all corners of science. Just to take one example: "Optimal dietary patterns for healthy aging" (Tessier et al., *Nature Medicine*, 2025) followed over 100,000 people for three decades and asked which of eight dietary patterns (the levers) predicted healthy ageing (the outcome). Serious data, serious journal, huge press coverage. And the models adjust for, among many other things, **BMI**.

![Causal diagram of the dietary study. The dietary patterns point to healthy ageing directly, but also run through BMI, which sits downstream of the levers and upstream of the outcome. Conditioning on BMI strips out the part of diet's benefit that works through body weight, for all eight levers at once.](/images/science-series/many-levers-6.png)

Your body weight and composition is substantially a *consequence* of some of the things you eat, and that itself is a key driver of how well you age. **BMI is a mediator for many of the model inputs**: it lies on the causal road from the levers to the outcome. Condition on it and you erase the part of each diet's effect that works by keeping weight healthy, arguably one of the biggest ways diet matters. The published numbers answer a strange question: "how much does eating better help, if your weight refuses to respond to what you eat?"

And with eight dietary levers plus dozens of covariates mutually adjusted for one another in one conditioning model, the mutual adjustment fallacy is built into the study design. If it can happen there, with that much scrutiny, it is happening in your quarterly lever-to-KPI analysis.

## A workaround hack: one model per lever (and why you'll hate it)

If you stay inside the conditioning-model paradigm, there is exactly one correct response to all this: **stop asking one model about many levers**. Build a separate model for each lever-outcome pair, each with its own bespoke conditioning set. For our retailer: an ads model that *excludes* traffic, and a discount model that *includes* it. Each one right, on its own terms. This is the orthodox advice, and it genuinely fixes the statistical problem.

But this doesn't really fix the business problem.

![A suite of per-lever conditioning models on the left, each with a different control set and no answer to the joint question, against one structural causal model on the right that answers any lever in any combination.](/images/science-series/many-levers-7.png)

Count the costs of the suite of models. You now maintain N models with N different, mutually contradictory assumptions, each valid only for its own question. Try maintaining and updating that correctly.

Worse, their answers don't combine: the ads model assumed discount behaviour stays as it was, the discount model assumed the same about ads, and the moment you move both levers, every model is describing a world that no longer exists. Interactions between levers, the very thing that makes multi-lever decisions hard, live in none of the models. And the question the business actually asked, *what's the best set of actions across everything?*, has no owner. You've traded one wrong answer for several right answers to questions nobody asked in isolation.

## The solution: model the structure, not the guest list

The way out is to stop shoehorning causal structure into models that don't represent the causal structure. A **structural causal model** (SCM, also called a graphical causal model) is the solution.

The SCM explicitly encodes the causal structure, so that the model's architecture and the causal diagram are the same object. It learns the relationships between all variables in the causal diagram for which you allow an arrow. You take the same diagram you were already drawing to classify good and bad controls, and build the model from it instead of using it as an off-model checklist.

That single change dissolves the whole fallacy, for three reasons:

**First, and most importantly, inclusion of a variable no longer automatically means conditioning/controlling for that variable.** In a conditioning model, adding the variable traffic (from our example) implicitly holds it fixed for every question. In a structural model, traffic is just a node: its role is stated by arrows. It's a function of the nodes that go into it, and the nodes it goes into are functions of traffic (and other nodes). Variables are no longer good or bad controls; they're parts of the mechanism, and they all *belong in the model*. Good controls are taken care of, because their causal impact is captured in the flow. Bad controls are no longer bad, and should be included. Traffic, for example, can be in there, both as the road from ads to sales and as the footprint of demand, and the structure represents both roles at once, correctly.

**Second, information can only flow the way causality flows.** When you ask a structural model "what if we raise ad spend?", the change propagates forward along arrows: ads to traffic, traffic to sales. It cannot leak backwards from effects to causes, which is precisely the leak that conditioning creates: "holding traffic fixed" quietly lets the model reason from an effect back to its causes. Structural models make that move impossible by construction.

**Third, and this is the big payoff for the business: it's one model that answers every question.** Ask it about ads: it gives you +£3k per £1k, the truth. Ask it about discounts: +£12k per point, the truth. Ask it about *both at once, under next quarter's budget, with demand looking soft*, and it answers, because moving multiple levers is just multiple interventions propagating through one coherent structure. The best combination of settings across the whole control panel is a single optimisation over a single model. That is the question the business was asking all along, and it is the one thing neither a kitchen-sink model nor a suite of per-lever models can ever answer.

## The take-aways

- **A conditioning model is usually only trustworthy about one lever: the one it was built around.** The numbers it shows for every other variable are not causal effects. If one model is being used to read several levers, some of its advice is wrong, and nothing in the output tells you which part.
- **With multiple levers, no single set of controls is always right for all of them.** The same variable can be a good control for one lever and a bad control for another, and in a business those variables are everywhere.
- **One model per lever fixes the statistics but not the business.** The answers don't combine, and the question that matters (what to do across all levers together) goes unanswered.
- **A structural causal model fixes the problem.** Including a variable no longer means holding it fixed. The old "bad controls" go in and get modelled correctly, and one model answers any lever, in any combination.


Most business decisions involve many levers. Most causal models get all but one of them wrong


Most customer segmentation approaches score behaviour. Recency, frequency, monetary value. Discount usage, channel preference, engagement metrics. The assumption is that behaviour tells you who's valuable and who isn't.

But behaviour without context is ambiguous. A customer who only buys on discount could be extracting value you should have captured, or responding exactly as intended to your yield management strategy. A high-frequency customer could be genuinely loyal, or gaming your promotions for maximum benefit at minimum cost to them.

The same behaviour can create or destroy value depending on when, where, and why it happens. Behavioural segmentation can't tell the difference.

## Scoring for intent

Value exchange is a framework for understanding what customer behaviour actually means. Instead of asking "what did they do?", it asks "what did they mean by it?"

Some customers give back proportionate to what they receive: they pay fair prices, use resources efficiently, generate referrals, provide useful data. Others extract value without reciprocating: they demand discounts where none are warranted, consume disproportionate service resources, game programmes designed to reward genuine loyalty.

By scoring for intent rather than just behaviour, you avoid two common mistakes:

## dual-card-section
card1Label: Mistake one
card1Heading: Penalising customers who look problematic but aren't
card1Points: ["Discount-seekers responding to your own yield strategy", "High-touch customers who generate referrals that justify the cost", "Low-frequency buyers who purchase at full margin"]
card2Label: Mistake two
card2Heading: Overvaluing customers who look good but aren't
card2Points: ["High-frequency buyers who only transact on promotion", "\"Loyal\" customers whose engagement masks declining spend", "Programme members who've learned to extract maximum benefit at minimum reciprocity"]

## What we've found

We've tested value exchange segmentation against transaction data and found material differences from behavioural classification. 37% of customers classified as value-destructive by behavioural metrics were reclassified when we scored for intent. Context - competition intensity, capacity utilisation, channel dynamics - changed what their behaviour meant.

We're continuing to test this across industries and contexts. The methodology is grounded in established theory but the application is new, and we're honest about what we don't yet know.


Scoring customers for intent, not just behaviour


*Two models. One predicts sales almost perfectly, yet gives disastrous business advice. The other is a terrible forecaster, yet gets the decision exactly right and boosts profit.*

*What's going on?*

Every data science model review seems to eventually arrive at the same slide: MAPE, R², backtesting and holdout accuracy. These are all measures of how well the model predicts. If the accuracy is good, the data scientists are happy. The model is accurate, so surely we can now use it to make data-driven business decisions? How much to spend, what to price, where to take action.

But accuracy metrics grade exactly one skill: *given what we observed, how close was the prediction to what happened?* That's the wrong exam for data science designed for decision making. It's passive.

Business decisions are active: *if we take action, what happens?* To those inexperienced in causal thinking, these might sound like the same question. But they're very different. And a model can ace one while failing the other completely.

Everyone knows that correlation does not imply causation. But data scientists often don't acknowledge the close cousin of that trap: *predictive power does not equal causation* either. A model can forecast beautifully and still be worse than useless for deciding what to do.

## dual-card-section
card1Label: Prediction accuracy
card1LabelColor: #A5502A
card1Heading: "What will revenue be next quarter?"
card1Body: Graded by MAPE, R², holdout accuracy. As long as no action is taken and the world stays as it is, the model is fine.
card2Label: Decision making ability
card2LabelColor: #1A4D5C
card2Heading: "What happens to revenue if we double the budget?"
card2Body: Graded by causal thinking and reality, after you act. The world changes because you moved a lever. The model needs to be able to handle that.

We can show this with a very simple case: just three variables and a simulation. Both examples that follow are deliberately *minimal toy models*, stripped down to abstract away the clutter of a real system, so the mechanism stands out sharply and nothing hides behind complexity.

If we *simulate* the world, we know what is actually going on. It's our ground truth. We can use the simulation to generate data from a known process, then fit our models to it and see how well they understand what's going on.

So when a model claims "acting on A does nothing to the outcome C", we can check it against a ground truth that no real dataset ever gives you. We know the real answer to check against.

A simple three-variable example shows that this is not because of complexity. Rather, it's a fundamental mismatch between predictive models and their understanding of the world in which we need to act to change outcomes, rather than just predict them given no action.

And if a mere three nodes can pull accuracy and causation this far apart, the problem only deepens in practice. Real business processes aren't three variables: they're dozens, wired together by many overlapping structures at once: mediators, confounders, colliders, feedback loops. Each is another way for a high-accuracy model to quietly misread the levers. Stripping the system down to three nodes doesn't understate the problem. It isolates it, and complexity only ever adds more places for it to hide.

**Case 1**

## The high-accuracy model that makes bad decisions

Our simulated world: marketing spend (**A**) genuinely drives sales (**C**). Both of them, independently, eat into the support team's free capacity (**B**): campaigns light up the enquiry line with pre-sale questions from people who haven't bought anything, while every completed order generates its own tickets: shipping, returns, how-do-I. (Assume the team flexes shifts, so slow service never chokes off sales.) Free capacity sits *downstream* of the very thing we want to predict, and it's on the ops dashboard in real time, making it a perfectly legitimate forecast input.

![Causal diagram of the simulated world. Marketing spend A has a positive arrow to sales C, and both A and C have negative arrows into B, the support team's free capacity, which therefore sits downstream of sales.](/images/science-series/accurate-model-2.png)

Now the data science team builds a model to predict sales, and quite reasonably throws in every signal available: spend *and* the support team's current free capacity. The fit is spectacular. Predicted sales hugs actual sales on every holdout set. MAPE is tiny. The data scientists rejoice.

![Scatter plot of predicted against actual sales for model 1. Every point sits tight on the diagonal, so by any accuracy metric the model is superb.](/images/science-series/accurate-model-3.png)

Then someone asks the model the question the business actually cares about: *if we raise marketing spend, what happens to sales?* The model's answer: sales will *fall*. Not just "no effect", the results are backwards. The team's busyness is explained by campaign questions and order tickets between them, so holding today's free capacity fixed (by simply including it in a predictive model), a world with more spend must be a world with fewer orders. The model reports that faithfully. In our simulated world we know the truth: spend has a strong, real, *positive* effect on sales.

![Bar comparison for raising spend. What actually happens: sales rise. What the model says: sales fall. Near-perfect accuracy, and the sign of the lever is backwards.](/images/science-series/accurate-model-4.png)

In causal inference we call B a variable on a **bad control path,** here a consequence of the outcome used as a predictor. Conditioning on something downstream of sales lets the model "peek at the answer", which makes accuracy look wonderful and makes every coefficient about levers meaningless. Here the wiring flips the estimated effect of A to the wrong sign outright; other wirings merely shrink it towards zero. There are other bad controls too, but the point here is that accuracy doesn't care about bad controls, and will actively use them if it gets better predictions. That creates bad decision-making algorithms.

**Case 2**

## The low-accuracy model that makes great decisions

New simulated world. A retailer's discount depth (**A**) drives units sold (**C**). So does something else entirely: demand shocks (**B**) such as weather, competitor moves, payday timing. Although we simulate the effect of B, nobody measures B in our example and so it's not included in the ML model. And, because discounts follow a pre-set test calendar, A moves independently of those shocks.

![Causal diagram of the second simulated world. Discount depth A has a positive arrow to units sold C, and unobserved demand shocks B also feed into C, with B unmeasured and independent of A.](/images/science-series/accurate-model-5.png)

The ML model fitted to the simulated data predicts units sold from discount depth alone. As a result its forecasts are poor. The scatter graph of predicted versus actual is a cloud. MAPE is terrible, R² is low, and in any accuracy / predictive-power bake-off this model finishes dead last. It's not a good forecaster.

![Scatter plot of predicted against actual units for model 2. The points form a wide cloud far from the diagonal, so as a forecaster the model is poor.](/images/science-series/accurate-model-6.png)

Now ask it the decision question: *if we deepen the discount, what happens to units sold?* Its answer matches the simulated ground truth exactly. The relationship it learned between discount and sales *is* the true causal effect, because the only thing it left out was independent noise that you do not need to control for anyway (if you only want to know the impact of changing A).

![Bar comparison for deepening the discount. What actually happens: units rise. What the model says: units rise. Terrible accuracy, and exactly right about the lever.](/images/science-series/accurate-model-7.png)

## Why accuracy can't grade the decision

For each case, the accuracy score told you nothing about the decision-making ability of the model. Case 1 would be near-perfect on MAPE, R², every holdout and backtest, yet got the lever's sign backwards. Case 2 was terrible on those metrics, yet would nail the best action to take exactly. High accuracy sat with a disastrous decision; low accuracy sat with the right one.

The gap has a name: **observation** ("what are sales when the support team is slammed?") versus **intervention** ("what will sales be if *we take action* to make spend higher?"). Taking action asks about a *different* world (one that you've changed based on the model's recommendation) and no amount of cross-validation on the old world certifies a model for the new world. Everything in a standard ML pipeline, from the loss function to the leaderboard, sits on the observation side of that line. Decisions live on the other side, and are where you need causal ML.

## You don't have to choose

Here's the part that gets lost in the "accuracy versus impact" framing: it is not a trade-off you're stuck with. Causal models can be designed to be both great predictors and great decision makers too. The trick is separating the two jobs: downstream variables like B are welcome in the forecast, so long as the lever's effect is estimated from the causal structure, never read off the kitchen-sink coefficients. But if the only interest is decision making, then accuracy doesn't tell you much.

## The question to ask instead

Next time a model is proposed for a decision, don't ask how accurate it is. Ask: *if we change the lever, does the model know what happens following a change?* That is a causal question, and it is answered with causal evidence. It isn't a prediction problem, and accuracy tells you nothing about how good the model is for this job.

An accurate model tells you what tomorrow looks like if you change nothing. Only a causal model tells you what tomorrow looks like *because of what you do*. Don't let the first one grade the second one's exam.


Why a highly accurate model can still make terrible decisions


In July I wrote about a hotel group where pricing, marketing and an easy year-on-year comparison all claimed the same good quarter. Add the claims up and they came to more than the actual gain. The question was how to tell which team was right.

That example was fictionalised, but I've sat through a version of it when I ran digital at Premier Inn. Every team had numbers and every set looked good. The easiest way for marketing to hit its targets was cheap traffic that barely converted, and nothing in our measurement would have flagged it, because traffic was up and traffic was the metric. Product, marketing and pricing each had data showing they'd driven the same revenue. Nobody was lying. Their tools measured correlation, and correlation is generous with credit. The loudest case won the budget, and when someone asked whether an investment had worked, the honest answer was that nobody knew.

The usual answer is to test. And a well-run A/B test does answer the causal question, for one lever, at one time.

The trouble is that each test knows nothing outside itself. It tells you what your change did, on that audience, in that quarter, with every other team's activity frozen exactly where it happened to be. What works at peak won't necessarily work off peak. What marketing's test measured shifts once pricing moves. So every team ends up holding a valid local answer, and the answers still don't fit together. There's nothing for them to fit into.

And there's a question tests can't reach at all. Looking back, we want to know how much of the result was down to us. Looking forward, we want to know what will happen if we act. If we cut prices again, does revenue rise, or have we trained customers to wait for discounts? A test run last quarter can't tell you what this quarter's change will do once every other team has moved too.

What's missing is the thing the tests plug into: a model of how your business works, built with the teams who run it and agreed by them. It maps how the moving parts drive the outcomes in your business, rather than a template of your industry. Each team's test and each team's claim feeds into the same structure. Every contribution comes with a range, the contributions have to add up to the result the business actually got, and you can see what teams achieve together as well as alone. It acts as the umpire, and the dashboards report into it.

That's also why twenty years of better tools haven't fixed this. You can't buy a model of your business off the shelf, because the structure is different in every business, and finding it means careful work with the people who run it. It's a discipline more than a product, and very few organisations have it yet.

So the question from July still stands: can you tell who's right, or are you backing the team with the best story? Testing gets you part of the way. The rest needs a structure that everyone's numbers have to agree with.


Every team's test is right but the answers still don't add up


*The bigger the dataset, the more likely we are to mistake correlation for causation. The good news is that causal thinking can be learned.*

Everyone knows that correlation does not imply causation. And proper causal thinking is a concept at the heart of any good business decision making.

Yet a recent paper highlights what it calls **the big data fallacy**: the bigger the dataset, the more likely decision-makers are to mistake correlation for causation. That leads to bad decisions, because no amount of data can do this job without a causal model.

It's the same instinct behind a question I get a lot from data scientists. After a talk on causal modelling, someone will ask: "Won't a predictive model be able to solve this if it just has more data?"

No, it won't.

(See [The simple task your AI can't learn](/articles/your-ai-cant-learn).)

So what did the paper find?

Across twelve preregistered experiments with 6,408 participants, they found the same pattern again and again:

- Show people a correlational finding from a small observational study, and they treat it with appropriate caution.
- Show them the same finding from a massive dataset, and suddenly they act on it as if it were causal.

This finding held even when participants were shown a well-powered randomised experiment showing no effect at all, or even the opposite effect.

Although the paper doesn't study this, I suspect that the same would be found if they studied model complexity. The bigger and more impressive the model sounds (a deep neural net rather than a humble regression, say), the more likely people are to wrongly treat it as causal.

## Experienced executives weren't immune

The uncomfortable part is who fell for it. In one experiment, 44 executives with an average of 19 years' business experience showed a **stronger** big data fallacy than a general online sample. Most of them answered a basic causal reasoning question correctly, and still made the wrong call. The very people making the decisions were no better protected.

And it survived every quick fix the authors tried: reciting the "correlation is not causation" mantra, prompting people to list confounding variables (which, I'm convinced, most people don't properly understand, but that's for another post). Awareness alone simply didn't move the needle.

> **A biased estimate from 200 million rows is still biased. It's just a very precise wrong answer, delivered with high confidence.**

## Why does this happen?

The paper shows that people substitute an easy question (*which dataset is bigger and better?*) for the harder but correct one: *do this data and this model actually support a causal conclusion?* Large samples genuinely give you precision (higher certainty) and representativeness. But they do nothing about confounding, bad controls, reverse causality, or selection effects.

## dual-card-section
heading: The same question, two methods
card1Label: Observational, 200m observations
card1LabelColor: #A5502A
card1Stat: +872%
card1Body: Estimated lift in brand searches from an ad campaign.
card2Label: Randomised experiment
card2LabelColor: #1A4D5C
card2Stat: +5.4%
card2Body: The actual causal effect, on the same question.
footer: The paper's opening example: an ad campaign's effect on brand searches.

## What to do about it

If one-off reminders and awareness campaigns don't fix the problem, what does? Causal thinking. Scientific thinking. A causal mindset. You need methods that force the causal question to be asked explicitly, and people who instinctively ask it.

This is why we build our work around *structural causal models*. Before touching the data, we map out the data-generating process: what drives what, where confounders sit, where selection creeps in. The model makes your causal assumptions visible, testable and open to scrutiny. It also tells you whether your data can answer the question at all.

The study closes with a call for far better training in causal inference across business and beyond. We agree. Teams need to be able to reason about the data-generating process itself: to sketch the causal structure behind a question, understand why correlation and prediction aren't causal, and know when a big dataset is genuinely informative and when it's confidently misleading.

Big data is a wonderful thing. But volume answers "*how certain?*" and "*how representative?*", but not "*why?*" If your organisation is making decisions on best actions from correlational dashboards or predictive models, however large the datasets, it's worth a rethink.

**Reference:** Vosgerau, J., Giambastiani, G., & Scopelliti, I. (2025). *The Big Data Fallacy: When N is Large, Correlation = Causation.* SSRN preprint 5553621.


The questions leaders bring us, and what we find when we answer them properly.

Company Details

Registered Address

Cookie Preferences