small_Blacktesty.png

thumbnail_Blacktesty.png

Blacktesty.png

large_Whitetesty.png

medium_Whitetesty.png

small_Whitetesty.png

thumbnail_Whitetesty.png

Whitetesty.png

Like most websites, we store cookies for analytics, and well, that's it. You can read all about this in our privacy policy.

The questions leaders bring us, and what we find when we answer them properly.

Bring us one live decision. We'll tell you what we'd need to answer it.


## questions-hero
label: What we work on
headline: The questions leaders bring us, and what we find when we answer them properly.
standfirst: Most businesses are complex, not complicated. The parts react to each other and change over time. These are the questions where that matters most, by the part of the business that asks them.

## questions-index

## questions-cta
heading: Got a question like these?
text: Bring us one live decision. We'll tell you what we'd need to answer it.
buttonLabel: Talk to us
buttonHref: mailto:hello@zebra.associates


What we work on


*Most models are built to answer "what's the best way to act on this one lever to change this one outcome in our favour?". This might be the effect of any one of ad budget, capex, price, or any other single lever. But good business decisions don't mean looking at each lever in isolation, one at a time. High-value decisions mean pulling several of those levers together in concert. They don't act in isolation: change the ad budget (lever 1) and the right discount (lever 2) changes too. Deploy capex (lever 1) on a hotel site, and the price (lever 2) has more headroom to change.*

Decision-making with data requires causal inference. But the standard approach to causal machine learning (ML) falls apart the moment you're weighing more than one lever at once. Here I'll discuss why, and what our solution is.

Quick recap, because everything below rests on it. There are two fundamentally different jobs we give machine learning, and they only look similar on the surface.

## dual-card-section
card1Label: Predictive ML
card1LabelColor: #A5502A
card1Heading: "What will happen if we change nothing?"
card1Body: Predicting the future of a business left on its current course. It can lean on any pattern in the data: it just has to get the number right. Judged on accuracy.
card2Label: Causal ML
card2LabelColor: #1A4D5C
card2Heading: "What will happen because we act?"
card2Body: Deciding what to change. You pull a lever and the business responds. Here the model has to tell real cause and effect apart from patterns that merely travel together. Judged on what actually happens after you act.

A model can be superb at the first job and horribly wrong for the second: accuracy tells you nothing about lever quality. For more on that, read [Why a highly accurate model can still make terrible decisions](/articles/accurate-model-terrible-decisions).

This post is about what happens next: the way most teams *try* to make predictive models causal, and the trap waiting inside it the moment your business has more than one lever.

Which, of course, it does.

## The standard approach: how causality usually gets shoehorned into a predictive model

Most modelling projects make no attempt at causality at all, and simply look at accuracy or predictive power. That answers the wrong question and is not about the impact of levers. So any attempt at causal inference should be lauded.

But when causal inference *is* attempted, here is the standard recipe. (To be clear, it can be a legitimate one for a single lever, and it is how most applied causal inference is actually done):

**1.** Draw a causal diagram of your business: what drives what.

**2.** Pick the lever you care about and the outcome you care about.

**3.** Use the diagram to sort every variable into **good controls** (include them; they block confounding) and **bad controls** (exclude them; they distort the very effect you're measuring).

**4.** Fit a predictive model (xgboost, doubleML, whatever) with the lever, the outcome, and *only* the good controls.

The crucial mechanic, and the one that is easiest to miss: **in a predictive model, including a variable in the model *is* controlling for that variable.** There is no separate dial marked "control for this one but not that one". Inclusion is controlling for the variable (through conditioning on it).

That's why I call these **conditioning models**: every variable included is conditioned on (controlled for), automatically and unavoidably. The only causal instrument you have is the model's guest list of good and bad controls.

Notice what this approach really is: the model itself does not structure itself causally. **All the causal reasoning lives outside the model**, in your head and your causal diagram. It gets expressed through a single crude choice: what to include and what to leave out. We are shoehorning causality into a machine built for a different job. Double machine learning, for all its sophistication and common use for causal inference, works in exactly the same way causally (if with better statistical hygiene). It's a conditioning model.

The causal structure every conditioning model **implicitly** assumes is that every input independently causes the outcome, but that no input causes another input. The structure of conditioning models looks like the causal diagram on the left, while your business problem might look more like the diagram on the right. The structure in the diagram on the right is smuggled into the model on the left by variable inclusion/exclusion alone.

![Two causal diagrams side by side. On the left, the diagram a conditioning model assumes: ads, discount and traffic each point straight at sales with no arrows between the inputs. On the right, the diagram the business actually has: inputs cause each other and unmeasured demand touches several of them.](/images/science-series/many-levers-2.png)

And this works. Until it doesn't.

Used carefully, with **one** lever, **one** outcome and **one** thoughtfully chosen control set, this recipe works. It can sometimes work if multiple lever-outcome pairs share exactly the same conditioning variables. But that's rare and restrictive.

Usually a trap springs when there is more than one lever, which is to say, in pretty much every business case we've worked on.

## Businesses don't have one lever. They have a control panel.

Almost no real decision is "take action on one lever/dial, and hold everything else sacred". A retailer sets advertising budgets *and* discount depth *and* price *and* stock levels. A subscription business tunes acquisition spend, onboarding, plan pricing and retention offers. A lender moves credit limits, interest rates and collections intensity. And the dials interact: the right discount depends on how much traffic your ads are driving; the right ad budget depends on what margin your discounts have left you. What the business needs is the **best combination of settings across the whole control panel**, and a stack of isolated effects can't give it that.

So let's take a simplified version of that: a retailer with just two levers. And because arguments about causality on real data always dissolve into "well, you can't know the truth", we'll simulate the world, so we *do* know the truth, exactly.

![The simulated ground truth as a causal diagram. Ads, lever one, drive traffic, and traffic drives sales. Discount, lever two, lifts sales directly. Unmeasured demand pushes traffic up and also pushes the trading team to discount deeper when times are lean.](/images/science-series/many-levers-3.png)

Every arrow here is business-mundane. Ads bring people to the site; people on the site buy things. Discounts convert. And when demand goes soft, any trading team reaches for deeper discounts. That is exactly how weak demand ends up statistically glued to big discounts and low sales. Ads, meanwhile, run on a pre-agreed test calendar, so they move independently of all that.

In this world the truth is: every £1 of advertising returns £3 of sales, and every extra point of discount adds £12k of weekly sales. Both levers work. Now let's see what the standard recipe reports.

## One model, two levers, one problem

The data science team fits a conditioning model for weekly sales. The only real choice they get to make is whether **traffic** goes in. Watch what each choice does, over 20,000 simulated weeks and an ordinary regression:

![Results table. Model A, which includes traffic, reports no effect of ads and the right discount effect. Model B, which excludes traffic, reports the right ad effect and a strongly negative discount effect. The simulated truth is that both levers work. Model A has the higher forecast R², and no accuracy metric flags either failure.](/images/science-series/many-levers-4.png)

The model that includes traffic (conditions on it) gets the discount decision right but the ads wrong. The model that excludes it gets the discount decision wrong, but the ads right.

## The mutual adjustment fallacy: the problem with conditioning models

Why does this happen?

Because **traffic plays opposite roles for each of the two levers**. For the *discount* lever, traffic is a **good control**: unmeasured demand reaches sales only through traffic, so traffic carries its fingerprint, and conditioning on it blocks the "weak demand → deep discounts → low sales" confounding that otherwise slanders discounts. For the *ads* lever, traffic is a **bad control**: it is the very channel through which ads work, and conditioning on it amounts to asking "how much do ads lift sales, among weeks with identical traffic?", to which the answer is "none". Not because ads do nothing, but because holding traffic fixed erases the only route by which they work.

![Pull quote: one variable, a good control for one lever and a bad control for the other. A conditioning model has a single guest list, so it cannot be right about both.](/images/science-series/many-levers-5.png)

Model A is the more accurate model on every fold, every backtest, every holdout, so a standard validation pipeline actively *prefers* the model that gets the ad lever backwards. Reaching for a more flexible learner (a boosted tree, a neural net) makes it worse, not better: the more freely a model can use traffic to predict sales, the harder it leans on the very variable that corrupts the ad effect. More data doesn't help either (see [The big data fallacy](/articles/big-data-fallacy)).

This reading error has a name: the **mutual adjustment fallacy**. The seductive line "all effects shown are adjusted for all other variables" describes precisely the thing that makes most of the numbers meaningless: each coefficient is adjusted for the others, but the adjustment that is right for one lever is wrong for the next.

Epidemiologists know the same problem as the **Table 2 fallacy** (Westreich & Greenland, 2013), after the habit of listing a study's full regression output in Table 2, whereupon readers interpret *every* row as a causal effect, when the model was at best designed to be causal for one variable.

In plain business terms: **a conditioning model can tell you the causal effect of one lever: the one it was built around. The numbers it shows for everything else are not usually causal.** They might land near the truth now and then, but only by luck.

## Scientists make this mistake too

Lest you think this only bites careless business analysis, this is common in all corners of science. Just to take one example: "Optimal dietary patterns for healthy aging" (Tessier et al., *Nature Medicine*, 2025) followed over 100,000 people for three decades and asked which of eight dietary patterns (the levers) predicted healthy ageing (the outcome). Serious data, serious journal, huge press coverage. And the models adjust for, among many other things, **BMI**.

![Causal diagram of the dietary study. The dietary patterns point to healthy ageing directly, but also run through BMI, which sits downstream of the levers and upstream of the outcome. Conditioning on BMI strips out the part of diet's benefit that works through body weight, for all eight levers at once.](/images/science-series/many-levers-6.png)

Your body weight and composition is substantially a *consequence* of some of the things you eat, and that itself is a key driver of how well you age. **BMI is a mediator for many of the model inputs**: it lies on the causal road from the levers to the outcome. Condition on it and you erase the part of each diet's effect that works by keeping weight healthy, arguably one of the biggest ways diet matters. The published numbers answer a strange question: "how much does eating better help, if your weight refuses to respond to what you eat?"

And with eight dietary levers plus dozens of covariates mutually adjusted for one another in one conditioning model, the mutual adjustment fallacy is built into the study design. If it can happen there, with that much scrutiny, it is happening in your quarterly lever-to-KPI analysis.

## A workaround hack: one model per lever (and why you'll hate it)

If you stay inside the conditioning-model paradigm, there is exactly one correct response to all this: **stop asking one model about many levers**. Build a separate model for each lever-outcome pair, each with its own bespoke conditioning set. For our retailer: an ads model that *excludes* traffic, and a discount model that *includes* it. Each one right, on its own terms. This is the orthodox advice, and it genuinely fixes the statistical problem.

But this doesn't really fix the business problem.

![A suite of per-lever conditioning models on the left, each with a different control set and no answer to the joint question, against one structural causal model on the right that answers any lever in any combination.](/images/science-series/many-levers-7.png)

Count the costs of the suite of models. You now maintain N models with N different, mutually contradictory assumptions, each valid only for its own question. Try maintaining and updating that correctly.

Worse, their answers don't combine: the ads model assumed discount behaviour stays as it was, the discount model assumed the same about ads, and the moment you move both levers, every model is describing a world that no longer exists. Interactions between levers, the very thing that makes multi-lever decisions hard, live in none of the models. And the question the business actually asked, *what's the best set of actions across everything?*, has no owner. You've traded one wrong answer for several right answers to questions nobody asked in isolation.

## The solution: model the structure, not the guest list

The way out is to stop shoehorning causal structure into models that don't represent the causal structure. A **structural causal model** (SCM, also called a graphical causal model) is the solution.

The SCM explicitly encodes the causal structure, so that the model's architecture and the causal diagram are the same object. It learns the relationships between all variables in the causal diagram for which you allow an arrow. You take the same diagram you were already drawing to classify good and bad controls, and build the model from it instead of using it as an off-model checklist.

That single change dissolves the whole fallacy, for three reasons:

**First, and most importantly, inclusion of a variable no longer automatically means conditioning/controlling for that variable.** In a conditioning model, adding the variable traffic (from our example) implicitly holds it fixed for every question. In a structural model, traffic is just a node: its role is stated by arrows. It's a function of the nodes that go into it, and the nodes it goes into are functions of traffic (and other nodes). Variables are no longer good or bad controls; they're parts of the mechanism, and they all *belong in the model*. Good controls are taken care of, because their causal impact is captured in the flow. Bad controls are no longer bad, and should be included. Traffic, for example, can be in there, both as the road from ads to sales and as the footprint of demand, and the structure represents both roles at once, correctly.

**Second, information can only flow the way causality flows.** When you ask a structural model "what if we raise ad spend?", the change propagates forward along arrows: ads to traffic, traffic to sales. It cannot leak backwards from effects to causes, which is precisely the leak that conditioning creates: "holding traffic fixed" quietly lets the model reason from an effect back to its causes. Structural models make that move impossible by construction.

**Third, and this is the big payoff for the business: it's one model that answers every question.** Ask it about ads: it gives you +£3k per £1k, the truth. Ask it about discounts: +£12k per point, the truth. Ask it about *both at once, under next quarter's budget, with demand looking soft*, and it answers, because moving multiple levers is just multiple interventions propagating through one coherent structure. The best combination of settings across the whole control panel is a single optimisation over a single model. That is the question the business was asking all along, and it is the one thing neither a kitchen-sink model nor a suite of per-lever models can ever answer.

## The take-aways

- **A conditioning model is usually only trustworthy about one lever: the one it was built around.** The numbers it shows for every other variable are not causal effects. If one model is being used to read several levers, some of its advice is wrong, and nothing in the output tells you which part.
- **With multiple levers, no single set of controls is always right for all of them.** The same variable can be a good control for one lever and a bad control for another, and in a business those variables are everywhere.
- **One model per lever fixes the statistics but not the business.** The answers don't combine, and the question that matters (what to do across all levers together) goes unanswered.
- **A structural causal model fixes the problem.** Including a variable no longer means holding it fixed. The old "bad controls" go in and get modelled correctly, and one model answers any lever, in any combination.


Most business decisions involve many levers. Most causal models get all but one of them wrong


Most customer segmentation approaches score behaviour. Recency, frequency, monetary value. Discount usage, channel preference, engagement metrics. The assumption is that behaviour tells you who's valuable and who isn't.

But behaviour without context is ambiguous. A customer who only buys on discount could be extracting value you should have captured, or responding exactly as intended to your yield management strategy. A high-frequency customer could be genuinely loyal, or gaming your promotions for maximum benefit at minimum cost to them.

The same behaviour can create or destroy value depending on when, where, and why it happens. Behavioural segmentation can't tell the difference.

## Scoring for intent

Value exchange is a framework for understanding what customer behaviour actually means. Instead of asking "what did they do?", it asks "what did they mean by it?"

Some customers give back proportionate to what they receive: they pay fair prices, use resources efficiently, generate referrals, provide useful data. Others extract value without reciprocating: they demand discounts where none are warranted, consume disproportionate service resources, game programmes designed to reward genuine loyalty.

By scoring for intent rather than just behaviour, you avoid two common mistakes:

## dual-card-section
card1Label: Mistake one
card1Heading: Penalising customers who look problematic but aren't
card1Points: ["Discount-seekers responding to your own yield strategy", "High-touch customers who generate referrals that justify the cost", "Low-frequency buyers who purchase at full margin"]
card2Label: Mistake two
card2Heading: Overvaluing customers who look good but aren't
card2Points: ["High-frequency buyers who only transact on promotion", "\"Loyal\" customers whose engagement masks declining spend", "Programme members who've learned to extract maximum benefit at minimum reciprocity"]

## What we've found

We've tested value exchange segmentation against transaction data and found material differences from behavioural classification. 37% of customers classified as value-destructive by behavioural metrics were reclassified when we scored for intent. Context - competition intensity, capacity utilisation, channel dynamics - changed what their behaviour meant.

We're continuing to test this across industries and contexts. The methodology is grounded in established theory but the application is new, and we're honest about what we don't yet know.


Scoring customers for intent, not just behaviour


*Two models. One predicts sales almost perfectly, yet gives disastrous business advice. The other is a terrible forecaster, yet gets the decision exactly right and boosts profit.*

*What's going on?*

Every data science model review seems to eventually arrive at the same slide: MAPE, R², backtesting and holdout accuracy. These are all measures of how well the model predicts. If the accuracy is good, the data scientists are happy. The model is accurate, so surely we can now use it to make data-driven business decisions? How much to spend, what to price, where to take action.

But accuracy metrics grade exactly one skill: *given what we observed, how close was the prediction to what happened?* That's the wrong exam for data science designed for decision making. It's passive.

Business decisions are active: *if we take action, what happens?* To those inexperienced in causal thinking, these might sound like the same question. But they're very different. And a model can ace one while failing the other completely.

Everyone knows that correlation does not imply causation. But data scientists often don't acknowledge the close cousin of that trap: *predictive power does not equal causation* either. A model can forecast beautifully and still be worse than useless for deciding what to do.

## dual-card-section
card1Label: Prediction accuracy
card1LabelColor: #A5502A
card1Heading: "What will revenue be next quarter?"
card1Body: Graded by MAPE, R², holdout accuracy. As long as no action is taken and the world stays as it is, the model is fine.
card2Label: Decision making ability
card2LabelColor: #1A4D5C
card2Heading: "What happens to revenue if we double the budget?"
card2Body: Graded by causal thinking and reality, after you act. The world changes because you moved a lever. The model needs to be able to handle that.

We can show this with a very simple case: just three variables and a simulation. Both examples that follow are deliberately *minimal toy models*, stripped down to abstract away the clutter of a real system, so the mechanism stands out sharply and nothing hides behind complexity.

If we *simulate* the world, we know what is actually going on. It's our ground truth. We can use the simulation to generate data from a known process, then fit our models to it and see how well they understand what's going on.

So when a model claims "acting on A does nothing to the outcome C", we can check it against a ground truth that no real dataset ever gives you. We know the real answer to check against.

A simple three-variable example shows that this is not because of complexity. Rather, it's a fundamental mismatch between predictive models and their understanding of the world in which we need to act to change outcomes, rather than just predict them given no action.

And if a mere three nodes can pull accuracy and causation this far apart, the problem only deepens in practice. Real business processes aren't three variables: they're dozens, wired together by many overlapping structures at once: mediators, confounders, colliders, feedback loops. Each is another way for a high-accuracy model to quietly misread the levers. Stripping the system down to three nodes doesn't understate the problem. It isolates it, and complexity only ever adds more places for it to hide.

**Case 1**

## The high-accuracy model that makes bad decisions

Our simulated world: marketing spend (**A**) genuinely drives sales (**C**). Both of them, independently, eat into the support team's free capacity (**B**): campaigns light up the enquiry line with pre-sale questions from people who haven't bought anything, while every completed order generates its own tickets: shipping, returns, how-do-I. (Assume the team flexes shifts, so slow service never chokes off sales.) Free capacity sits *downstream* of the very thing we want to predict, and it's on the ops dashboard in real time, making it a perfectly legitimate forecast input.

![Causal diagram of the simulated world. Marketing spend A has a positive arrow to sales C, and both A and C have negative arrows into B, the support team's free capacity, which therefore sits downstream of sales.](/images/science-series/accurate-model-2.png)

Now the data science team builds a model to predict sales, and quite reasonably throws in every signal available: spend *and* the support team's current free capacity. The fit is spectacular. Predicted sales hugs actual sales on every holdout set. MAPE is tiny. The data scientists rejoice.

![Scatter plot of predicted against actual sales for model 1. Every point sits tight on the diagonal, so by any accuracy metric the model is superb.](/images/science-series/accurate-model-3.png)

Then someone asks the model the question the business actually cares about: *if we raise marketing spend, what happens to sales?* The model's answer: sales will *fall*. Not just "no effect", the results are backwards. The team's busyness is explained by campaign questions and order tickets between them, so holding today's free capacity fixed (by simply including it in a predictive model), a world with more spend must be a world with fewer orders. The model reports that faithfully. In our simulated world we know the truth: spend has a strong, real, *positive* effect on sales.

![Bar comparison for raising spend. What actually happens: sales rise. What the model says: sales fall. Near-perfect accuracy, and the sign of the lever is backwards.](/images/science-series/accurate-model-4.png)

In causal inference we call B a variable on a **bad control path,** here a consequence of the outcome used as a predictor. Conditioning on something downstream of sales lets the model "peek at the answer", which makes accuracy look wonderful and makes every coefficient about levers meaningless. Here the wiring flips the estimated effect of A to the wrong sign outright; other wirings merely shrink it towards zero. There are other bad controls too, but the point here is that accuracy doesn't care about bad controls, and will actively use them if it gets better predictions. That creates bad decision-making algorithms.

**Case 2**

## The low-accuracy model that makes great decisions

New simulated world. A retailer's discount depth (**A**) drives units sold (**C**). So does something else entirely: demand shocks (**B**) such as weather, competitor moves, payday timing. Although we simulate the effect of B, nobody measures B in our example and so it's not included in the ML model. And, because discounts follow a pre-set test calendar, A moves independently of those shocks.

![Causal diagram of the second simulated world. Discount depth A has a positive arrow to units sold C, and unobserved demand shocks B also feed into C, with B unmeasured and independent of A.](/images/science-series/accurate-model-5.png)

The ML model fitted to the simulated data predicts units sold from discount depth alone. As a result its forecasts are poor. The scatter graph of predicted versus actual is a cloud. MAPE is terrible, R² is low, and in any accuracy / predictive-power bake-off this model finishes dead last. It's not a good forecaster.

![Scatter plot of predicted against actual units for model 2. The points form a wide cloud far from the diagonal, so as a forecaster the model is poor.](/images/science-series/accurate-model-6.png)

Now ask it the decision question: *if we deepen the discount, what happens to units sold?* Its answer matches the simulated ground truth exactly. The relationship it learned between discount and sales *is* the true causal effect, because the only thing it left out was independent noise that you do not need to control for anyway (if you only want to know the impact of changing A).

![Bar comparison for deepening the discount. What actually happens: units rise. What the model says: units rise. Terrible accuracy, and exactly right about the lever.](/images/science-series/accurate-model-7.png)

## Why accuracy can't grade the decision

For each case, the accuracy score told you nothing about the decision-making ability of the model. Case 1 would be near-perfect on MAPE, R², every holdout and backtest, yet got the lever's sign backwards. Case 2 was terrible on those metrics, yet would nail the best action to take exactly. High accuracy sat with a disastrous decision; low accuracy sat with the right one.

The gap has a name: **observation** ("what are sales when the support team is slammed?") versus **intervention** ("what will sales be if *we take action* to make spend higher?"). Taking action asks about a *different* world (one that you've changed based on the model's recommendation) and no amount of cross-validation on the old world certifies a model for the new world. Everything in a standard ML pipeline, from the loss function to the leaderboard, sits on the observation side of that line. Decisions live on the other side, and are where you need causal ML.

## You don't have to choose

Here's the part that gets lost in the "accuracy versus impact" framing: it is not a trade-off you're stuck with. Causal models can be designed to be both great predictors and great decision makers too. The trick is separating the two jobs: downstream variables like B are welcome in the forecast, so long as the lever's effect is estimated from the causal structure, never read off the kitchen-sink coefficients. But if the only interest is decision making, then accuracy doesn't tell you much.

## The question to ask instead

Next time a model is proposed for a decision, don't ask how accurate it is. Ask: *if we change the lever, does the model know what happens following a change?* That is a causal question, and it is answered with causal evidence. It isn't a prediction problem, and accuracy tells you nothing about how good the model is for this job.

An accurate model tells you what tomorrow looks like if you change nothing. Only a causal model tells you what tomorrow looks like *because of what you do*. Don't let the first one grade the second one's exam.


Why a highly accurate model can still make terrible decisions


In July I wrote about a hotel group where pricing, marketing and an easy year-on-year comparison all claimed the same good quarter. Add the claims up and they came to more than the actual gain. The question was how to tell which team was right.

That example was fictionalised, but I've sat through a version of it when I ran digital at Premier Inn. Every team had numbers and every set looked good. The easiest way for marketing to hit its targets was cheap traffic that barely converted, and nothing in our measurement would have flagged it, because traffic was up and traffic was the metric. Product, marketing and pricing each had data showing they'd driven the same revenue. Nobody was lying. Their tools measured correlation, and correlation is generous with credit. The loudest case won the budget, and when someone asked whether an investment had worked, the honest answer was that nobody knew.

The usual answer is to test. And a well-run A/B test does answer the causal question, for one lever, at one time.

The trouble is that each test knows nothing outside itself. It tells you what your change did, on that audience, in that quarter, with every other team's activity frozen exactly where it happened to be. What works at peak won't necessarily work off peak. What marketing's test measured shifts once pricing moves. So every team ends up holding a valid local answer, and the answers still don't fit together. There's nothing for them to fit into.

And there's a question tests can't reach at all. Looking back, we want to know how much of the result was down to us. Looking forward, we want to know what will happen if we act. If we cut prices again, does revenue rise, or have we trained customers to wait for discounts? A test run last quarter can't tell you what this quarter's change will do once every other team has moved too.

What's missing is the thing the tests plug into: a model of how your business works, built with the teams who run it and agreed by them. It maps how the moving parts drive the outcomes in your business, rather than a template of your industry. Each team's test and each team's claim feeds into the same structure. Every contribution comes with a range, the contributions have to add up to the result the business actually got, and you can see what teams achieve together as well as alone. It acts as the umpire, and the dashboards report into it.

That's also why twenty years of better tools haven't fixed this. You can't buy a model of your business off the shelf, because the structure is different in every business, and finding it means careful work with the people who run it. It's a discipline more than a product, and very few organisations have it yet.

So the question from July still stands: can you tell who's right, or are you backing the team with the best story? Testing gets you part of the way. The rest needs a structure that everyone's numbers have to agree with.


Every team's test is right but the answers still don't add up


Earlier in the year I published two pieces on causal mapping in football. The first used basic causal analysis across 2,140 Premier League matches to separate myth from signal. The second plotted every factor by how much pundits discuss it against how much it actually affects results.

From this we learned quickly that most football chat is baseless. Form, weather, travel distance, fixture congestion all clustered top-left: talked about constantly, causally irrelevant. Wage bill gap and crowd size sat bottom-right: barely discussed, strongest effects in the data. The gap between where the bubbles were big and where they sat on the x-axis was the gap between received wisdom and evidence.

The same chart could be drawn for most commercial organisations. Most of the effort is spent focusing on the things that the data would tell us just don't matter. Meanwhile there's quiet neglect of the things that do. And then a harder question nobody's really asking: who owns the budget for finding out the difference?

## The intelligence budget: the people v the tokens

Every commercial organisation invests in intelligence. They just don't call it that, and nobody owns it as a whole.

Data science teams, analytics platforms, econometric studies, agency reports, dashboards, forecasting tools, AI pilots. All of these exist to improve commercial decisions. But they sit in different budgets, report to different leaders, and get judged by different standards. The net result: no single person owns intelligence as a whole.

That wasn't a pressing question when intelligence meant hiring people who analyse things. Analysts sat inside functions and produced what those functions could afford. The budget was headcount. The governance was line management.

AI changes the economics. A Bayesian model that continuously updates commercial priors costs a fraction of a permanent analyst. An LLM that explains model outputs in plain language replaces hours of report writing. But the model often sits in a technology budget whereas the analyst sits in a headcount budget. Different owners, different approval processes, different expectations of return.

And the question most organisations aren't asking: for each commercial decision, who decides whether it should be supported by a person, a model, a test, or nothing at all? And where does the budget for that come from?

## Three questions every commercial leader should be asking

The same phrase from three different leaders means three different things. A CFO asking for "better data" wants forecasts they can defend. A CMO asking for "better data" wants to know which spend is working. A CIO asking for "better data" wants a cleaner architecture. While all three are reasonable, they're all different questions, and the budget that answers one won't answer the others.

In my experience when a commercial leader says "we need better data" or "we need to understand what's working," they're usually conflating three separate things.

The first is **where to invest in understanding**. The impact of many commercial decisions are assumed but never fully measured. A business *believes* promotions lift demand, or that price changes affect mix, or that brand spend improves conversion. Until those relationships have been isolated and validated, they're still assumptions. Closing those gaps requires hard analytical investment to run a test, build a model, fix a data pipeline. So not a marketing question or a technology question, but a question about how much the organisation is willing to invest in actually knowing how its commercial system works.

The second is **where to invest in commercial activity**. Once you understand the system better, you can decide which levers to pull, how hard, and in what sequence. Price, promotion, channel mix, investment allocation. This is where most organisations focus their attention and their money. The quality of these decisions depends entirely on the intelligence underneath them. Strong execution on weak understanding compounds error faster.

The third - and probably the newest question - is **where to invest in automation**. Which decisions could eventually be made by a system rather than a person? This is where most of the current AI excitement sits. But it's *downstream of the other two*. **You can't automate a decision you don't understand causally**.

Most organisations conflate these three, or only think seriously about the second. Plenty of spending on commercial activity. Minimal investment in understanding whether that activity works. And premature automation of decisions nobody properly understands.

Then there's a second problem underneath this. Even the spend that does get classified as "commercial activity" is often justified by benefits nobody has tested. Initiative A will deliver +3%. Initiative B will deliver +2%. Initiative C another +2%. Stacked together, the board's plan now shows a headline uplift that depends on projections nobody validated, then stacked together as if they don't overlap. When the numbers don't land, nobody can say which initiative actually worked, which was noise, and which was confounded by something else happening at the same time. The budget gets built on assumptions dressed up as commitments. Mark wrote about this in [The problem with business cases](/articles/the-problem-with-business-cases-and-a-better-way-to-make-decisions) and the pattern hasn't got any better since.

## Moving up the rungs of causation

Judea Pearl makes a critical distinction in his Ladder of Causation between **seeing, doing and imagining**. Most commercial analytics sits on the first rung: what correlates with what. To move up the ladder and begin to understand not just 'what happened when' but 'what caused it' we need a **causal map**.

A causal map of a commercial system does for business what those Premier League DAGs did for football. It shows which relationships are understood, which are guessed at, and which haven't been looked at.

That matters because each type implies a different investment.

Where relationships are understood, backed by validated evidence, they can potentially be monitored or automated. A Bayesian model can watch them continuously and flag when something shifts. Human judgment adds less marginal value here.

Where relationships are guessed at, where the organisation assumes a connection but hasn't tested it, the investment should go into understanding. Run an experiment. Build a model. Fix a broken data pipeline. Someone has to decide that's worth doing.

Where nobody has looked, the causal map surfaces the blind spot. Then the question is: do we invest in finding out, or accept the uncertainty and the risk that comes with it?

Once those relationships are visible, the organisation can prioritise intelligence investment the way it prioritises any other scarce resource, by asking "where does additional understanding create the most value?".

That question matters most where relationships are interconnected. A gap at a high-leverage node, one with many downstream dependencies, is worth more to close than a gap at the edge of the system. The causal map tells you which is which.

## The operating model underneath

Having identified that intelligence is a genuine enterprise resource, it needs to be addressed and actively managed by the **commercial operating model**.

AI creates a category of resource that doesn't fit existing budget structures. A Bayesian system that continuously updates beliefs isn't a technology cost in the traditional sense. It's doing work that previously required a person. An LLM that translates technical findings into language commercial teams can act on isn't a software licence. It's replacing a communication function.

Azeem Azhar has been making this point forcefully on his *Exponential View* podcast. His argument: organisations should start thinking about their token budget the way they think about their people budget. He goes as far as suggesting a rough 1:1 ratio, which feels punchy, but the underlying point holds. Tokens are a productive input, not an IT line item. Treating them as infrastructure cost to be rationalised away from the business misses what's actually happening: a category of work is moving from headcount to compute, and nobody is managing that allocation as a single decision.

So the question for commercial leaders needs to be: for each commercial decision, what's the right blend of human judgement and machine inference, and who is accountable for that allocation over time?

That allocation will change. As data improves, models get stronger, and trust builds, some decisions will move from **human-heavy** to **machine-heavy**. Others shouldn't, because they involve strategic judgment, ambiguity, or trade-offs that models can inform but not resolve. Leadership's role is to decide where that boundary sits and how it shifts.

Right now, nobody owns that decision. Technology teams buy tools. Commercial teams hire people. Data science teams build models. Finance approves each line separately. Nobody sits above all of it and asks: is this the right allocation of intelligence for the decisions we need to make?

![A two-by-two of commercial decisions, plotting how well a decision is understood against the volume and speed required. Expert judgement sits bottom left, strategic choices bottom right, the danger zone of automating assumptions top left, and automated monitoring top right.](/images/who-owns-the-intelligence-budget-quadrant.png)


## What this looks like in practice

Take marketing effectiveness and yield management. Most organisations have a view on each. A marketing mix model that tells them roughly what spend drives what revenue. A pricing or revenue management system that optimises yield. Both can be good in isolation. But they almost never share a view of how they affect each other. Promotional pricing pulls forward demand that the MMM then attributes to the campaign running at the same time. Marketing spend shifts price elasticity in ways the yield model doesn't see. Each team optimises their own model. Nobody owns the interaction between them.

That's exactly where a **causal intelligence layer** earns its keep. Not replacing either model. Sitting above them and asking: are these two views of the world consistent, and where are they conflicting?

*The broader practical version is simpler than it sounds.*

Start by mapping the commercial system causally. Not "what metrics do we track" but "what actually drives what, and how confident are we in each relationship?" That map shows immediately where intelligence is thin, fragmented, or missing. Some of those gaps are worth closing. Others aren't. The map helps you decide.

For each gap worth closing, the next question is what form the investment should take. A person dedicated to ongoing analysis. A model that monitors continuously. A one-off experiment to validate an assumption. Each has different costs, different timelines, different budget implications.

Over time, as understanding improves, some of those decisions can be progressively automated. Not because automation is the goal, but because certain relationships become well-understood enough that a system can manage them reliably, freeing human judgment for where it actually matters.

The sequence is: understanding before activity, activity before automation. Most organisations do it backwards. They spend heavily on commercial activity and AI tools while underinvesting in whether they understand the system those tools are supposed to optimise. The football equivalent: buying a striker because the goals column looks light, without understanding why the midfield isn't creating chances. The symptom gets treated. The cause doesn't.

## The leave behind question

Next time your data science director asks for more headcount, or your technology team proposes an AI pilot, or your agency recommends a new measurement framework, ask this:

Does this improve our understanding of how the commercial system actually works? Does it improve the quality of a specific decision? Or does it automate something we haven't properly understood yet?

If nobody can answer that clearly, you probably don't have an intelligence budget. You have intelligence spending, and in most organisations nobody owns the difference.


The questions leaders bring us, and what we find when we answer them properly.

Company Details

Registered Address

Cookie Preferences