Location Intelligence Retail Site Selection Automation

Measuring Forecast Error After a Store Opens

This page solves one exact task: taking the sales forecast a site was given before approval, comparing it against what the store achieved once mature, and decomposing the difference into the parts of the model that produced it.

The decomposition is what makes this worth doing. Knowing a forecast missed by eighteen per cent is mildly interesting; knowing that the catchment population was right, the capture rate was right and the basket assumption was wrong tells the modelling team exactly what to fix and tells the committee that the site was not misjudged.

Prerequisites

  • The archived forecast with its components: reachable population, expected capture rate, expected basket or spend per visit, and the assumptions applied.
  • Realized performance at maturity, per backtesting site scores.
  • Observed catchment evidence — a mobility panel or loyalty origins — so the catchment component can be measured rather than inferred.
  • Python packages: pandas, numpy. Install with pip install pandas numpy.

Configuration and execution parameters

Parameter Value for this task Notes
maturity_months 18 Same definition as the backtest
components catchment, capture, basket, market Multiplicative decomposition
market_index applied Separates the model’s error from the economy’s
attribution_order fixed, documented Order changes the split; pick one and keep it
report_sign over or under, kept separate Systematic bias hides in an absolute mean
review_threshold ±20% Beyond this, a written post-mortem
feedback to the model card, quarterly Errors that do not reach the model are lost

Keeping the sign is more important than it sounds. A model that misses by fifteen per cent in both directions is noisy; one that misses by fifteen per cent upward on every store is systematically optimistic, and the two demand completely different responses. Reporting mean absolute error alone makes them indistinguishable.

A 22 per cent miss, attributed A store forecast at 5.4 million achieved 4.2, a miss of 22 per cent. The catchment population was 4 per cent lower than modelled, the capture rate 9 per cent lower, the basket 6 per cent lower and the market 3 per cent weaker — with capture the largest single contributor and the one the model can act on. The miss has four owners, and only two are the model's component forecast observed contribution to the miss catchment population 61,000 58,600 −4% capture rate 14.0% 12.7% −9% — the biggest piece basket / spend per visit £31.40 £29.60 −6% market movement index 1.00 index 0.97 −3% — not the model total £5.4M £4.2M −22% Capture is where the work is: the model expected this site to convert more of its catchment than it did, which usually means an underweighted competitor or an access problem the geometry did not see.

Annotated implementation

python
from __future__ import annotations

import numpy as np
import pandas as pd

COMPONENTS = ["catchment", "capture", "basket", "market"]


def decompose(forecast: pd.Series, observed: pd.Series) -> pd.Series:
    """Multiplicative decomposition, applied in a FIXED order.

    Sales = catchment × capture × basket × market. Swapping one factor at a
    time from forecast to observed attributes the change to that factor; the
    order matters when factors interact, so it is fixed and documented rather
    than chosen per store."""
    values = forecast[COMPONENTS].astype(float).to_numpy()
    target = observed[COMPONENTS].astype(float).to_numpy()

    contributions = {}
    running = values.copy()
    baseline = float(np.prod(running))
    for i, name in enumerate(COMPONENTS):
        running[i] = target[i]
        after = float(np.prod(running))
        contributions[name] = (after - baseline) / float(np.prod(values))
        baseline = after

    contributions["total"] = (float(np.prod(target)) - float(np.prod(values))) / float(np.prod(values))
    return pd.Series(contributions).round(4)


def portfolio_error(store_errors: pd.DataFrame) -> pd.DataFrame:
    """Bias and dispersion, per format — both, and never only the absolute."""
    return (store_errors.groupby("format")
            .agg(stores=("total", "size"),
                 mean_signed=("total", "mean"),
                 median_signed=("total", "median"),
                 mean_absolute=("total", lambda s: float(s.abs().mean())),
                 within_10pct=("total", lambda s: float((s.abs() < 0.10).mean())))
            .round(3).reset_index())

The fixed attribution order deserves the comment it gets in the code. When factors interact, the amount attributed to each depends on the order in which they are swapped, so a team that varies the order — or lets each analyst choose — will produce decompositions that cannot be compared across stores.

Failure modes and debugging

Comparing against a forecast that was revised. Sites are often re-forecast during negotiation, and comparing against the final, lower number flatters the model. Archiving the forecast at the point of the investment decision, and comparing against that one, is the only version that answers the question the committee asked.

Attributing market movement to the model. A store that opened into a downturn missed its forecast for reasons no site model could have known, and correcting the model for it teaches it to distrust perfectly good sites. The market index is what separates the two, and it should be a published index rather than the chain’s own performance, which is partly caused by the same decisions.

Only reviewing the misses. Post-mortems triggered by underperformance produce a systematically biased learning set: the model is corrected downward every time it is optimistic and never upward when it is pessimistic. Reviewing the largest errors in both directions keeps the correction balanced.

Feeding every error straight back into the weights. A single store’s miss is one observation, and adjusting weights in response is how a model acquires a coefficient for last quarter’s disappointment. Errors accumulate into the validation record and inform the scheduled refit; they do not trigger one.

A cohort that is systematically optimistic Across twenty-four stores opened in one year, nineteen underperformed their forecast and five exceeded it, with a mean signed error of minus eleven per cent and a mean absolute error of fourteen. The absolute figure alone would suggest ordinary noise; the sign shows a consistent bias. Nineteen of twenty-four missed low — that is not noise forecast −30% +12% Mean signed error −11%, mean absolute 14%. Reporting only the second would describe this cohort as reasonably accurate rather than as consistently optimistic by a tenth. Each bar is one store's signed error against its decision-point forecast.

Verification

  • Confirm the forecast archive predates the decision. A forecast recorded after approval is not a forecast.
  • Check the decomposition multiplies back to the total error within rounding. If it does not, a component is missing or double-counted.
  • Compare the observed catchment against the modelled one directly, rather than inferring it from the residual. Where a panel exists, this converts an assumption into a measurement.
  • Test the bias detection by simulating a cohort with a known systematic error and confirming the report identifies it.
Which component fails, by format For supermarkets the largest error component is basket, at 6 per cent; for convenience it is capture, at 11; for city-centre stores it is catchment, at 14, because the drive-time model describes the wrong population. Each format's weakest component points at a different fix. Different formats fail in different places format largest error component size what it points at supermarket basket 6% spend assumptions convenience capture 11% competitor set city centre catchment 14% wrong travel mode The city-centre row is a specific, fixable finding: a car isochrone is describing a population that mostly arrives on foot or by transit, which is a modelling choice rather than an unlucky cohort. Without decomposition all three formats look like "the forecast was out by about ten per cent".

Frequently Asked Questions

How many openings are needed before the error report means anything?

Around twenty for a signed-bias conclusion and considerably more for anything per format. With fewer, the honest report is the individual decompositions rather than a statistic — five stores with detailed attributions are more useful to a modelling team than a mean computed from five numbers. Estates that open a handful of stores a year should expect this to be a rolling three-year view rather than an annual one.

Should the review include sites that were rejected?

Where they were subsequently developed by somebody else, yes, and they are among the most valuable observations available. A site the model scored highly, the committee declined, and a competitor opened successfully is direct evidence about a decision rule rather than about the model — which is a conversation worth having separately from the modelling one.

What if the store underperformed for an operational reason?

Record it and exclude it from the modelling feedback, with the reason. A store that lost its anchor tenant, suffered eighteen months of roadworks or opened with a staffing problem did not test the site model, and folding its miss into the error distribution teaches the model something false. The exclusions themselves should be visible and countable, because a review process that excludes half its misses is not a review process.

How does this connect to the model card?

Directly: the error distribution from this exercise is the accuracy claim the card publishes. Regenerating the card from the validation table rather than editing it by hand means the published accuracy is always the measured one, and it removes the situation where a model card describes a performance the model achieved two years ago.

Should the error review be shared with the property team?

Yes, and it usually improves the input data more than the model. Property teams hold context the pipeline cannot see — a delayed anchor tenant, an access change during construction, a competitor that opened two months before — and the decomposition gives them a specific question rather than a general one. In practice these reviews are also where the most useful additions to the criterion set originate, because the property team can name the factor that explained the miss.

How does this differ from an ordinary variance analysis?

An ordinary variance analysis explains a store’s performance against its plan; this explains it against the model’s forecast, and attributes the difference to model components rather than to trading decisions. The two are complementary and answer to different people: the variance analysis belongs to the operator, and the decomposition belongs to whoever will use the model to approve the next site.

What is the right cadence for the review?

Quarterly for the aggregate report and per store as each one matures. Batching the individual reviews into an annual exercise loses the detail — nobody remembers eighteen months later why a particular store underperformed — while reviewing the portfolio statistics more often than quarterly produces movement that is mostly sampling. Running the two on different clocks is more work to schedule and considerably more informative.

Should the decomposition drive the analogue set as well as the model?

It should. If capture is the component that consistently misses, the analogue matching features probably under-represent competition; if basket misses, they under-represent the demographic mix. The decomposition is therefore a maintenance signal for both the fitted model and the analogue baseline, and updating only the first is how the baseline quietly stops being a fair comparison.

Does the review need a formal format?

A short structured record beats a written narrative: the forecast, the outcome, the four component contributions, an exclusion flag where one applies, and two or three sentences of context from whoever knows the store. Structure makes the reviews aggregatable, which is what turns twenty individual post-mortems into a portfolio finding — and the free-text field is where the reason that eventually becomes a new criterion tends to appear first.

← Back to Validating & Backtesting Site Selection Models