Validating & Backtesting Site Selection Models
A suitability model produces a ranking whether or not it works. This section is about establishing whether it does: designing holdouts, backtesting scores against realized store performance, detecting the overfitting that makes a model look excellent and travel badly, tracking decay as markets move, and recording assumptions in a form an investment committee can interrogate.
It is the stage that separates a model from an opinion with arithmetic attached. Everything upstream — the catchments, the demographic joins, the weighted scores — is machinery for producing a number. Validation asks the only question that matters about that number: when it said a site would trade well, did it?
Concept: Three Different Questions Called “Validation”
Teams use one word for three exercises with different designs and different answers, and conflating them is the commonest reason a validation effort produces reassurance rather than information.
Does the model reproduce what we already know? Fitting the score against stores currently trading and checking the correlation. This is the weakest test — the stores exist because previous models and previous judgement approved them, so the sample is selected on the outcome — and it is the one most often presented as proof.
Does the model predict stores it did not see? Holding out stores from the fit and predicting their performance. Considerably stronger, and it requires discipline about what “did not see” means: a store held out of the regression but used to choose the criteria is not held out in any meaningful sense.
Does the model predict the future? Scoring sites before they open and comparing against what they actually did. The only test that answers the business question, and the slowest — it takes as long as a store takes to mature.
A serious programme runs all three, weights them in that order, and never reports the first alone.
The Survivorship Problem
Every store in an estate was approved. That single fact undermines naive validation more than any modelling error, because the sample contains no examples of the sites the model would have rejected and that would have traded well, and few of the sites that were approved against the model’s advice.
Three partial remedies are available and worth combining. Rejected candidates that a competitor subsequently opened on provide a natural experiment: the model scored them, somebody else built there, and their performance is often observable through a mobility panel. Closed stores are the estate’s own negative examples, and excluding them — which happens by default when the analysis starts from the current estate — throws away the most informative rows in the dataset. And acquired estates, where a chain has taken on stores it did not choose, are the closest thing to an unselected sample most retailers ever have.
None of these fully solves the problem, and stating it plainly is part of the validation. A model validated only on stores it or its predecessors approved should be described as such, because the alternative is a confidence interval that is narrower than the evidence supports.
Metrics: What to Measure Against What
| Metric | What it answers | When to prefer it |
|---|---|---|
| Rank correlation | Does the model order sites correctly? | The default; ranking is the deliverable |
| Decile lift | Do the top-scored sites outperform? | Communicating to a committee |
| Mean absolute percentage error | How wrong is a forecast? | When the output is a sales number |
| Hit rate at the cut | Of the sites we would have opened, how many worked? | Directly matches the decision |
| Cost of errors | What did the mistakes cost? | When error costs are asymmetric |
The last row is the one that changes behaviour. A model whose errors are symmetric in rank can still be badly wrong in money if its misses are concentrated among expensive sites — and a chain that opens twenty stores a year cares far more about avoiding two failures than about ordering the middle of the list correctly.
Architecture: Validation as a Pipeline Stage
Validation belongs in the pipeline rather than in a notebook, for the same reason every other check does: an exercise that runs when somebody remembers is an exercise that stops running. The shape is straightforward.
A scored-run archive keeps every scoring run with its inputs, weights and versions — which the versioning discipline already provides. An outcomes table records what each opened store actually did, joined on the site identifier the score used. A validation job runs quarterly, computing the metrics above across every design, and writes them to a small, dated table. And a model card — the assumptions register described in documenting model assumptions for investment committees — is regenerated from that table so the published description of the model’s accuracy is never older than the last measurement.
The single most valuable property of this arrangement is that the score is archived before the outcome is known. A prospective validation is impossible to reconstruct after the fact, because nobody can prove what the model would have said before it saw the answer.
Designing a Holdout That Actually Holds
Every validation design comes down to one question: what does the model not know when it makes the prediction being scored? Getting that wrong is the most common way a validation programme produces a number that nobody should have believed, and there are four distinct leaks worth naming.
Spatial leakage. Two stores in the same market share origin zones, competitors and often customers, so a model fitted on one has effectively seen the other’s catchment. A random store-level split therefore reports an error much lower than the model will achieve on a site in a market it has never scored. The remedy is to hold out whole spatial blocks larger than any modelled catchment, which is the design used for cross-validating choice-model parameters and applies equally to the composite score.
Temporal leakage. A model fitted on stores that opened across a decade and validated on a random subset of them has been told about the future: it knows how the market developed. Splitting on opening date instead — fit on everything before a cutoff, test on everything after — reproduces the situation the model is actually used in, and it is the only split that tests whether the relationship it found is stable over time.
Selection leakage. The criteria themselves were chosen by people who had seen the estate perform. Even with a perfect split, a criterion set selected on the full dataset carries information about the holdout, which is why serious designs put the criterion selection inside the fold and report how much the selected set varies between folds. In practice it varies more than anyone expects, and that variability is itself a finding about how well-determined the model is.
Outcome leakage. The subtlest of the four. A criterion that is partly a consequence of performance rather than a cause of it — a store’s staffing level, its refit history, its opening hours — will predict sales beautifully and cannot be known for a candidate site. Auditing the criterion list for anything that would not exist before the store opened is a five-minute exercise that occasionally invalidates a whole model.
A design that addresses all four will report a lower number than one that addresses none. That is the point: the lower number is the one that will be reproduced when the model is used, and the gap between the two is a measure of how much a naive validation would have overstated.
What to Do When Validation Says the Model Is Weak
A validation programme that only ever confirms the model is not a validation programme, so it is worth deciding in advance what happens when the answer is disappointing. Four responses cover almost every case, and choosing between them is a judgement about where the weakness sits rather than how large it is.
Narrow the claim. A model that ranks suburban supermarkets well and city-centre convenience poorly is not a broken model — it is a model with a stated scope. Publishing the segmented accuracy and restricting the model’s use to the segments where it performs is both honest and immediately actionable, and it usually preserves most of the value while removing most of the risk.
Improve an input. A large share of apparent model weakness is measurement weakness upstream: a competitor set missing a discounter, a catchment computed on a stale road network, a demographic layer whose vintage predates a market’s growth. The error decomposition is what distinguishes this case, and when it applies, no amount of refitting will help — the model is faithfully reproducing a wrong input.
Simplify. Where the gap between in-sample and holdout performance is large, the answer is usually fewer criteria rather than better ones. A model with three well-measured criteria that generalises beats one with eight that memorises, and the simplification also makes the weights defensible to a committee, which has its own value.
Change the target. Occasionally the model is predicting the wrong thing. A score fitted against gross sales in a chain that opens infill stores is being asked to predict a number the business does not actually want to maximise, and refitting against net new sales — after transfer — changes both the weights and the ranking. This is the rarest of the four responses and the one with the largest effect when it applies.
What should not happen is the fifth response, which is to keep the model and quietly stop measuring it. That is the outcome a validation programme exists to prevent, and it is reached not by decision but by the report becoming inconvenient and then optional.
Frequently Asked Questions
How long before a new store’s performance can be used?
Twelve to eighteen months for most formats. The first quarter is an opening surge, the second is still distorted by launch marketing, and seasonality means a partial year cannot be compared against a full one. Using early performance because it is available is the most common way a backtest ends up measuring opening promotions rather than site quality — and it systematically favours sites in dense areas, where the surge is largest.
What if the estate is too small for a holdout?
Then borrow across time rather than across stores. A model fitted on stores opened before a cutoff and tested on those opened after gives a temporal holdout that works with far fewer observations than a random split, and it happens to test the property that matters — whether the model transfers forward. With very small estates, comparing against an analogue-based forecast rather than a fitted model is often the more honest design.
Should the model be re-fitted every time it is validated?
No. Validation and fitting are separate acts, and interleaving them destroys the holdout: a model re-fitted after seeing its holdout error has seen the holdout. Fit on a schedule, validate on a schedule, and keep the two apart with the discipline that the fitted parameters are frozen artifacts rather than something regenerated on demand.
What does a failed validation look like in practice?
Rarely a model that is wrong everywhere; usually one that has stopped working for a segment. The pattern is a decile chart that is still monotonic overall and flat across the top three deciles, or a rank correlation that holds nationally and has collapsed in one market. Reporting the metrics by segment as well as in aggregate is what makes that visible, and it converts “the model is degrading” into “the model no longer separates urban infill sites”, which is actionable.
Who should own validation?
Somebody who does not own the model. This is the one place where organisational separation earns its cost: a team that builds a scoring model and also reports its accuracy will, without any bad intent, choose designs that flatter it. An analyst or a finance partner running the validation against an archived scored run — with no ability to change the model — produces a number the committee can rely on and the modelling team can learn from.
What does a minimal validation programme look like?
Three things, and none of them requires a data-science team. Archive every scored run with its inputs before decisions are made. Record what each opened store did once it matured, joined on the same identifier. And once a quarter, compute a rank correlation and a decile table over the matured stores, segmented by format, and write the two numbers into a dated table. That is a day of engineering to set up and an afternoon a quarter to run, and it is the difference between a model whose accuracy is known and one whose accuracy is asserted.
Everything else in this section — spatial holdouts, overfitting diagnostics, error decomposition, decay tracking — is an elaboration on those three, added when the estate is large enough to support it. The order matters, though: teams that begin with sophisticated cross-validation on a model whose scores were never archived have built the roof before the foundation, and the resulting number describes a reconstruction rather than a decision.
How does validation change the relationship with the committee?
More than the accuracy figure itself does. A model presented without measured accuracy is argued about on intuition, and the argument is unwinnable in both directions — the modelling team cannot prove it works and the sceptics cannot prove it does not. A model that arrives with a stated design, a measured correlation, a decile table and a named list of segments where it is weak converts that argument into a series of specific, answerable questions. The result is usually that the model is trusted more in the places it deserves and less in the places it does not, which is precisely the outcome a validation programme is for.
Where should a team start if none of this exists today?
With the archive, before anything else. Recording every scored run — the site, the score, the criterion values, the input versions and the date — costs one table and a few lines in the scoring job, and it is the prerequisite for every measurement in this section. A team that starts archiving today can run its first honest prospective validation in eighteen months; a team that waits until it wants the answer will find the evidence was never captured, and reconstructing it is not possible.
The second step is the outcomes table, joined on the same identifier, and the third is a quarterly job that computes two metrics into a dated row. Those three together are a validation programme; everything else is refinement.
Conclusion
A site-selection model earns trust by being wrong in public and improving. Archive every scored run before outcomes exist, keep closed stores and rejected candidates in the dataset, report the design alongside every accuracy figure, prefer prospective tests over in-sample fits, and segment the metrics so degradation is visible before it costs a store. The result is not a model that is always right — no site model is — but one whose errors are known, bounded and shrinking, which is the only honest basis for spending capital on its recommendations.
Related
- Backtesting Site Scores Against Realized Store Sales — the core exercise.
- Detecting Overfitting in Site Suitability Weights — the gap between the first two designs.
- Tracking Model Decay Across Refresh Cycles — noticing degradation before it costs a store.
- Documenting Model Assumptions for Investment Committees — the register a committee can interrogate.