Location Intelligence Retail Site Selection Automation

Detecting Overfitting in Site Suitability Weights

This page solves one exact task: determining whether a set of fitted criterion weights describes something real about how sites trade, or has memorised the particular stores it was fitted on.

The risk is structural rather than careless. A retail estate offers a few hundred observations at most, criteria are heavily correlated with one another, and the temptation to add “just one more” criterion is constant — which is a recipe for weights that reproduce the training stores beautifully and rank new sites no better than a coin.

Prerequisites

Configuration and execution parameters

Parameter Value for this task Notes
stores_per_criterion ≥ 20 The rule of thumb that prevents most of this
gap_threshold 0.15 in correlation In-sample minus holdout
learning_curve_points 5 sample sizes Shows whether more data would help
permutation_repeats 30 For importance stability
correlation_flag 0.8 between criteria Above this, one is redundant
regularisation ridge, tuned on the holdout The standard remedy
report gap, curve, importance spread All three; each catches a different case

The stores_per_criterion rule does more work than every diagnostic below it. An estate of a hundred stores supports about five criteria; fitting eight is asking a hundred observations to determine eight parameters plus their interactions, and the result will fit well and travel badly whatever the diagnostics later say.

The gap opens as criteria are added With three criteria the in-sample correlation is 0.61 and the holdout 0.57. At five it is 0.71 and 0.59. At eight it is 0.83 and 0.54, and at twelve it is 0.91 and 0.46 — the in-sample figure improving all the way while the holdout peaks at five and then declines. In-sample always improves; the holdout is where the truth is 0.95 0.70 0.45 in-sample holdout peak 3 criteria 5 8 12 A model reported on its in-sample figure gets better forever. The same model tested properly is worse at twelve criteria than at three — and it is the twelve-criterion version that usually reaches the committee.

Annotated implementation

python
from __future__ import annotations

import numpy as np
import pandas as pd
from sklearn.inspection import permutation_importance
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import GroupKFold, learning_curve

GAP_THRESHOLD = 0.15


def fit_and_gap(x: pd.DataFrame, y: pd.Series, groups: pd.Series) -> dict:
    """In-sample versus holdout correlation, with spatial grouping."""
    model = RidgeCV(alphas=np.logspace(-3, 3, 25))
    splitter = GroupKFold(n_splits=5)

    in_sample, held_out = [], []
    for train, test in splitter.split(x, y, groups=groups):
        model.fit(x.iloc[train], y.iloc[train])
        in_sample.append(np.corrcoef(model.predict(x.iloc[train]), y.iloc[train])[0, 1])
        held_out.append(np.corrcoef(model.predict(x.iloc[test]), y.iloc[test])[0, 1])

    gap = float(np.mean(in_sample) - np.mean(held_out))
    return {
        "in_sample": round(float(np.mean(in_sample)), 3),
        "holdout": round(float(np.mean(held_out)), 3),
        "gap": round(gap, 3),
        # A gap this size means the weights describe these stores, not the world.
        "overfitting_suspected": gap > GAP_THRESHOLD,
        "alpha": float(model.alpha_),
    }


def importance_stability(x: pd.DataFrame, y: pd.Series,
                         groups: pd.Series, repeats: int = 30) -> pd.DataFrame:
    """Permutation importance with its spread — a criterion whose importance
    swings wildly between folds is not a criterion the model relies on."""
    model = RidgeCV(alphas=np.logspace(-3, 3, 25))
    splitter = GroupKFold(n_splits=5)
    frames = []
    for train, test in splitter.split(x, y, groups=groups):
        model.fit(x.iloc[train], y.iloc[train])
        result = permutation_importance(model, x.iloc[test], y.iloc[test],
                                        n_repeats=repeats, random_state=0)
        frames.append(pd.Series(result.importances_mean, index=x.columns))

    imp = pd.concat(frames, axis=1)
    return pd.DataFrame({
        "mean_importance": imp.mean(axis=1).round(4),
        "fold_spread": (imp.max(axis=1) - imp.min(axis=1)).round(4),
        "sign_flips": (imp > 0).sum(axis=1).sub(len(frames) / 2).abs().lt(len(frames) / 2).astype(int),
    }).sort_values("mean_importance", ascending=False)

The sign_flips column is the quiet one. A criterion whose measured importance is positive in three folds and negative in two is not contributing a stable signal, however respectable its average looks — and it is exactly the criterion whose weight a stakeholder will defend most vigorously.

Failure modes and debugging

Correlated criteria splitting a weight. Two criteria correlated at 0.9 share the signal between them arbitrarily, so their individual weights are unstable across folds even when the pair’s joint contribution is solid. The diagnostic looks like overfitting and the remedy is different: combine them, drop one, or regularise and stop interpreting the individual coefficients.

Selecting criteria on the full dataset. Choosing which criteria to include by looking at their correlation with the outcome, then cross-validating the chosen set, leaks the outcome into the selection. The whole pipeline — selection included — has to sit inside the fold, which usually reveals that the “obvious” criterion set varies between folds.

A holdout that is not held out. Tuning the regularisation strength on the same holdout used to report the error makes that error optimistic. A nested arrangement — an inner loop for tuning, an outer for reporting — is the correct structure, and with small estates the simpler alternative is to fix the regularisation from theory and keep the holdout clean.

Concluding “not overfitted” from a small gap on a random split. As covered in the cross-validation guide, a random split of a spatial dataset leaks, so a small gap may just mean the folds were not independent. Spatial grouping first, then interpret the gap.

Mean importance is not enough — the spread decides Reachable population has a mean importance of 0.31 with a fold spread of 0.06; demographic fit 0.18 with spread 0.05; competition 0.11 with spread 0.09; rent 0.04 with spread 0.14 and two sign flips; co-tenancy 0.03 with spread 0.16 and three sign flips. The last two are noise wearing a coefficient. Two criteria are carrying the model; two are carrying nothing criterion mean importance fold spread sign flips reachable population 0.31 0.06 0 demographic fit 0.18 0.05 0 competition 0.11 0.09 0 rent 0.04 0.14 2 co-tenancy 0.03 0.16 3 Dropping the last two costs 0.01 of holdout correlation and removes a third of the model's parameters.

Verification

  • Run a learning curve. If holdout performance is still rising with sample size, the model is data-limited rather than overfitted, and the remedy is more stores rather than fewer criteria.
  • Permute the outcome. Fit the same pipeline against a shuffled outcome; any holdout correlation materially above zero means the pipeline is leaking.
  • Refit with the two weakest criteria removed and compare holdout performance. A model that does not get worse has just become simpler and more defensible.
  • Check the criteria correlation matrix before interpreting any individual weight.
Would more stores help, or fewer criteria? Holdout correlation rises from 0.41 at fifty stores to 0.52 at a hundred, 0.58 at a hundred and fifty, and 0.59 at two hundred, where it flattens. The flattening says the model is no longer data-limited, so further improvement has to come from better criteria rather than more observations. The curve flattens, so more stores will not rescue it 0.62 0.38 50 stores 100 150 200 A curve still climbing at the right edge means waiting for more stores will improve the model; one that has flattened means the next gain has to come from a better measurement, not a bigger sample.

Frequently Asked Questions

Is regularisation enough on its own?

It helps and it does not substitute for judgement about the criterion set. Ridge regularisation shrinks correlated coefficients toward each other and stabilises the fit, which improves holdout performance and makes the individual weights less interpretable — a trade that is fine for prediction and awkward when a committee wants to know why rent is weighted as it is. Combining modest regularisation with a deliberately small criterion set gives both stability and an explanation.

What if the estate is genuinely too small to fit weights?

Then do not fit them. Expert weights from a structured process, validated for rank agreement against the stores that do exist, are more defensible than a regression on forty observations — and they can be revisited annually as the estate grows. A model that admits its weights are judgements is stronger than one that presents forty stores’ noise as evidence.

Does adding a criterion always risk overfitting?

Adding a criterion that measures something genuinely new, and that theory says should matter, is usually worth the parameter — especially if it replaces rather than joins an existing one. Adding a fourth measure of affluence to a model that already has three is spending a parameter on a signal already present, which is where the risk concentrates. The correlation matrix answers this in seconds and is rarely consulted.

How often should this diagnostic run?

Whenever the weights are refitted, and once between refits as the estate grows. The second run matters because a criterion set that was appropriate for a hundred stores may be under-specified for two hundred — overfitting is a relationship between model complexity and sample size, and one side of that relationship changes every time a store opens.

Does this apply to expert-assigned weights too?

Yes, in a different form. Expert weights cannot overfit a dataset they were never fitted to, but they can overfit an organisation’s recent experience — the weight on competition rises after a bad opening next to a discounter and never comes back down. Running the same holdout evaluation on expert weights measures whether that accumulated judgement predicts anything, and it occasionally reveals that a carefully negotiated weight vector performs no better than equal weights.

What is the single most useful diagnostic if only one can be run?

The gap between in-sample and holdout performance under a spatial split. It is one number, it requires no additional data, and it catches the failure that matters most — a model that describes the estate rather than the world. The learning curve and the importance stability tell you what to do about it, but the gap tells you whether there is anything to do.

What should be reported to the committee about overfitting?

One sentence, in the model card: how many criteria the model uses, how many stores it was fitted on, and the gap between its in-sample and held-out accuracy. Committees do not need the learning curve, and they are entirely capable of understanding that eight parameters fitted on a hundred observations deserves more caution than four fitted on three hundred. Stating it also pre-empts the more damaging conversation that happens when somebody discovers the ratio independently.

Is there a rule of thumb worth remembering?

Twenty stores per criterion, and treat any model that breaks it as provisional until a holdout says otherwise. It is crude, it ignores correlation structure, and it will keep most teams out of trouble — which is more than can be said for the usual alternative of adding criteria until the in-sample fit looks convincing.

← Back to Validating & Backtesting Site Selection Models