Location Intelligence Retail Site Selection Automation

Tracking Model Decay Across Refresh Cycles

This page solves one exact task: measuring a site-selection model’s accuracy repeatedly over time, in a way that distinguishes a model whose inputs have drifted from one whose relationship to the world has genuinely changed.

Site models decay slowly and invisibly. Nothing breaks, no gate fires, and the ranking continues to look reasonable — while the competitive landscape, the format’s customer base and the demographic layers underneath it all move. The failure is noticed when a store underperforms badly enough to prompt a question, by which point the model has been quietly wrong for two years.

Prerequisites

  • A validation history: metrics from every quarterly run, dated, as produced by validating and backtesting site selection models.
  • Input version records for every scored run — graph build, demographic vintage, competitor extract, weight version.
  • Python packages: pandas, numpy. Install with pip install pandas numpy.

Configuration and execution parameters

Parameter Value for this task Notes
cadence quarterly Matches the pace at which stores mature
tracked_metrics rank correlation, decile lift, signed bias Three, not one
segment_by format, market density Decay is rarely uniform
input_drift_metrics criterion distributions per run Separates drift from decay
alert_rule 3 consecutive declines, or a step Not a single quarter
refit_trigger documented threshold Decided in advance, not in the moment
retain_history indefinitely The series is the asset

Alerting on three consecutive declines rather than one is what keeps the signal usable. Quarterly metrics computed on a handful of newly matured stores are noisy, and a rule that fires on any decline produces an alert most quarters and is ignored within a year.

Gradual decay and a step change look completely different Gradual decay shows rank correlation drifting from 0.61 to 0.52 over eight quarters as markets slowly change. A step change shows it holding near 0.60 for five quarters and then dropping to 0.44 in one — which is almost always an input change rather than the world moving. A slope is the market; a cliff is your pipeline 0.65 0.40 gradual decay step change Q1 Q8 The step lands in the quarter a new demographic vintage was adopted — which the input version record confirms in seconds and which no amount of model retuning would have fixed. Rank correlation between archived score and matured outcome, computed each quarter on all matured stores.

Annotated implementation

python
from __future__ import annotations

import numpy as np
import pandas as pd

DECLINE_QUARTERS = 3
STEP_THRESHOLD = 0.08


def decay_report(history: pd.DataFrame, metric: str = "spearman") -> pd.DataFrame:
    """Per segment, classify the recent trajectory of a validation metric.

    history: quarter, segment, metric columns, plus the input version columns.
    """
    out = []
    for segment, group in history.sort_values("quarter").groupby("segment"):
        series = group[metric].to_numpy()
        if len(series) < 4:
            out.append({"segment": segment, "verdict": "insufficient history"})
            continue

        diffs = np.diff(series)
        consecutive = int(np.all(diffs[-DECLINE_QUARTERS:] < 0))
        step = float(diffs[-1])
        # A single large drop is a step; several small ones are decay.
        verdict = ("step change" if step <= -STEP_THRESHOLD
                   else "decaying" if consecutive
                   else "stable")
        out.append({
            "segment": segment,
            "latest": round(float(series[-1]), 3),
            "change_4q": round(float(series[-1] - series[-4]), 3),
            "verdict": verdict,
        })
    return pd.DataFrame(out)


def attribute_step(history: pd.DataFrame, quarter: str,
                   version_cols: list[str]) -> pd.DataFrame:
    """What changed in the quarter a step occurred? Usually exactly one thing."""
    window = history.loc[history["quarter"].isin([quarter, _previous(quarter)])]
    changed = {}
    for col in version_cols:
        values = window[col].dropna().unique()
        changed[col] = "changed" if len(values) > 1 else "unchanged"
    return pd.DataFrame([changed], index=[quarter])


def _previous(quarter: str) -> str:
    year, q = quarter.split("Q")
    return f"{year}Q{int(q) - 1}" if q != "1" else f"{int(year) - 1}Q4"

Attributing a step to an input change takes seconds when the versions are recorded and is close to impossible when they are not — which is the practical argument for the versioning discipline applied throughout this pipeline.

Failure modes and debugging

Reading noise as decay. Quarterly metrics computed on ten newly matured stores move substantially by chance. Reporting the number of stores alongside every metric, and requiring a consecutive-decline pattern before alerting, keeps attention on real movement.

A metric that improves while the model degrades. As an estate matures, its stores become more homogeneous, and a rank correlation computed over a narrower range of sites can rise even as the model’s ability to distinguish genuinely different candidates falls. Tracking decile lift alongside correlation catches this, since lift is less sensitive to the range of the sample.

Mistaking a segment failure for a general one. Aggregate decay usually starts in one segment — a format, a market type — and the aggregate figure is a weighted average that hides it for several quarters. Segmenting the report is the fix, and it also makes the finding actionable when it arrives.

Refitting on every decline. A model refitted whenever its metric falls tracks noise, and its weights become a moving average of recent disappointments. Deciding the refit threshold in advance, and requiring the decay verdict rather than a single reading, keeps refits deliberate.

The aggregate hid a segment that stopped working Over eight quarters the aggregate rank correlation fell from 0.60 to 0.55, which reads as mild noise. Split by format, supermarkets held at 0.63 and convenience fell from 0.56 to 0.38 — a collapse that the aggregate diluted for six quarters. One segment failed; the average called it noise segment Q1 Q8 change verdict aggregate 0.60 0.55 −0.05 stable-ish supermarket 0.62 0.63 +0.01 stable convenience 0.56 0.38 −0.18 decaying Convenience is the format where a discounter entered three markets over the period — a competitive change the model's competitor weighting never absorbed. Segmented reporting turns "the model is drifting" into "the convenience competitor term needs refitting", which is a week of work rather than a programme.

Verification

  • Plot the series with the input version changes marked. Most steps line up with a marker, and the ones that do not are the genuinely interesting findings.
  • Confirm the store counts behind each point. A metric computed on fewer than ten stores should be shown but not alerted on.
  • Re-compute an old quarter’s metric from the archive and confirm it matches the recorded value. If it does not, something in the metric pipeline has changed and the series is not comparable along its length.
  • Test the alert rule against a synthetic series with an injected decline.
Annotating the series is most of the diagnosis The rank-correlation series is marked with three input events: a new demographic vintage in quarter three, a weight refit in quarter five, and a competitor extract change in quarter six. The only visible step coincides with the demographic vintage, and the refit produced no measurable improvement. Mark the events and most questions answer themselves new ACS vintage weight refit competitor extract Q1 Q7 The refit at Q5 moved nothing, which is worth knowing: the model was not mis-weighted, it was being fed a different demographic base — and the correct response was a re-baseline, not a retune.

Frequently Asked Questions

How fast do site models actually decay?

Slowly where markets are stable and abruptly where they are not. A model in a mature market with little competitive entry can hold its accuracy for years; the same model in a market where a discounter is expanding will degrade within a handful of quarters, because the competitive term it was fitted with no longer describes the field. That variation is the argument for segmenting the tracking by market as well as by format.

Should decay trigger a refit or a rebuild?

Decay in one segment usually means a refit; decay everywhere usually means something structural. A pattern where every segment declines together and no input version changed points at the outcome measure or the estate rather than at the weights — for instance a chain that has shifted its opening strategy toward a site type the model was never fitted on.

Can decay be predicted rather than detected?

Partially. Input drift can be monitored directly — the distribution of criterion values across scored candidates, quarter over quarter — and a large shift there usually precedes a decline in accuracy. That gives a leading indicator on the input side, which is worth having. What cannot be predicted is a change in how shoppers behave, and that is what the outcome-based tracking exists for.

Who should see this report?

The modelling team quarterly and the investment committee annually, in different forms. The team needs the segmented series with input markers; the committee needs one line about whether the model’s accuracy is holding, and the honest answer when it is not. A committee that learns of a two-year decline at the same time as a disappointing store has been badly served by the reporting rather than by the model.

Should the tracking include models that are no longer in use?

Keeping a retired model’s series running for a year is surprisingly informative, because it shows whether the replacement actually improved anything. Comparing the two series across the same quarters, on the same stores, is the only clean test of a model change — and it frequently shows that a refit which looked better in development performs identically in the field, which is worth knowing before the next one.

How does decay tracking relate to input drift monitoring?

They watch the two halves of the same failure. Drift monitoring sees the inputs move; decay tracking sees the model’s accuracy fall. Drift usually precedes decay by a quarter or two, which makes it the leading indicator, but drift without decay is common and harmless — an input can shift substantially without changing the ranking. Watching both, and correlating them, is what distinguishes a change that matters from one that merely happened.

What is the first thing to build?

The dated table. A single row per quarter per segment, holding the metric, the store count and the input versions, is the entire foundation — every chart, alert and attribution in this page is a query over it. Teams often begin instead with a dashboard, which produces a live view of a number nobody has any history for, and the history is the part that makes decay visible at all.

How does this fit into a quarterly rhythm?

Comfortably. The validation job runs, the decay report is generated, the modelling team reads the segmented series with the input markers, and the model card is regenerated from the result. That is an afternoon a quarter and it produces the one thing a site-selection programme otherwise lacks: a documented, dated record of whether the model is still working.

What is the most common mistake in decay tracking?

Changing the metric definition mid-series. A team refines its outcome measure, adopts a better market index or switches from correlation to lift, and the series before and after are no longer comparable — which destroys exactly the property that made it valuable. When a definition genuinely has to change, recompute the history under the new definition and keep both series, rather than splicing them and hoping nobody looks at the join.

← Back to Validating & Backtesting Site Selection Models