Location Intelligence Retail Site Selection Automation

Cross-Validating Huff Model Parameters with Holdout Stores

This page solves one exact task: choosing the attractiveness and distance-decay parameters of a Huff-style choice model by fitting them on part of the store estate and measuring their error on stores the fit never saw — with a spatial split, because ordinary random cross-validation leaks badly in this setting.

The leakage is the interesting part. Two stores four kilometres apart share origin zones, so a model fitted on one has effectively seen the other’s catchment. A random split therefore reports an error far lower than the model will achieve on a genuinely new site, which is precisely the situation it will be used in.

Prerequisites

  • A choice model implementation producing patronage probabilities from attractiveness and travel time.
  • Observed patronage by origin zone — from loyalty transactions or a mobility panel — for the stores in the fit.
  • A travel-time matrix between origin zones and every store in the competitive set, not only your own.
  • Python packages: numpy, pandas, scikit-learn for the splitter. Install with pip install numpy pandas scikit-learn.

Configuration and execution parameters

Parameter Value for this task Notes
split spatial blocks, 5 folds Never a random store-level split
block_size_km 40 Larger than any modelled catchment
parameters_fitted beta, attractiveness exponent Two is usually enough
objective weighted log-loss on zone shares Matches how the model is used
competitive_set all stores, both networks A partial set biases beta upward
min_zones_per_store 30 Stores below this are predicted, not fitted
report per-fold error, not just the mean The spread is the finding

Fitting beta against an incomplete competitive set is the most common way a Huff calibration goes wrong. Missing competitors leave demand unexplained, and the optimiser compensates by lowering the distance penalty so your stores reach further — producing a parameter that is wrong in a direction that flatters every subsequent site.

Why a random split flatters a spatial model Under a random split, training and holdout stores are interleaved across the same markets and share origin zones, giving a held-out error of 0.061. Under a spatial split, whole blocks are held out so no holdout store shares a catchment with a training store, and the error rises to 0.104 — which is what a new market will actually produce. The honest error is nearly twice the flattering one random split holdout stores sit between training stores held-out error 0.061 spatial block split a whole block is held out at once held-out error 0.104 Solid markers: training stores. Faded markers: holdout stores.

Annotated implementation

python
from __future__ import annotations

import numpy as np
import pandas as pd
from scipy.optimize import minimize
from sklearn.model_selection import GroupKFold

BLOCK_KM = 40.0


def spatial_blocks(stores: pd.DataFrame, block_km: float = BLOCK_KM) -> pd.Series:
    """Assign each store to a grid block larger than any modelled catchment."""
    x = (stores.geometry.x / (block_km * 1000)).astype(int)
    y = (stores.geometry.y / (block_km * 1000)).astype(int)
    return (x.astype(str) + "_" + y.astype(str)).rename("block")


def huff_shares(attract: np.ndarray, times: np.ndarray,
                beta: float, alpha: float) -> np.ndarray:
    """Probability of each store, per origin zone. Rows sum to one."""
    utility = np.power(attract, alpha) / np.power(np.maximum(times, 0.5), beta)
    return utility / utility.sum(axis=1, keepdims=True)


def fit_parameters(attract: np.ndarray, times: np.ndarray,
                   observed: np.ndarray, weights: np.ndarray) -> tuple[float, float]:
    """Weighted log-loss on observed zone shares; weights are zone demand."""
    def loss(params: np.ndarray) -> float:
        beta, alpha = params
        pred = np.clip(huff_shares(attract, times, beta, alpha), 1e-9, 1.0)
        return float(-(weights[:, None] * observed * np.log(pred)).sum())

    result = minimize(loss, x0=np.array([1.8, 1.0]),
                      bounds=[(0.5, 4.0), (0.3, 2.0)], method="L-BFGS-B")
    return float(result.x[0]), float(result.x[1])


def cross_validate(stores: pd.DataFrame, data: dict, n_splits: int = 5) -> pd.DataFrame:
    """One fit per fold, holding out whole spatial blocks."""
    blocks = spatial_blocks(stores)
    splitter = GroupKFold(n_splits=n_splits)
    rows = []
    for fold, (train_idx, test_idx) in enumerate(
            splitter.split(stores, groups=blocks)):
        beta, alpha = fit_parameters(
            data["attract"][train_idx], data["times"][train_idx],
            data["observed"][train_idx], data["weights"][train_idx])
        pred = huff_shares(data["attract"][test_idx], data["times"][test_idx],
                           beta, alpha)
        mae = float(np.abs(pred - data["observed"][test_idx]).mean())
        rows.append({"fold": fold, "beta": round(beta, 3),
                     "alpha": round(alpha, 3), "holdout_mae": round(mae, 4),
                     "blocks_held_out": blocks.iloc[test_idx].nunique()})
    return pd.DataFrame(rows)

Reporting per-fold parameters rather than only their mean is what makes this diagnostic rather than decorative. Folds that produce betas of 1.4, 1.6, 1.9, 3.2 and 1.5 have not produced a parameter of 1.9 — they have revealed one market where the model behaves differently, and that market is worth understanding before an average is taken.

Failure modes and debugging

Blocks smaller than the catchments. A twenty-kilometre block with thirty-minute catchments still leaks, because a held-out store’s trade area extends into the training blocks. The block has to exceed the largest modelled catchment radius, which for regional formats can mean blocks of sixty kilometres or more.

Optimising an objective that does not match the use. Fitting on unweighted zone shares treats a zone of forty households as equal to one of four thousand. Weighting the loss by zone demand aligns the fit with what the model is for, which is estimating captured demand rather than reproducing shares.

Fitting alpha and beta with too little data. The two parameters trade off — a larger attractiveness exponent can be partly offset by a larger distance penalty — so with few stores the optimiser wanders along a ridge and returns whichever point it happened to reach. Fixing alpha at one and fitting beta alone is the sensible fallback, recorded as a constraint rather than a result.

Ignoring format in the competitive set. Treating every nearby store as a substitute, regardless of format, inflates the denominator and depresses your stores’ shares. A similarity weight on the competitive set — even a crude one by format — usually improves the held-out error more than any amount of parameter tuning.

Five folds, four agreements and one outlier Folds one, two, four and five fit betas of 1.62, 1.71, 1.58 and 1.66 with held-out errors near 0.10. Fold three fits 3.14 with an error of 0.19 — a coastal market with a single access corridor, where the geometry rather than the model is unusual. The mean would hide the one fold worth investigating fold fitted beta held-out error character of the held-out block 1 1.62 0.098 suburban grid 2 1.71 0.105 mixed urban 3 3.14 0.190 coastal, one access corridor 4 and 5 1.58, 1.66 0.101, 0.096 rural and small town The coastal fold is not a bad fold; it is a market where a single road makes distance behave differently. The right response is a market-specific parameter, recorded as such, not a mean that suits neither.

Verification

  • Confirm no holdout store’s catchment overlaps a training store’s. A direct geometric check on the blocks, run once per estate.
  • Compare against a random split deliberately, and report both. The gap is the leakage, and quantifying it is what justifies the extra machinery.
  • Check the parameters are identified. Perturb alpha and refit beta; if the held-out error barely moves, the two are trading off and one should be fixed.
  • Re-run with the competitive set truncated to your own stores. Beta should fall noticeably; if it does not, the competitive set was probably incomplete to begin with.
An incomplete competitive set moves beta, not just the fit Fitting with only own stores in the denominator gives a beta of 1.02; adding the major chains takes it to 1.44; adding independents and discounters takes it to 1.66, and the held-out error falls from 0.171 to 0.101 across the same range. Missing competitors are absorbed into the distance parameter competitive set fitted beta held-out error own stores only 1.02 0.171 + major chains 1.44 0.126 + independents, discounters 1.66 0.101 A beta of 1.02 says shoppers barely mind distance, which nobody believes — it says the model had no other way to explain where the demand went. Competitor data quality is therefore a modelling input, not a reporting nicety.

Frequently Asked Questions

How many folds, and how large should blocks be?

Five folds and blocks larger than the largest modelled catchment. More folds give a smoother error estimate and, with spatial blocking, quickly run out of geography — at ten folds each block is small enough that leakage returns. The block size is the parameter that actually matters, and it should be derived from the catchments rather than chosen for convenience.

Should parameters differ by market?

Only where the evidence supports it and the difference has an explanation. A fold whose fitted beta is double the others usually reflects real geography — a coastal market, a valley, a river with few crossings — and a market-specific parameter is defensible when it can be attributed to something visible. Fitting per market by default produces parameters that mostly encode each market’s competitive accidents and transfer to nothing.

Can this validate attractiveness measures rather than parameters?

Yes, and it is often more valuable. Swapping selling area for a composite attractiveness — category count, anchor tenancy, parking — and comparing held-out error under the same folds is a clean comparison of two measurement choices. Because the split is fixed, the difference is attributable to the measure rather than to the fit.

What relationship does this have to distance-decay calibration?

They are the same estimation problem approached from two directions. Distance-decay calibration fits a curve to observed patronage against distance for one store at a time; the choice model fits a parameter that has to work across every store simultaneously in the presence of competition. Where both are available, the choice-model parameter is the more transferable and the decay curve is the easier to explain.

How does this interact with the composite suitability score?

The choice model supplies one criterion — expected captured demand — to a score that also weighs rent, co-tenancy and demographic fit. Cross-validating the choice model therefore validates an input rather than the whole ranking, and both need testing: a well-fitted beta feeding a poorly-weighted composite still produces a bad shortlist. Running the two validations separately also localises a failure, since a decline in ranking accuracy with a stable choice model points squarely at the weights.

Can the folds be reused across model versions?

They should be, and fixing them is what makes version comparisons meaningful. If version A is evaluated on one random set of folds and version B on another, the difference between their held-out errors includes the difference between the folds. Freezing the block assignment — storing it as a column on the store table — means every future model is measured on the same geography, and the comparison is about the model.

How long does a full cross-validation take to run?

Minutes for a regional estate and an hour or two nationally, dominated by the travel-time matrix rather than the optimisation. That makes it cheap enough to run on every parameter change, which is the right cadence — a parameter adjusted without a cross-validation is a parameter whose effect on unseen markets is unknown, however carefully it was reasoned about.

← Back to Validating & Backtesting Site Selection Models