Estimating Visit Frequency Decay from Foot Traffic Panels

This page solves one exact task: fitting the relationship between travel time and visit frequency from observed panel data, and separating it into the two effects it actually contains — how many households in a zone visit at all, and how often the visitors among them return.

That separation is the substance of the page. A single decay curve blends both effects and hides the one that matters commercially: a zone twenty minutes away may contribute few customers who shop weekly, or many who shop twice a year, and those two markets need completely different treatment in a suitability score and in a media plan.

Prerequisites #

• Corrected visit aggregates by origin zone and period, from correcting panel bias in mobility datasets.
• A travel-time matrix from each origin zone to the store, from the routing layer.
• Visitor counts as well as visit counts — the vendor’s distinct-device figure per zone, which is what makes the two effects separable.
• Python packages: pandas, numpy, scipy for the fit. Install with pip install pandas numpy scipy.

Configuration and execution parameters #

Splitting the measurement into penetration and frequency costs nothing — both come from figures the vendor already provides — and it is the difference between a curve that describes what happens and one that explains it.

Annotated implementation #

The fit is applied to each effect separately, on bins with enough observations to support it.

from __future__ import annotations

import numpy as np
import pandas as pd
from scipy.optimize import curve_fit

MIN_VISITORS = 50


def exponential(t: np.ndarray, a: float, lam: float) -> np.ndarray:
    return a * np.exp(-lam * t)


def power_law(t: np.ndarray, a: float, beta: float) -> np.ndarray:
    return a * np.power(1.0 + t / 10.0, -beta)


def split_effects(visits: pd.DataFrame, households: pd.DataFrame,
                  travel: pd.DataFrame) -> pd.DataFrame:
    """Penetration and frequency, per origin zone, from the same aggregates."""
    df = (visits.merge(households, on="zone_id")
                .merge(travel, on="zone_id"))
    df["penetration"] = df["visitors"] / df["households"]
    df["frequency"] = df["visits"] / df["visitors"].clip(lower=1)
    df["fittable"] = df["visitors"] >= MIN_VISITORS
    return df


def fit_curve(df: pd.DataFrame, column: str) -> dict:
    """Fit both families on the fittable rows; choose on held-out error."""
    data = df.loc[df["fittable"]]
    t, y = data["travel_minutes"].to_numpy(), data[column].to_numpy()

    # Simple holdout: every third row, ordered by travel time, is held out so
    # the split spans the range rather than sampling one end of it.
    order = np.argsort(t)
    holdout = order[::3]
    train = np.setdiff1d(order, holdout)

    results = {}
    for name, fn in (("exponential", exponential), ("power", power_law)):
        params, _ = curve_fit(fn, t[train], y[train], maxfev=10_000)
        pred = fn(t[holdout], *params)
        results[name] = {
            "params": [round(float(p), 4) for p in params],
            "holdout_mae": float(np.abs(pred - y[holdout]).mean()),
        }
    results["chosen"] = min(("exponential", "power"),
                            key=lambda k: results[k]["holdout_mae"])
    return results

Holding out every third row by travel-time order rather than at random matters more than it looks. A random holdout in a dataset dominated by nearby zones tests the fit almost entirely on the near bands, which is where every functional form agrees; spreading the holdout across the range tests the tail, which is where they differ and where the decay parameter is actually decided.

Failure modes and debugging #

Fitting on zones with a handful of visitors. A zone with four visitors produces a frequency figure of 1.0 or 3.0 depending on one person’s habits. The visitor floor keeps those rows out of the fit while still reporting them, which is the right treatment: they are data, they are not evidence about the curve.

Confusing visits with visitors. The two columns look interchangeable and are not. Dividing visits by households conflates the effects and produces a curve that is neither penetration nor frequency, and no amount of careful fitting recovers the distinction afterwards.

Ignoring the competitive field. A zone’s penetration is low both because it is far away and because there is a rival between it and your store. Fitting decay without any competition term attributes the rival’s effect to distance, which then transfers badly to a market with a different competitive structure — the same argument that motivates a gravity model over a pure distance decay.

Fitting one curve for all formats. Convenience and destination formats have genuinely different decay, and a pooled fit describes neither. Fit per format, and where a format has too few stores, borrow explicitly and record the borrowing.

Verification #

• Check the fit against a held-out band, not against the data it was fitted on. Every reasonable family fits the near bands well.
• Compare the fitted decay with one derived from loyalty data where both exist. Agreement is strong evidence; disagreement usually points at the panel’s home-zone attribution rather than at the fit.
• Confirm the two effects multiply back to the observed visits per household within tolerance. If they do not, a denominator is mismatched somewhere.
• Re-fit with a competition term on a sample of stores and see how much the distance parameter moves. A large move means the pure decay was absorbing competitive structure.

Frequently Asked Questions #

Related #

• Calibrating Distance-Decay Functions for Trade Areas — the same fit from loyalty trips.
• Correcting Panel Bias in Mobility Datasets — the correction this fit assumes.
• Mobility & Foot Traffic Data Integration — the pipeline context.

← Back to Mobility & Foot Traffic Data Integration