Location Intelligence Retail Site Selection Automation

Handling Privacy Thresholds in Visit Aggregations

This page solves one exact task: enforcing minimum-count suppression on visit aggregations so that no published figure — cell, margin or derived total — can be used to infer something about a small group of people, and so that suppressed values never reappear as zeros downstream.

Vendors apply their own thresholds before data reaches you, and that protection ends the moment your pipeline starts combining, differencing and re-aggregating their tables. Suppression is not a property of a file; it is a property of everything computed from it, and preserving it is a design requirement rather than a compliance checkbox.

Prerequisites

  • Vendor aggregates with a stated suppression threshold and a way to distinguish suppressed cells from genuine zeros.
  • The contractual limits on publication geography and retention, which are usually stricter than the technical threshold.
  • Python packages: pandas and numpy. Install with pip install pandas numpy.
  • The parent context. Mobility and foot traffic data integration covers where these tables sit and how they are used.

Configuration and execution parameters

Parameter Value for this task Notes
min_cell 5 visitors, or the contract’s figure Whichever is higher
suppress_margins true Row and column totals as well as cells
complementary_suppression true Suppress a second cell so the first cannot be derived
null_semantics withheld ≠ zero Two distinct states, never merged
export_check mandatory A gate on the publication path, not a convention
aggregate_rule suppress if any input suppressed Or report a bounded range
retention per contract Enforced by storage lifecycle

complementary_suppression is the rule that is most often missed and most necessary. Suppressing one cell in a row whose total is published leaves that cell computable by subtraction; suppressing a second cell in the same row is what actually protects the first.

Primary suppression alone is arithmetic, not protection A row of visits from four zones totals 214. Three cells are published at 96, 74 and 41; the fourth is suppressed because it holds three visitors. Subtracting the published cells from the published total recovers it exactly, so a second cell must also be suppressed for the first to be protected. 214 − 96 − 74 − 41 = 3 primary suppression only zone A 96 zone B 74 zone C 41 zone D withheld row total 214 — published recoverable by subtraction: 3 visitors with complementary suppression zone A 96 zone B 74 zone C withheld zone D withheld row total 214 — now only a bound of 44 is recoverable The complementary cell is chosen as the smallest publishable one, so the least information is lost.

Annotated implementation

The suppressor operates on a table and returns a table with two states — a value or a withheld marker — and it refuses to produce an output where a withheld value is recoverable.

python
from __future__ import annotations

import numpy as np
import pandas as pd

MIN_CELL = 5


def suppress(table: pd.DataFrame, value: str = "visitors",
             by: str = "store_id", across: str = "home_zone",
             min_cell: int = MIN_CELL) -> pd.DataFrame:
    """Primary and complementary suppression, per group."""
    out = table.copy()
    out["withheld"] = out[value] < min_cell
    out.loc[out["withheld"], value] = np.nan

    for _, idx in out.groupby(by).groups.items():
        group = out.loc[idx]
        if group["withheld"].sum() != 1:
            continue  # zero suppressed cells is fine; two or more already protect
        # Exactly one suppressed cell is recoverable from the published total,
        # so suppress the smallest remaining cell as well.
        publishable = group.loc[~group["withheld"], value]
        if publishable.empty:
            continue
        victim = publishable.idxmin()
        out.loc[victim, "withheld"] = True
        out.loc[victim, value] = np.nan
    return out


def safe_aggregate(table: pd.DataFrame, value: str = "visitors",
                   by: str | list[str] = "store_id") -> pd.DataFrame:
    """Aggregate without laundering suppressed cells into a total.

    A sum over a group containing withheld values is a LOWER BOUND, and it must
    be labelled as one — silently summing the non-null cells is exactly how a
    suppressed figure becomes a published number."""
    grouped = table.groupby(by)
    return pd.DataFrame({
        "value": grouped[value].sum(min_count=1),
        "withheld_cells": grouped["withheld"].sum(),
        "is_lower_bound": grouped["withheld"].any(),
    }).reset_index()


def export_gate(frame: pd.DataFrame, value: str = "visitors",
                min_cell: int = MIN_CELL) -> pd.DataFrame:
    """Last line of defence on the publication path."""
    below = frame[value].dropna() < min_cell
    if below.any():
        raise ValueError(f"{int(below.sum())} cell(s) below the threshold in an export")
    return frame

safe_aggregate carries the property that matters most downstream. A total computed over a group containing withheld cells is a lower bound rather than a total, and labelling it as such is what stops a suppressed figure being reconstructed two joins later by someone who had no idea it was ever suppressed.

Failure modes and debugging

Zero-filling on load. A single fillna(0) anywhere in the pipeline converts every withheld value into a published claim that no visits occurred. It is one line, it looks like tidiness, and it undoes the entire protection while also biasing every trade area toward dense zones.

Suppression applied at read and lost at write. A pipeline that suppresses when loading and then aggregates freely can publish a total from which individual cells are derivable. The gate belongs on the export path as well as the ingest path, and it should raise rather than warn.

Differencing two published tables. Two monthly tables, each correctly suppressed, can reveal a suppressed cell when subtracted — a change of exactly three visitors in a zone where one month was withheld. Where period-on-period differences are published, they need their own suppression pass rather than inheriting the inputs’.

Thresholds applied inconsistently across geographies. Suppressing at block-group level and publishing an unsuppressed tract total that contains only one populated block group protects nothing. The threshold has to be evaluated at every published geography, not only the finest one.

Where suppression bites, and how much of the total it hides In dense urban markets 6 per cent of origin cells are suppressed, holding 1 per cent of visits. In rural markets 41 per cent of cells are suppressed, holding 14 per cent of visits — which is why a rural trade area built from published cells alone is materially incomplete. Suppression is a rural phenomenon, and it hides real demand market type cells suppressed visits hidden effect dense urban 6% 1% negligible suburban 14% 3% minor rural 41% 14% material Reporting the suppressed share beside every trade area is what stops a rural store looking like it draws from a smaller area than it does — the missing visits are spread across many tiny zones. Aggregating rural origins to a coarser geography before analysis recovers most of them legitimately, at the cost of resolution nobody could use anyway.

Verification

  • Attempt the subtraction attack on your own published output. Take a row with one suppressed cell and a published total and check the cell cannot be recovered. If it can, complementary suppression is not running.
  • Assert no export contains a value below the threshold, as a raising gate rather than a log line.
  • Confirm nulls survive a round trip through storage and reload as withheld rather than as zero, which is a real risk with formats that do not distinguish them.
  • Difference two consecutive periods and re-run the suppression check on the result.
Three places the threshold has to be enforced Suppression is applied on ingest when vendor data arrives, re-applied after any aggregation or differencing that could create a small cell, and enforced as a raising gate on every export path. Applying it only at ingest protects the input and not the output. Once at the door is not enough 1 · on ingest primary + complementary 2 · after aggregation sums, differences, joins 3 · on export raises, never warns Stage two is the one teams skip, and it is where a legitimate analysis creates an illegitimate cell: a month-on-month difference, a filter to one demographic band, a join that splits a zone. Stage three exists because stage two will eventually be forgotten in some new code path, and a gate that raises is the only kind that survives that.

Frequently Asked Questions

Is the vendor’s suppression not sufficient on its own?

It is sufficient for the file they send and says nothing about what you compute from it. Every aggregation, difference and filter creates new cells with new counts, and some of those are small even when every input cell was large. Treating the vendor’s threshold as a property to be maintained rather than a step already completed is the difference between a compliant pipeline and one that happens to have started compliant.

What should a report say about suppressed data?

That it exists, how much of the total it represents, and that the published figures are therefore a lower bound. One line does it, and its absence is what allows a reader to treat a partial total as a complete one. In markets where the suppressed share is large — rural ones, mostly — the line belongs next to the headline number rather than in a footnote.

Does aggregating to a coarser geography solve the problem?

It reduces it legitimately and does not remove the obligation. Rolling small zones up until the cells clear the threshold recovers most of the hidden visits and is usually the right analytical choice, since the finer resolution was not supportable anyway. What it does not do is remove the need to check the coarser table for cells that are still small, which happens more often than expected in sparsely populated regions.

How does this interact with data retention?

They are two halves of the same obligation and are usually specified together in the vendor contract. Suppression governs what may be published; retention governs how long the underlying data may be held. Both belong enforced by the platform — a lifecycle rule on the storage and a gate on the export — because both are the kind of policy that a person will observe carefully for a year and then forget during a busy quarter.

Should the threshold be higher than the contract requires?

Often yes, and it costs less than it appears. Contracts specify a floor rather than a target, and small cells contribute almost nothing to a trade-area analysis while carrying most of the disclosure risk. Raising the threshold from five to ten typically hides another few per cent of visits in rural markets and none of the analytical signal, which is a favourable trade in both directions — more protection and a cleaner dataset.

How does suppression interact with the calibration against transactions?

It biases the panel estimate downward for stores with many small origin zones, which are disproportionately rural. If the calibration is fitted without accounting for that, the resulting factor absorbs the suppression bias and then over-corrects urban stores. Including the suppressed share as a covariate in the calibration, or fitting separately by market density, keeps the factor describing the panel rather than describing the suppression.

Who should own the suppression rules?

The data platform, with the vendor contract as the input and a named person accountable for keeping the two aligned. Leaving suppression to individual analyses guarantees inconsistency, and leaving it to the vendor guarantees it stops at the file boundary. A single implementation used by every consumer, updated when a contract changes, is both the safest and the least effort.

← Back to Mobility & Foot Traffic Data Integration