Versioning Network Extracts for Reproducible Catchments

This page solves one exact task: constructing a build identifier that captures everything which can change a route, attaching it to every artifact computed from the graph, and using it to answer questions about why numbers moved.

Without it, a catchment is a polygon with a date. With it, a catchment is a polygon with a provenance — and the difference shows up the first time a real-estate committee asks why a site that scored 0.81 last quarter scores 0.78 today.

Prerequisites #

• A build pipeline producing graphs at versioned locations, per preparing OSM network extracts for routing.
• Artifact storage that can carry metadata — a column in the contour table, a field in the GeoParquet schema, a tag on the cache key.
• Python packages: hashlib from the standard library is genuinely all that is required; pandas for the comparison queries.

Configuration and execution parameters #

The rule that makes this work: anything that can change a returned duration goes into the identifier, and nothing else does. Adding the build host or the wall-clock time produces a new version for every build even when nothing changed, which destroys the property that makes the identifier useful — that two identical builds are recognisably identical.

Annotated implementation #

The identifier is a hash over the things that matter, with a human-readable prefix so it can be recognised at a glance.

from __future__ import annotations

import hashlib
import json
from pathlib import Path


def _file_hash(path: Path, chunk: int = 1 << 20) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        while block := fh.read(chunk):
            h.update(block)
    return h.hexdigest()


def build_identifier(extract_path: Path, extract_date: str, profile_path: Path,
                     clip_geojson: Path, engine_version: str,
                     mode: str = "ch") -> tuple[str, dict]:
    """Deterministic: the same inputs always produce the same identifier, and
    a change to any of them produces a different one."""
    components = {
        "extract_date": extract_date,
        "extract_sha256": _file_hash(extract_path),
        "profile_sha256": _file_hash(profile_path),
        "clip_sha256": _file_hash(clip_geojson),
        "engine_version": engine_version,
        "mode": mode,
    }
    digest = hashlib.sha256(
        json.dumps(components, sort_keys=True).encode()).hexdigest()

    profile_name = profile_path.stem
    build_id = f"{extract_date}_{profile_name}_{digest[:8]}"
    # The manifest is what makes the id reproducible rather than merely unique.
    return build_id, {"build_id": build_id, **components}


def write_manifest(build_dir: Path, manifest: dict) -> Path:
    out = build_dir / "BUILD_MANIFEST.json"
    out.write_text(json.dumps(manifest, indent=2, sort_keys=True))
    return out

The manifest matters as much as the identifier. A hash tells you two builds differ; the manifest tells you which component differs, which turns “the catchments moved” into “the profile changed and the map did not” in the time it takes to diff two small files.

Failure modes and debugging #

A version that changes on every build. Usually caused by including a timestamp, a temporary path or an unsorted dictionary in the hash. The symptom is a cache that never hits and a comparison that always reports change. Sorting keys before hashing, as above, removes the commonest cause.

A version that does not change when it should. More dangerous. If the profile is hashed by filename rather than contents, an edited speed table produces the same identifier and two genuinely different graphs become indistinguishable. Hash contents, always.

Artifacts that lose the stamp. A contour written with its build id, aggregated into a reachable-population figure that drops it, then fed into a score that has no idea which graph it came from. The stamp has to propagate through every aggregation, which means the aggregation code has to carry it deliberately — usually as a column that is grouped by rather than dropped.

Two versions in one result set. A batch that spans a promotion, as covered in the refresh pipeline. The stamp makes it visible after the fact; a lease held during the batch prevents it in the first place, and both are worth having.

Verification #

• Hash the same inputs twice and confirm identical identifiers. If they differ, something non-deterministic is in the hash.
• Change one input at a time and confirm the identifier changes each time — particularly the profile contents, which is the one most often hashed incorrectly.
• Query artifacts by version. Every contour, cache entry and scored row should be filterable by build id; if any stage cannot answer “which build produced this?”, the stamp is not propagating.
• Reproduce an old contour. Take a stored contour, read its build id, restore that build from retention, re-run the request and compare geometry. This is the end-to-end proof, and it is worth doing once per quarter rather than assuming.

Frequently Asked Questions #

Related #

• Preparing OSM Network Extracts for Routing — the build this identifier describes.
• Automating Monthly OSM PBF Refresh for Routing Engines — where the identifier is produced.
• Invalidating Isochrone Caches on Traffic Updates — version-scoped keys in the cache.

← Back to Preparing OSM Network Extracts for Routing