Automating Monthly OSM PBF Refresh for Routing Engines

This page solves one exact task: running the download, clip, build and promotion of a routing graph as a scheduled, restartable pipeline, with a gate between “built” and “serving” so an unvalidated graph never answers a query.

The refresh is deceptively simple — download a file, run an importer, restart a service — and every one of those three steps has a failure mode that produces a working service with wrong answers. A truncated download imports cleanly and yields a smaller network; an importer interrupted halfway leaves partial graph files that some engines load without complaint; and a restart mid-batch means one scoring run used two different graphs.

Prerequisites #

• An orchestrator. Airflow, a workflow runner, or a scheduled job with retry semantics — the same orchestration layer that runs the rest of the pipeline.
• Object storage for the source file and the built graph, addressed by version.
• A clip geometry, produced as in clipping OSM extracts to retail market boundaries.
• A validation suite — counts, connectivity and a fixed route set — that the promotion gate can run.

Configuration and execution parameters #

drain_before_switch prevents the subtlest failure in the list. Switching the pointer while a batch is running means some contours in one scoring run were computed on the old graph and some on the new, with nothing in the output distinguishing them. Draining — or simply refusing to promote while a batch holds a lease — keeps a run internally consistent.

Annotated implementation #

The tasks below are written as plain functions so they can run under any orchestrator. What matters is the shape: every step writes to a version-scoped location, and promotion is a separate, last step.

from __future__ import annotations

import hashlib
import subprocess
from datetime import date
from pathlib import Path

SOURCE_URL = "https://download.example.org/region-latest.osm.pbf"
GRAPH_ROOT = Path("/srv/graphs")
POINTER = GRAPH_ROOT / "current"


def build_version(source_date: str, profile: str, config_hash: str) -> str:
    return f"{source_date}_{profile}_{config_hash[:8]}"


def download(target: Path, expected_sha256: str) -> Path:
    """A truncated download imports cleanly and yields a smaller network,
    so the checksum is not optional — it is the only thing that catches it."""
    subprocess.run(["curl", "-fsSL", "-o", str(target), SOURCE_URL], check=True)
    digest = hashlib.sha256(target.read_bytes()).hexdigest()
    if digest != expected_sha256:
        target.unlink(missing_ok=True)
        raise ValueError(f"checksum mismatch: {digest[:12]} != {expected_sha256[:12]}")
    return target


def build_graph(clipped_pbf: Path, profile: Path, version: str) -> Path:
    """Build into a version-scoped directory. Nothing writes to the live path."""
    out_dir = GRAPH_ROOT / version
    out_dir.mkdir(parents=True, exist_ok=True)
    subprocess.run(["osrm-extract", "-p", str(profile),
                    "-o", str(out_dir / "graph.osrm"), str(clipped_pbf)], check=True)
    subprocess.run(["osrm-contract", str(out_dir / "graph.osrm")], check=True)
    (out_dir / "BUILD_OK").write_text(version)     # completion marker, written last
    return out_dir


def promote(version: str, checks_passed: bool, batch_lease_held: bool) -> bool:
    """Pointer swap. Refuses while a batch holds the graph, and refuses on red."""
    if not checks_passed:
        return False
    if batch_lease_held:
        return False                               # try again after the batch
    target = GRAPH_ROOT / version
    if not (target / "BUILD_OK").exists():
        raise RuntimeError("refusing to promote a build with no completion marker")
    tmp = POINTER.with_suffix(".tmp")
    tmp.symlink_to(target)
    tmp.replace(POINTER)                           # atomic on the same filesystem
    return True

The BUILD_OK marker written last is the cheap version of a transaction. An import killed partway leaves a directory full of plausible files and no marker, and the promotion step refuses it — which is exactly the case that otherwise produces a service that starts, answers queries, and returns nonsense for half the region.

Failure modes and debugging #

A source file that moved or changed format. Extract providers reorganise, and a URL that has worked for two years starts returning an error page that a naive download saves as a .pbf. The checksum catches this immediately; without one, the importer’s error message is about malformed data and the cause is three steps upstream.

Disk exhaustion mid-import. Graph builds need several times the extract size in temporary space, and the failure arrives as a cryptic error from a tool that was halfway through writing. Checking free space against a multiple of the input size before starting turns an obscure failure into a clear one.

Promotion while a batch is running. The most damaging and least obvious. A contour batch that spans a promotion produces a mixed-version result set, and the artifacts carry a single build version that is now wrong for half of them. A lease held by the batch and checked by the promoter is a few lines and removes the possibility.

A build that never promotes. The gate fails, the alert goes to a channel nobody reads, and three months later the graph is a quarter old while the pipeline reports success every month. Alerting on a skipped promotion, not only on a failure, is what catches the slow version of this.

Verification #

• Confirm the pointer moved and the engine reloaded. Query the service for its build version after promotion. A pointer that changed while the engine still holds the old graph in memory is the classic half-promoted state.
• Re-run the route regression against the promoted graph, not only against the built one. The two should be identical; if they are not, the service is serving something other than what was validated.
• Check that old builds are still readable for the retention window, so a rollback is a pointer change rather than a rebuild.
• Assert the completion marker exists before every promotion, and test the assertion by promoting a deliberately incomplete build in a staging environment.

Frequently Asked Questions #

Related #

• Preparing OSM Network Extracts for Routing — the surrounding architecture and the pin.
• Writing Idempotent Airflow DAGs for Geospatial Refresh — the restartability this pipeline relies on.
• Benchmarking Graph Build Times for Large Extracts — sizing the build window this schedule assumes.

← Back to Preparing OSM Network Extracts for Routing