Location Intelligence Retail Site Selection Automation

Benchmarking Graph Build Times for Large Extracts

This page solves one exact task: measuring the time, memory and disk a routing graph build consumes at each stage, so the refresh window can be sized from evidence and the right bottleneck gets the money.

Build time decides whether a network refresh is a routine monthly job or an event that needs planning, and it is the number most often guessed at. Teams size a build host by doubling whatever the last one had, discover the import fails at three in the morning on a disk that filled, and conclude that graph builds are fragile. They are not — they are simply resource-hungry in a way that is easy to measure and rarely measured.

Prerequisites

  • A build host you can instrument — measuring inside a shared cluster with noisy neighbours produces numbers you cannot reason about.
  • Two or three extracts of different sizes, so the scaling behaviour is visible rather than inferred from one point.
  • Python packages: psutil for resource sampling and pandas for the summary. Install with pip install psutil pandas.
  • The parent context. Preparing OSM network extracts for routing describes the stages being measured and why they exist.

Configuration and execution parameters

Parameter Value for this task Notes
sample_interval_s 2 Resource sampling; finer is noise, coarser misses peaks
stages_measured parse, edge build, contract Each has a different limiting resource
extract_sizes 3 points, 4× apart Enough to see whether scaling is linear
repeat_runs 2 Filesystem caching makes the first run pessimistic
record wall time, peak RSS, peak disk, cores used All four; the missing one is always the one that fails
isolate dedicated host Shared hosts produce unrepeatable numbers

Recording all four resources matters because the bottleneck moves between stages. Parsing is I/O-bound and single-threaded in most importers; the edge build is CPU-bound and parallel; the contraction phase is memory-bound and the one that fails on a machine that seemed adequate through the first two.

Three stages, three different limiting resources Parsing takes 22 per cent of the wall clock, uses one core and modest memory but heavy disk throughput. The edge build takes 34 per cent, saturates every core and peaks at 14 gigabytes. Contraction takes 44 per cent, uses two cores and peaks at 41 gigabytes — the stage that decides how much memory the host needs. Buying more cores does not help the stage that fails share of wall clock parse 22% edge build 34% contract 44% stage cores used peak memory limiting resource parse 1 3 GB disk throughput edge build all 16 14 GB CPU contract 2 41 GB memory A host with thirty-two gigabytes completes the first two stages comfortably and dies in the third, which is why "it got most of the way through" is such a common description of a failed import. Measured on a 1.9 GB clipped extract, sixteen cores, NVMe storage.

Annotated implementation

The harness wraps the build commands, samples resource use while they run, and produces a per-stage record that can be compared across extracts and hosts.

python
from __future__ import annotations

import shutil
import subprocess
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path

import psutil


@dataclass
class StageResult:
    name: str
    wall_s: float
    peak_rss_gb: float
    peak_disk_gb: float
    samples: list[float] = field(default_factory=list)


def _sample(proc: psutil.Process, out: StageResult, workspace: Path,
            stop: threading.Event, interval: float = 2.0) -> None:
    """Sample the process tree; a build spawns children that hold most of the RAM."""
    while not stop.is_set():
        try:
            rss = proc.memory_info().rss
            for child in proc.children(recursive=True):
                rss += child.memory_info().rss
            out.peak_rss_gb = max(out.peak_rss_gb, rss / 1e9)
            used = sum(f.stat().st_size for f in workspace.rglob("*") if f.is_file())
            out.peak_disk_gb = max(out.peak_disk_gb, used / 1e9)
            out.samples.append(rss / 1e9)
        except psutil.NoSuchProcess:
            return
        stop.wait(interval)


def run_stage(name: str, cmd: list[str], workspace: Path) -> StageResult:
    result = StageResult(name=name, wall_s=0.0, peak_rss_gb=0.0, peak_disk_gb=0.0)
    started = time.monotonic()
    popen = subprocess.Popen(cmd)
    stop = threading.Event()
    watcher = threading.Thread(
        target=_sample, args=(psutil.Process(popen.pid), result, workspace, stop))
    watcher.start()
    try:
        popen.wait()
    finally:
        stop.set()
        watcher.join()
    result.wall_s = round(time.monotonic() - started, 1)
    if popen.returncode != 0:
        raise RuntimeError(f"{name} failed with code {popen.returncode}")
    return result


def free_space_guard(workspace: Path, extract_gb: float, multiple: float = 6.0) -> None:
    """Refuse to start rather than fail at 80%: builds need several times the
    extract size in scratch, and the failure at that point is expensive."""
    free_gb = shutil.disk_usage(workspace).free / 1e9
    needed = extract_gb * multiple
    if free_gb < needed:
        raise RuntimeError(f"need ~{needed:.0f} GB free, have {free_gb:.0f} GB")

Sampling the process tree rather than the parent is the detail that makes the memory figure real. Importers spawn workers that hold nearly all of the allocation, so a naive measurement of the parent reports a few hundred megabytes for a build that peaked at forty gigabytes.

Failure modes and debugging

Measuring a warm run and planning for a cold one. The second build of the same extract benefits from filesystem cache and can be twenty per cent faster. Running twice and reporting the slower figure — or dropping caches between runs where the environment permits — gives a number that survives contact with the scheduled job.

A shared host. Benchmarks on a machine that also runs a database produce numbers that vary by a factor of two between runs, and the conclusion drawn from them is usually about the wrong resource. If a dedicated host is not available, at minimum record the host’s other load alongside the measurement so the numbers can be discounted appropriately.

Extrapolating from one extract. Build time is not linear in extract size; the contraction phase in particular grows faster than the input. Measuring three sizes and plotting them shows the shape, and the shape is what tells you whether a national extract is a longer build or an impossible one on that host.

Ignoring disk. The scratch requirement during a build is several times the extract size, and it peaks in the middle of the run rather than at the end. A guard that refuses to start when free space is inadequate turns a three-in-the-morning failure into a clear message at the start.

Scaling is worse than linear, and memory is the wall A 0.5 gigabyte extract builds in 4 minutes using 11 gigabytes of memory; 1.9 gigabytes takes 14 minutes and 41 gigabytes; 7.4 gigabytes takes 71 minutes and 148 gigabytes. Time grows a little faster than the input, and memory grows fast enough to decide which extracts are buildable at all. Time is a schedule problem; memory is a hard limit extract edges build time peak memory scratch disk 0.5 GB · one metro 2.4M 4 min 11 GB 3 GB 1.9 GB · four markets 9.4M 14 min 41 GB 11 GB 7.4 GB · half a country 38M 71 min 148 GB 44 GB A fourfold extract costs five times the time and three and a half times the memory, which means the largest build is decided by what fits rather than by how long anyone is willing to wait. This table is also the clipping argument in numbers: the four-market extract builds on commodity hardware, and the unclipped alternative needs a machine bought for the purpose. Same engine, same profile, same host class; only the extract differs.

Verification

  • Run each configuration twice and report the slower run, so the scheduled job is not sized against a cached best case.
  • Confirm the peak memory figure includes child processes by comparing against the operating system’s own accounting for the build.
  • Check the free-space guard fires by running it against a deliberately small volume; a guard that has never triggered has not been tested.
  • Repeat the benchmark after an engine upgrade. Import performance changes between versions, occasionally by a lot, and a build window sized on the previous version is exactly the kind of assumption that fails silently until it does not.
Sizing the maintenance window from the measurement A 14-minute build plus 3 minutes of clipping, 4 minutes of validation and 2 minutes of promotion fits in a 30-minute window with 7 minutes of headroom. The same sequence on an unclipped extract needs 96 minutes and no longer fits, which is a scheduling decision rather than a technical one. The window is the sum, not the build clipped extract · fits a 30-minute window clip 3 build 14 min validate 4 promote headroom 7 min unclipped · needs its own arrangement build 96 min — validation and promotion do not fit at all Headroom is not slack, it is what absorbs a slow download or a retried stage without the job overrunning into the hours when someone is running a scoring batch against the graph it is trying to replace.

Frequently Asked Questions

Is a faster host or a smaller extract the better investment?

Almost always the smaller extract, because it improves every stage at once and reduces the serving footprint as well. Doubling the cores helps only the edge build, which is a third of the time; halving the extract helps parsing, building, contraction, memory and the serving instance. The exception is an organisation that genuinely needs a national graph, where the extract cannot shrink and the host is the only lever.

Can the build run in a container on a shared cluster?

Yes, with the memory limit set from the measured peak plus a margin, and with the understanding that the numbers will be less repeatable. The common failure is a container limit set from the average rather than the peak, which produces a build killed during contraction with a message about memory that looks like a code problem. Measuring first turns that into a configuration line.

What is a reasonable failure budget for a monthly build?

One failure a year is comfortable; one a quarter suggests the host is sized at the edge of what the build needs. Because a failed build leaves the previous graph serving, the operational impact is small — but a build that fails regularly stops being investigated, and eventually the graph is a year old with a green dashboard reporting monthly attempts.

Should build metrics be kept?

Yes, alongside the build version. The trend in build time is an early warning that the extract is growing past the host, and the trend in peak memory is the one that predicts the failure before it happens. Both are two numbers per build and they turn a capacity question into a chart rather than an argument.

How does the benchmark change the refresh design?

Usually in one of three ways, and knowing which is the point of measuring. If the build comfortably fits the window, nothing changes and the measurement becomes a baseline to watch. If it fits only just, the answer is normally to clip harder rather than to buy hardware, because the clip improves the serving footprint too. And if it does not fit at all, the honest options are a bigger host, a split into per-region builds that run in parallel, or a longer window — all of which are decisions someone can make from a table of numbers and none of which can be made from an impression that the import takes a while.

Does preprocessing mode change the picture?

Substantially. Contraction-style preprocessing is the expensive stage in these measurements and it is what buys the query speed a batch of ten thousand contours depends on; a mode that skips it builds far faster and answers queries far slower. The right comparison is therefore not build time alone but build time plus the query time of the batch that follows, since a graph that takes twenty minutes longer to build and halves a four-hour batch is straightforwardly worth it.

← Back to Preparing OSM Network Extracts for Routing