Location Intelligence Retail Site Selection Automation

Clipping OSM Extracts to Retail Market Boundaries

This page solves one exact task: producing a smaller OpenStreetMap extract covering only the markets a chain operates in, without cutting the road network in a way that breaks routing at the market edge.

Clipping is worth doing — a graph a fifth the size imports in a fifth the time and serves from a fifth the memory — and it is the stage where an entire market’s edge quietly becomes unroutable. The failure is silent by construction: a severed road still produces contours, they are simply smaller than they should be, and only at the boundary where nobody is looking.

Prerequisites

  • Command-line tools: osmium-tool for the extract operations, which handles complete ways and referenced nodes correctly. Install through your package manager or a container image.
  • Python packages: geopandas and shapely for building the clip geometry, pyproj for buffering in metres. Install with pip install geopandas shapely pyproj.
  • A market boundary layer — the administrative or trade geography the business plans on, as polygons with a declared CRS.
  • The parent context. Preparing OSM network extracts for routing covers where clipping sits and what the build version records.

Configuration and execution parameters

Parameter Value for this task Notes
buffer_km 30 Larger than the longest contour you will compute
buffer_crs EPSG:5070 Buffering in degrees produces a distorted shape
strategy complete ways Keeps ways whole rather than cutting at the boundary
simplify_tolerance_m 500 The clip polygon, not the roads
union_markets true One clip geometry, not one extract per market
output_format pbf Compact and what every importer expects
keep_metadata false Version and timestamp attributes are not needed for routing

The strategy row is the one that decides whether this works. Cutting geometry at the boundary produces ways that end in mid-air; keeping complete ways means any way that touches the region is included in full, along with the nodes it references, which is what preserves connectivity. Every serious extract tool offers this, and it is not always the default.

Cutting at the boundary versus keeping complete ways On the left the extract is cut exactly at the market boundary, so a highway crossing it ends abruptly and a store near the edge can only route within the market. On the right complete ways are retained, so the highway continues past the boundary and the store's contour extends naturally across it. The same boundary, two extracts, two different catchments cut at the boundary store the contour stops at the extract edge reachable population understated by 34% complete ways, 30 km buffer store the contour extends past the boundary and reaches the population it should Dashed rectangle is the extract limit; the shaded polygon is a fifteen-minute contour from the store.

Annotated implementation

The clip geometry is built in Python — buffered in a projected CRS, unioned across markets, simplified enough to keep the extract tool fast — and applied with an extract tool that understands ways and their referenced nodes.

python
from __future__ import annotations

import subprocess
from pathlib import Path

import geopandas as gpd
from pyproj import CRS

METRIC_CRS = CRS.from_epsg(5070)
BUFFER_M = 30_000
SIMPLIFY_M = 500


def build_clip_polygon(markets: gpd.GeoDataFrame, out_path: Path) -> Path:
    """Union the operating markets, buffer in metres, and write GeoJSON in WGS84."""
    assert markets.crs is not None, "market layer has no CRS"
    metric = markets.to_crs(METRIC_CRS)

    # Buffer BEFORE union so narrow gaps between adjacent markets close, then
    # simplify: the clip shape only needs to be roughly right, and a polygon
    # with 200,000 vertices makes the extract tool far slower than the buffer.
    buffered = metric.geometry.buffer(BUFFER_M)
    merged = buffered.union_all().simplify(SIMPLIFY_M, preserve_topology=True)

    out = gpd.GeoSeries([merged], crs=METRIC_CRS).to_crs("EPSG:4326")
    out.to_file(out_path, driver="GeoJSON")
    return out_path


def clip_extract(source_pbf: Path, clip_geojson: Path, out_pbf: Path) -> Path:
    """Complete ways keeps every way that touches the region intact, plus the
    nodes it references — which is what preserves routability at the edge."""
    cmd = [
        "osmium", "extract",
        "--polygon", str(clip_geojson),
        "--strategy", "complete_ways",
        "--overwrite",
        "-o", str(out_pbf),
        str(source_pbf),
    ]
    subprocess.run(cmd, check=True)
    assert out_pbf.stat().st_size > 0, "empty extract — check the clip geometry"
    return out_pbf

Buffering before the union rather than after it is a small ordering detail with a real consequence: two adjacent markets separated by a few hundred metres of unincorporated land produce a clip polygon with a gap through the middle if unioned first, and the roads in that gap are exactly the ones connecting them.

Failure modes and debugging

A clip polygon with holes. Markets defined as separate polygons that share boundaries can, after a union of unbuffered geometry, leave slivers and holes where the boundaries did not quite coincide — the same sliver problem that afflicts spatial joins. Buffering first hides it; checking the polygon’s interior-ring count before use catches it.

Buffering in degrees. A buffer of 0.3 degrees is roughly thirty kilometres at the equator and twenty at fifty degrees north, so a degree-buffered clip is systematically tighter in the north of a country than the south. The symptom is edge truncation that correlates with latitude, which is a memorable afternoon of debugging if the buffer units are not checked first.

An over-detailed clip polygon. Passing a market boundary at full resolution to an extract tool can make the clip take longer than the import. Simplifying to a few hundred metres has no effect on which roads are included — the buffer is thirty kilometres wide — and it makes the operation fast.

Silently empty output. A clip polygon in the wrong CRS, or one that does not intersect the source extract at all, produces a valid, tiny output file rather than an error. Asserting on the output size and on the node count is the cheap guard, and it belongs in the pipeline rather than in a person’s memory.

What clipping buys on the import and the serving instance A national extract of 11.4 gigabytes imports in 96 minutes and needs 62 gigabytes of memory to serve. Clipped to four operating regions it is 1.9 gigabytes, imports in 14 minutes and serves from 11 gigabytes — a difference that changes what hardware the pipeline needs. The clip is what makes a rebuild a coffee break rather than an evening extract size import time memory to serve national 11.4 GB 96 min 62 GB four regions, unclipped 4.8 GB 38 min 26 GB clipped to markets + 30 km 1.9 GB 14 min 11 GB A fourteen-minute rebuild can run monthly without ceremony; a ninety-six-minute one acquires a change process and then stops happening. The memory figure is the one that decides infrastructure: 11 gigabytes fits a modest instance, 62 does not. Times from a single import host; ratios travel better than absolutes.

Verification

  • Count nodes and ways in the output and compare against the previous clip. A large unexplained drop is a clip geometry problem, not a map change.
  • Snap every store to the clipped graph and confirm all of them land in the largest connected component. A store in a small component means its neighbourhood was severed.
  • Compare edge-market contour areas against the unclipped extract for a sample of stores near the boundary. They should match; any that shrink identify exactly where the buffer is too tight.
  • Confirm the clip polygon covers every store with room to spare — a store within the buffer distance of the clip edge will produce a truncated contour even when the clip itself was correct.
The check that proves the clip did no harm For 120 stores within fifty kilometres of the clip boundary, contour areas from the clipped extract are compared against the national reference. At a thirty-kilometre buffer, 118 match within one per cent and two differ by more than five, both within ten kilometres of the edge — which tells you exactly where to widen. Compare against the unclipped extract once, then trust the clip buffer match within 1% differ over 5% where the misses are 5 km 74 / 120 31 anywhere near the edge 15 km 106 / 120 9 within 15 km of the edge 30 km 118 / 120 2 within 10 km of the edge The two remaining misses are stores that should not be scored from this extract at all — they belong to a neighbouring region's graph, and routing them here was the real error. Run this comparison once per geography, not per build: it validates the buffer, and the buffer rarely changes.

Frequently Asked Questions

Should the buffer scale with the longest contour computed?

Yes, and with a comfortable multiple. A thirty-minute drive-time contour can reach fifty kilometres on open highway, so a buffer smaller than that guarantees truncation for any store near the edge. The working rule is a buffer at least as large as the furthest distance the longest contour could plausibly reach, plus enough margin that the network beyond it is complete rather than fraying — which is where thirty kilometres comes from for fifteen-minute urban work and considerably more for regional analysis.

Can several markets share one extract?

They should, wherever their buffers overlap. Separate extracts per market duplicate the shared boundary region, multiply the import cost and — worse — produce two different graphs covering the same roads, so a store on the boundary gets different contours depending on which extract answered. One unioned clip covering all markets in a region avoids the whole class of problem.

How does clipping interact with the build version?

The clip geometry is part of the build configuration, so a change to markets or buffer produces a new build version exactly as a profile change would. That matters because a widened clip changes contours at the edge, and those changes need to be attributable. Hashing the clip polygon into the version identifier makes the connection automatic rather than remembered.

Is it worth clipping at all with cheap storage?

The storage was never the point — import time and serving memory are. A clipped extract turns a rebuild into something that can run monthly without negotiation and lets the engine run on modest instances, which in turn makes it affordable to keep several builds available for comparison. Where an organisation genuinely operates nationally, a national graph is simpler; for anyone else, clipping buys agility rather than disk.

What happens when a new market is opened?

The clip geometry changes, which means a new build version, a new graph and — importantly — a full re-score of anything that compared sites across the affected region. The mistake is treating a market addition as a configuration tweak: the roads that were previously outside the extract are now inside it, so catchments near the old boundary genuinely change. Handling it as a deliberate rebuild with a comparison against the previous build makes those changes visible instead of surprising.

Does the clip need to follow administrative boundaries at all?

No, and there is often a better shape. The clip only needs to cover every store plus the furthest reach of the longest contour, so a buffered convex hull around the estate can be simpler, smaller and less prone to the sliver problems that unioned administrative polygons produce. Administrative boundaries earn their place when the business plans in them and wants the extract to align with reporting geography, which is a legitimate reason — just not a routing one.

← Back to Preparing OSM Network Extracts for Routing