Batch Geocoding with a Self-Hosted Service

This page solves one exact task: geocoding a large batch of normalized addresses against a reference dataset you control, so that the coordinates are reproducible months later and the reference version behind each one is recorded.

Reproducibility is the whole argument. A hosted geocoder is accurate and effortless, and it improves continuously — which means the coordinate it returns for a store today may differ from the one it returned last quarter, with no record of why. For a site-selection model that a committee will question, that is a problem: the suitability score rests on a catchment, the catchment rests on a coordinate, and if the coordinate cannot be reproduced then neither can the decision.

Prerequisites #

• Python packages: httpx for a connection-pooled client, pandas for the batch frame, and tenacity for bounded retries. Install with pip install httpx pandas tenacity.
• A running geocoding service with a reference dataset imported — typically an open-source stack fed by national address points and a street network. The specific engine matters less than the ability to pin and record the dataset build.
• Normalized addresses. The typed components produced by parsing and standardizing US store addresses, each with a stable key.
• The parent context. Geocoding and address normalization covers the precision tiers this page records and the result store it writes to.

Configuration and execution parameters #

Sending a structured query rather than a flattened string is the setting most often left at the wrong value. Having carefully parsed an address into components, re-joining them into one line asks the geocoder to parse it again with a different parser, and the two will disagree on exactly the awkward records that motivated the parsing work. Passing components through preserves the decisions already made and usually improves the match rate by a few points.

Annotated implementation #

The client below sends structured batches, records the reference build with every result, and separates a transport failure from a no-match. That distinction matters: the first should be retried, the second should never be.

from __future__ import annotations

import asyncio
from dataclasses import dataclass

import httpx
import pandas as pd
from tenacity import retry, stop_after_attempt, wait_exponential

GEOCODER = "http://geocoder.internal:4000/v1/search/structured"
REFERENCE_BUILD = "openaddresses-2026-07-31"     # pinned, recorded on every row
BATCH_SIZE = 500
CONCURRENCY = 8
MIN_SCORE = 0.72


@dataclass(frozen=True)
class Result:
    address_key: str
    lon: float | None
    lat: float | None
    tier: str
    score: float
    reference_build: str


@retry(stop=stop_after_attempt(3),
       wait=wait_exponential(multiplier=1, min=1, max=16))
async def _post_batch(client: httpx.AsyncClient, payload: list[dict]) -> list[dict]:
    """Transport-level retries only. A 4xx is a request bug, not a blip."""
    resp = await client.post(GEOCODER, json={"queries": payload}, timeout=30.0)
    if 400 <= resp.status_code < 500:
        resp.raise_for_status()          # do not retry a malformed request
    resp.raise_for_status()
    return resp.json()["results"]


def _to_result(key: str, hit: dict | None) -> Result:
    if not hit:                          # a genuine no-match, recorded as such
        return Result(key, None, None, "none", 0.0, REFERENCE_BUILD)
    lon, lat = hit["geometry"]["coordinates"]
    return Result(key, lon, lat,
                  hit["properties"].get("accuracy", "unknown"),
                  float(hit["properties"].get("confidence", 0.0)),
                  REFERENCE_BUILD)


async def geocode_frame(df: pd.DataFrame) -> pd.DataFrame:
    """Geocode a frame of parsed components, preserving one row per input row."""
    sem = asyncio.Semaphore(CONCURRENCY)
    batches = [df.iloc[i:i + BATCH_SIZE] for i in range(0, len(df), BATCH_SIZE)]

    async def run(batch: pd.DataFrame) -> list[Result]:
        payload = [{"address": r.number + " " + r.street, "locality": r.city,
                    "region": r.region, "postalcode": r.postcode,
                    "country": "US"} for r in batch.itertuples()]
        async with sem:
            async with httpx.AsyncClient(http2=True) as client:
                hits = await _post_batch(client, payload)
        return [_to_result(k, h) for k, h in zip(batch["address_key"], hits)]

    nested = await asyncio.gather(*(run(b) for b in batches))
    out = pd.DataFrame([r.__dict__ for group in nested for r in group])
    out["usable"] = (out["score"] >= MIN_SCORE) & out["lon"].notna()
    return out

Two properties of this client are worth stating explicitly. Results are returned one per input row, in order, including the no-matches — a client that silently drops failures produces a frame shorter than its input and a join that quietly loses stores. And the reference build is stamped on every row rather than recorded once for the job, so a table assembled from several runs still says which coordinate came from which build.

Failure modes and debugging #

A match rate that falls after an import. The usual cause is a reference build that imported partially — a state’s address points missing, a street file that failed to load — rather than anything about the addresses. Compare match rate by region against the previous build before accepting a new one; a uniform drop is a query change, and a drop concentrated in one region is an import problem.

Retries on a permanent failure. A malformed structured query returns a client error for every attempt, and retrying it three times with backoff turns an instant failure into a slow one. Separating client errors from server and network errors, as the code above does, is the difference between a batch that fails in seconds and one that fails in minutes.

Memory growth on large batches. Accumulating every result before writing is fine for a store estate and fatal for a customer file. Writing each batch’s results as they complete keeps the footprint flat, and it means an interrupted job has already persisted most of its work — the same streaming discipline that batch routing needs.

Coordinate order. Almost every geocoding API returns longitude first, and almost every mapping library expects latitude first. The symptom is an estate that plots in the wrong hemisphere, which is at least obvious; the dangerous variant is a market near the diagonal where the transposed points still land on the map. Unpack coordinates by name at the boundary and assert the values are in their expected ranges.

Verification #

• Re-run a sample against the same build. Ten thousand addresses geocoded twice must return identical coordinates. Any variation means something in the service is non-deterministic — usually a tie between equally-scored candidates broken by iteration order — and that is worth fixing before the results are trusted.
• Compare against a hosted service on a sample. Not to decide which is right, but to find the records where they disagree by more than a few hundred metres. Those are the addresses worth looking at by hand, and there are usually few enough to read.
• Check the tier distribution against the previous build. A shift toward weaker tiers is the clearest signal that a reference import was incomplete.
• Assert every input row produced an output row. Cheap, and it catches the class of bug where a batch failure silently shortened the frame.

Frequently Asked Questions #

Related #

• Geocoding & Address Normalization — precision tiers, the result store, and where this stage sits.
• Caching and Rate Limiting Geocoding Requests — the store that keeps this batch from re-running.
• Writing Idempotent Airflow DAGs for Geospatial Refresh — running this batch as a restartable task.

← Back to Geocoding & Address Normalization