GitHub Actions for Scheduled Spatial Data Refresh

This page solves one exact task: standing up a scheduled GitHub Actions workflow that runs a lightweight spatial refresh on a cron — pulling demographic updates, revalidating coordinates, and publishing checkable artifacts — without provisioning an orchestration cluster.

Not every refresh needs Airflow. A daily freshness check that re-pulls changed ACS rows, revalidates a coordinate set, and publishes a small artifact is a single linear job, and a scheduled GitHub Actions workflow runs it for free on managed runners while doubling as the CI that gates code changes. The heavy, multi-stage production graph belongs in Airflow — see orchestrating spatial pipelines with Airflow — but this cron-triggered workflow is the right tool for the lightweight end of the spectrum.

Prerequisites #

Before wiring up the workflow you need:

• A repository with Actions enabled and permission to add workflow files under .github/workflows/.
• Repository or environment secrets for the credentials the job needs — a Census API key and the object-store keys the publish step writes with. Never hard-code these in the YAML.
• Python packages the job installs: geopandas, shapely, pyproj, pandas, and requests. Pin them in a requirements.txt so the cached environment is reproducible.
• A defined artifact target. The job publishes to the same GeoParquet data lake the rest of the platform reads, or attaches artifacts to the run for inspection.

Configuration and execution parameters #

The table lists the workflow settings that govern when the job runs, how it fans out, and how it fails. GitHub-hosted runners impose hard limits that shape the design.

Two facts about hosted runners drive the design. Scheduled crons run in UTC only — a “6 AM” job means 06:00 UTC, not local time — and the cron scheduler can lag by several minutes under load, so never assume exact-minute execution. And a job that runs longer than timeout-minutes is killed with no partial credit, which is why heavy routing work stays in Airflow and only lightweight validation and publishing runs here.

Annotated implementation #

The workflow below runs daily, fans out over regions with a matrix, caches the pip environment so geospatial wheels are not rebuilt each run, injects secrets as environment variables, runs a Python refresh step, and uploads a checksummed artifact. Secrets are referenced through the secrets context and never echoed.

name: scheduled-spatial-refresh

on:
  schedule:
    - cron: "0 6 * * *"        # daily 06:00 UTC (cron is always UTC)
  workflow_dispatch: {}         # allow manual runs for backfills

permissions:
  contents: read                # least privilege

concurrency:
  group: refresh-${{ github.ref }}
  cancel-in-progress: false     # let an in-flight refresh finish

jobs:
  refresh:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false          # one region failing must not cancel the rest
      max-parallel: 4
      matrix:
        region: [midwest, south, west, northeast]
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"          # cache the pip download layer

      - name: Cache geo wheels
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: geo-${{ runner.os }}-${{ hashFiles('requirements.txt') }}

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run refresh
        env:
          CENSUS_API_KEY: ${{ secrets.CENSUS_API_KEY }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: python -m refresh.run --region "${{ matrix.region }}"

      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: refresh-${{ matrix.region }}
          path: out/${{ matrix.region }}.parquet
          retention-days: 14

The Python step the workflow invokes is a small, single-purpose refresh: pull the changed demographic rows, revalidate the coordinates against WGS84 bounds and geometry validity, and write a GeoParquet artifact with a checksum for later verification. It asserts a CRS before any spatial operation, exactly as the data validation rules for store coordinates require.

# refresh/run.py
import argparse
import hashlib
import os
import geopandas as gpd
from pyproj import CRS

STORAGE_CRS = CRS.from_epsg(4326)


def refresh_region(region: str) -> str:
    api_key = os.environ["CENSUS_API_KEY"]        # injected from repo secrets
    acs = pull_changed_acs(region, api_key)       # small delta, not a full load
    coords = load_coordinates(region)

    if coords.crs is None:                        # fail loudly on missing CRS
        raise ValueError(f"{region}: coordinates missing CRS")
    coords = coords.to_crs(STORAGE_CRS)

    in_bounds = coords.geometry.x.between(-180, 180) & coords.geometry.y.between(-90, 90)
    valid = coords[coords.geometry.is_valid & in_bounds].copy()
    dropped = len(coords) - len(valid)
    if dropped:
        print(f"{region}: dropped {dropped} invalid/out-of-bounds coordinates")

    attributed = valid.merge(acs, on="geoid", how="left")
    out = gpd.GeoDataFrame(attributed, geometry="geometry", crs=STORAGE_CRS)

    path = f"out/{region}.parquet"
    os.makedirs("out", exist_ok=True)
    out.to_parquet(path)

    digest = hashlib.sha256(open(path, "rb").read()).hexdigest()
    print(f"{region}: wrote {len(out)} rows, sha256={digest}")
    return digest


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--region", required=True)
    args = ap.parse_args()
    refresh_region(args.region)

The demographic delta this job pulls is the incremental counterpart to the full ACS sync; the scheduled workflow keeps the coordinate set fresh between the heavier Airflow runs rather than re-attributing every catchment nightly.

Secrets and credential handling #

Credentials are the part most workflows get wrong. Three rules keep the refresh from leaking or over-privileging.

• Never inline a secret in YAML. Reference it through ${{ secrets.NAME }} and inject it as a step-scoped env value, as the Run refresh step does. Anything committed to the workflow file is in the repository history forever, including on public forks.
• Scope the token to the job. The top-level permissions: contents: read strips the default write-heavy GITHUB_TOKEN down to what a read-and-publish job actually needs. Grant additional scopes only on the step that requires them.
• Prefer short-lived credentials over static keys. For the object-store write, an OIDC trust relationship (permissions: id-token: write plus a cloud role assumption) issues a token that expires with the run, so there is no long-lived AWS_SECRET_ACCESS_KEY to rotate or leak. Static keys in secrets are acceptable for lower-risk endpoints like the Census API, whose key is rate-limited rather than write-capable.

The fork boundary matters for scheduled and pull-request workflows alike: a workflow triggered from a fork cannot read the repository’s secrets by design, so a KeyError on CENSUS_API_KEY in a PR check is expected, not a misconfiguration. Gate the credentialed publish step behind the schedule and workflow_dispatch triggers, and let fork PRs run only the validation that needs no secrets.

Caching geospatial dependencies #

Geospatial wheels are heavy — geopandas drags in pyproj, shapely, and fiona, each carrying compiled GEOS/PROJ/GDAL binaries — and reinstalling them on every scheduled run wastes minutes of runner time. Two cache layers cut that cost. The cache: "pip" option on setup-python caches pip’s download directory, and the explicit actions/cache step keyed on hashFiles('requirements.txt') persists the resolved wheels across runs so they are only re-fetched when a pinned version changes. Keying on the requirements hash is what makes the cache correct: a dependency bump changes the hash, invalidates the cache, and forces a clean install, while an unchanged manifest reuses the warm cache. Pin exact versions rather than ranges so the cache key is stable and the environment is reproducible from run to run.

Failure modes and debugging #

The most surprising failure is the paused schedule: GitHub disables scheduled workflows in a repository with no activity for 60 days, and the cron simply stops firing with no error. A weekly commit, a keepalive job, or an occasional manual workflow_dispatch keeps it live.

Verification #

Confirm the workflow did what it should before trusting its output.

1. Workflow status. The run is green across every matrix leg. A single red region under fail-fast: false means the others succeeded but that region needs triage — check its logs, not the whole run.
2. Artifact checksum. Compare the sha256 printed by each region against the prior run. An unchanged checksum on a day with no upstream change confirms determinism; an unexpected change flags drift worth investigating.
3. Row-count log line. Each region logs rows written and coordinates dropped. A sudden spike in drops signals an upstream schema or geometry problem.
4. Reproduce locally. Run python -m refresh.run --region midwest locally with the same pinned requirements.txt and confirm the checksum matches the runner’s — proof the environment is reproducible and not runner-dependent.

# locally, against the same pinned requirements
python -m refresh.run --region midwest
# -> midwest: wrote 4213 rows, sha256=9f2c...  (must match the workflow's log)

A matching checksum between the runner and a local run is the strongest signal: it proves the cached environment is reproducible and that the refresh depends only on its inputs, not on the runner. Feeding that checksum into a cross-run comparison is exactly how the drift monitoring gate turns a scheduled job into an early-warning system for upstream data problems.

Frequently Asked Questions #

Related #

• Orchestrating Spatial Pipelines with Airflow — where the heavy, multi-stage production refresh lives.
• Syncing US Census ACS Data via API — the full demographic sync this job incrementally refreshes.
• Monitoring Spatial Data Drift in CI Pipelines — turning the artifact checksum into a drift gate.

← Back to Orchestrating Spatial Pipelines with Airflow