Verifying CRS Metadata Integrity in GeoParquet Archives

GeoParquet stores its coordinate reference system as PROJJSON inside the geo key of the Parquet file footer, and across a partitioned archive that single block is where CRS integrity quietly fails: one partition carries a full PROJJSON definition, its neighbour a legacy WKT1 string, a third a bare EPSG:4326 shorthand, and a fourth an axis order that disagrees with the coordinates it labels. This guide is for the data engineers who own a partitioned GeoParquet tree and need an automated gate that reads every footer, confirms each geo block names the intended archival CRS in the correct form, and catches the WKT1-versus-WKT2 and axis-order defects that read fine locally but corrupt a spatial join at scale. It is the columnar-format specialisation of CRS Synchronization in Pipelines under Format Conversion & Pipeline Automation, and it treats the geo block as a contract every partition must satisfy identically.

What the geo Block Must Guarantee #

The GeoParquet specification requires the geo metadata to name a version, a primary_column, and per-column encoding and CRS. The CRS should be PROJJSON — a machine-readable object that pins the datum, coordinate system, and axis order unambiguously. Integrity breaks when a writer substitutes a lossy form: a WKT1 string cannot express axis order reliably, a bare EPSG:4326 shorthand defers axis order to the reader, and a null CRS means “assume longitude/latitude” — three different ways of losing the very information the archive exists to preserve. Because these defects live in the footer and not the data, a file opens and previews correctly while its CRS is wrong.

The verification therefore inspects the footer of every partition and asserts four independent properties against it — the CRS is PROJJSON, it equals the intended archival CRS, its axis order is the WKT2 longitude/latitude convention, and the geo key is present on every leaf, not just the one you sampled:

Validating the geo Block Across a Partitioned Tree #

The checks read footers only — never the geometry payload — so a full audit of a terabyte-scale tree costs seconds per thousand files. The gate runs after any write that could touch metadata, including the conversions described in Converting Legacy Shapefiles to GeoParquet at Scale.

Phase 1: Read and Structurally Validate Every Footer #

Walk the partition tree and pull the geo block from each footer with pyarrow, which reads Parquet metadata without materialising a single row group. Assert the required keys exist before inspecting the CRS — a missing geo key or absent primary_column is a hard structural failure.

import json, pathlib
import pyarrow.parquet as pq

TREE = pathlib.Path("datasets/vector/archive")
findings = []
for path in TREE.rglob("*.parquet"):
    md = pq.read_metadata(path)
    if b"geo" not in md.metadata:
        findings.append({"file": str(path), "error": "MISSING_GEO_KEY"})
        continue
    geo = json.loads(md.metadata[b"geo"])
    col = geo.get("primary_column")
    if not col or col not in geo.get("columns", {}):
        findings.append({"file": str(path), "error": "NO_PRIMARY_COLUMN"})
        continue
    findings.append({"file": str(path), "geo": geo["columns"][col]})

Phase 2: Assert PROJJSON Form and Target Equality #

Each surviving crs value must be PROJJSON — a JSON object — and must equal the archival target when parsed. A string value signals a legacy WKT1 or bare-code substitution; a None value signals the implicit-lon/lat default. Both fail. Use pyproj.CRS.equals for the comparison so EPSG:4326, its WKT2 form, and its PROJJSON compare equal by identity rather than by text.

import pyproj

TARGET = pyproj.CRS.from_epsg(4326)

def check_crs(entry):
    crs = entry.get("crs")
    if crs is None:
        return "NULL_CRS_IMPLICIT_LONLAT"
    if not isinstance(crs, dict):
        return "NOT_PROJJSON"                      # WKT1 string or "EPSG:xxxx" shorthand
    parsed = pyproj.CRS.from_json_dict(crs)
    if not parsed.equals(TARGET):
        return f"WRONG_CRS:{parsed.to_epsg()}"
    return "OK"

Phase 3: Catch Axis-Order and WKT1-vs-WKT2 Defects #

Axis order is the defect PROJJSON exists to prevent and WKT1 cannot express. Inspect the coordinate-system axes in the PROJJSON and confirm the first axis is easting/longitude, then cross-check against the actual coordinate envelope: if the declared order says lon/lat but the sampled xmin behaves like a latitude, the geometry was written transposed.

import duckdb

def axis_order_ok(crs_dict):
    axes = crs_dict.get("coordinate_system", {}).get("axis", [])
    if not axes:
        return False
    first = axes[0].get("direction")               # "east"/"north" or geographic dir
    return first in ("east", "geodeticEast")

con = duckdb.connect(); con.execute("INSTALL spatial; LOAD spatial;")
def envelope_matches_lonlat(path):
    xmin, ymin = con.execute(f"""
        SELECT min(ST_XMin(geom)), min(ST_YMin(geom))
        FROM (SELECT geom FROM ST_Read('{path}') USING SAMPLE 2000 ROWS)
    """).fetchone()
    # Longitude range is wider than latitude; a swapped file inverts this.
    return abs(xmin) <= 180 and abs(ymin) <= 90

Phase 4: Fan the Gate Across the Whole Tree #

Combine the checks into one pass and fail the archive if a single partition disagrees. A tree is only trustworthy when uniform: a query engine that reads the geo block from the first file it opens will apply that CRS to the whole dataset, so one deviant partition mislabels everything read after it.

import sys

verdicts = {}
for f in findings:
    if "error" in f:
        verdicts[f["file"]] = f["error"]; continue
    crs = check_crs(f["geo"])
    axis = "OK" if axis_order_ok(f["geo"].get("crs", {})) else "BAD_AXIS_ORDER"
    verdicts[f["file"]] = crs if crs != "OK" else axis

failures = {k: v for k, v in verdicts.items() if v != "OK"}
json.dump(verdicts, open("geo_integrity_report.json", "w"), indent=2)
sys.exit(1 if failures else 0)                     # non-zero blocks promotion in CI

Confirming the Gate on a Sample Partition #

Spot-check any partition with the DuckDB Parquet reader to read the raw footer key, and confirm the crs renders as a PROJJSON object naming the target authority.

duckdb -c "SELECT decode(value) FROM parquet_kv_metadata(
  'datasets/vector/archive/region=north/part-0007.parquet') WHERE decode(key) = 'geo'"

Annotated expected output — the crs is a nested object (PROJJSON), not a string, and its id names the target authority:

{"version":"1.1.0","primary_column":"geometry","columns":{"geometry":{
  "encoding":"WKB","geometry_types":["MultiPolygon"],
  "crs":{"type":"GeographicCRS","name":"WGS 84",
         "coordinate_system":{"axis":[{"direction":"east"},{"direction":"north"}]},
         "id":{"authority":"EPSG","code":4326}}}}}

If the crs prints as "EPSG:4326" in quotes or is absent, the writer emitted a lossy form and the partition fails the PROJJSON check even though pyproj can still parse it — the archive standard is the full object, because only it survives a future reader that does not share your EPSG database. Record the passing CRS form in the Metadata Cataloging & Discovery index so the catalog and the footer never disagree.

Troubleshooting #

Operational Execution Checklist #

• Read the geo block from every partition footer with pyarrow, without materialising row groups.
• Fail structurally on any partition missing the geo key or a resolvable primary_column.
• Require the crs to be PROJJSON; reject WKT1 strings, bare EPSG shorthands, and null.
• Compare against the archival target with pyproj.CRS.equals, not string matching.
• Assert first-axis east/longitude and cross-check the sampled envelope for transposed geometry.
• Fan the gate across the entire tree and block promotion if a single partition disagrees.
• Record the confirmed PROJJSON form in the catalog so footer and index stay in lockstep.

Related #

• Up: CRS Synchronization in Pipelines — the parent reference for the projection contract this gate enforces on columnar output.
• Detecting and Fixing CRS Drift in Archived Datasets — the sibling audit that uses these footer checks as its declared-CRS inventory step.
• Managing EPSG Datum Shifts in Long-Term Archives — the sibling that adds realization and coordinate-epoch fields the PROJJSON must also carry.
• Converting Legacy Shapefiles to GeoParquet at Scale — the neighbouring conversion path that writes the geo block this gate validates.
• Metadata Cataloging & Discovery — the archival-side catalog that must mirror the confirmed PROJJSON so index and footer never drift apart.

Part of the Spatial Data Archival knowledge base.