Spatial Data Archival · Cold Storage

Verifying CRS Metadata Integrity in GeoParquet Archives

GeoParquet stores its coordinate reference system as PROJJSON inside the geo key of the Parquet file footer, and across a partitioned archive that single block is where CRS integrity quietly fails: one partition carries a full PROJJSON definition, its neighbour a legacy WKT1 string, a third a bare EPSG:4326 shorthand, and a fourth an axis order that disagrees with the coordinates it labels. This guide is for the data engineers who own a partitioned GeoParquet tree and need an automated gate that reads every footer, confirms each geo block names the intended archival CRS in the correct form, and catches the WKT1-versus-WKT2 and axis-order defects that read fine locally but corrupt a spatial join at scale. It is the columnar-format specialisation of CRS Synchronization in Pipelines under Format Conversion & Pipeline Automation, and it treats the geo block as a contract every partition must satisfy identically.

What the geo Block Must Guarantee

The GeoParquet specification requires the geo metadata to name a version, a primary_column, and per-column encoding and CRS. The CRS should be PROJJSON — a machine-readable object that pins the datum, coordinate system, and axis order unambiguously. Integrity breaks when a writer substitutes a lossy form: a WKT1 string cannot express axis order reliably, a bare EPSG:4326 shorthand defers axis order to the reader, and a null CRS means “assume longitude/latitude” — three different ways of losing the very information the archive exists to preserve. Because these defects live in the footer and not the data, a file opens and previews correctly while its CRS is wrong.

The verification therefore inspects the footer of every partition and asserts four independent properties against it — the CRS is PROJJSON, it equals the intended archival CRS, its axis order is the WKT2 longitude/latitude convention, and the geo key is present on every leaf, not just the one you sampled:

Anatomy of the GeoParquet geo block and its four integrity checks The geo metadata block in the Parquet footer with version, primary_column, encoding, and a highlighted PROJJSON crs field, connected to four checks: CRS is PROJJSON, CRS equals the archival target, axis order is longitude/latitude per WKT2, and the geo key exists on every partition. Parquet footer · “geo” key {   "version": "1.1.0",   "primary_column": "geometry",   "columns": { "geometry": {     "encoding": "WKB",     "crs": { …PROJJSON… }   } } one footer per partition file CRS is PROJJSON not a WKT1 string or bare EPSG code CRS equals archival target declared ↔ intended EPSG Axis order lon/lat (WKT2:2019) no WKT1 lat/lon ambiguity geo key on EVERY partition fan across the whole tree

Validating the geo Block Across a Partitioned Tree

The checks read footers only — never the geometry payload — so a full audit of a terabyte-scale tree costs seconds per thousand files. The gate runs after any write that could touch metadata, including the conversions described in Converting Legacy Shapefiles to GeoParquet at Scale.

Walk the partition tree and pull the geo block from each footer with pyarrow, which reads Parquet metadata without materialising a single row group. Assert the required keys exist before inspecting the CRS — a missing geo key or absent primary_column is a hard structural failure.

import json, pathlib
import pyarrow.parquet as pq

TREE = pathlib.Path("datasets/vector/archive")
findings = []
for path in TREE.rglob("*.parquet"):
    md = pq.read_metadata(path)
    if b"geo" not in md.metadata:
        findings.append({"file": str(path), "error": "MISSING_GEO_KEY"})
        continue
    geo = json.loads(md.metadata[b"geo"])
    col = geo.get("primary_column")
    if not col or col not in geo.get("columns", {}):
        findings.append({"file": str(path), "error": "NO_PRIMARY_COLUMN"})
        continue
    findings.append({"file": str(path), "geo": geo["columns"][col]})

Phase 2: Assert PROJJSON Form and Target Equality

Each surviving crs value must be PROJJSON — a JSON object — and must equal the archival target when parsed. A string value signals a legacy WKT1 or bare-code substitution; a None value signals the implicit-lon/lat default. Both fail. Use pyproj.CRS.equals for the comparison so EPSG:4326, its WKT2 form, and its PROJJSON compare equal by identity rather than by text.

import pyproj

TARGET = pyproj.CRS.from_epsg(4326)

def check_crs(entry):
    crs = entry.get("crs")
    if crs is None:
        return "NULL_CRS_IMPLICIT_LONLAT"
    if not isinstance(crs, dict):
        return "NOT_PROJJSON"                      # WKT1 string or "EPSG:xxxx" shorthand
    parsed = pyproj.CRS.from_json_dict(crs)
    if not parsed.equals(TARGET):
        return f"WRONG_CRS:{parsed.to_epsg()}"
    return "OK"

Phase 3: Catch Axis-Order and WKT1-vs-WKT2 Defects

Axis order is the defect PROJJSON exists to prevent and WKT1 cannot express. Inspect the coordinate-system axes in the PROJJSON and confirm the first axis is easting/longitude, then cross-check against the actual coordinate envelope: if the declared order says lon/lat but the sampled xmin behaves like a latitude, the geometry was written transposed.

import duckdb

def axis_order_ok(crs_dict):
    axes = crs_dict.get("coordinate_system", {}).get("axis", [])
    if not axes:
        return False
    first = axes[0].get("direction")               # "east"/"north" or geographic dir
    return first in ("east", "geodeticEast")

con = duckdb.connect(); con.execute("INSTALL spatial; LOAD spatial;")
def envelope_matches_lonlat(path):
    xmin, ymin = con.execute(f"""
        SELECT min(ST_XMin(geom)), min(ST_YMin(geom))
        FROM (SELECT geom FROM ST_Read('{path}') USING SAMPLE 2000 ROWS)
    """).fetchone()
    # Longitude range is wider than latitude; a swapped file inverts this.
    return abs(xmin) <= 180 and abs(ymin) <= 90

Phase 4: Fan the Gate Across the Whole Tree

Combine the checks into one pass and fail the archive if a single partition disagrees. A tree is only trustworthy when uniform: a query engine that reads the geo block from the first file it opens will apply that CRS to the whole dataset, so one deviant partition mislabels everything read after it.

import sys

verdicts = {}
for f in findings:
    if "error" in f:
        verdicts[f["file"]] = f["error"]; continue
    crs = check_crs(f["geo"])
    axis = "OK" if axis_order_ok(f["geo"].get("crs", {})) else "BAD_AXIS_ORDER"
    verdicts[f["file"]] = crs if crs != "OK" else axis

failures = {k: v for k, v in verdicts.items() if v != "OK"}
json.dump(verdicts, open("geo_integrity_report.json", "w"), indent=2)
sys.exit(1 if failures else 0)                     # non-zero blocks promotion in CI

Anatomy of the geo Metadata Block

The geo key in a GeoParquet file’s footer is a JSON document with a small number of required members, and validation is mostly a matter of checking that each is present, well-formed, and consistent with the data it describes. Laid out, the block makes obvious which fields a reader depends on and which a careless writer omits.

The geo metadata block, member by member A nested structure diagram of the GeoParquet geo block showing required top-level members and the per-column members, with notes on which are commonly omitted and what breaks when they are. key: “geo” · file-level Parquet metadata version “1.1.0” — required primary_column “geometry” — required columns map of column name → descriptor — required columns[“geometry”] encoding “WKB” — required geometry_types [“Polygon”, “MultiPolygon”] crs · PROJJSON omitted → reader assumes OGC:CRS84 bbox optional · drives discovery orientation optional · ring winding epoch optional — but required for dynamic frames Highlighted members are the two whose omission is silently tolerated by readers and materially wrong for an archive.

The highlighted pair is where a validator earns its keep. An omitted crs is not an error under the specification — it means longitude-latitude on WGS84 — so a file written from projected data with the member missing is technically valid and geographically wrong. An omitted epoch is legal for a static frame and a defect for a dynamic one. Neither omission raises a warning in any writer, which is precisely why the gate has to assert on their presence rather than on the file’s validity.

Confirming the Gate on a Sample Partition

Spot-check any partition with the DuckDB Parquet reader to read the raw footer key, and confirm the crs renders as a PROJJSON object naming the target authority.

duckdb -c "SELECT decode(value) FROM parquet_kv_metadata(
  'datasets/vector/archive/region=north/part-0007.parquet') WHERE decode(key) = 'geo'"

Annotated expected output — the crs is a nested object (PROJJSON), not a string, and its id names the target authority:

{"version":"1.1.0","primary_column":"geometry","columns":{"geometry":{
  "encoding":"WKB","geometry_types":["MultiPolygon"],
  "crs":{"type":"GeographicCRS","name":"WGS 84",
         "coordinate_system":{"axis":[{"direction":"east"},{"direction":"north"}]},
         "id":{"authority":"EPSG","code":4326}}}}}

If the crs prints as "EPSG:4326" in quotes or is absent, the writer emitted a lossy form and the partition fails the PROJJSON check even though pyproj can still parse it — the archive standard is the full object, because only it survives a future reader that does not share your EPSG database. Record the passing CRS form in the Metadata Cataloging & Discovery index so the catalog and the footer never disagree.

Running the Gate Across a Partitioned Tree Efficiently

Validating metadata does not require reading the data, and treating it as if it did is what makes teams run the check quarterly instead of on every write. A Parquet footer sits at the end of the file and is a few kilobytes; two ranged reads — one for the footer length, one for the footer itself — retrieve everything the gate needs.

Whole-file validation versus footer-only validation Two bars comparing bytes transferred and elapsed time for validating metadata across a 2,240-partition archive: 640 gigabytes and 95 minutes reading whole files, against 41 megabytes and 70 seconds reading only footers. Metadata validation across 2,240 partitions read whole files 640 GB transferred · ~95 min · egress billed read footers only 41 MB transferred · ~70 s Two ranged requests per file 1. GET the last 8 bytes → footer length + “PAR1” magic. 2. GET that many bytes from the end → the full footer, including the geo block. Neither request touches a data page, so nothing is decompressed and no archive-tier restore is triggered for instant-retrieval classes.

The cost profile is what makes this gate practical to run on every write and cheap enough to sweep the whole archive nightly. The one class it cannot reach is data already in a non-instant archive tier, where even a footer read requires a restore — for those partitions, validate at write time and rely on the recorded result, since an immutable object’s metadata cannot have changed since.

Troubleshooting

Symptom Root cause Diagnostic & fix
crs prints as a quoted "EPSG:4326" string Writer stored the shorthand instead of PROJJSON Rewrite the footer with table.replace_schema_metadata, injecting pyproj.CRS(...).to_json_dict()
Some partitions pass, others report MISSING_GEO_KEY Non-spatial writer (plain pyarrow/pandas) touched part of the tree Re-emit affected partitions through a GeoParquet writer; add the gate to CI so it cannot recur
BAD_AXIS_ORDER with a valid target CRS WKT1 lineage or a lat/lon writer under a lon/lat CRS Re-encode PROJJSON with explicit east/north axes; verify the envelope with the DuckDB sample
Gate passes locally, fails in the reader Reader took the CRS from the first footer; a later partition differs Enforce tree-wide uniformity; block promotion on any single non-OK verdict

Operational Execution Checklist

Frequently Asked Questions

Does a valid geo block guarantee the coordinates are correct?

No. Validation proves the file describes itself consistently; it says nothing about whether the transformation that produced the coordinates was the right one. The two checks are complementary: metadata integrity catches missing and contradictory declarations, and the drift audit catches coordinates that do not match an independent reference. An archive needs both, and they typically run at different points — integrity at write time, drift on a schedule.

What should the gate do when a partition fails?

Fail the whole write, not the partition. A dataset whose partitions disagree about their reference system is worse than one that failed to load, because the disagreement is discoverable only by a reader who compares partitions — which nobody does until a query returns features in two places. Treat metadata parity across partitions as an atomic property of the dataset version.

Can the geo block be repaired without rewriting the data?

Only by rewriting the file, because Parquet metadata lives in the footer and the footer is written last, as part of the file. Some tooling offers an in-place footer rewrite, but on object storage that still means uploading a new object; there is no partial update. The practical consequence is that metadata defects cost the same as data defects once the object has been tiered, which is the argument for asserting before the write rather than after.

Part of the Spatial Data Archival knowledge base.

Related pages

Automating CRS Transformations in ETL Pipelines for Spatial Data Archival Detecting and Fixing CRS Drift in Archived Datasets Managing EPSG Datum Shifts in Long-Term Archives