Spatial Data Archival · Cold Storage

Validating Attribute Schema Parity After Format Conversion

Attribute schema parity validation is the automated gate that proves a converted GeoParquet dataset carries the same field names, types, and column count as its source shapefile or GeoPackage — and fails the pipeline when it does not. Format conversion silently mutates schemas: DBF truncates field names to ten characters, coerces 64-bit integers to floats, and drops columns that exceed driver limits without raising an error. A pipeline that trusts an exit code of zero will publish an archive that looks complete but has lost a survey identifier or rounded a parcel area. This guide is for the data engineers and GIS archivists who need a deterministic parity diff — one that compares source and target schemas field by field, classifies every difference, and blocks promotion on mismatch — under the Schema Mapping & Attribute Validation framework rather than trusting the converter’s own warnings.

Why Successful Conversion Still Loses Attributes

The failure mode is asymmetric: ogr2ogr reports success while quietly reshaping the attribute table. Three mechanisms account for nearly every silent loss. First, the DBF format underlying shapefiles caps field names at ten bytes, so population_density and population_count both collapse toward populatio and one overwrites the other. Second, DBF has no native 64-bit integer or boolean type, so wide identifiers become float64 and lose their least-significant digits past 2^53. Third, drivers apply column limits and encoding substitutions per feature layer, dropping or renaming fields when a target format is stricter than the source. Parity validation exists precisely because none of these raise a non-zero exit. The companion procedure on handling attribute loss during spatial format conversion covers preventing the loss; this page covers proving it did not happen.

The Parity Diff Model

A parity check reduces both schemas to a canonical descriptor — an ordered set of (name, logical_type, nullability) tuples plus a field count — and diffs them. Every difference is classified so the gate can distinguish a benign, expected coercion from a data-destroying one:

Attribute schema parity diff and gate Two schema descriptors — the source shapefile schema and the converted GeoParquet schema — are normalized to (name, type, count) tuples and compared field by field. Each field produces a verdict: match passes, a coerced type is reviewed, and a truncated or dropped name fails the gate before promotion. Source schema shapefile / GPKG parcel_id · int64 area_ha · float64 land_use · string is_zoned · bool population_density Parity comparator Converted schema GeoParquet parcel_id · int64 area_ha · double land_use · string is_zoned · bool populatio (!) 3 fields → PASS (type coercion audited) population_density → FAIL (truncated)

The classifier maps each field difference to one of four verdicts. An exact match on name and logical type passes. A tolerated coercion — int64 widened to a Parquet INT64, DBF Real mapped to double — passes but is written to an audit record so the change is never invisible. A name change, including truncation and case folding, fails hard. A field present in the source but absent in the target fails hard and is the single most important case to catch, because it is the one no converter warns about.

The Four Comparisons a Parity Gate Makes

Parity is not one comparison but four, and each catches defects the others cannot see. Running them as a single pass over both sides keeps the gate cheap enough to sit in the write path.

The four comparisons that make up a parity gate Four comparisons between source and converted output — field set, type mapping, null density and value distribution — each annotated with the defect class it detects and the ones it cannot. source table vs converted output · one pass, four assertions 1 · field set every mapped source field present, no extras catches: dropped columns, silent additions blind to: everything about the values 2 · type mapping each type matches the declared coercion catches: unintended widening or narrowing blind to: values that fit the wrong type fine 3 · null density per-column null rate within tolerance of source catches: columns that arrived empty, failed joins blind to: wrong-but-present values 4 · value distribution min, max, distinct count, checksum of sorted values catches: encoding damage, unit changes, truncation the only one that sees the values themselves

The fourth comparison is the expensive one and the only one that catches a column whose values changed while its shape did not — a measurement rescaled from feet to metres, or a text column re-encoded. A checksum over sorted distinct values is usually enough and costs one aggregate per column on each side, which is affordable even on a large table.

Building the Parity Gate

1. Extract the source descriptor deterministically

Read the source schema without loading geometry so the check is fast even on large layers. ogrinfo emits a stable JSON field list for shapefiles and GeoPackages alike:

ogrinfo -ro -json -so \
  s3://spatial-archive/parcels/2023/parcels_region_north.gpkg parcels \
  | jq '{count: (.layers[0].fields | length),
         fields: [.layers[0].fields[] | {name: .name, type: .type}]}' \
  > /tmp/source_schema.json

The -so (summary-only) flag skips per-feature reads, and jq reduces the output to the canonical count plus a (name, type) array. Capture this descriptor from the raw source before conversion so a corrupted intermediate cannot poison the baseline.

2. Extract the converted descriptor from Parquet metadata

The GeoParquet writer records the Arrow schema in the file footer; read it without scanning row groups:

import pyarrow.parquet as pq

pf = pq.ParquetFile("s3://spatial-archive/parcels/2023/parcels_region_north.parquet")
schema = pf.schema_arrow

converted = {
    "count": sum(1 for f in schema if f.name != "geometry"),
    "fields": [
        {"name": f.name, "type": str(f.type)}
        for f in schema
        if f.name != "geometry"
    ],
}

Exclude the primary geometry column from the attribute count so the comparison is apples-to-apples with the DBF field list, which never includes geometry as an attribute.

3. Normalize and diff

Both descriptors pass through the same normalizer before comparison. Normalization lowercases names, maps driver-specific type spellings to a shared logical vocabulary, and sorts by name so field ordering never causes a false mismatch:

import json

TYPE_ALIASES = {
    "Integer64": "int64", "int64": "int64",
    "Integer": "int32", "int32": "int32",
    "Real": "double", "double": "double", "float": "double",
    "String": "string", "large_string": "string",
    "Date": "date32", "DateTime": "timestamp[us]",
}

def normalize(desc):
    out = {}
    for f in desc["fields"]:
        key = f["name"].lower()
        out[key] = TYPE_ALIASES.get(f["type"], f["type"])
    return out, desc["count"]

src = json.load(open("/tmp/source_schema.json"))
src_fields, src_count = normalize(src)
dst_fields, dst_count = normalize(converted)

missing = sorted(set(src_fields) - set(dst_fields))   # dropped or renamed
extra   = sorted(set(dst_fields) - set(src_fields))    # unexpected additions
coerced = sorted(
    n for n in src_fields.keys() & dst_fields.keys()
    if src_fields[n] != dst_fields[n]
)

report = {
    "count_match": src_count == dst_count,
    "missing_fields": missing,
    "extra_fields": extra,
    "coerced_types": {n: [src_fields[n], dst_fields[n]] for n in coerced},
}
print(json.dumps(report, indent=2))

fatal = bool(missing) or bool(extra) or not report["count_match"]
raise SystemExit(1 if fatal else 0)

The exit code is the contract: any missing field, unexpected field, or count mismatch is fatal, while a coercion is reported and allowed. Truncation surfaces as a missing_fields entry (population_density) paired with an extra_fields entry (populatio), which is exactly the fingerprint of a DBF name collision. Preventing that truncation in the first place is covered under preserving field names across shapefile to GeoPackage conversion.

Confirming the Gate Fires

Run the gate against a known-good pair and inspect the report. A clean parity check on a dataset whose only difference is the audited Real → double coercion should read:

{
  "count_match": true,
  "missing_fields": [],
  "extra_fields": [],
  "coerced_types": {
    "area_ha": ["double", "double"]
  }
}

Then run a DuckDB cross-check that counts non-null values per column on both sides, catching the rarer case where a field survives structurally but its values were nulled during coercion:

duckdb -c "
  SELECT 'parcel_id' AS field, count(parcel_id) AS non_null
  FROM read_parquet('s3://spatial-archive/parcels/2023/parcels_region_north.parquet')"

Expected output — the non-null count must equal the source feature count when the field has no legitimate nulls:

┌───────────┬──────────┐
│   field   │ non_null │
│  varchar  │  int64   │
├───────────┼──────────┤
│ parcel_id │  318204  │
└───────────┴──────────┘

A non-null count below the source feature count on an identifier column means values were dropped even though the column exists — a failure the structural diff alone cannot see.

Tolerances That Make the Gate Usable

A parity gate with no tolerances fails on legitimate differences and gets disabled within a month. Setting them explicitly, per comparison, is what keeps it running.

Tolerances applied to each parity comparison Five comparisons with their tolerance: exact for field set, type mapping and row count; half a percentage point for null density; and a relative floating-point tolerance for distribution bounds. comparisontolerancewhy row countexact any difference is a lost or duplicated record field setexact the mapping declares every field that should exist type mappingexact vs declared coercions are chosen, never discovered null density±0.5 percentage points absorbs sampling and late-arriving rows float min/maxrelative 1e-9 rounding differs between engines, not values

The two tolerated rows are tolerated for specific, documented reasons — not because exactness proved inconvenient. That distinction is what a reviewer should look for: a tolerance with a stated cause is a design decision, and one without is a suppressed failure.

Troubleshooting Parity Failures

Symptom Cause Fix
missing_fields and extra_fields both list one entry with a shared prefix DBF ten-character truncation renamed the field on write Apply an explicit field-name mapping before conversion and re-run; never let the driver auto-truncate
count_match false, target has fewer fields Driver silently dropped a column exceeding a format limit or with an unsupported type Coerce the offending type ahead of conversion and add the field to an explicit -select list
coerced_types shows int64 → double on an ID column DBF has no 64-bit integer; the ID was widened and may lose precision past 2^53 Store the identifier as a string, or convert via GeoPackage/GeoParquet which preserve int64 natively
Parity passes but non-null counts differ Values coerced to null on type mismatch (dates, booleans) Fix the source encoding, re-convert, and re-run the DuckDB non-null cross-check

Wire the gate into the same job that promotes staging output to the canonical archive prefix, so a non-zero exit blocks promotion. Because CRS metadata is a schema invariant that travels alongside the attribute schema, run this parity gate immediately after the projection assertions in CRS Synchronization in Pipelines so a single validation stage certifies both attribute and reference-system fidelity. Low-cardinality categorical fields that survive parity are also the best candidates for dictionary encoding for GIS attributes, so record the parity report’s field list where the compression stage can consume it.

Operational Execution Checklist

Frequently Asked Questions

Should the parity gate run before or after the data is written?

After the write and before the object is published or tiered. Running it earlier means validating something other than the artefact readers will get; running it later means the defective object is already in the catalogue. The sequence that works is write to a staging key, run parity, then copy to the final key on success — which also gives the idempotency marker the pipeline needs.

What is compared when the source is no longer available?

Nothing external, so the gate degrades to internal consistency: declared schema versus actual schema, null densities against the previous generation, and value distributions against the last known-good archive version. That is weaker than true parity and should be labelled as such in the evidence record, because it cannot detect a defect introduced before the first archived generation.

How long should parity evidence be kept?

As long as the data it describes. The evidence is small — a few kilobytes of counts and checksums per generation — and it is the artefact that answers “how do we know this archive is faithful?” years later. Storing it with the archive rather than in a CI system is what makes it survive the tooling that produced it.

Part of the Spatial Data Archival knowledge base.

Related pages

Handling Attribute Loss During Spatial Format Conversion Preserving Field Names Across Shapefile to GeoPackage Conversion