Spatial Data Archival · Cold Storage

Handling Attribute Loss During Spatial Format Conversion

Attribute loss during spatial format conversion operates as a silent failure mode that compromises archival integrity, breaks downstream analytics, and triggers compliance violations. When migrating legacy shapefiles, GeoJSON, or PostGIS exports to columnar or cloud-optimized formats, implicit type coercion, field-name truncation, and metadata stripping routinely drop critical attributes. This guide delivers exact configuration steps, validation rules, and pipeline controls for data engineers, GIS archivists, cloud architects, and compliance/ops teams operating within spatial data archival and cold storage optimization workflows.

Attribute-Loss Failure Modes

Each common loss mode maps to a deterministic mitigation:

flowchart LR
  A["Conversion"] --> B{"Failure mode"}
  B -->|"Name truncation"| M1["Deterministic aliasing"]
  B -->|"Type coercion"| M2["Explicit casts"]
  B -->|"Metadata stripping"| M3["Sidecar manifest"]

Deterministic Failure Modes & Root Cause Analysis

Attribute loss originates from three deterministic failure points: schema incompatibility, unmanaged type coercion, and metadata serialization gaps. Shapefile DBF specifications restrict field names to 10 ASCII characters and cap numeric precision at 18 digits. GeoJSON lacks strict typing, forcing downstream converters to infer types from the first row. When targeting GeoParquet or FlatGeobuf, unhandled null semantics, mixed-type arrays, and unregistered coordinate reference systems (CRS) trigger silent column drops. Proper Schema Mapping & Attribute Validation must precede any batch migration to prevent irreversible data degradation.

Pre-Conversion Schema Profiling & Audit

Execute a strict schema audit before conversion to establish a baseline for attribute parity.

  1. Extract Source Schema: Use GDAL/OGR or DuckDB to dump the exact schema.
ogrinfo -al -so input.shp
# OR for GeoJSON/PostGIS (the spatial extension provides ST_Read)
duckdb -c "INSTALL spatial; LOAD spatial; DESCRIBE SELECT * FROM ST_Read('input.geojson');"
  1. Map Field Types to Target Specifications: Reject implicit String → Integer or Float → Decimal promotions. Define explicit casting rules in a YAML manifest. Cross-reference against the Apache Parquet Type System to ensure target compatibility.
  2. Validate Field Name Constraints: GeoParquet and FlatGeobuf both accept long UTF-8 field names, but round-tripping through Shapefile DBF caps names at 10 characters. Implement deterministic aliasing whenever a target format imposes a length limit:
import hashlib
def alias_field(name: str, max_len: int = 64) -> str:
    if len(name) <= max_len: return name
    return f"{name[:max_len-12]}_{hashlib.md5(name.encode()).hexdigest()[:8]}"
  1. Enforce Null-Tolerance Thresholds: If a column exceeds 15% null values in the source, flag it for explicit NULLABLE declaration in the target schema. Columns exceeding 85% nulls should be dropped pre-conversion to optimize cold storage footprint.
  2. Verify CRS Metadata Presence: Missing EPSG codes or WKT strings cause geometry columns to be dropped during serialization. Validate with ogrinfo -so input.shp | grep -i "srs".

Pipeline Configuration & Execution Controls

Automated conversion pipelines require deterministic type mapping and explicit schema enforcement. Disable automatic type promotion and enforce UTF-8 encoding across all vector drivers.

GDAL/OGR Execution:

ogr2ogr -f "Parquet" output.parquet input.shp \
  -dialect SQLITE \
  -lco GEOMETRY_NAME=geometry \
  -lco COMPRESSION=ZSTD \
  -nln target_layer \
  -sql "SELECT CAST(field_a AS REAL) AS field_a, CAST(field_b AS TEXT) AS field_b, geometry FROM input"

DuckDB/PyArrow Execution: For cloud-native workflows, register explicit column types before write operations to bypass inference engines.

import duckdb
import pyarrow.parquet as pq

con = duckdb.connect()
con.execute("""
    INSTALL spatial;
    LOAD spatial;
""")

# Explicit schema enforcement during read
query = """
    SELECT
        CAST(id AS BIGINT) AS id,
        CAST(measurement AS DOUBLE) AS measurement,
        CAST(status AS VARCHAR) AS status,
        geom
    FROM ST_Read('input.shp')
"""

# Write with strict schema and GeoParquet v1.0 compliance
con.execute(f"COPY ({query}) TO 'output.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")

Refer to Format Conversion & Pipeline Automation for orchestration patterns that enforce idempotent execution and rollback triggers on schema mismatch.

Post-Conversion Validation & Integrity Checks

Validation must occur immediately after write operations. Implement a three-tier verification sequence:

  1. Row & Column Parity Check:
# Source vs Target row count
src_rows=$(ogrinfo -ro -al -so input.shp | grep "Feature Count:" | awk '{print $NF}')
tgt_rows=$(duckdb -noheader -list -c "SELECT count(*) FROM read_parquet('output.parquet');")
[ "$src_rows" -eq "$tgt_rows" ] || echo "ROW COUNT MISMATCH: $src_rows vs $tgt_rows"
  1. Schema Delta Verification: Dump the target column names and types and compare them against the source schema (ogrinfo -so input.shp).
duckdb -c "DESCRIBE SELECT * FROM read_parquet('output.parquet');"
  1. Null & Type Drift Analysis: SUMMARIZE reports per-column type and null percentage in one pass, surfacing unexpected promotions or null injection.
SUMMARIZE SELECT * FROM read_parquet('output.parquet');
  1. Geometry Integrity Validation: Ensure spatial attributes survived serialization without coordinate degradation. The geometry column is stored as WKB, so load the spatial extension and decode it first.
duckdb -c "INSTALL spatial; LOAD spatial; SELECT count(*) FROM read_parquet('output.parquet') WHERE NOT ST_IsValid(ST_GeomFromWKB(geom));"

Archival Compliance & Cold Storage Optimization

Attribute loss triggers compliance violations in regulated archival environments. Implement cryptographic checksums and audit trails for every conversion batch. Store schema manifests alongside cold storage objects (S3 Glacier, Azure Archive) to enable deterministic reconstruction.

  • Manifest Generation: Export target schema and row statistics to JSON before archival.
  • Checksum Verification: Generate SHA-256 hashes for both source and target files. Store in an immutable ledger.
  • Retention Mapping: Map deprecated attributes to a legacy_metadata JSON column if they fail target schema validation but require regulatory retention.

Enforce strict version control on conversion pipelines. Any schema drift detected during cold storage retrieval must trigger an automated rollback to the last verified manifest. Consult the OGC GeoParquet Specification for compliance requirements regarding spatial metadata serialization and attribute preservation in long-term archival formats.