Spatial Data Archival · Cold Storage

Preserving Field Names Across Shapefile to GeoPackage Conversion

Preserving field names across a shapefile-to-GeoPackage conversion means defeating the DBF ten-character column-name limit that silently truncates and collides descriptive attribute names before they ever reach the long-name-capable GeoPackage schema. A shapefile’s DBF header stores every field name in ten bytes; population_density and population_total both arrive as populatio, and the second write clobbers the first. GeoPackage and GeoParquet impose no such ceiling, so the conversion is an opportunity to restore intent — but only if you carry an explicit field-name mapping instead of trusting the driver to reverse a lossy truncation it cannot undo. This guide is for the data engineers and GIS archivists who need round-trip-safe attribute names when lifting legacy shapefile archives into GeoPackage under the Schema Mapping & Attribute Validation framework.

Where the Ten-Character Ceiling Bites

The dBASE III format that backs the shapefile .dbf reserves exactly eleven bytes for a field name, one of which is a null terminator, leaving ten usable characters. GDAL enforces this on write by truncating, and — critically — by resolving collisions with numeric suffixes only within a single write session. Convert parcel_owner_name and parcel_owner_id into a shapefile and you get parcel_own and parcel_o_1; the suffix is positional, not semantic, so a second export in a different field order produces parcel_o_1 for a different column. Once names have degraded this way, no downstream tool can recover the originals, because the mapping from long name to truncated stub is many-to-one and unlabeled. GeoPackage stores column names as SQLite identifiers with no practical length limit, which is why it is the right target — but the truncation happens on the shapefile side, so the fix must live in the conversion step, not after it. This is the specific mechanism behind much of the loss catalogued in handling attribute loss during spatial format conversion.

The Alias Table Model

The durable fix is a field-name mapping table — an explicit, version-controlled record of truncated_stub → canonical_name — captured either from the original data dictionary or from a metadata sidecar carried alongside the shapefile. The conversion applies the table to rename columns to their full form as it writes GeoPackage, and the same table drives round-trip verification:

Field-name truncation, collision, and alias restoration A four-column matrix: the canonical source field name, its ten-character DBF truncation, whether that truncation collides with another field, and the GeoPackage name restored from the alias table. Two fields collide on the stub populatio and are separated again by the alias mapping. Source field DBF stub (10 char) Collision Restored (GeoPackage) population_density populatio yes (!) population_density population_total populatio yes (!) population_total land_use_code land_use_c no land_use_code

The highlighted band marks the collision: both population fields share the populatio stub, so a naive shapefile-to-GeoPackage copy cannot know which restored name belongs to which column. The alias table breaks the ambiguity by keying restoration on something other than the truncated name — position, or better, a stable sidecar written when the shapefile was first created.

How Ten Characters Become a Collision

The DBF field-name limit does not merely shorten names; it merges distinct ones. Truncation is applied per field in table order, and a writer that finds a duplicate resolves it by appending digits — so the surviving names depend on the order the fields happened to be in.

Truncation collisions and the names they leave behind Four descriptive source field names truncated to ten characters, producing two collisions that the writer resolves with numeric suffixes. The resulting names do not indicate which source field each came from. source fieldtruncatedwritten asrecoverable? installation_dateinstallati installationly via the alias table installation_typeinstallati — collision install_1no — order-dependent maintenance_intervalmaintenanc maintenanconly via the alias table maintenance_ownermaintenanc — collision mainten_1no — order-dependent Reorder the source fields and the suffixes land on different columns — which is why the alias table must be written at conversion time, not reconstructed later.

The order dependency is what makes reconstruction impossible after the fact: two exports of the same dataset can assign _1 to different fields, so a later reader cannot map names back by rule. The alias table has to be produced by the same job that did the truncation, stored alongside the output, and carried forward into every downstream format — which is exactly what the GeoPackage conversion should restore.

Building and Applying the Mapping

1. Capture the canonical names before truncation happens

The only reliable source of full names is upstream of the shapefile. If the archive still holds the originating GeoPackage, PostGIS table, or a documented data dictionary, extract the canonical list there. When only the shapefile survives, write a mapping sidecar the first time a human confirms the intended names, and version it with the data:

{
  "layer": "parcels_region_north",
  "field_map": {
    "populatio":  "population_density",
    "populat_1":  "population_total",
    "land_use_c": "land_use_code",
    "parcel_own": "parcel_owner_name",
    "parcel_o_1": "parcel_owner_id"
  }
}

The keys are the exact truncated stubs GDAL produced, including its _1 collision suffixes; the values are the canonical names. This file is the contract, checked into the pipeline repository next to the dataset manifest.

2. Apply the map during conversion

Rename as you write GeoPackage so the long names land in the SQLite schema directly. ogr2ogr with an SQL SELECT ... AS clause renames per column without a second pass:

ogr2ogr -f GPKG \
  s3://spatial-archive/parcels/2023/parcels_region_north.gpkg \
  /vsis3/spatial-archive/parcels/2023/parcels_region_north.shp \
  -sql "SELECT populatio AS population_density,
               populat_1 AS population_total,
               land_use_c AS land_use_code,
               parcel_own AS parcel_owner_name,
               parcel_o_1 AS parcel_owner_id,
               GEOMETRY
        FROM parcels_region_north" \
  -nln parcels_region_north

For programmatic pipelines that already hold the sidecar, drive the rename from the JSON so the mapping stays single-sourced:

import json
import geopandas as gpd

field_map = json.load(open("parcels_region_north.field_map.json"))["field_map"]

gdf = gpd.read_file("/vsis3/spatial-archive/parcels/2023/parcels_region_north.shp")
gdf = gdf.rename(columns=field_map)

gdf.to_file(
    "/vsis3/spatial-archive/parcels/2023/parcels_region_north.gpkg",
    layer="parcels_region_north",
    driver="GPKG",
)

Because GeoPackage preserves these long names cleanly, the same renamed frame flows on into columnar storage without further mapping — the naming discipline established here is what makes a later GeoParquet migration workflow round-trip-safe.

Verifying the Round Trip

Confirm every canonical name is present and no stub leaked through. List the GeoPackage schema straight from SQLite and check it against the mapping’s values:

ogrinfo -so \
  s3://spatial-archive/parcels/2023/parcels_region_north.gpkg \
  parcels_region_north | grep -E ':' | grep -iv geometry

Expected output — full names, no ten-character stubs, no _1 collision suffixes:

population_density: Real (0.0)
population_total: Real (0.0)
land_use_code: String (0.0)
parcel_owner_name: String (0.0)
parcel_owner_id: Integer64 (0.0)

Then run a strict assertion in code that fails if any restored name is missing or any truncated stub survived, which is the round-trip guarantee the pipeline gate depends on:

import json
import fiona

expected = set(json.load(open("parcels_region_north.field_map.json"))["field_map"].values())
with fiona.open("/vsis3/spatial-archive/parcels/2023/parcels_region_north.gpkg") as src:
    actual = set(src.schema["properties"].keys())

missing = expected - actual
leaked = {n for n in actual if len(n) == 10 or n[-2:] in ("_1", "_2")}
assert not missing, f"canonical names missing: {missing}"
assert not leaked, f"truncated stubs leaked into GeoPackage: {leaked}"

This paired schema-and-assertion check is the natural sibling of the field-count-and-type diff in validating attribute schema parity after format conversion; run both so name preservation and type parity are certified in one gate.

Where the Alias Table Travels

The alias table is only useful if it reaches every consumer of the data, which means treating it as part of the dataset rather than as a job artefact. Three destinations matter, and each has a different lifetime.

Where the alias table has to be published One conversion job writing the alias mapping to three destinations: a sidecar file beside the data, column comments inside the GeoPackage, and the catalogue entry used for discovery. conversion job one source of truth sidecar · field_aliases.json lives beside the object; survives format changes; easiest to read inside the container · column comments travels with the file even when the sidecar is lost in transit catalogue entry makes the original vocabulary searchable, which is how users find fields Write all three from the same mapping in the same job — a sidecar that disagrees with the column comments is worse than either alone.

The catalogue destination is the one that changes user behaviour. People search for maintenance_owner, not for mainten_1, and a catalogue that indexes only the truncated names makes a well-preserved field effectively invisible.

Troubleshooting Truncation and Collisions

Symptom Cause Fix
Two columns map to one populatio stub, one is missing DBF truncation collided the names and the last write won Recover canonical names from the sidecar or data dictionary and rename with an explicit SELECT ... AS, keyed on the _1-suffixed stubs
Restored name still ten characters long Rename step skipped or the sidecar key did not match GDAL’s actual stub Print the shapefile’s real field list with ogrinfo -so and align sidecar keys to the exact stubs, suffixes included
Integer64 field arrives as Real after rename The rename preserved the name but not the type; DBF lacks 64-bit integers Coerce the type during the GeoPackage write, which supports Integer64 natively
Non-ASCII field name mangled Shapefile field names outside ASCII depend on LDID/encoding Set SHAPE_ENCODING=UTF-8 before reading and confirm the sidecar stores the intended Unicode name

Keep the sidecar and the CRS definition together as the two things that must survive every hop; the projection half of that pairing is enforced by CRS Synchronization in Pipelines. Descriptive, un-truncated names also make categorical columns self-documenting for the compression stage, where dictionary encoding for GIS attributes keys on stable field identities. For the field name rules themselves, the authoritative reference is the OGC GeoPackage Encoding Standard.

Operational Execution Checklist

Frequently Asked Questions

Can the original names be restored automatically on the way out?

Only from a recorded mapping. Given the alias table, restoration is mechanical and should be part of the conversion to any format without the length limit. Without it, a human has to decide which source field install_1 was, and that decision is a guess unless the source is still available to compare against.

What if the alias table itself was never written?

Recover what you can from the source and document the rest. Where the original geodatabase or its schema documentation survives, the mapping can usually be rebuilt by comparing value distributions field by field. Where it does not, record the truncated names as the archive’s canonical names rather than inventing plausible expansions — an honest short name is better than a confident wrong one.

Do other formats have similar limits?

Most modern ones do not, which is why the limit propagates rather than compounds: once names have been truncated by a shapefile hop, every downstream format faithfully preserves the truncation. That asymmetry is the argument for keeping the alias table with the data permanently, since it is the only artefact that carries the original vocabulary forward.

Part of the Spatial Data Archival knowledge base.

Related pages

Handling Attribute Loss During Spatial Format Conversion Validating Attribute Schema Parity After Format Conversion