Spatial Data Archival · Cold Storage

Managing EPSG Datum Shifts in Long-Term Archives

A datum is not a fixed thing over the lifetime of an archive: the label “NAD83” hides a family of successive realizations whose coordinates for the same physical point differ by up to one to two metres, and “WGS84” is a moving ensemble tied to shifting ITRF epochs that drift with plate motion at roughly two centimetres a year. This guide is for GIS archivists and geodesists who must retrieve a survey stored in 2004 and reproject it in 2035 to within its original accuracy, which is impossible if the archive recorded only the datum name and not the realization and coordinate epoch. It extends CRS Synchronization in Pipelines inside the Format Conversion & Pipeline Automation discipline into the time dimension, where an EPSG code alone is no longer enough to reconstruct where a coordinate really sits on the Earth.

Why a Datum Label Decays Over Decades

Reference frames are re-realized as the geodetic network improves and the crust moves. NAD83 has been realized as NAD83(1986), NAD83(CORS96), and NAD83(2011) at epoch 2010.0, with the 2022 modernization of the National Spatial Reference System replacing it entirely. Each realization is a distinct EPSG datum, yet legacy files declare the bare ensemble code and leave the reader to assume a realization — an assumption that silently introduces a metre-scale shift. Global frames are worse: ITRF2014, ITRF2020, and their WGS84 approximations are time-dependent, so a coordinate is only fully specified by a frame and the epoch at which it was observed.

An archive that plans to survive decades therefore stores three things per dataset, not one: the specific realization, the coordinate epoch, and the transformation pipeline used to bring it to the archival frame. Those fields must ride into the columnar footer during the GeoParquet Migration Workflows write, because a datum name that is not carried into the file is lost the first time the data is repacked. The timeline below shows why the bare name is insufficient — the ensemble spans the full spread of its realizations, and only the epoch pins a point within it:

NAD83 realizations across four decades and why the epoch matters A horizontal timeline with four realization nodes — NAD83(1986), NAD83(CORS96), NAD83(2011) at epoch 2010.0, and NATRF2022 at epoch 2020.0 — under a bracket labelled the NAD83 datum ensemble spanning one to two metres. The caption states that retrieval must pin the realization and coordinate epoch, not just the datum name. datum ensemble “NAD83” · ~1–2 m spread across realizations NAD83(1986) 1986 NAD83(CORS96) ~1998 NAD83(2011) epoch 2010.0 NATRF2022 epoch 2020.0 retrieval must pin realization + coordinate epoch → the bare name is ambiguous

Preserving Realization and Epoch Through the Archive

The workflow captures the full frame specification at ingest, pins a reproducible transformation, and — for time-dependent frames — propagates coordinates by epoch. Each phase writes to an immutable lineage record so a retrieval decades later is deterministic rather than a guess.

Phase 1: Capture the Realization and Coordinate Epoch at Ingest

Resolve the source to a specific realization EPSG code, never the ensemble. Record the coordinate epoch alongside it; for a static-frame product the epoch is the frame’s reference epoch, and for an observed dataset it is the survey date expressed as a decimal year.

import json, pyproj

# Resolve to the realization, not the ensemble. EPSG:6318 = NAD83(2011) geographic 2D;
# EPSG:4269 is the bare NAD83 ensemble and must NOT be stored as the archival frame.
src = pyproj.CRS.from_epsg(6318)
assert not src.is_deprecated

record = {
    "file": "datasets/survey/2011/control_region_north.gpkg",
    "realization_epsg": src.to_epsg(),          # 6318, not 4269
    "realization_name": src.name,               # "NAD83(2011)"
    "coordinate_epoch": 2010.0,                  # decimal year — mandatory for time-dependent frames
    "datum_ensemble": src.datum.name,            # human-readable ensemble for discovery
}
with open("frame_lineage.jsonl", "a") as fh:
    fh.write(json.dumps(record) + "\n")

Storing coordinate_epoch as a first-class field is the single change that makes long-term retrieval reproducible; without it a future reader cannot choose the correct time-dependent transformation and will fall back to a static shift that is wrong by the accumulated plate motion.

Phase 2: Pin a Reproducible Transformation Pipeline

PROJ can produce several candidate operations between two frames, ranked by accuracy, and the winner can change when the PROJ or EPSG database is upgraded. For an archive that must reproduce a result identically in ten years, capture the exact pipeline string and the versions that produced it, then replay that pipeline rather than re-solving.

# Enumerate candidate operations and their accuracies, then freeze the chosen one.
projinfo -s "EPSG:6318" -t "EPSG:4326" --summary
projinfo -s "EPSG:6318" -t "EPSG:4326" -o PROJ --hide-ballpark > pinned_pipeline.txt

# Record the toolchain that produced the pipeline, so a rebuild is reproducible.
projinfo --version >> pinned_pipeline.txt   # PROJ + EPSG database versions

Bake every grid shift file the pipeline references into the container image and forbid runtime downloads, exactly as the ETL guidance in Automating CRS Transformations in ETL Pipelines requires. A pipeline that depends on us_noaa_nadcon5_nad83_1986_nad83_2011.tif is only reproducible if that grid is present and its checksum is recorded:

export PROJ_NETWORK=OFF                       # no silent runtime grid fetches
GRID="us_noaa_nadcon5_nad83_1986_nad83_2011.tif"
test -f "$PROJ_DATA/$GRID" || { echo "FATAL: datum grid $GRID not baked in"; exit 1; }
sha256sum "$PROJ_DATA/$GRID" >> frame_lineage.jsonl

Phase 3: Propagate Time-Dependent Coordinates by Epoch

Between a global frame and a plate-fixed frame, the transformation is a function of epoch. PROJ carries the coordinate epoch as a fourth ordinate, so a point observed in ITRF2014 at epoch 2015.7 transforms into NAD83(2011) through a time-dependent Helmert plus a plate-motion or deformation model. Supply the epoch explicitly — omitting it makes PROJ assume the frame’s reference epoch and reintroduces the drift you are trying to remove.

# Transform ITRF2014 lon/lat/h observed at epoch 2015.7 into NAD83(2011).
# The 4th input value is the coordinate epoch (decimal year).
echo "-104.9903 39.7392 1609.0 2015.7" | \
  cs2cs "EPSG:7912" "EPSG:6318" -d 6

For bulk archives, drive the same operation through pyproj.Transformer, passing a tt array of epochs so every feature is propagated to the archival reference epoch before it is written:

from pyproj import Transformer
tr = Transformer.from_crs("EPSG:7912", "EPSG:6318", always_xy=True)
lon, lat, h, epoch = -104.9903, 39.7392, 1609.0, 2015.7
x, y, z = tr.transform(lon, lat, h, epoch)   # epoch drives the time-dependent step

How Far a Point Moves While Standing Still

Plate motion is the part of datum management that surprises teams new to long-term archives. A survey monument that has not moved relative to the ground beneath it acquires new coordinates every year in a global frame, because the plate carrying it is moving. Over a retention window of decades the accumulated displacement exceeds the accuracy the survey was commissioned to deliver.

Accumulated coordinate displacement from plate motion over a 30-year retention window Four plates plotted as displacement against time over thirty years. Every plate crosses a typical ten-centimetre survey tolerance within a few years, and the fastest accumulate more than two metres by year thirty. 2.2 m1.5 m0.7 m0 0.1 m survey tolerance Pacific · 70 mm/yr → 2.1 m Australian · 68 mm/yr Eurasian · 25 mm/yr North American · 20 mm/yr year 0year 10year 20year 30 Displacement in a global frame for a monument that has not moved relative to local ground. Without a recorded epoch, none of this is recoverable.

The archival consequence is narrow and important: an epoch is not optional metadata. A coordinate pair plus a datum name identifies a position only when the epoch is known, and an archive that records the first two but not the third has stored a value whose accuracy silently degrades with the age of the file. Record the epoch at ingest, carry it through every conversion, and store it in the same manifest field that holds the reference-system definition, so a reader in twenty years can transform to whatever frame is then current instead of guessing when the measurement was made.

Verifying the Transformation Is Reproducible

Reproducibility here means two things: the pipeline resolves to the pinned operation, and a known control point round-trips within the operation’s stated accuracy. Check both, because a matching pipeline string over a changed grid still yields wrong coordinates.

projinfo -s "EPSG:6318" -t "EPSG:4326" -o PROJ --hide-ballpark

Annotated expected output — the operation and its accuracy must match pinned_pipeline.txt exactly:

+proj=pipeline
  +step +proj=axisswap +order=2,1
  +step +proj=unitconvert +xy_in=deg +xy_out=rad
  +step +proj=noop            # NAD83(2011) -> WGS84 is sub-metre; ballpark hidden
  +step +proj=unitconvert +xy_in=rad +xy_out=deg
  +step +proj=axisswap +order=2,1

If instead the output names a grid-based step with a different accuracy figure, the EPSG database was upgraded under you and the archive’s transformations are no longer the ones its lineage claims — freeze the toolchain and re-pin deliberately. Because these pinned pipelines and their grids are part of the legal record of what a coordinate meant, govern their retention under the Retention Policy Frameworks that keep the rest of the archive defensible.

Troubleshooting

Symptom Root cause Diagnostic & fix
Coordinates off by ~1 m between two “NAD83” datasets Different realizations stored under the bare ensemble code Recover each realization from the survey date; store realization_epsg (e.g. 6318), transform through NADCON5 grids
Global-frame points drift ~2 cm/year vs expected Coordinate epoch omitted; static transform assumed Record the observation epoch; use a time-dependent transform with the epoch as the 4th ordinate
projinfo returns a different pipeline than last year PROJ/EPSG database upgraded, changing the ranked winner Replay the pinned pipeline string; freeze PROJ + EPSG versions in the image and re-pin only on review
Cannot find grid at transform time Required .tif grid not baked in under PROJ_NETWORK=OFF Stage the grid, verify its recorded checksum, treat a missing grid as a hard failure

Realizations Are Not Interchangeable

“WGS84” names a family, not a single frame. Its realizations — G730, G873, G1150, G1674, G1762, G2139 — differ by centimetres to decimetres, and each was current for a period of years. A file labelled only EPSG:4326 therefore identifies its frame to about a metre, which is adequate for a web map and inadequate for a cadastral archive.

WGS84 realizations and the offsets between them A timeline of six WGS84 realizations between 1994 and 2021 with the positional offset introduced at each step, ranging from a few centimetres to about twenty centimetres. All six are labelled EPSG:4326 in most metadata. G730 1994 G873 1997 ~20 cm G1150 2002 ~8 cm G1674 2012 ~5 cm G1762 2013 ~2 cm G2139 2021 ~3 cm All six are reported as “WGS84 / EPSG:4326” by most tooling Record the realization, not the family — the difference between the oldest and newest exceeds a third of a metre.

Two practices follow. Record the realization string as written by the source system rather than normalising it away, and prefer a plate-fixed national frame as the archive’s storage system where one exists, because its coordinates do not move with the plate and therefore need no epoch bookkeeping for domestic use. Transformation to a global frame then becomes an explicit, dated operation at read time rather than an implicit assumption baked into the stored bytes.

Operational Execution Checklist

Frequently Asked Questions

Is it safe to treat NAD83 and WGS84 as equivalent?

Not for anything survey-grade. They agreed to about a metre when both were introduced and have diverged since, by roughly one to two metres in the continental United States. Tooling that treats them as identical — and several defaults do — introduces an error larger than the precision most cadastral and utility archives are contracted to preserve. Where the difference does not matter, record that you decided it does not, so a later reader does not have to infer it.

What should a pipeline do when the source datum is unknown?

Refuse the ingest rather than assume. An unlabelled dataset can be assigned a working assumption for analysis, but writing that assumption into an archive converts a known unknown into a false certainty that no later audit can distinguish from a measurement. Quarantine the dataset, record what is known about its provenance, and archive it with an explicit “datum undetermined” marker if it must be kept.

Do datum shifts affect raster archives as much as vector?

They affect the georeferencing identically, but the remedy is more expensive. Correcting a vector dataset is a coordinate transformation; correcting a raster requires resampling, which alters pixel values and therefore produces a derived product rather than a corrected original. For that reason, raster archives should retain the original georeferencing and record the correction as transformation metadata wherever the accuracy requirement allows.

Part of the Spatial Data Archival knowledge base.

Related pages

Automating CRS Transformations in ETL Pipelines for Spatial Data Archival Detecting and Fixing CRS Drift in Archived Datasets Verifying CRS Metadata Integrity in GeoParquet Archives