Archive Integrity Verification for Spatial Data
An archive’s promise is that the file you retrieve in fifteen years is the file that was written today. Nothing about storing bytes cheaply delivers that promise on its own: objects can be silently truncated by a failed multipart upload, corrupted in transit by a client that skipped checksum validation, replaced by a job that wrote the wrong version, or left byte-perfect and structurally unreadable because the writer produced an invalid file in the first place. This topic is for the GIS archivist and platform engineer responsible for proving — not assuming — that a spatial archive is still faithful, and it covers the three checks that together constitute that proof.
The Failure Mode: Trusting Durability Figures
Storage platforms publish eleven-nines annual durability, and teams reasonably conclude that corruption is not their problem. The figure is accurate and answers a narrower question than it appears to: it describes the probability that the platform loses a stored object, not the probability that the object was correct when it arrived, that a later job did not overwrite it with something wrong, or that its format is still readable. In a decade of running spatial archives, media failure is the rarest cause of a bad object; ingest defects and process errors are the common ones, and durability guarantees are silent about both.
Prerequisite Context
Verification needs three inputs that must exist before an object is archived. A checksum computed by the producer and recorded independently of the object — because a checksum stored only inside the file proves nothing about the file. An inventory that enumerates the archive without listing it object by object, since per-object API calls do not scale past a few hundred thousand. And a structural validator per format the archive accepts, which for a spatial archive means at least a GeoTIFF/COG checker, a Parquet footer reader, and a LAZ/COPC header parser.
Objects already in the archive without a recorded checksum can be brought into the scheme retrospectively, but the checksum then attests only that the object has not changed since the audit began — not that it was correct on arrival. Recording that distinction honestly is part of the archive’s provenance.
Concept & Design Decisions
Which checksum. Prefer a checksum the storage platform can compute and return as part of an inventory report — SHA-256 where available, otherwise the platform’s own additional checksum. That choice removes the need to read object bytes for the routine audit, which is what makes continuous verification affordable. A checksum that only your pipeline can compute forces a full read of the archive on every pass.
How often, and how much. Full-archive verification annually is a reasonable target, achieved by rotating through a fraction each cycle rather than doing it all at once. A quarterly cadence over a quarter of the archive, weighted so that older and colder material is checked at least as often as new material, spreads the cost evenly and finds problems within months rather than years.
What counts as a failure. A checksum mismatch is unambiguous. A structural validation failure needs judgement: a file that fails a strict COG validator may still be a perfectly readable GeoTIFF, and treating advisory findings as failures makes the audit noisy enough to be ignored. Separate the two levels explicitly.
Implementation
The routine audit is a join between two tables: the checksums the archive recorded at ingest, and the checksums the storage platform reports in its inventory. Neither side requires reading an object.
-- Athena: quarterly fixity pass over the rotating slice
WITH recorded AS (
SELECT object_key, sha256, ingested_at
FROM archive_manifest
WHERE audit_slice = 3 -- one of four rotating slices
),
reported AS (
SELECT key AS object_key, checksum_algorithm, checksum, size, storage_class
FROM spatial_archive_inventory
WHERE dt = date_format(current_date, '%Y-%m-%d')
)
SELECT r.object_key,
r.sha256 AS expected,
p.checksum AS observed,
p.storage_class,
r.ingested_at
FROM recorded r
JOIN reported p USING (object_key)
WHERE p.checksum IS NULL -- platform never computed one: gap, not failure
OR p.checksum <> r.sha256; -- genuine mismatchStructural validation is a separate job that reads a sample. For a spatial archive it is three validators behind one interface:
# archive/validate.py — structural checks per format, run on ingest and on the annual sample
import subprocess, json
from osgeo import gdal
import pyarrow.parquet as pq
import laspy
def validate_cog(path: str) -> dict:
# rio-cogeo returns advisory warnings separately from hard errors
out = subprocess.run(["rio", "cogeo", "validate", path],
capture_output=True, text=True)
ds = gdal.Open(path) # proves the file opens at all
return {"format": "COG", "valid": ds is not None,
"advisory": "warning" in out.stdout.lower(),
"bands": ds.RasterCount if ds else 0}
def validate_geoparquet(path: str) -> dict:
pf = pq.ParquetFile(path) # reads the footer only
geo = pf.schema_arrow.metadata.get(b"geo")
return {"format": "GeoParquet", "valid": geo is not None,
"row_groups": pf.metadata.num_row_groups,
"rows": pf.metadata.num_rows}
def validate_copc(path: str) -> dict:
with laspy.open(path) as f: # header + VLRs, not the points
h = f.header
return {"format": "COPC", "valid": h.point_count > 0,
"points": h.point_count, "crs": str(h.parse_crs())}Each validator reads headers and footers rather than whole files, which keeps the annual sample affordable even when the sampled objects have to be restored first.
Validation Gate
The audit is itself verified by planting a known-bad object and confirming the pass detects it. An audit nobody has seen fail is an audit nobody should trust.
# Plant a deliberately corrupted copy in the audit slice, run the pass, expect a failure
aws s3 cp corrupted_scene.tif s3://spatial-archive/audit-canary/scene_bad.tif
python -m archive.audit run --slice 3 --expect-failures 1Expected output ends with 1 mismatch detected (audit-canary/scene_bad.tif) and a non-zero exit. A pass that reports zero failures against a planted canary is broken — most often because the manifest join dropped rows the inventory did not contain, which silently narrows coverage without narrowing the report.
Cost & Performance Trade-offs
| Approach | Cost per pass, 400 TB | What it detects | What it misses |
|---|---|---|---|
| Inventory-based fixity | negligible — no object reads | changed or truncated objects | files that were never valid |
| Full-read fixity | full retrieval of the slice | the same, more slowly | the same |
| Structural sample, 5% | restore + read of the sample | invalid files, format drift | defects outside the sample |
| Validate on ingest | one read at write time | invalid files, at zero marginal cost | later corruption |
The table argues for a specific combination: validate everything at ingest when the object is already in memory, use inventory-based fixity for continuous assurance, and sample structurally to catch what neither covers. Full-read fixity buys nothing over the inventory-based version and costs a retrieval of the archive.
Failure Modes & Edge Cases
Multipart ETags are not checksums. An ETag for a multipart upload is a hash of hashes and depends on the part size used, so two byte-identical objects uploaded with different part sizes have different ETags. Compare content checksums, and record the part size if ETags must be used.
Cold objects cannot be structurally validated in place. Any check that opens a file requires a restore for Flexible Retrieval and Deep Archive. Sample accordingly, and prefer validating at ingest — before the object ever reaches an archive class — for all material.
Checksums recorded after the fact prove less. A checksum computed from the archived object attests to stability, not to correctness. Where that is all that exists, label it as such in the provenance record rather than presenting it as an ingest checksum.
A passing archive with no readers is not preserved. Fixity and structure can both pass on a format nobody can open. That is what the annual format review exists for, and it is the check most archives have no owner for.
Operational Execution Checklist
Reporting Integrity to People Who Are Not Engineers
An integrity programme that produces log lines produces nothing an institution can act on. The audience for the results is a preservation officer, a funder, or an auditor, and each needs the same three facts expressed without reference to storage classes or batch jobs: how much of the archive has been checked recently, what was found, and what happened to it.
Publish it annually on a fixed date, whether or not anything was found, and keep the series. A single year’s report says the archive was checked; ten years of reports say the institution has a preservation practice — and the trend in the findings column is what tells an auditor whether the practice is working.
Who Owns Integrity, and What They Do Weekly
Integrity programmes fail through diffusion of responsibility more often than through technical inadequacy: the checks run, the results land somewhere, and nobody is answerable for what they say. Naming an owner and a small recurring routine is what converts a set of jobs into a practice.
The owner is usually the archive’s operational lead rather than a preservation specialist, because the actions the results call for — investigating a mismatch, scheduling a repair, adjusting a schedule — are operational. What the role needs is authority to stop an ingest that is producing invalid objects and a route to whoever can approve a restore budget when a repair requires one.
The weekly routine is short. Check that each scheduled pass ran and completed; look at the count of findings rather than the findings themselves; confirm that anything found last week has moved to a resolution. Anything beyond that belongs to the quarterly review, where the interesting questions live: is the coverage rotation actually covering everything, is the read-required backlog shrinking, and has the advisory queue grown to the point where a rewrite pass is worth scheduling.
The quarterly review is also where the format surveillance question gets asked, because it is the check with no automated component. It takes an hour: list the formats the archive holds, confirm each still has a maintained open-source reader, and note anything that has changed. Most years the answer is that nothing has changed, and recording that is the point — a decade of “no change” entries is what makes the one year with a change legible as a signal rather than an alarm.
None of this is heavy, and its lightness is deliberate. An integrity practice that demands significant time every week will be skipped in the weeks that matter; one that demands ten minutes weekly and an hour quarterly survives staff changes, reorganisations and the long stretches in which it finds nothing at all.
Frequently Asked Questions
How is a corrupted object actually repaired?
From the replica, if one exists and passes its own check; from the retained source, if the archive kept it; and otherwise not at all. That ordering is why integrity verification and replication are designed together — verification without a good copy to restore from produces a precise description of an unrecoverable loss. Where neither exists, the honest response is to record the defect in the catalogue so future readers know, which is covered in the drift-correction pattern used for reference-system errors.
Does enabling checksums slow down ingest?
Marginally, and less than the alternative. Computing SHA-256 costs a few percent of the time spent writing a large object, and the platform can compute it during upload rather than as a separate pass. Compared with reading the archive back to verify it later, the cost is not close.
Should the replica be verified independently?
Yes, and against the same recorded checksums rather than against the primary. Verifying the replica by comparing it with the primary tests only that they agree — including on a defect they both inherited. Both copies should be checked against the value recorded at ingest, which is the only reference that predates either of them.
What cadence catches a problem before it spreads?
Ingest-time validation is what actually prevents propagation: an object that never enters the archive cannot be replicated, catalogued, or locked. The periodic passes exist to find what the gate missed and to detect change over time, and quarterly is frequent enough for that purpose in every archive the author has measured. Increasing the frequency of the periodic pass is a poor substitute for a gate at the door.
How does this programme interact with an institutional preservation policy?
It supplies the evidence such a policy asserts. A preservation policy states that holdings are checked, that defects are remediated and that formats are monitored; this programme is what makes each of those statements true and auditable. Where the institution has no formal policy, the programme’s own annual report is a reasonable substitute — it answers the same questions in the same order, from measurements rather than intentions.
What is the minimum viable version for a small archive?
Checksums recorded at ingest, an annual inventory-based comparison, and validation at write time. That covers the two cheapest checks and the one that prevents most defects, and it can be implemented in a day against any storage platform with an inventory report. The rotating slices, batch backlog and structural sampling are refinements that matter at scale and can wait until the archive has one.
Does any of this change for an archive held on-premise?
The mechanisms differ and the structure does not. An on-premise archive has no inventory report, so the equivalent is a filesystem walk producing the same table, and it has no per-object API charges, which makes read-based verification affordable enough to do more often. The three checks, their cadences and the separation between hard failures and advisories all transfer unchanged.
How is the programme’s own health monitored?
By tracking coverage rather than findings. A pass that reports nothing may be working perfectly or may have silently stopped matching objects, and the difference is visible only in how many objects it actually checked. Alert on a coverage figure that falls, not on a finding count that rises.
Related
- Verifying Checksums Across Spatial Archive Tiers — the procedure for running fixity against objects in every storage class.
- Detecting Bit Rot in Long-Term Raster Archives — what silent corruption looks like in imagery and how to distinguish it from a bad read.
- Automating Fixity Audits with S3 Batch Operations — running the rotating audit as a scheduled batch job.
- Validating GeoTIFF Structure After a Cold Restore — the structural check applied to restored imagery.
- Multi-Region Replication & Disaster Recovery — the copy that makes a detected corruption repairable.
- Validating Attribute Schema Parity After Format Conversion — the conversion-side gate that keeps invalid objects out of the archive in the first place.
Up one level: Spatial Archival Architecture & Tiering Strategy.