Detecting Bit Rot in Long-Term Raster Archives
Bit rot is the failure archivists worry about most and encounter least — but when it does occur in imagery it is uniquely insidious, because a raster with a handful of flipped bits opens cleanly, renders plausibly, and reports no error anywhere. This walkthrough is for the GIS archivist responsible for a decade-scale imagery holding who needs to distinguish genuine storage-level corruption from the far more common causes of a bad file, and to have a defensible response when a mismatch is confirmed.
Distinguishing Rot from Everything Else
A checksum mismatch is a symptom, not a diagnosis. Four causes produce one, and they call for different responses — so the first job after a mismatch is triage, not recovery.
Only the first row is bit rot, and in audited archives it accounts for well under a twentieth of mismatches. Running the triage before the recovery avoids restoring from a replica to fix a problem the replica shares — which is exactly what happens when an overwrite is misdiagnosed as corruption.
Step-by-Step Procedure
Step 1 — Re-read before believing the mismatch
Transient read errors are common enough that a single mismatch should never trigger a response. Re-read the object, ideally through a different client and network path.
KEY=archive/imagery/2014/ortho_n5432_e0871.tif
for i in 1 2; do
aws s3api get-object --bucket spatial-archive --key "$KEY" /dev/stdout \
| sha256sum | awk -v n=$i '{print "read " n ": " $1}'
doneTwo identical hashes that both differ from the recorded value confirm the object on disk has changed. Two differing hashes indicate a transport problem, not a stored one.
Step 2 — Compare size and last-modified against the manifest
These two fields separate three of the four causes without reading a byte.
aws s3api head-object --bucket spatial-archive --key "$KEY" \
--query '{size:ContentLength,modified:LastModified,version:VersionId}'A smaller size means truncation. A last-modified date after the manifest’s ingest timestamp means something wrote over it. Matching size and date with a differing hash is the signature of genuine corruption.
Step 3 — Localise the damage inside the raster
For a confirmed corruption, finding where the bytes changed tells you whether the file is salvageable. A GeoTIFF’s damage may sit in a single tile, in an overview, or in the header — and only the last makes the file unopenable.
# Compare the archived object against a known-good copy, tile by tile
from osgeo import gdal
import hashlib
def tile_digests(path, band=1):
ds = gdal.Open(path); b = ds.GetRasterBand(band)
bx, by = b.GetBlockSize()
out = {}
for y in range(0, b.YSize, by):
for x in range(0, b.XSize, bx):
w, h = min(bx, b.XSize - x), min(by, b.YSize - y)
buf = b.ReadRaster(x, y, w, h)
out[(x, y)] = hashlib.sha256(buf).hexdigest()
return out
bad, good = tile_digests("/vsis3/spatial-archive/" + KEY), tile_digests("replica_copy.tif")
damaged = [k for k in good if bad.get(k) != good[k]]
print(f"{len(damaged)} of {len(good)} tiles differ: {damaged[:5]}")Step 4 — Replace, do not patch
Restore the whole object from a verified copy and re-record its checksum. Rebuilding a damaged file tile by tile produces a new artefact that matches neither the original checksum nor the replica’s, which destroys the audit trail the archive exists to maintain.
Validation & Verification
After replacement, confirm three things: the object’s hash matches the manifest, its structure validates, and the version history records what happened.
aws s3api get-object --bucket spatial-archive --key "$KEY" /dev/stdout | sha256sum
rio cogeo validate "/vsis3/spatial-archive/$KEY"
aws s3api list-object-versions --bucket spatial-archive --prefix "$KEY" \
--query 'Versions[].[VersionId,LastModified,Size]' --output tableExpected output is a hash matching the manifest, a valid verdict from the COG validator, and a version list showing both the damaged version and its replacement — the damaged one retained deliberately, because an incident record that deletes its own evidence is not a record.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| Mismatch that never reproduces | Transient read or client-side error | Log and move on; do not restore |
| Every object in a prefix mismatches | Manifest hashes computed before a reprojection step | Fix the manifest, not the data; re-record from the current objects with a noted caveat |
| Damage in the header only | Failed metadata rewrite by an older tool | Restore from replica; check what tool wrote metadata in place |
| Replica has the same damage | Corruption preceded replication | Recover from the retained source; if none, quarantine and document |
| Validator passes but pixels differ | Bit flips inside a compressed tile, decoding to plausible values | Compare tile digests against a known-good copy — validation cannot see this |
The final row is the reason a structural validator is not a substitute for fixity. A compressed tile with altered bytes usually still decompresses, producing pixels that are wrong and entirely plausible.
Operational Execution Checklist
Recording the Incident So It Stays Explicable
A confirmed corruption is a provenance event. The object changes, the archive’s record of it must change too, and a future reader comparing the file against a published checksum needs to find an explanation rather than a discrepancy.
Store the record with the archive rather than in a ticketing system, because tickets are retired and archives are not. Where the object is publicly catalogued, link the record from the catalogue entry: a reader who fetched the file before the repair has a copy whose checksum no longer matches anything, and the incident record is the only thing that will ever explain why.
Frequently Asked Questions
How often does genuine bit rot actually occur?
Rarely enough that a single-digit number of confirmed cases across a multi-petabyte archive over a decade is typical, and often enough that an archive with no detection would never know. The reason to run the audit is not the expected rate but the asymmetry: detection is nearly free from inventory metadata, and an undetected corruption in a preservation archive is permanent.
Does compression make rot worse?
It concentrates the consequences. In an uncompressed raster a flipped bit alters one pixel; inside a compressed tile it can alter the whole tile’s decoded output, because the corrupted symbol changes everything after it in that block. That argues for smaller internal tiles in preservation copies rather than against compression — the ratio saving remains worth far more than the blast-radius difference.
Should imagery be stored with per-tile checksums?
For archives where partial recovery is genuinely useful, yes — but note that neither GeoTIFF nor COG carries per-tile digests natively, so it means a sidecar. The pragmatic middle ground is a sidecar listing per-tile digests generated at ingest, which costs a few kilobytes per scene and turns “this file is damaged” into “these three tiles are damaged”, which is exactly the information the salvage decision needs.
Does the same triage apply to vector and point-cloud objects?
The four causes are identical; only the localisation step differs. For a GeoParquet object, the equivalent of comparing tiles is comparing per-row-group statistics and column checksums against a known-good copy, which narrows the damage to specific row groups. For a COPC file it is comparing octree node digests. In every case the principle holds: find where the difference is before deciding whether the object is salvageable.
Should a corrupted object be deleted once it is replaced?
No — retain it as a noncurrent version and mark it. It is the evidence that the incident happened, it is what a future auditor will ask to see, and its storage cost is trivial. Expire it on the same schedule as other noncurrent versions rather than deleting it by hand, so the retention is a policy rather than a decision.
How often should a known-good copy be re-verified?
On the same rotating schedule as everything else, and with one addition: after any object is used as a repair source, verify it immediately. A replica that supplied a repair has just been read in full, which is the cheapest possible moment to confirm it is itself intact, and the worst possible outcome is repairing one object from another that shares the defect.
Related
- Archive Integrity Verification for Spatial Data — the parent topic covering fixity, structure and format surveillance together.
- Verifying Checksums Across Spatial Archive Tiers — the audit that surfaces the mismatch this procedure triages.
- Validating GeoTIFF Structure After a Cold Restore — the structural check that runs alongside fixity on restored imagery.
- Multi-Region Replication & Disaster Recovery — the copy a confirmed corruption is repaired from.
Up one level: Archive Integrity Verification.