Spatial Data Archival · Cold Storage

Validating GeoTIFF Structure After a Cold Restore

A restored scene that hashes correctly can still be unusable. Fixity proves the bytes are unchanged; it says nothing about whether those bytes were a valid GeoTIFF when they were written, whether the internal tiling and overviews that make it cloud-optimised survived a conversion, or whether the georeferencing still places the pixels where the catalogue claims. This walkthrough is for the archivist or engineer validating imagery as it comes back from an archive tier, and it separates the checks that must pass from the advisories that should merely be recorded.

Three Levels of “Valid”

A restored raster can fail at three different depths, and conflating them produces an audit that is either too permissive or too noisy to use.

Three levels at which a restored raster can fail validation Opening the file, cloud-optimised layout, and georeferencing correctness, with the severity each level should be treated as. levelwhat it checksseverity 1 · does it open header, IFD chain, band count the file parses and reports its dimensions hard failure 2 · is it cloud-optimised tiling, overviews, layout order range reads can fetch a window efficiently advisory — unless served by range read 3 · is it georeferenced right CRS, geotransform, footprint pixels land where the catalogue says hard failure Level 2 is advisory because a plain GeoTIFF is a perfectly good preservation copy; it is a failure only where the access pattern depends on it.

Step-by-Step Procedure

Step 1 — Confirm the restore actually completed

A get against an object whose restore is still in progress fails in a way that is easy to mistake for corruption. Check the restore state before validating.

KEY=archive/imagery/2014/ortho_n5432_e0871.tif
aws s3api head-object --bucket spatial-archive --key "$KEY" \
  --query '{restore:Restore,class:StorageClass,size:ContentLength}'
# {"restore": "ongoing-request=\"false\", expiry-date=\"Tue, 19 Aug 2026 00:00:00 GMT\"", ...}

Step 2 — Validate structure without downloading the scene

GDAL’s virtual filesystem reads only the headers it needs, so level-one and level-three checks cost a few kilobytes rather than a full transfer.

gdalinfo -json "/vsis3/spatial-archive/$KEY" | jq '{
  size, bands: (.bands | length),
  driver: .driverShortName,
  crs: .coordinateSystem.wkt | split("\n")[0],
  transform: .geoTransform,
  blocks: .bands[0].block,
  overviews: (.bands[0].overviews // [] | length)
}'

Expected output reports the driver as GTiff, a non-empty coordinate system, a block size indicating internal tiling (typically [512, 512], not [width, 1]), and a non-zero overview count.

Step 3 — Run the cloud-optimised check and read its two verdicts separately 

rio cogeo validate "/vsis3/spatial-archive/$KEY"
# The file is a valid Cloud Optimized GeoTIFF
# ...or, with advisories:
#   The following warnings were found:
#   - The offset of the main IFD should be < 300. It is 1148

Treat “is a valid COG” as the level-two result and the warning list as advisory. An archive that only ever retrieves whole scenes can carry those warnings indefinitely; one that serves range reads should fix them at the next rewrite opportunity.

Step 4 — Reconcile the footprint against the catalogue

The check that catches misplaced imagery compares the file’s own footprint with what discovery advertises. A mismatch means either the raster or the catalogue is wrong, and both are worth knowing about.

from osgeo import gdal, osr
import json, math

ds = gdal.Open(f"/vsis3/spatial-archive/{KEY}")
gt, (w, h) = ds.GetGeoTransform(), (ds.RasterXSize, ds.RasterYSize)
corners = [(gt[0], gt[3]), (gt[0] + w * gt[1], gt[3] + h * gt[5])]

src = osr.SpatialReference(wkt=ds.GetProjection())
dst = osr.SpatialReference(); dst.ImportFromEPSG(4326)
dst.SetAxisMappingStrategy(osr.OAMS_TRADITIONAL_GIS_ORDER)
tx = osr.CoordinateTransformation(src, dst)
(lon0, lat0, _), (lon1, lat1, _) = (tx.TransformPoint(x, y) for x, y in corners)
observed = [min(lon0, lon1), min(lat0, lat1), max(lon0, lon1), max(lat0, lat1)]

declared = json.load(open("stac_item.json"))["bbox"]
drift_m = max(abs(o - d) for o, d in zip(observed, declared)) * 111_320
print(f"observed={observed}\ndeclared={declared}\nmax drift ≈ {drift_m:.1f} m")
Validation outcomes across a restored collection Outcome counts for 4,200 restored scenes: most pass fully, a smaller number carry advisories, fewer lack overviews, and a handful fail on missing reference systems or fail to open. 4,200 scenes restored and validated pass all three levels 4,061 scenes · 96.7% advisory warnings only 96 · record, do not block no overviews 31 · blocks range-read delivery only missing CRS 9 · hard failure — pixels are unplaced fails to open 3 · hard failure — triage as corruption or bad write

Troubleshooting

Symptom Root cause Fix
gdalinfo reports block size [width, 1] Stripped rather than tiled TIFF Not corruption — a valid GeoTIFF written without tiling; rewrite when convenient
Overview count zero Overviews never built, or stripped by a conversion Rebuild overviews on the next rewrite; record which scenes lack them
No coordinate system reported GeoTIFF keys absent; often an .aux.xml sidecar held it Recover from the sidecar or the catalogue; never guess
Footprint drift of thousands of metres Axis order confusion between the file and the catalogue Compare in a known axis order; fix the catalogue entry, not the raster
Opens locally but not via /vsis3/ Restore expired between the check and the read Re-check the restore state; extend the availability window

The axis-order row is worth expecting: it produces a drift that looks alarming and is a metadata bug rather than a data one, and correcting the raster instead of the catalogue would make a good file wrong.

Operational Execution Checklist

Turning Advisories Into a Rewrite Plan

Advisory findings accumulate quietly and are worth acting on in batches, when an object has to be rewritten anyway. Keeping them as a queue rather than a log is what turns them from noise into a plan.

Advisory backlog grouped by the rewrite that clears it Four advisory categories totalling 2,766 scenes, almost all of which are cleared by a single re-conversion pass with corrected creation options. Advisory backlog · 2,766 scenes across four categories no overviews 1,840 · re-convert with OVERVIEWS=IGNORE_EXISTING suboptimal IFD offset 620 · same re-conversion non-standard tile size 210 · same re-conversion older compression setting 96 · opportunistic One re-conversion pass with corrected creation options clears 96% of the backlog — which is why the advisories are worth grouping rather than fixing individually.

Schedule the pass when the affected material is warm for another reason — a migration, a replication backfill, a format review — so the restore is paid once for both purposes. Rewriting cold objects solely to clear advisories is the case where the cure costs more than the condition.

Frequently Asked Questions

Should a scene that fails the cloud-optimised check be rewritten immediately?

Only if something reads it by range. A plain, valid GeoTIFF is a perfectly acceptable preservation copy, and rewriting it costs a restore, a conversion and a new object — plus an early-deletion charge if the original has not met its minimum duration. Collect the advisories, and act on them the next time the object has to be rewritten for another reason.

Can structural validation replace fixity?

No, and the two catch different things. Validation proves the file is well-formed today; fixity proves it is the same file that was written. A corrupted tile inside a compressed raster typically passes validation, because it decompresses to plausible values — which is exactly the case fixity catches and validation cannot.

How large a sample is enough?

Five percent annually across each collection, stratified by age and by the tool that wrote the files, catches systematic defects reliably — and systematic defects are what matter, because they affect thousands of scenes rather than one. Validating every scene at ingest is what makes the sample sufficient later: the sample is looking for drift and format decay, not for a bad write it should have caught years ago.

Does validation need to run on every restored object or only on a sample?

Every object, when the restore is for a recovery — you are about to depend on those files, and the check costs a header read. A sample is appropriate only for the periodic assurance pass, where the goal is detecting systematic drift rather than certifying a specific set of files for use.

What should happen when validation fails during a recovery?

Treat it as a repair task rather than a blocker for the whole recovery. Restore continues, the failing object is recorded, and its repair is attempted from another copy in parallel. A recovery that halts on the first bad file serves nobody; one that quietly skips bad files serves them badly. The middle path is to continue and report.

Can validation be run against objects still in an archive class?

Only the parts that live in metadata — size, storage class, checksum where recorded. Anything that requires opening the file requires a restore, which is why the write-time validation result is so valuable: it is the only structural check that was ever free, and recording it means the archive knows the file was valid even for objects it cannot currently open.

Should validation results expire?

They should carry a date rather than expire. A structural check from three years ago is still evidence that the file was valid then, which is exactly what a preservation record needs; treating it as stale and discarding it loses information. What should have an expiry is the assumption that the result still holds, which is what the periodic sampling exists to refresh.

Does a failed validation always mean the object is unusable?

No, and the distinction matters for triage. A scene that fails a strict COG check may open perfectly in every tool that will ever read it, whereas one that fails to open at all is a genuine loss. Reporting the two under one heading is what makes validation reports hard to act on.

Up one level: Archive Integrity Verification.

Related pages

Automating Fixity Audits with S3 Batch Operations Detecting Bit Rot in Long-Term Raster Archives Verifying Checksums Across Spatial Archive Tiers