Validating COG Structure Before Archival
The cheapest moment to reject a defective raster is before it is written to the archive; the most expensive is after it has been tiered, locked and catalogued. This walkthrough is for the engineer building the write gate that stands between a conversion job and the archive — a check that runs in seconds per scene, blocks what must be blocked, records what is merely advisory, and never lets a scene through on the strength of “it opened”.
What the Gate Must Refuse
A write gate that refuses everything imperfect will be switched off within a month. The distinction that keeps it alive is between defects that make the archive wrong and defects that make it suboptimal.
Step-by-Step Procedure
Step 1 — Check that it opens and carries georeferencing
Two conditions, both cheap, both non-negotiable. A raster without a reference system is a picture, not spatial data.
from osgeo import gdal, osr
gdal.UseExceptions()
def opens_and_georeferenced(path: str) -> dict:
ds = gdal.Open(path) # raises if it cannot open
wkt = ds.GetProjection()
gt = ds.GetGeoTransform()
identity = gt == (0.0, 1.0, 0.0, 0.0, 0.0, 1.0) # the "no geotransform" default
return {"opens": True, "has_crs": bool(wkt), "has_transform": not identity,
"bands": ds.RasterCount, "size": [ds.RasterXSize, ds.RasterYSize]}Step 2 — Compare against the source, not against nothing
Most conversion defects are only visible as a difference. Band count, data type, nodata and dimensions should all survive a lossless conversion unchanged.
def parity(src_path: str, out_path: str) -> list[str]:
src, out = gdal.Open(src_path), gdal.Open(out_path)
problems = []
if src.RasterCount != out.RasterCount:
problems.append(f"band count {src.RasterCount} -> {out.RasterCount}")
if (src.RasterXSize, src.RasterYSize) != (out.RasterXSize, out.RasterYSize):
problems.append("dimensions changed")
for i in range(1, src.RasterCount + 1):
s, o = src.GetRasterBand(i), out.GetRasterBand(i)
if s.DataType != o.DataType:
problems.append(f"band {i} dtype changed")
if s.GetNoDataValue() is not None and o.GetNoDataValue() is None:
problems.append(f"band {i} lost its nodata value")
return problemsStep 3 — Run the cloud-optimised check and split its output
rio cogeo validate "$OUT" > cog.txt 2>&1 || true
grep -q 'is a valid Cloud Optimized GeoTIFF' cog.txt && echo "cog=yes" || echo "cog=no"
sed -n '/warnings were found/,$p' cog.txt # advisory list, recorded not blockingStep 4 — Compare the footprint against the collection’s declared extent
A scene whose footprint falls outside its collection is either mis-georeferenced or filed in the wrong collection, and both are worth blocking.
def within_collection(out_path: str, collection_bbox: list[float]) -> bool:
ds = gdal.Open(out_path)
gt, w, h = ds.GetGeoTransform(), ds.RasterXSize, ds.RasterYSize
src = osr.SpatialReference(wkt=ds.GetProjection())
dst = osr.SpatialReference(); dst.ImportFromEPSG(4326)
dst.SetAxisMappingStrategy(osr.OAMS_TRADITIONAL_GIS_ORDER)
tx = osr.CoordinateTransformation(src, dst)
pts = [tx.TransformPoint(gt[0], gt[3])[:2],
tx.TransformPoint(gt[0] + w * gt[1], gt[3] + h * gt[5])[:2]]
lons, lats = [p[0] for p in pts], [p[1] for p in pts]
return (collection_bbox[0] <= min(lons) and max(lons) <= collection_bbox[2]
and collection_bbox[1] <= min(lats) and max(lats) <= collection_bbox[3])Validation & Verification
The gate is validated by fixtures: one file per defect it claims to catch, run on every change to the gate’s own code.
python -m archive.gate test --fixtures tests/fixtures/
# no_crs.tif -> BLOCKED (has_crs=false) ok
# lost_nodata.tif -> BLOCKED (band 1 lost nodata) ok
# outside_extent.tif -> BLOCKED (footprint outside bbox) ok
# truncated.tif -> BLOCKED (does not open) ok
# suboptimal_ifd.tif -> PASS with 1 advisory ok
# clean.tif -> PASS okExpected output is ok on every fixture. A gate whose fixtures have never failed has never been proven to fire, and a validation step that silently stopped working is indistinguishable from a data source that suddenly became clean.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| Gate blocks scenes that are genuinely fine | Collection bbox too tight, or in the wrong axis order | Widen the collection extent; compare in a known axis order |
| Advisory count climbing over time | Conversion job drifting from the collection standard | Pin creation options in the job definition |
| Gate passes a file that later fails validation | Gate ran on the temporary key, archive holds a different object | Validate the promoted object, or validate then promote atomically |
| Nodata check fires on every scene | Source declares nodata in a sidecar the converter never read | Read the sidecar during conversion; do not relax the check |
| Gate slow on large scenes | Reading pixel data rather than headers | Every check here reads headers only; profile for an accidental full read |
Operational Execution Checklist
What the Gate Should Emit Besides a Verdict
A gate that returns pass or fail wastes most of what it computed. The same header read that decides the verdict also produces everything the catalogue and the integrity programme need, and emitting it once removes a later pass over the same objects.
The integrity row is the one with the most leverage: a structural validity result recorded at write time is the baseline every later check compares against, and it is the only moment when the object is guaranteed to be warm. Capturing it later costs a restore per scene.
Frequently Asked Questions
Should the gate run inside the conversion job or as a separate step?
Inside the job, as a phase before promotion. A separate nightly validator runs after the object is in the archive and possibly already catalogued, which turns a preventable rejection into an incident. Keeping the gate inside the job is also what makes the temporary-key-then-promote pattern work: nothing reaches the final key without passing.
What happens to a scene the gate blocks?
It goes to quarantine with its input intact and its specific failure recorded, and it is somebody’s work item. Deleting it destroys evidence; retrying it blindly wastes compute on the same defect. Most blocked scenes turn out to be fixable at the source — a missing .prj, a sidecar that did not travel — and become archivable after one intervention.
Does the gate need to run on scenes converted years ago?
Not routinely, because the same checks form part of the periodic structural validation described under archive integrity. What is worth doing once is a retrospective pass over material converted before the gate existed, since that population is where the archive’s undetected defects live.
Should the gate block on advisory findings if they become widespread?
No — change the conversion instead. A rising advisory rate means the pipeline has drifted from the collection standard, and the fix belongs in the job configuration rather than in a stricter gate. Blocking on advisories converts a configuration problem into a stalled pipeline without addressing the cause.
How long should the gate take per scene?
A second or less, because every check reads headers rather than pixels. A gate taking tens of seconds is almost always doing a full read somewhere — often a statistics computation that GDAL performs lazily on first access — and that cost multiplied across a large conversion is significant.
What happens to the gate’s results after the object is published?
They become the object’s first integrity record. The structural verdict, the checksum and the footprint captured at write time are the baseline every later check compares against, so they belong with the object in the manifest rather than in the job’s logs. An archive that discards them has to pay a restore to establish the same facts later.
What should happen when the gate itself has a bug?
Treat it as an incident affecting everything written since the bug was introduced. That is why the gate’s version belongs in the record it writes: it makes the affected population a query rather than a guess. Re-validating that population is usually cheap, because the objects are recent and therefore still warm.
Should the gate be able to fix what it finds?
No. A gate that repairs silently removes the signal that something upstream is producing defective output, and the repair itself becomes an undocumented transformation. Blocking and reporting keeps the responsibility where it belongs — with the conversion job, which can be corrected and re-run.
Should the gate run on delivered COGs as well as converted ones?
Yes, on the same terms. A delivered file is subject to the same failure modes and has not been through your conversion, so it has never been checked against your collection’s standards at all.
Related
- Cloud-Optimized GeoTIFF Conversion Pipelines — the parent topic and the conversion parameters this gate checks against.
- Converting GeoTIFF to Cloud-Optimized GeoTIFF at Scale — the job this gate runs inside.
- Validating GeoTIFF Structure After a Cold Restore — the same checks applied years later, where they cost a restore.
- Validating Attribute Schema Parity After Format Conversion — the vector-side equivalent of this gate.
Up one level: Cloud-Optimized GeoTIFF Conversion Pipelines.