Spatial Data Archival · Cold Storage

Converting GeoTIFF to Cloud-Optimized GeoTIFF at Scale

Converting one scene is a command; converting sixty thousand is a system. This walkthrough is for the data engineer running a raster migration across a multi-terabyte imagery holding, where the job will be interrupted, some inputs will be malformed, and the conversion must be restartable without producing duplicates or half-written objects. The pipeline below fans out per scene, validates before and after, quarantines what it cannot handle, and leaves a manifest that explains every output it produced.

Sizing the Job Before Starting It

Raster conversion is CPU-bound on compression and I/O-bound on transfer, and the ratio between them decides whether to run it on many small workers or few large ones. Measure on a representative sample rather than extrapolating from one scene, because compression time varies by a factor of five across imagery types.

Conversion throughput by imagery type Four imagery types with measured single-core conversion throughput, from 46 megabytes per second for 8-bit RGB down to 11 for 16-bit radar, and the resulting fleet sizing for a sixty-thousand-scene job. Measured single-core throughput, DEFLATE level 9 with predictor 8-bit RGB ortho 46 MB/s 4-band multispectral 31 MB/s 16-bit elevation 18 MB/s 16-bit radar, speckled 11 MB/s 60,000 scenes · 38 TB mixed → about 2,900 core-hours → 12 hours on 240 cores. Transfer, not CPU, becomes the limit above ~400 cores.

Step-by-Step Procedure

Step 1 — Inventory and triage the inputs

Enumerate what is actually there before converting any of it. Legacy raster holdings routinely contain files that are not what their extension claims, scenes with no georeferencing, and duplicates under different names.

# Probe every input's essential properties without reading pixel data
find datasets/imagery/raw -name '*.tif' -print0 \
  | xargs -0 -P 16 -I{} sh -c '
      gdalinfo -json "{}" 2>/dev/null \
      | jq -c --arg p "{}" "{path:\$p, size:.size, bands:(.bands|length),
                           dtype:.bands[0].type, crs:(.coordinateSystem.wkt!=null),
                           blocks:.bands[0].block}" \
      || echo "{\"path\":\"{}\",\"error\":\"unreadable\"}"' \
  > inventory.ndjson

jq -s 'group_by(.dtype) | map({dtype: .[0].dtype, n: length})' inventory.ndjson

Step 2 — Convert with idempotent output keys

Each scene’s output key is a pure function of its input identity, and a completion marker is written last so a restart can skip finished work safely.

# convert.py — one scene; the fleet runs this concurrently
import hashlib, subprocess, boto3, os, json

s3 = boto3.client("s3")
BUCKET, PREFIX = "spatial-archive", "archive/imagery/2026/"

def output_key(src: str) -> str:
    stem = os.path.splitext(os.path.basename(src))[0]
    return f"{PREFIX}{stem}.tif"

def already_done(key: str) -> bool:
    try:
        s3.head_object(Bucket=BUCKET, Key=key + ".done")
        return True
    except s3.exceptions.ClientError:
        return False

def convert(src: str, dtype: str) -> dict:
    key = output_key(src)
    if already_done(key):
        return {"src": src, "key": key, "status": "skipped"}

    predictor = "3" if dtype.startswith("Float") else "2"
    resampling = "NEAREST" if "classified" in src else "AVERAGE"
    tmp_key = key + ".tmp"

    subprocess.run([
        "gdal_translate", src, f"/vsis3/{BUCKET}/{tmp_key}", "-of", "COG",
        "-co", "BLOCKSIZE=512", "-co", "COMPRESS=DEFLATE",
        "-co", f"PREDICTOR={predictor}", "-co", "LEVEL=9",
        "-co", "OVERVIEWS=IGNORE_EXISTING",
        "-co", f"OVERVIEW_RESAMPLING={resampling}",
        "-co", "BIGTIFF=YES", "-co", "NUM_THREADS=ALL_CPUS",
    ], check=True)

    # Promote only after the object is complete: a key that exists is always whole
    s3.copy_object(Bucket=BUCKET, Key=key,
                   CopySource={"Bucket": BUCKET, "Key": tmp_key},
                   ChecksumAlgorithm="SHA256")
    s3.delete_object(Bucket=BUCKET, Key=tmp_key)
    s3.put_object(Bucket=BUCKET, Key=key + ".done",
                  Body=json.dumps({"src": src, "predictor": predictor,
                                   "resampling": resampling}).encode())
    return {"src": src, "key": key, "status": "converted"}

Step 3 — Validate every output before it counts as done

Validation runs inside the same job, so a scene that fails never gets a completion marker and will be retried.

rio cogeo validate "/vsis3/spatial-archive/archive/imagery/2026/ortho_n5432_e0871.tif" \
  && gdalinfo -json "/vsis3/spatial-archive/archive/imagery/2026/ortho_n5432_e0871.tif" \
     | jq -e '.bands[0].block == [512,512] and (.bands[0].overviews | length) >= 4'
Fan-out structure of the conversion job An inventory sharded across many workers, each converting to a temporary key, validating, then promoting and writing a completion marker, with failures routed to quarantine and a final reconciliation. inventory 60,000 scenes worker 1 … 240 convert → .tmp validate promote + .done key exists ⇒ object is whole quarantine input + error, nothing deleted reconcile markers vs inventory · nothing missing Restarting the job re-runs only the shards whose markers are missing — no duplicates, no partially written objects.

Validation & Verification

Reconciliation at the end is what proves the job finished, and it is a count comparison rather than a spot check.

INPUTS=$(wc -l < inventory.ndjson)
DONE=$(aws s3api list-objects-v2 --bucket spatial-archive \
        --prefix archive/imagery/2026/ --query 'length(Contents[?ends_with(Key, `.done`)])')
QUARANTINED=$(aws s3api list-objects-v2 --bucket spatial-archive \
        --prefix quarantine/imagery/ --query 'length(Contents)')
echo "inputs=$INPUTS done=$DONE quarantined=$QUARANTINED  gap=$((INPUTS - DONE - QUARANTINED))"

Expected output has a gap of zero. Every input either produced a validated output or is sitting in quarantine with its error — no third state exists, which is what makes the job’s completion assertable rather than assumed.

Troubleshooting

Symptom Root cause Fix
Output larger than the input Predictor wrong for the data type Predictor 3 for floating point, 2 for integer; re-convert affected scenes
Overviews missing on some outputs Source contained overviews and the driver preserved them Use OVERVIEWS=IGNORE_EXISTING so they are always regenerated
Job fails near completion on large scenes BigTIFF threshold crossed unexpectedly Set BIGTIFF=YES unconditionally for archival conversions
Workers idle while transfer saturates Fleet sized on CPU alone Cap concurrency at the point network throughput plateaus
Duplicate outputs after a restart Output key derived from a timestamp or job id Derive it from the input identity only

Operational Execution Checklist

Watching the Job Without Watching the Logs

A conversion fleet running for twelve hours produces more log output than anyone reads, and the useful signals are four counters rather than a stream. Publishing them as the job runs turns “is it working?” into a glance.

The four counters worth publishing during a conversion run Throughput, quarantine rate, validation failure rate and worker utilisation, each with its target and what a deviation means. throughput 84 scenes/min · target 80 low → check transfer quarantine rate 0.26% threshold 2% high → a bad source batch validation failures 0.4% retried automatically high → wrong creation options worker utilisation 91% CPU-bound, as intended low → network is the limit A rising quarantine rate is the one that should page someone: it usually means a whole source batch differs from what the pipeline expects. Everything else is a tuning signal that can wait until the run finishes.

Emit the counters to whatever the team already watches rather than building a dashboard for one job. The point is that a long-running conversion should be legible to someone who was not there when it started — including the person who has to decide, at hour nine, whether to let it finish.

Frequently Asked Questions

Should conversion happen before or after the imagery is tiered?

Before, always. Converting a scene already in an archive class costs a restore, and if that object has not met its minimum storage duration, an early-deletion charge on top. Convert at ingest — or, for a backlog, while the material is still in an instant-access class — and let the lifecycle rules tier the converted result.

How are multi-file scenes with sidecars handled?

Fold the sidecar content into the output before discarding it. World files, .aux.xml statistics and .prj files all carry information the COG can hold internally, and a conversion that leaves them behind loses georeferencing or statistics silently. Where the sidecar carries free-text provenance, move it into the metadata record rather than into the raster.

Can the conversion run against imagery already in object storage?

Yes, and it is usually the right arrangement — GDAL reads and writes through the virtual filesystem, so no scene is ever staged on local disk. Run the workers in the same region as both buckets, or the job pays egress on every byte it reads and writes.

How should the job handle scenes that are already valid COGs?

Detect and skip them rather than re-converting. A re-conversion of an already-compliant scene costs compute and produces a byte-different file with a new checksum, which breaks any integrity record referencing the original. Validate first, convert only what fails, and record the skip so the reconciliation still accounts for every input.

Does the fleet need to run in the same account as the archive?

It needs credentials into the archive and network proximity to it; the account is a matter of preference. Running conversion in a separate account with write access to a staging prefix is a reasonable separation, and it means a runaway job cannot touch the published archive directly. What matters more is the region, since cross-region conversion pays egress on every byte read and written.

What is a reasonable failure rate to accept before stopping the job?

Around one to two percent for legacy holdings, and near zero for a modern feed. The useful discipline is a rising-rate alarm rather than an absolute threshold: a job that has been quarantining 0.3% for six hours and suddenly quarantines 40% has met a batch that differs from everything before it, and stopping to look is cheaper than converting it wrongly.

How is the job’s output verified against the inputs at the end?

By the reconciliation described above — inputs equals outputs plus quarantine, with no third state. That single assertion is what makes the job’s completion a fact rather than an impression, and it is worth failing the run on rather than reporting as a warning.

Should the conversion write to the final bucket or to a staging one?

Either works provided the promotion step is atomic. Writing to a staging prefix in the same bucket is simpler and keeps the copy local; a separate staging bucket adds isolation at the cost of a cross-bucket copy. What matters is that no key in the published prefix ever holds a partially written or unvalidated object.

Up one level: Cloud-Optimized GeoTIFF Conversion Pipelines.

Related pages

Choosing Overview Levels and Resampling for Archived Imagery Tuning COG Internal Tile Size for Range Requests Validating COG Structure Before Archival