Cloud-Optimized GeoTIFF Conversion Pipelines
Imagery is where most spatial archives keep their volume, and the difference between a directory of ordinary GeoTIFFs and a properly built Cloud-Optimized GeoTIFF archive is the difference between downloading a scene to look at a corner of it and fetching sixty kilobytes. This topic is for the data engineer or GIS archivist converting raster holdings at scale — thousands of scenes, terabytes at a time — and it covers the decisions that are fixed at write time and cannot be revisited once the objects are tiered and locked.
The Failure Mode: A Valid File That Reads Like a Tape
A GeoTIFF is valid whether or not it is organised for partial reads, and every tool will open it either way. The organisation is invisible until a reader tries to fetch a window: a stripped, overview-less scene forces a full download for any request, so a 900 MB orthophoto costs 900 MB to answer a question about one field. The same pixels written as a COG cost a few hundred kilobytes for the same question.
For an archive the consequence is doubled, because the wasted bytes are also retrieval charges from an archive class. Conversion is therefore not a nicety applied to the delivery copy — it changes the economics of every future read of the preservation copy too.
Prerequisite Context
Three things should be settled before a conversion pipeline is written. The reference system must be correct and declared in the GeoTIFF keys — a COG with wrong georeferencing is a well-organised error, and the CRS synchronization discipline applies here exactly as it does to vector conversion. The compression choice must be made against the imagery’s bit depth and content, since a predictor setting that suits 8-bit RGB is wrong for 16-bit elevation. And the intended access pattern must be known, because tile size and overview levels are chosen against the window sizes readers will actually request.
Concept & Design Decisions
Internal tile size. 512×512 is the sensible default for archives: large enough that a window request fetches few tiles, small enough that fetching one does not pull half a megabyte of unwanted pixels. 256×256 suits interactive map delivery where many small windows are requested; 1024×1024 suits bulk analytical reads that consume whole neighbourhoods.
Overview levels and resampling. Build overviews down to where the whole scene fits in roughly one tile — typically five or six levels for a large orthophoto. The resampling method matters more than teams expect: average for continuous data, nearest for categorical rasters where interpolating class codes invents categories that do not exist, and mode where a categorical overview should show the dominant class.
Compression. DEFLATE with a horizontal predictor remains the safest choice for archival imagery — universally readable, lossless, and effective on the smooth gradients typical of aerial photography. ZSTD compresses better and faster where the reader ecosystem supports it. Lossy options belong in delivery copies, never in the preservation copy.
Layout order. A COG’s overviews and the image file directory must precede the full-resolution data, so a reader can discover the structure from the first few kilobytes. This is what the validator’s “offset of the main IFD” warnings are about, and it is produced automatically by the COG driver and easily destroyed by a later in-place edit.
Implementation
A production conversion sets every parameter explicitly. Defaults change between GDAL versions, and an archive whose layout depends on a default is an archive whose layout is a function of when it was written.
# Convert a source orthophoto to an archival COG, all parameters pinned
gdal_translate \
datasets/imagery/raw/ortho_n5432_e0871.tif \
/vsis3/spatial-archive/archive/imagery/2026/ortho_n5432_e0871.tif \
-of COG \
-co BLOCKSIZE=512 \
-co COMPRESS=DEFLATE \
-co PREDICTOR=2 \
-co LEVEL=9 \
-co OVERVIEWS=IGNORE_EXISTING \
-co OVERVIEW_RESAMPLING=AVERAGE \
-co BIGTIFF=YES \
-co NUM_THREADS=ALL_CPUS \
--config GDAL_CACHEMAX 2048For a fleet, the same parameters belong in a job definition rather than a shell history, and the conversion should be idempotent on the output key so a restarted batch skips what it already produced — the discipline described under conversion pipeline automation applies unchanged.
Validation Gate
Every converted scene passes through the same three-part check before it is published: it opens, it is cloud-optimised, and its footprint matches what the catalogue will advertise.
rio cogeo validate /vsis3/spatial-archive/archive/imagery/2026/ortho_n5432_e0871.tif
gdalinfo -json /vsis3/spatial-archive/archive/imagery/2026/ortho_n5432_e0871.tif \
| jq '{blocks: .bands[0].block, overviews: (.bands[0].overviews | length),
crs: (.coordinateSystem.wkt | split("\n")[0])}'Expected output is is a valid Cloud Optimized GeoTIFF, a block size of [512, 512], five or more overview levels, and a non-empty coordinate system. A scene that validates but reports zero overviews is the most common near-miss, and it happens whenever a source file already contained overviews that the driver was told to preserve.
Cost & Performance Trade-offs
| Choice | Storage effect | Read effect | Notes |
|---|---|---|---|
| Tile 256 vs 512 | +2–4% metadata | more requests, smaller each | favours interactive map delivery |
| Tile 1024 | −1% metadata | fewer, larger fetches | favours bulk analytical reads |
| Overviews on | +33% size | window reads 100× cheaper | the single best archival trade |
| DEFLATE → ZSTD 12 | −12–18% size | equal | needs a reader that supports it |
| Predictor 2 on 8-bit RGB | −20–30% size | equal | wrong for floating-point data |
The overview row is the one that decides the archive’s economics. Overviews add roughly a third to stored size and remove two orders of magnitude from the cost of every windowed read — which for imagery that is browsed far more often than it is analysed in full is not a close call.
Failure Modes & Edge Cases
Predictor mismatch inflates rather than shrinks. A horizontal predictor on floating-point elevation data can make files larger. Use predictor 3 for floating point, 2 for integer imagery, and measure rather than assume.
Overviews inherited from a bad source. OVERVIEWS=FORCE_USE_EXISTING carries across whatever the source contained, including overviews built with the wrong resampling for categorical data. Regenerate rather than inherit unless the source is known-good.
Nodata lost in conversion. A source whose nodata is declared only in a sidecar loses it silently, and averaged overviews then blend nodata into real pixels along scene edges. Assert the nodata value survives.
BigTIFF thresholds. Files crossing 4 GB require BigTIFF, and a conversion that discovers this at the end fails after doing all the work. Set BIGTIFF=YES for archival conversions regardless of expected size.
Operational Execution Checklist
Nodata, Masks and the Edges of a Scene
Every aerial or satellite scene has edges where there is no data, and how those pixels are represented decides whether an archive’s mosaics have visible seams a decade later. The choice is between a nodata sentinel value and an explicit mask band, and it interacts with compression, overviews and every downstream mosaic.
Declare whichever is chosen in the file rather than in a convention, and carry it through the conversion explicitly — a mask that exists in the source and not in the archived copy is a silent loss that appears as a grey halo in the first mosaic somebody builds. Where the source uses a sentinel and the archive standardises on masks, do the translation once during conversion and record it in the manifest.
Where COG Sits Among the Archive’s Formats
An archive holding imagery, vector features and point clouds ends up with three cloud-optimised formats, and the similarity between them is the useful part: each is a self-describing, range-readable file whose internal organisation is fixed at write time and whose layout decisions govern every later read. Recognising that as one pattern rather than three unrelated conversions simplifies both the pipeline and the operational model considerably.
The shared properties are worth naming. Each format puts an index or a directory near the front so a reader can discover the structure in a few kilobytes. Each makes partial reads possible without a server, so the archive needs storage and not software to serve them. Each fixes its layout at write time, so the conversion is the last opportunity to make those decisions cheaply. And each carries its own reference-system declaration, so a file separated from its catalogue is still interpretable.
The differences that matter operationally are narrower than they appear. Tile size in a COG plays the role that row-group size plays in GeoParquet and octree node size plays in COPC: it is the granularity at which a reader fetches, and it is chosen against the query’s shape rather than against a general rule. Overviews play the role that a coarser octree level plays for point clouds — a cheap approximation for zoomed-out reads — and vector formats have no equivalent, which is why simplified geometry is sometimes stored as a separate column.
That symmetry is what allows one validation discipline to cover all three: does it open, is it organised for partial reads, and is it georeferenced as the catalogue claims. The specific checks differ, the structure of the gate does not, and an archive that implements it once per format behind a common interface gets consistent behaviour across its whole holding rather than three pipelines with three different notions of what “valid” means.
The corollary for planning is that a raster conversion project and a vector migration are the same project shape. Both need an inventory and triage step, an idempotent per-object conversion, a write gate, a quarantine path and a reconciliation. Building that scaffolding for imagery and reusing it for point clouds is far less work than the first project suggests, which is worth knowing when the imagery conversion is being scoped.
Frequently Asked Questions
Should the source imagery be kept after conversion?
Yes, tiered to the cheapest class the retention policy allows, at least until the converted archive has been validated and used. COG conversion is lossless when compression is lossless, so the source is not strictly needed — but a conversion defect discovered later is only fixable if the input still exists, and the storage cost of a deep-archived source is small against that insurance.
Is a COG appropriate for the preservation copy or only for delivery?
For both, and using one file for both is the main practical benefit. A COG is an ordinary GeoTIFF with a particular internal arrangement, readable by every tool that reads GeoTIFF, so nothing is given up by preserving in that form. The alternative — a plain preservation copy plus a COG delivery copy — doubles storage for no gain in readability.
How do overviews interact with archive storage classes?
They make partial reads possible only where the class supports ranged reads. In Glacier Flexible Retrieval or Deep Archive the whole object must be restored regardless, so overviews buy nothing until the restore completes — after which they make the restored copy far cheaper to work with. That asymmetry is one reason to keep frequently browsed imagery in an instant-retrieval class rather than the deepest one.
What to Do With an Existing Imagery Holding
Most archives arrive at this topic with imagery already stored — often years of it, in whatever form the producing systems delivered. The question is not how to convert new scenes but what to do about the existing holding, and the answer depends on a triage that takes a day.
Sort the holding into three groups. Scenes that are already valid COGs need nothing beyond a validation record. Scenes that are valid GeoTIFFs but not cloud-optimised are candidates for conversion, and their priority follows how often they are read: material nobody has opened in five years can wait indefinitely, while a browsed collection pays for its conversion within months in avoided transfer. Scenes that fail to open, lack georeferencing, or carry their reference system only in a sidecar need attention regardless of the conversion question, because they are the archive’s actual preservation risk.
The third group is the one to act on first, and it is usually small — a few percent of a legacy holding. Recovering a reference system from a sidecar or a delivery document while the people who produced the data are still reachable is a different exercise from attempting it a decade later, and the window for doing it cheaply closes quietly.
For the second group, convert opportunistically rather than as a campaign. Any scene that has to be restored for another reason — an integrity sample, a user request, a replication backfill — is already warm, and converting it then costs only the compute. Archives that adopt that habit find a large fraction of their browsed material converted within a couple of years without a project ever being scheduled.
A full conversion campaign is justified where the holding is browsed heavily enough that transfer costs dominate, or where a delivery obligation requires range-readable access. Price it against the measured transfer saving rather than against a general preference for modern formats; the saving is usually large for served collections and near zero for the long tail, and treating the two the same way is what makes such campaigns look expensive.
Should the archive standardise on one compression for all imagery?
Per collection rather than per archive, chosen from the data type. DEFLATE with the right predictor is the safe universal choice; ZSTD is better where the reader ecosystem supports it; and imagery that is genuinely lossy at source — compressed aerial deliveries, for instance — should not be recompressed at all. Recording the choice in the collection profile is what keeps a fleet from drifting.
How does COG interact with mosaicking?
Well, and it is one of the strongest arguments for the format. A mosaic reader fetches the overview level matching the requested zoom from each contributing scene, so a hundred-scene mosaic at low zoom transfers a few megabytes rather than gigabytes. That behaviour depends entirely on the overviews being present and correctly resampled, which is why the overview decision is not an internal detail.
Does the archive need to keep the pre-COG originals?
Only where the conversion was lossy, which for a correctly configured pipeline it is not. A losslessly converted COG contains the same pixels, so the original adds provenance rather than information — worth keeping cheaply, not worth a warm copy.
Related
- Converting GeoTIFF to Cloud-Optimized GeoTIFF at Scale — the fleet-scale conversion procedure built on these decisions.
- Choosing Overview Levels and Resampling for Archived Imagery — how far down to build, and which method to use for which data.
- Tuning COG Internal Tile Size for Range Requests — measuring tile size against the windows readers actually request.
- Validating COG Structure Before Archival — the write-gate check that keeps defective scenes out of the archive.
- Automating STAC Catalog Generation for Archived Imagery — publishing the converted scenes so they can be found.
- Compression Tuning & Storage Optimization — the codec reasoning behind the compression column above.
Up one level: Format Conversion & Pipeline Automation.