Selecting the ZSTD Long-Window Size for Raster Archives
Above level 15, ZSTD engages its long-distance matcher, which finds repeated byte sequences much further apart than the ordinary window allows. For raster archives that matters more than for vector data: imagery repeats itself at the scale of whole tile rows, not at the scale of adjacent values. This walkthrough is for the engineer tuning the window parameter for a raster or point-cloud holding, and it covers the trade nobody mentions — the window size chosen at write time must be allocated by every reader, including the constrained ones.
What the Window Actually Bounds
The window is the distance over which the compressor can find a match. Anything repeated further apart than the window is compressed as if it were new. In a tiled raster, the natural repetition distance is one tile row — which for a large scene can be tens of megabytes.
Step-by-Step Procedure
Step 1 — Measure the natural repetition distance
For imagery, it is roughly the compressed size of one tile row. Compute it rather than guessing, because it varies by an order of magnitude between a small aerial frame and a national mosaic tile.
python3 - <<'PY'
from osgeo import gdal
ds = gdal.Open("datasets/imagery/raw/ortho_n5432_e0871.tif")
b = ds.GetRasterBand(1)
bx, by = b.GetBlockSize()
tiles_per_row = -(-b.XSize // bx)
bytes_per_tile = bx * by * ds.RasterCount * (gdal.GetDataTypeSize(b.DataType) // 8)
print(f"tile row ≈ {tiles_per_row * bytes_per_tile / 1e6:.1f} MB uncompressed")
PY
# tile row ≈ 36.9 MB uncompressedStep 2 — Set the window to cover it, not to maximise it
# zstd CLI: --long=N sets a 2^N byte window. 26 = 64 MB, comfortably over a 37 MB tile row
zstd --long=26 -19 -T0 \
datasets/imagery/raw/ortho_n5432_e0871.tif \
-o staging/ortho_n5432_e0871.tif.zst
# For Parquet writers, the window follows the level; set it explicitly where the API allows
python3 - <<'PY'
import pyarrow.parquet as pq, pyarrow as pa
# pyarrow exposes level; window is derived, so for raster blobs prefer the CLI or libzstd directly
PYStep 3 — Confirm every reader can allocate it
This is the step that turns a good ratio into an unreadable archive. A frame written with a 256 MB window requires a 256 MB allocation on read, which a serverless function with a 512 MB limit may not survive alongside its own working set.
# Fails on a memory-constrained reader unless the window is allowed explicitly
zstd -d --long=26 staging/ortho_n5432_e0871.tif.zst -o /tmp/out.tif
# What a reader sees if it does not opt in to the long window
zstd -d staging/ortho_n5432_e0871.tif.zst -o /tmp/out.tif
# zstd: error 11 : Allocation error : not enough memoryValidation & Verification
# Inspect the frame header: window size is recorded in the frame itself
zstd -l --long staging/ortho_n5432_e0871.tif.zst
# Frames Skips Compressed Uncompressed Ratio Check Filename
# 1 0 198 MiB 872 MiB 4.404 XXH64 ortho_n5432_e0871.tif.zst
# Decompress in the most constrained environment the archive must support
docker run --memory=512m --rm -v "$PWD:/w" alpine:3 \
sh -c 'apk add -q zstd && zstd -d --long=26 /w/staging/ortho_n5432_e0871.tif.zst -o /dev/null'Expected output is a ratio matching the tuning target and a clean decompression inside the memory limit. A decompression that succeeds on a workstation and fails in the container is the failure this step exists to catch, and it is invisible in any measurement taken on the machine that wrote the file.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| No ratio improvement from a larger window | Repetition distances are short — vector data | Do not use a long window; it costs memory for nothing |
| Reader fails with an allocation error | Window larger than the reader will allocate by default | Readers must pass --long; lower the window to what they allow |
| Ratio improves but write time explodes | Level raised at the same time as the window | Change one variable at a time; the window is cheap, the level is not |
| Window setting appears ignored | Writer API derives the window from the level | Use the zstd CLI or libzstd directly for whole-file raster compression |
| Different ratios for identical scenes | Tile size differs, so tile-row distance differs | Standardise internal tiling before tuning the window |
Operational Execution Checklist
Where the Window Belongs in the Archive Record
The window is the one compression parameter a reader may have to act on, and an archive that records the level but not the window leaves constrained readers to discover the requirement through a failure. Three places it should appear, each for a different consumer.
Where the archive standardises one window across a collection, the profile is the authority and the other three are copies of it. That arrangement makes an inconsistent file detectable — its recorded window differs from its collection’s — rather than merely unusual.
Frequently Asked Questions
Does the long window help already-compressed data?
No. LAZ point clouds, JPEG-compressed imagery and any DEFLATE-compressed payload present essentially incompressible bytes, and a larger search window finds nothing in them. Applying one wastes writer memory and imposes an allocation on readers for zero benefit — one of several reasons not to layer a general-purpose codec over a format-aware one.
How does this interact with COG’s internal compression?
It does not directly: a COG compresses each tile independently with its own codec, so there is no long-distance matching across tiles at all. The long window applies when a whole file is compressed as one stream — an archival .tif.zst, for instance — which is a different arrangement from a COG and gives up range-readability in exchange for the better ratio.
Should the window be recorded in the archive’s metadata?
Yes, alongside the codec and level. It is the one compression parameter a reader may need to act on, since decompression requires opting in to a large window explicitly. An archive that records the level but not the window leaves constrained readers to discover the requirement through an allocation failure.
Does a long window change decompression speed?
Barely — the throughput difference across window sizes is within measurement noise. What changes is the memory the decoder must allocate up front, which is why a constrained reader fails rather than slows. Treating the window as a memory parameter rather than a speed one is the framing that avoids most confusion here.
Should the window differ between collections?
It should follow each collection’s repetition distance, which for imagery follows its tile row size. A collection of small aerial frames and a collection of large mosaic tiles genuinely want different windows, and standardising both to the larger value imposes the memory cost on readers of the smaller collection for no benefit.
What if the reader ecosystem is unknown?
Choose a window at or below 32 MB, which every mainstream reader handles without special configuration, and accept the ratio you get. An archive whose readers are unknown is by definition an archive with public or long-term consumers, and a file that some of them cannot open is a worse outcome than one that is a few percent larger.
Is the window worth setting for vector archives at all?
No. Vector data’s repetition distances are short — within a row group, usually within a page — so the long-distance matcher finds nothing and the window is pure reader-side cost. Leave it unset for columnar vector files and reserve it for whole-file compression of large raster or point-cloud payloads.
How is the window recorded so a reader knows to pass it?
In the object’s user metadata and in the collection profile, as described above. Readers that fail on a large window fail with an allocation error rather than a helpful message, so the archive has to supply the information out of band — which means recording it where a reader can find it before attempting the read.
Does the window need to match between the writer and the reader exactly?
The reader must permit at least the writer’s window; permitting more is harmless. That asymmetry is why recording the writer’s value is sufficient — a reader configured above it will always succeed.
Related
- ZSTD Level Configuration for Spatial Files — the parent topic where level and window are chosen together.
- Benchmarking ZSTD Decompression Cost on Restore — measuring what the read path actually pays.
- Cloud-Optimized GeoTIFF Conversion Pipelines — the per-tile alternative that trades ratio for range-readability.
- ZSTD vs LZ4 vs Snappy Compression Trade-offs for Spatial Files — the codec comparison this tuning assumes has already been settled.
Up one level: ZSTD Level Configuration for Spatial Files.