Spatial Data Archival · Cold Storage

Benchmarking ZSTD Decompression Cost on Restore

The standard defence of aggressive compression is that decompression is cheap and level-independent. That is true, and it is worth verifying against your own archive rather than repeating — because the restore path has characteristics a benchmark on a laptop does not reproduce, and because “cheap” is a claim about a ratio between decompression time and everything else in the path. This walkthrough measures that ratio properly for a cold spatial archive.

What the Restore Path Is Made Of

Decompression is one term in a sum, and on an archive-class restore it is rarely the largest. Measuring it in isolation answers a question nobody asked.

Where the time goes on a cold restore Four components of the time to first usable feature: restore latency dominating at three hours forty minutes, then transfer at eleven seconds, decompression at 2.4 seconds and parsing at 0.9 seconds. 1.2 GB GeoParquet object, GLACIER, standard restore tier — to first usable feature restore latency 3 h 40 m · 99.9% of the elapsed time transfer 11 s decompression (level 19) 2.4 s — 0.02% of the total parse + predicate 0.9 s Choosing level 3 over level 19 would save 1.4 seconds and cost roughly 40% more storage for the object's whole retention window.

Step-by-Step Procedure

Step 1 — Produce the same data at several levels 

for L in 3 9 12 15 19; do
  python3 - "$L" <<'PY'
import sys, pyarrow.parquet as pq, pyarrow.dataset as ds
level = int(sys.argv[1])
tbl = ds.dataset("staging/parcels_2026.parquet").to_table()
pq.write_table(tbl, f"bench/parcels_l{level}.parquet",
               compression="zstd", compression_level=level,
               row_group_size=500_000, write_statistics=True)
PY
done
ls -l bench/ | awk '{print $5, $9}'

Step 2 — Measure decompression alone, warm

Isolate the codec by reading from a local cache with the file already in the page cache, so the number is the CPU cost and nothing else.

import pyarrow.parquet as pq, time, statistics

for level in (3, 9, 12, 15, 19):
    path = f"bench/parcels_l{level}.parquet"
    pq.read_table(path)                              # warm the cache
    runs = []
    for _ in range(5):
        t0 = time.perf_counter()
        pq.read_table(path)                          # full decode, all columns
        runs.append(time.perf_counter() - t0)
    size = __import__("os").path.getsize(path) / 1e6
    print(f"level {level:>2}  {size:7.1f} MB  decode {statistics.median(runs):.3f} s  "
          f"{size / statistics.median(runs):6.1f} MB/s")

Step 3 — Measure through the real path, cold

Then repeat against the archive as it is actually served — same region, same client, no local cache — because that is the number that describes a restore.

for L in 3 19; do
  aws s3 cp "bench/parcels_l${L}.parquet" "s3://spatial-archive/bench/" --quiet
  /usr/bin/time -f "level $L end-to-end %e s" python3 -c "
import pyarrow.dataset as ds
ds.dataset('s3://spatial-archive/bench/parcels_l${L}.parquet').to_table()"
done
Decompression throughput and stored size across levels Across levels 3 to 19, decompression throughput falls only eleven percent while stored size falls thirty-one percent. levelstored sizedecode throughputdecode time, 1.2 GB 3 412 MB 640 MB/s1.9 s 9 331 MB 610 MB/s2.0 s 12 302 MB 598 MB/s2.0 s 15 291 MB 586 MB/s2.1 s 19 284 MB 571 MB/s2.4 s

Validation & Verification

The benchmark is only meaningful if the decode is real and complete. A reader that lazily skips columns will report a throughput figure for work it did not do.

import pyarrow.parquet as pq
t = pq.read_table("bench/parcels_l19.parquet")
print(t.num_rows, t.num_columns, t.nbytes / 1e6, "MB materialised")

Expected output shows every column materialised and a byte count matching the uncompressed size. A nbytes far below expectation means the read was projected or lazy, and the timing describes a partial decode.

Troubleshooting

Symptom Root cause Fix
Throughput varies wildly between runs Page cache and CPU frequency scaling Warm the cache, take the median of five runs, pin frequency if possible
Higher level decodes much slower Long window engaged, memory pressure on the reader Check the window size; the level itself barely affects decode
End-to-end time unchanged across levels Network, not CPU, is the bottleneck Expected — this is the normal result and the point of the exercise
Decode faster than the disk can supply Reading from page cache Fine for isolating the codec; do not quote it as an end-to-end figure
Single-threaded decode much slower Reader not parallelising across column chunks Configure reader threads; decode parallelism is per column chunk

Operational Execution Checklist

What Changes the Answer, and What Does Not

The finding that decompression is a rounding error on the restore path is stable across levels, codecs and object sizes. It is not stable across every reader configuration, and knowing which variables actually move it prevents both false alarms and false confidence.

Which variables move the decompression share Six variables with whether each materially changes decompression’s share of the read path. variable moves the share? why compression level barely decode speed is level-independent codec (ZSTD vs Snappy) barely both far exceed network throughput object size no the ratio is scale-invariant reader thread count substantially single-threaded decode can bottleneck long window engaged substantially memory allocation, not CPU column projection substantially fewer columns, less to decode Every variable that matters is on the reader side, which is where a slow read should be investigated first.

That pattern is worth remembering when someone reports that a high compression level made their reads slow. The level is almost never the cause; a single-threaded reader, a memory-constrained worker or an unprojected scan of a wide table almost always is.

Frequently Asked Questions

Is there any level at which decompression cost becomes material?

Not from the level alone. What does become material is the long-distance window that high levels can engage, because it imposes a memory allocation on every reader and can push a constrained one into swapping or failure. That is a memory question rather than a throughput one, and it is settled by choosing the window explicitly instead of inheriting it.

Does the benchmark change for raster data?

The shape of the answer does not — decompression stays a small fraction of a restore — but the absolute numbers differ, because imagery decodes per tile and a windowed read decodes only the tiles it needs. For a COG the relevant measure is time per window request rather than per whole object, which usually makes the codec’s share smaller still.

Should the archive standardise one level everywhere?

Standardise per column family rather than per archive: geometry and coordinate columns flatten out around level 12, while categorical and temporal columns keep improving to 19. A single level is a compromise between two curves, and splitting the write costs nothing at read time because decompression is level-independent — which is exactly what this benchmark demonstrates.

Should the benchmark include the catalogue lookup and predicate planning?

For an end-to-end figure, yes — those are part of what a user waits for, and on a large partitioned dataset the planning step can exceed the decode. Keep them separate in the reporting, though, since they respond to different fixes: planning is improved by better statistics and fewer files, decoding by reader threads and projection.

How does the result change for a cold read on a small object?

The restore latency dominates even more completely, because it is per-object and largely independent of size. A 40 MB object and a 4 GB object from the same class both wait hours; the smaller one then decodes in a fraction of a second. That is the arithmetic behind consolidating small objects before tiering.

Is the measurement worth repeating for each collection?

Once per data shape rather than per collection. Vector attribute tables, imagery and point clouds have genuinely different decode profiles; two vector collections with similar schemas do not. Measuring three representative shapes and reusing the conclusions is enough for the purpose, which is confirming that the codec is not on the critical path.

Should the benchmark be re-run after a hardware change?

Only if the conclusion is close. The finding that decompression is a small share of the read path holds across an enormous range of hardware, because the comparison is against network and restore latency rather than against a fixed budget. Re-run it when the reader environment changes qualitatively — a move to serverless workers with tight memory, say — rather than on every instance-type refresh.

What should be published from the benchmark?

Three numbers: decode throughput per core, decode time as a share of the end-to-end read, and the level the archive settled on. Those are what a future engineer needs to decide whether to revisit the choice, and they fit in one line of the collection profile.

Does the conclusion hold for readers on constrained hardware?

For CPU, yes — even a modest core decodes far faster than a network delivers. For memory it does not, which is the one caveat worth carrying: a frame written with a large long-distance window imposes an allocation that a small worker may not be able to satisfy, and that failure looks like a codec problem while being a configuration one. Test the smallest reader the archive must support rather than assuming the finding generalises.

Is the benchmark worth automating?

As a scheduled job, no; as a script kept beside the collection profile, yes. It is run when something changes rather than continuously, and the value is in being able to re-run it in five minutes when someone questions the level rather than in a dashboard nobody reads.

Up one level: ZSTD Level Configuration for Spatial Files.

Related pages

Selecting the ZSTD Long-Window Size for Raster Archives Tuning ZSTD Compression for GeoParquet Archives