ZSTD vs LZ4 vs Snappy: Compression Trade-offs for Spatial Files
Choosing a compression codec for spatial columnar and vector archives is a trade among storage footprint, decompression latency, and CPU cost, and the right answer flips completely between a hot analytical tier and a cold archive tier. This guide puts ZSTD, LZ4, and Snappy head-to-head on GeoParquet and Arrow-backed spatial files, measuring compression ratio, compress and decompress throughput, CPU burn, cold-retrieval decompression latency, and level tunability — then gives a concrete recommendation by scenario. It is written for the data engineers and cloud architects who set compression= once and pay for it on every scan and every restore, under the Compression Tuning & Storage Optimization framework. The default Parquet codec (Snappy) is tuned for query speed, not archival density, so leaving it in place on a cold tier quietly overpays for storage for years.
The Decision Framing
Every codec choice trades three quantities that cannot all be maximized at once: the ratio (how small the file gets), the compression throughput (how fast you can write it), and the decompression throughput (how fast a reader can rehydrate it). Snappy and LZ4 sit at the fast-but-loose end — they compress and decompress at gigabytes per second but leave twenty to forty percent more bytes on disk than ZSTD. ZSTD sits at the dense end, with a single tunable level knob that spans a range from “faster than gzip, denser than Snappy” to “maximum ratio, slow to write.” For spatial archives the load profile is asymmetric: you write a partition once at ingest and read it rarely but under a latency SLA when a restore is triggered. That asymmetry is what makes the codec choice tier-dependent, and it is why a blanket default is almost always wrong for at least one of your tiers.
The numbers above are representative of GeoParquet with WKB geometry plus mixed attribute columns; your exact ratios depend on data entropy and are why you benchmark on a real sample rather than trusting a table. Highly repetitive attributes such as categorical land-use codes compress far better than dense floating-point coordinate columns, which is also why dictionary encoding for GIS attributes often matters more than the codec choice for the attribute side of the file.
Per-Dimension Analysis
Compression ratio. ZSTD’s larger window and entropy stage consistently beat LZ4 and Snappy on spatial data by twenty to forty percent, and the gap widens at higher levels. On a cold tier that ratio is money — thirty percent fewer bytes is thirty percent off every GB-month for the life of the archive, plus proportionally less cross-region replication bandwidth. Snappy and LZ4 were designed to be “good enough” ratio at maximum speed, and on coordinate-heavy geometry they leave real bytes on the table.
Compress throughput. This matters only at ingest, which for an archive happens once. LZ4 leads, Snappy close behind, and ZSTD is competitive at low levels but drops sharply as you climb toward level 19–22. Because ingest is a batch job you can parallelize across workers, compress throughput is rarely the binding constraint for a cold archive — you can afford ZSTD level 15 on a write that happens once and is read once a year.
Decompress throughput and cold-retrieval latency. This is the dimension that trips up archives. When a cold object is restored and a query engine reads it, decompression sits on the critical path of time-to-first-byte. LZ4 decompresses several times faster than ZSTD, which is why it wins for warm data that is read often. But ZSTD decompression is roughly constant regardless of the compression level used to write the file — a level-19 file decompresses about as fast as a level-3 file — so you get maximum ratio without a decompression-latency penalty. Snappy decompresses fast but, because its files are larger, more bytes have to move over the network from storage, which can erase its CPU advantage on a bandwidth-bound cold read.
CPU cost. LZ4 and Snappy are cheap on both ends. ZSTD costs meaningfully more CPU to compress at high levels and modestly more to decompress. In a serverless restore path where you pay per GB-second, ZSTD’s extra decompression CPU is usually dwarfed by the storage and egress savings from the smaller file, but you should model it explicitly against the spatial archive cost model.
Level tunability. Snappy has no knob — you get one operating point. LZ4 exposes an acceleration factor that trades ratio for speed. ZSTD exposes 22 levels, which makes it the only codec you can dial precisely onto a tier’s latency and density budget; the mechanics of choosing that number are the subject of ZSTD Level Configuration for Spatial Files.
The Three Codecs on One Axis
The codecs are not three points on a single quality scale; they occupy different positions in a two-dimensional space of ratio against speed, and the archive question is which corner of that space the workload actually lives in. Snappy and LZ4 sit at the fast, low-ratio end and differ from each other mainly in decompression speed. ZSTD spans a wide arc: at level 1 it is close to LZ4 in speed with a better ratio, and at level 19 it trades an order of magnitude of write speed for the best ratio of the three by a wide margin.
Read the shaded band as the constraint that decides the archive case. Nothing below a ratio of about four is worth choosing for cold storage, because the per-gigabyte saving compounds over a retention window measured in years while the write cost is paid once — which puts Snappy and LZ4 out of contention regardless of how fast they are. The remaining question is where inside the ZSTD arc to sit, and that is answered by the ingest window rather than by the archive: if the nightly load must finish in six hours, the level is the highest one whose throughput multiplied by the available cores clears the backlog.
Benchmarking on Your Own Spatial Sample
Never adopt a codec from a table. Round-trip a representative partition through all three and measure ratio and decompress time on your actual data. The following writes one partition three ways and reports the trade.
import time, os
import pyarrow.parquet as pq
import pyarrow as pa
table = pq.read_table("s3://spatial-archive/vector/sample/parcels_la_h3_res7.parquet")
raw_bytes = table.nbytes
for codec, level in [("snappy", None), ("lz4", None), ("zstd", 3), ("zstd", 15)]:
out = f"/tmp/bench_{codec}_{level}.parquet"
kw = {"compression": codec}
if level is not None:
kw["compression_level"] = level
pq.write_table(table, out, row_group_size=100_000, **kw)
size = os.path.getsize(out)
t0 = time.perf_counter()
for _ in range(5):
_ = pq.read_table(out) # cold-read decompression proxy
dt = (time.perf_counter() - t0) / 5
tag = f"{codec}-{level}" if level else codec
print(f"{tag:10s} ratio={raw_bytes/size:5.2f}:1 file={size/1e6:7.1f}MB read={dt*1000:6.1f}ms")For non-Python pipelines the same choice is a single GDAL layer-creation flag, so you can A/B two codecs directly at conversion time:
# Dense cold-tier write with ZSTD level 15
ogr2ogr -f Parquet parcels_zstd.parquet parcels.fgb \
-lco COMPRESSION=ZSTD -lco COMPRESSION_LEVEL=15 -lco ROW_GROUP_SIZE=100000
# Fast warm-tier write with LZ4 for comparison
ogr2ogr -f Parquet parcels_lz4.parquet parcels.fgb \
-lco COMPRESSION=LZ4 -lco ROW_GROUP_SIZE=100000Validating the Trade You Chose
Confirm the on-disk codec is what you intended and that the ratio justifies it. Read the Parquet column metadata directly rather than trusting the write call.
python -c "
import pyarrow.parquet as pq
m = pq.ParquetFile('parcels_zstd.parquet').metadata
rg = m.row_group(0)
for i in range(rg.num_columns):
c = rg.column(i)
print(f'{c.path_in_schema:20s} codec={c.compression:8s} '
f'ratio={c.total_uncompressed_size/max(c.total_compressed_size,1):.2f}')
"Expected output — every column reports the intended ZSTD codec, and the geometry column shows a lower ratio than the categorical attribute columns, which is normal for dense WKB coordinates:
geometry codec=ZSTD ratio=2.31
land_use_code codec=ZSTD ratio=8.74
parcel_area_m2 codec=ZSTD ratio=3.02
owner_type codec=ZSTD ratio=9.51If the geometry column’s ratio is the one dragging the file down and it dominates the byte budget, the codec is doing its job and further gains come from geometry encoding, not from switching codecs.
Decompression: The Number That Does Not Vary
Write throughput separates the codecs sharply; read throughput does not. All three decompress fast enough that the network, not the CPU, is the bottleneck for any realistic cloud read path, and the differences between them shrink to a rounding error against the latency of an archive restore. This is the fact that makes the write-side trade acceptable: an archive can be written slowly at a high ratio and still be read at line rate for its entire life.
Two qualifications keep this honest. First, per-core numbers assume the reader parallelises across column chunks; a single-threaded reader on a wide table can become CPU-bound on any codec. Second, ZSTD frames written with a large long-distance window allocate that window on read, so a memory-constrained reader — a Lambda function with 512 MB, say — can fail on a frame that a workstation handles easily. Both are configuration facts to record in the manifest, not reasons to prefer a weaker codec.
Recommendation by Scenario
The codec should follow the tier, and in a multi-tier archive it is entirely reasonable to re-encode as data ages — the pipeline that transitions an object from warm to cold can also rewrite it from LZ4 to ZSTD level 15. Coordinate the choice with the Hot/Warm/Cold Tier Design for Geospatial Data model so the codec matches the tier’s read pattern:
- Hot / interactive tier — keep Snappy (or ZSTD level 1). Reads are frequent and latency-sensitive, storage volume is small, and the default’s low decompression overhead is exactly right. Density does not pay off here.
- Warm / frequently-restored tier — use LZ4. When objects are read often but you still want some size reduction, LZ4’s very high decompression throughput keeps restore latency low while trimming bytes over Snappy.
- Cold / archive tier — use ZSTD level 12–15. Written once, read rarely, held for years: maximize ratio. Decompression stays fast because ZSTD decompression is level-independent, so a restore SLA is met while storage and replication bandwidth drop sharply.
- Deep archive / legal hold — use ZSTD level 19+. These objects may never be read; every byte saved compounds over a decade-long retention window, and the one-time high compress cost is negligible amortized across that lifespan.
Codec Selection Failure Modes
| Symptom | Cause | Fix |
|---|---|---|
| Cold-tier storage bill higher than modeled | Left the Snappy default on archived partitions | Re-encode aging objects to ZSTD level 12–15 during the warm-to-cold transition |
| Restore SLA missed on a bandwidth-bound read | Snappy’s larger files move more bytes over the network from cold storage | Switch to ZSTD; smaller files cut transfer time even after decompression CPU |
| Ingest job saturates CPU and stalls | ZSTD level 19+ applied at write time on a hot ingest path | Ingest at ZSTD level 3 or LZ4, then re-compress to a high level asynchronously as data cools |
| Codec flag silently ignored, file stays Snappy | Engine or GDAL build lacks the codec, falls back to default | Verify with parquet metadata that compression matches intent before publishing |
Operational Execution Checklist
Frequently Asked Questions
Is there a case for Snappy or LZ4 in a spatial archive?
Yes, at the edges of the pipeline rather than in the archive itself. Intermediate shuffle files, spill-to-disk during a large conversion job, and short-lived staging copies all benefit from a codec whose write cost is negligible, because those bytes exist for minutes and are never billed for a year. What does not justify them is the archive object: there, the ratio is paid for once and recovered every month of the retention window.
What about newer codecs such as Brotli or LZMA?
Both achieve ratios in the same region as high-level ZSTD, and both decompress substantially more slowly — LZMA by roughly an order of magnitude. For an archive whose defining read event is a bulk restore of many gigabytes, decompression speed is not a detail. The bigger objection is ecosystem: ZSTD is supported by every Parquet writer, GDAL build, and query engine a spatial archive is likely to meet over a decade, and a format that a future reader cannot open is a preservation failure regardless of its ratio.
Does the codec choice affect the ability to read part of a file?
Not directly — Parquet compresses each column chunk and page independently, so any of these codecs supports reading one column of one row group without touching the rest. What does affect it is the frame layout used for whole-file compression outside a columnar format: a single ZSTD frame over an entire GeoPackage must be decoded from the start, which is one of several reasons archives prefer columnar objects over compressed monoliths.
How should the codec decision be recorded?
In the dataset manifest, alongside the level, the writer version, and the date. Codec choice is not self-evident from the file — Parquet records which codec compressed each column chunk, but not why, nor what alternatives were measured. A one-line record turns a future “should we re-encode this archive?” conversation from a re-benchmarking exercise into a lookup, and it is the same record an auditor asks for when reproducing the archive from its source.
Does the choice change for raster archives?
The reasoning transfers but the numbers do not. Cloud-optimised GeoTIFF supports DEFLATE, LZW, ZSTD and lossy options, and the ratios on imagery are dominated by the predictor setting and the bit depth rather than by the codec family. Run the same two-axis measurement on representative tiles before committing, and expect the gap between codecs to be narrower than it is on vector attribute data.
Can two codecs be mixed inside one archive?
Yes, per column and per file, and it is sometimes the right answer. Parquet records the codec per column chunk, so a file can compress geometry with ZSTD at a high level and a rarely read audit column at a low one. Mixing across files in the same dataset is also legal and mainly costs explanation: the manifest should say which generation used which settings, or a later size comparison between partitions will be inexplicable.
Related
- Up: Compression Tuning & Storage Optimization — the parent framework for codec, level, encoding, and row-group decisions across the archive.
- ZSTD Level Configuration for Spatial Files — the sibling deep-dive on picking the exact ZSTD level once you have chosen the codec.
- When to Use Dictionary Encoding for Categorical GIS Fields — attribute-side gains that often outweigh the codec choice for categorical columns.
- Hot/Warm/Cold Tier Design for Geospatial Data — the tier read patterns that determine which codec each partition should carry.
- Spatial Archive Cost Modeling — quantify the storage, egress, and decompression-CPU trade of each codec choice.
Part of the Spatial Data Archival knowledge base.