Spatial Data Archival · Cold Storage

Measuring Dictionary Encoding Ratios for GIS Attribute Columns

Quantifying the exact byte savings dictionary encoding delivers on each categorical GIS attribute — land-use codes, administrative names, sensor identifiers — is the only defensible way to decide which columns to encode and which to leave in PLAIN. This guide is for data engineers and GIS archivists who need per-column, evidence-backed numbers rather than a blanket use_dictionary=True. Default writer settings apply dictionary encoding opportunistically and report nothing back, so an archive can carry columns whose dictionaries cost more than they save while genuinely repetitive columns silently fall back. The procedure below reads the actual encoded and uncompressed sizes out of Parquet column-chunk metadata, computes a true per-column ratio and cardinality profile, and turns those numbers into a keep-or-drop decision under the Compression Tuning & Storage Optimization framework, extending the qualitative rules in When to Use Dictionary Encoding for Categorical GIS Fields with hard measurement.

What the Ratio Actually Measures

Dictionary encoding replaces each value in a column with a small integer index into a per-column-chunk dictionary of distinct values. The saving is real only when the average value is wider than its index and the value repeats often. For a land-use column of 4 million rows holding roughly 40 distinct codes, each ~12-byte string collapses to a single-byte RLE-packed index plus a 40-entry dictionary — a large win. For a parcel-ID column where every value is unique, the dictionary grows to the full column, the indices are as wide as the pointers, and you pay dictionary overhead for zero repetition.

The number that matters is the encoded-to-plain ratio: the on-disk dictionary-encoded size (dictionary page plus data pages) divided by what the same column would occupy under PLAIN encoding, both measured before the page-level codec (ZSTD, Snappy) runs. Measuring pre-codec isolates the encoding decision from the compression decision covered in ZSTD Level Configuration for Spatial Files; the two stack, but they are tuned independently.

Per-column dictionary encoding measurement matrix A four-row table comparing GIS attribute columns across cardinality, plain size, dictionary-encoded size, encoded-to-plain ratio, and the resulting encode decision. Low-cardinality categorical columns show ratios well below one and are encoded; high-cardinality identifier columns show ratios at or above one and are left in plain encoding. Column Cardinality PLAIN Dict size Ratio Decision land_use_code 40 46 MB 5 MB 0.11 encode admin_name 3,100 88 MB 19 MB 0.22 encode sensor_id 210,000 64 MB 61 MB 0.95 review parcel_uid 4,000,000 120 MB 138 MB 1.15 leave PLAIN

Step-by-Step Measurement Procedure

1. Write two probe copies of the sample

To get a true PLAIN baseline you must disable dictionary encoding explicitly for one write, because pyarrow defaults to dictionary-on. Take a representative row-group-sized sample (not the whole archive) so the probe is fast and the cardinalities match production.

import pyarrow.parquet as pq
import pyarrow.dataset as ds

sample = ds.dataset("s3://spatial-archive/parcels/2023/", format="parquet")
tbl = sample.head(2_000_000)  # one row-group-scale representative slice

# Probe A: dictionary encoding forced ON, codec OFF to isolate encoding
pq.write_table(tbl, "/tmp/probe_dict.parquet",
               use_dictionary=True, compression="none",
               row_group_size=2_000_000)

# Probe B: PLAIN baseline, codec OFF
pq.write_table(tbl, "/tmp/probe_plain.parquet",
               use_dictionary=False, compression="none",
               row_group_size=2_000_000)

Turning the codec off for both probes is deliberate: ZSTD can mask a bad dictionary by compressing the redundant indices anyway, so measuring pre-codec exposes the encoding decision cleanly.

2. Read encoded sizes from column-chunk metadata

Parquet records total_compressed_size, total_uncompressed_size, and the encoding list for every column chunk in the footer. Iterate the row groups and pull them per column.

import pyarrow.parquet as pq

def column_sizes(path):
    md = pq.ParquetFile(path).metadata
    out = {}
    for rg in range(md.num_row_groups):
        for c in range(md.num_columns):
            col = md.row_group(rg).column(c)
            name = col.path_in_schema
            enc = tuple(col.encodings)
            agg = out.setdefault(name, {"bytes": 0, "encodings": set()})
            agg["bytes"] += col.total_uncompressed_size  # codec off, so == on-disk
            agg["encodings"].update(enc)
    return out

dict_sizes  = column_sizes("/tmp/probe_dict.parquet")
plain_sizes = column_sizes("/tmp/probe_plain.parquet")

With the codec off, total_uncompressed_size equals the bytes actually written, so it is the honest encoded footprint including the dictionary page.

3. Compute cardinality and the per-column ratio

Pair the byte figures with distinct-value counts. Use DuckDB for the cardinality pass so you never materialize the whole column in Python memory.

import duckdb

card = dict(duckdb.sql("""
    SELECT column_name, approx_count_distinct(v) AS ndv FROM (
        SELECT 'land_use_code' AS column_name, land_use_code::VARCHAR AS v
          FROM read_parquet('s3://spatial-archive/parcels/2023/*.parquet')
        UNION ALL SELECT 'admin_name', admin_name FROM read_parquet('s3://spatial-archive/parcels/2023/*.parquet')
        UNION ALL SELECT 'sensor_id',  sensor_id::VARCHAR FROM read_parquet('s3://spatial-archive/parcels/2023/*.parquet')
    ) GROUP BY column_name
""").fetchall())

for name in dict_sizes:
    d = dict_sizes[name]["bytes"]
    p = plain_sizes[name]["bytes"]
    ratio = d / p if p else 1.0
    enc = "DICT" if "RLE_DICTIONARY" in dict_sizes[name]["encodings"] else "PLAIN(fell back)"
    print(f"{name:16s} ndv={card.get(name,'—'):>10} ratio={ratio:5.2f}  {enc}")

A ratio near 0.1–0.3 is a clear encode; a ratio at or above 1.0 means the dictionary is pure overhead and the column should stay PLAIN. Watch the encoding flag: if a column you forced to use_dictionary=True still reports PLAIN, the writer hit the dictionary page-size ceiling and fell back mid-chunk — the failure mode addressed in Configuring Parquet Dictionary Page Size for Spatial Attributes.

Interpreting the Ratio in Context

A raw ratio is necessary but not sufficient — three spatial-specific factors decide whether a low ratio is truly worth acting on. First, the dictionary is per-column-chunk, so cardinality is measured at row-group scale, not archive scale. A district_name column with 3,100 distinct values nationwide might present only 40 distinct values inside a single region-partitioned row group; that local cardinality is what the encoder sees, and it can make a column that looks marginal archive-wide an easy win per chunk. Always measure on a slice matching your production row-group size, which is why the probes above use a row-group-scale sample rather than a random scatter of rows.

Second, physical sort order changes the ratio for the run-length side of the encoding. Dictionary indices are further compressed with RLE, so a column sorted or clustered such that identical codes sit contiguously packs far tighter than the same column in random ingest order. If your archive applies a Hilbert or H3 sort before writing — as most spatially partitioned pipelines do — categorical attributes that correlate with location (soil class, zoning code, sensor platform) inherit that locality and their measured ratio improves accordingly. Measure after the sort you actually ship, not before.

Third, a favorable ratio still carries a decode cost at read time: every scan of a dictionary-encoded column reconstructs values through an indirection the reader must resolve. For a cold archive read rarely, that cost is irrelevant against the storage saving; for a hot vector index scanned on every request, a marginal 0.85 ratio may not justify the per-query overhead. Weigh the ratio against access frequency, not in isolation — the same discipline that governs tier placement across the archive.

Validation & Verification

Confirm the encoding actually chosen per column with the standalone parquet-tools reader, which prints the encoding list straight from the footer without your own metadata code in the loop.

parquet-tools inspect /tmp/probe_dict.parquet | grep -E 'path_in_schema|encodings'

Expected output — categorical columns must list RLE_DICTIONARY; the identifier column must show it fell back to PLAIN:

path_in_schema: land_use_code
encodings:      ['PLAIN', 'RLE', 'RLE_DICTIONARY']
path_in_schema: admin_name
encodings:      ['PLAIN', 'RLE', 'RLE_DICTIONARY']
path_in_schema: parcel_uid
encodings:      ['PLAIN']

The PLAIN that appears alongside RLE_DICTIONARY is the dictionary page’s own encoding, not a fallback — presence of RLE_DICTIONARY in the list is the signal that data pages were dictionary-encoded. A column showing only PLAIN was never dictionary-encoded at all. Cross-check the ratios against the Apache Parquet encodings specification so the numbers you report map to documented behavior.

Troubleshooting

Symptom Cause Fix
Column forced on still reports PLAIN only Dictionary exceeded dictionary_pagesize_limit and the writer fell back for the whole chunk Raise the limit or accept PLAIN; see the page-size configuration guide
Ratio ≈ 1.0 on a text column you expected to shrink High real cardinality (free-text or per-feature identifiers), not categorical Leave PLAIN; dictionary indices are as wide as the values
Ratio worse when measured on full archive than on sample Cardinality grows with scale — codes concatenated with region suffixes Split the composite column or measure on a full-scale sample, not head()
Dictionary size larger than PLAIN Values shorter than the index width (e.g. single-char flags) Encode as a narrow native type instead of dictionary-encoding strings

Operational Execution Checklist

Part of the Spatial Data Archival knowledge base.

Related pages

Configuring Parquet Dictionary Page Size for Spatial Attributes When to Use Dictionary Encoding for Categorical GIS Fields