Spatial Data Archival · Cold Storage

When to Use Dictionary Encoding for Categorical GIS Fields

Categorical GIS attributes — land_use_code, admin_level, sensor model IDs, regulatory compliance flags — routinely dominate the metadata overhead of a spatial archive, yet engine defaults make the encoding decision silently and inconsistently across row groups. This page gives data engineers, GIS archivists, and cloud architects a deterministic rule for when dictionary encoding earns its keep on a categorical column and when it backfires, plus the exact profiling, writer, and validation commands to enforce that decision per column. Default behaviour fails because writers auto-enable dictionaries up to a size ceiling, then fall back to plain encoding mid-row-group when cardinality drifts — leaving you paying for a populated dictionary page that buys nothing while inflating decode latency on cold retrieval. This procedure sits one level under the parent Dictionary Encoding for GIS Attributes decision framework.

Decision at a Glance

Encode only when cardinality and null rate both stay within budget; otherwise fall back to plain encoding so downstream codecs see clean integer arrays:

When to enable dictionary encoding on a categorical GIS column A top-down flow. Start by profiling the column. First gate: 500 or fewer unique values? If no, disable the dictionary and fall back to plain encoding. If yes, second gate: null rate under 40 percent? If no, fall back to plain encoding. If yes, enable dictionary encoding, then cap the dictionary page at 1 megabyte. Profile column ≤ 500 unique values? Null rate < 40%? Disable dictionary plain encoding + ZSTD Enable dictionary Cap dictionary page at 1 MB Yes No Yes No

The Encoding Decision: Five Quantitative Gates

Enable dictionary encoding on a categorical field only when all five conditions hold simultaneously. A field failing any one criterion should bypass the dictionary and route directly to page-level compression — the entropy-reduction job is then handed to ZSTD Level Configuration for Spatial Files, which compresses the raw column far more cheaply than a saturated dictionary page would.

Gate Operational limit Failure root-cause Fallback strategy
Cardinality-to-volume ≤ 500 unique values per 10M rows Dictionary page exceeds 1 MB, negating columnar gains Plain encoding + ZSTD level 3
Average string length ≥ 4 characters Short codes (Y/N, 0/1) compress better via RLE / bit-packing Direct ZSTD or RLE
Repetition density ≥ 60% of non-null values repeat ≥ 3× per row group Sparse dictionaries waste a 4-byte pointer per row Numeric surrogate keys
Query access pattern =, IN, GROUP BY, dimension joins Range operators (>, <, BETWEEN) force a full dictionary decode at query time Leave unencoded; index separately
Format compatibility Parquet / GeoParquet v1.0+ Shapefile and GeoJSON lack native dictionary pages Pre-serialize to Parquet before archival

The format gate assumes your data is already columnar; if it still lives in Shapefile or GeoPackage, run the GeoParquet Migration Workflows pipeline first, because dictionary encoding is a column-chunk property that only exists in Parquet-family layouts.

How Many Columns Survive the Gates

Running the gates over a real attribute table is a filtering exercise, and the attrition is steeper than most teams expect. A typical municipal asset table exposes forty-odd columns; by the time cardinality, value width, null density, query usage, and growth rate have each been applied, four or five columns are left carrying almost all of the achievable saving. Knowing that shape in advance stops the encoding configuration from turning into a forty-line list of guesses.

Column attrition across the five encoding gates A funnel narrowing from forty-two candidate columns through five successive gates — cardinality ratio, value width, null density, query usage and growth rate — leaving five columns that are worth dictionary encoding. 42 columns candidates all non-geometry 23 columns ratio < 20% 19 identifiers dropped 14 width ≥ 8 B short codes dropped 11 nulls < 60% sparse columns dropped 7 queried never-read dropped 5 stable growth drifting columns dropped Gate attrition on a 42-column municipal asset table 5 survivors carry ~91% of the achievable saving Each gate is cheap to evaluate; run them in this order so the expensive query-usage lookup sees the fewest columns.

The order matters as much as the gates. Cardinality and value width come from the file itself and cost one scan; null density comes from the same scan; query usage requires access-log analysis, which is the expensive step and should therefore see the smallest possible candidate set. Growth rate needs history, so it comes last and often turns into a monitoring rule rather than a hard gate on the first load.

Step-by-Step Procedure

Phase 1 — Profile the source column

Run a cardinality, null-rate, and string-length scan before touching any writer configuration. The DuckDB CLI reads GeoParquet in place, so you can profile cold-tier objects without a full load.

# Profile a categorical column straight from a GeoParquet archive object.
duckdb -c "
SELECT
  land_use_code,
  COUNT(*)                       AS freq,
  AVG(LENGTH(land_use_code))     AS avg_len,
  COUNT(*) FILTER (land_use_code IS NULL) AS nulls
FROM read_parquet('s3://geo-archive/parcels/2024/region_north.parquet')
GROUP BY land_use_code
ORDER BY freq DESC;
"

Phase 2 — Assert the gates in code

Encode the five thresholds as hard assertions so a drifting partition fails the pipeline loudly instead of silently falling back to plain encoding at write time.

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq

table = pq.read_table("datasets/parcels/2024/region_north.parquet")
col = table.column("land_use_code")

unique_count = len(pc.value_counts(col))
null_pct = pc.sum(pc.is_null(col)).as_py() / len(table) * 100

assert unique_count <= 500, f"Cardinality {unique_count} exceeds dictionary threshold"
assert null_pct < 40, f"Null rate {null_pct:.1f}% triggers fallback encoding"

Phase 3 — Configure the writer per column

Set use_dictionary explicitly as a per-column list — never rely on the engine default — and cap the dictionary page so a cardinality leak can never blow past 1 MB and fragment.

import pyarrow.parquet as pq

pq.write_table(
    table,
    "datasets/parcels/2024/region_north.dict.parquet",
    use_dictionary=["land_use_code", "admin_jurisdiction"],  # gated columns only
    dictionary_pagesize_limit=1_048_576,  # 1 MB hard cap on the dictionary page
    data_page_size=1_048_576,
    write_statistics=True,
    compression="zstd",
)

For GDAL/OGR pipelines the Parquet driver enables dictionary encoding by default; control layout through row-group sizing instead:

ogr2ogr -f "Parquet" datasets/parcels/2024/region_north.dict.parquet \
  datasets/parcels/2024/region_north.shp \
  -lco COMPRESSION=ZSTD \
  -lco ROW_GROUP_SIZE=1000000

Phase 4 — Cluster values before the write

Dictionary efficiency decays when a row group spans heterogeneous spatial partitions, because the same values get re-materialized in every group. Sort by the categorical key so each row group holds contiguous values and the dictionary stays compact. Pick the row count with the formula in Calculating Optimal Row Group Size for Spatial Queries so groups land in the 128–256 MB band without straddling a partition.

# Contiguous categorical runs keep one dictionary per row group, not many.
table = table.sort_by("land_use_code")

Validation & Verification

Confirm the dictionary was actually written, sits under the 1 MB cap, and that integer pointers replaced the bulk of the original string bytes before promoting the file to an archival tier.

import pyarrow.parquet as pq

meta = pq.read_metadata("datasets/parcels/2024/region_north.dict.parquet")
for rg in range(meta.num_row_groups):
    col = meta.row_group(rg).column(0)  # land_use_code
    dpo = col.dictionary_page_offset
    # The dictionary page sits between its own offset and the first data page,
    # so its on-disk size is the difference of the two offsets.
    dict_size = (col.data_page_offset - dpo) if dpo is not None else 0
    overhead = dict_size / col.total_compressed_size
    print(f"RG {rg}: dict {dict_size/1024:.1f} KiB, overhead {overhead:.1%}")
    assert dict_size <= 1_048_576, f"RG {rg} dictionary page exceeds 1 MB cap"

Annotated expected output for a healthy low-cardinality column:

RG 0: dict 3.2 KiB, overhead 1.4%   # dictionary well under cap, pointers dominate
RG 1: dict 3.1 KiB, overhead 1.3%   # consistent across groups → no mid-file fallback

Dictionary overhead climbing above roughly 10% of compressed column size, or sizes that differ wildly between row groups, signals cardinality drift and a partial fallback — re-profile that partition before archiving.

Where the Break-Even Actually Sits

Dictionary encoding is not free: the dictionary page itself occupies bytes, and those bytes are paid per row group whether or not the encoding earns them back. For a low-cardinality column the overhead vanishes against the saving; as cardinality climbs, the two lines converge and then cross, and past the crossover the encoding costs more than plain storage before the codec has even run.

Dictionary saving against dictionary overhead as cardinality rises A line chart with cardinality ratio on the horizontal axis. The saving line falls from high to low while the overhead line rises, crossing at about twenty-two percent. The area before the crossing is shaded as net saving, the area after as net cost. break-even ≈ 22% bytes saved by indices dictionary page overhead net saving net cost 1%5%12% 22%45%100% cardinality ratio (distinct values ÷ rows in the row group) Measured on 24-byte string values in 128 MB row groups; wider values push the crossing right, narrower values pull it left.

The crossing point is not a constant — it moves with value width and row-group size, which is why the gates use a ratio rather than an absolute distinct-value count. Wide values push the break-even further right, because each avoided repetition saves more; small row groups pull it left, because the dictionary is rebuilt more often. When a column sits within a few points of the crossing, measure the two writes rather than reasoning about them: the difference is usually small enough that write-time simplicity and predictable behaviour under growth should decide it.

Troubleshooting

Symptom Root cause Fix
Storage grows after encoding High-cardinality leak or string-length variance > 32 chars saturates the dictionary Drop the column to use_dictionary=False; let ZSTD level 4 handle it
Query latency spikes on range filters >, <, BETWEEN predicates force a full dictionary decode at runtime Materialize a numeric sort key; keep the string column unencoded
Cold-retrieval timeouts Dictionary pages fragmented across row groups that straddle spatial partitions Increase row_group_size; sort by the categorical key and coalesce partitions before the write

High-cardinality identifiers, UUIDs, free-text survey notes, and already-numeric classification codes consistently fail the gates and should never be dictionary-encoded; they belong with direct ZSTD compression. Before committing mappings to immutable cold storage, stabilize attribute taxonomies through Schema Mapping & Attribute Validation so the dictionary you freeze still matches the data a year later.

Operational Execution Checklist

Frequently Asked Questions

Can a column pass the gates on one partition and fail on another?

Routinely, and it is the normal case for spatially partitioned archives. A contractor column over a dense metropolitan partition may hold six thousand distinct names, and eighty over a rural one at the same row count. Because encoding is chosen per write job, the honest answer is to evaluate the gates against the densest partition and apply one decision uniformly — a column encoded everywhere behaves predictably, whereas a column encoded in some partitions and not others produces confusing per-partition size reports and complicates any later audit of the archive.

Do the gates apply to numeric columns as well as strings?

They apply, but the saving is much smaller and the width gate usually rejects them. A four-byte integer replaced by a two-byte dictionary index saves at most half its width, and Parquet’s bit-packing and delta encodings already capture most of the redundancy in a numeric column without a dictionary. Reserve dictionary encoding for strings and for fixed-width binary keys, and let the numeric columns take the codec’s own encodings.

What is the right decision for a column that is never queried?

Leave it plain unless it is wide and highly repetitive. An unqueried column is still stored and still paid for per gigabyte, so a genuinely repetitive one is worth encoding for footprint alone. What it does not justify is spending the writer memory and dictionary overhead on a marginal case: the query-usage gate exists precisely to break ties in favour of the columns whose decode speed a reader will actually notice.

Up one level: Dictionary Encoding for GIS Attributes · Compression Tuning & Storage Optimization

Related pages

Configuring Parquet Dictionary Page Size for Spatial Attributes Measuring Dictionary Encoding Ratios for GIS Attribute Columns