Spatial Data Archival · Cold Storage

Configuring Parquet Dictionary Page Size for Spatial Attributes

High-cardinality spatial attribute columns — H3 cell IDs, tile keys, gazetteer place names — routinely overflow Parquet’s default dictionary page budget and fall back to PLAIN encoding partway through a column chunk, silently erasing the compression you expected. This guide is for data engineers tuning the pyarrow and GDAL Parquet writers so that fallback happens by design, never by accident. The default dictionary_pagesize_limit of 1 MiB was chosen for narrow analytics columns, not for the wide, semi-repetitive string attributes typical of a spatial archive; left unchanged, it caps the dictionary long before it has captured the column’s real value set. Below is the exact configuration and verification loop that keeps intentional columns dictionary-encoded end to end, extending the measurement discipline in Measuring Dictionary Encoding Ratios for GIS Attribute Columns within the Compression Tuning & Storage Optimization framework.

Why the Dictionary Falls Back

When a Parquet writer dictionary-encodes a column chunk, it accumulates distinct values into an in-memory dictionary page. That page has a hard byte ceiling set by dictionary_pagesize_limit. The moment the growing dictionary would exceed the ceiling, the writer stops adding new entries, abandons dictionary encoding for the remainder of that column chunk, and re-emits the already-buffered rows plus everything after as PLAIN. The result is a column that is dictionary-encoded for its first slice and PLAIN for the rest — usually reported in the footer as PLAIN only, because the fallback dominates.

For spatial attributes this bites hard. A h3_cell column at H3 resolution 8 can hold tens of thousands of distinct 15-character hex strings inside a single large row group; the dictionary page fills, the writer falls back, and the archive stores full 15-byte strings instead of two-byte indices. The fix is not to disable the ceiling but to size it against the column’s real cardinality, or to reshape the write so each column chunk sees a value set that fits.

Dictionary page-size fallback state machine Two states connected by a threshold decision. The writer builds a dictionary and stays dictionary-encoded while distinct values fit within dictionary_pagesize_limit; when a new value would exceed the limit it transitions irreversibly to a PLAIN fallback for the rest of the column chunk. Raising the limit keeps the writer in the dictionary-encoded state longer. DICTIONARY-ENCODED indices + growing dictionary page two bytes per value new value fits → add entry dict page > limit? next value PLAIN FALLBACK whole column chunk re-emitted PLAIN full-width strings on disk yes no · raise limit widens this path

Step-by-Step Configuration Procedure

1. Size the ceiling against measured cardinality

Estimate the dictionary footprint per column chunk: distinct_values × (avg_value_bytes + index_overhead). For an h3_cell column with 60,000 distinct values at 15 bytes each, budget roughly 60000 × 18 ≈ 1.08 MB — already over the 1 MiB default. Set the limit with headroom.

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

tbl = ds.dataset("s3://spatial-archive/features/2023/", format="parquet").to_table()

pq.write_table(
    tbl,
    "s3://spatial-archive/features/2023/tuned.parquet",
    use_dictionary=["h3_cell", "tile_key", "place_name"],  # target only wide categoricals
    dictionary_pagesize_limit=4 * 1024 * 1024,             # 4 MiB, up from 1 MiB default
    data_page_size=1 * 1024 * 1024,
    compression="zstd",
    compression_level=6,
    row_group_size=1_000_000,                              # smaller chunks = smaller per-chunk dict
    write_statistics=True,
)

use_dictionary accepts a column list, so you enable it precisely where measurement said it pays and leave true identifier columns in PLAIN. Note the interaction with row_group_size: a smaller row group means each column chunk sees fewer rows and therefore fewer distinct values, which can keep a borderline column under the ceiling without raising the limit at all — a trade-off that couples this decision to Calculating Optimal Row-Group Size for Spatial Queries.

Read the budget as three bands rather than one number. The values themselves are the irreducible part; the index and offset overhead scales with entry count; and the headroom is what absorbs the densest partition you have not measured yet. A ceiling set exactly at the estimate fails on the first metropolitan tile:

Dictionary page byte budget against the default and tuned ceilings A single horizontal budget bar split into raw values of 900 kilobytes and index overhead of 180 kilobytes, totalling 1.08 megabytes. The 1 mebibyte default ceiling is marked just short of that total, showing why the writer falls back. A second marker at 4 mebibytes shows the tuned ceiling and the 2.9 megabytes of headroom it leaves for denser partitions. h3_cell · 60,000 distinct values × 15 B + 3 B index values · 900 KB index 180 KB headroom for denser partitions · 2.9 MB 1 MiB default ceiling dictionary is abandoned here 4 MiB tuned ceiling required: 1.08 MB — already past the default before headroom is considered Budget per column chunk, not per file: every row group rebuilds its own dictionary page.

2. Configure the same limit through the GDAL writer

Pipelines built on ogr2ogr expose the ceiling through a layer-creation option, so CLI conversions get the same protection as the Python path.

ogr2ogr -f Parquet features_tuned.parquet features.gpkg \
  -lco COMPRESSION=ZSTD \
  -lco COMPRESSION_LEVEL=6 \
  -lco ROW_GROUP_SIZE=1000000 \
  -lco "PARQUET_DICTIONARY_PAGE_SIZE_LIMIT=4194304" \
  -progress

Keep the GDAL and pyarrow limits identical so an archive written by two different tools reads back with consistent encoding. Confirm the option name against the GDAL Parquet driver documentation for your installed GDAL version, as layer-creation options are version-gated.

3. Force a full-column dictionary where cardinality is stable

For columns whose value set is closed and known — a fixed land-cover legend, a national admin roster — you can pin the dictionary so the writer never fallbacks even under a large row group, by keeping the row group small enough that the closed set always fits.

# Closed-vocabulary column: cap row group so its full dictionary always fits the ceiling
pq.write_table(
    tbl.select(["land_cover_class", "geometry"]),
    "legend_bound.parquet",
    use_dictionary=["land_cover_class"],
    dictionary_pagesize_limit=2 * 1024 * 1024,
    row_group_size=500_000,
    compression="zstd",
)

Balancing the Ceiling Against Row-Group Size and Memory

Two knobs move the same fallback boundary, and choosing between them is the crux of tuning this well. Raising dictionary_pagesize_limit lets a single column chunk hold a bigger dictionary, which keeps a high-cardinality column encoded even in a large row group — at the cost of writer memory, because every actively written column keeps its dictionary page resident. Lowering row_group_size shrinks the number of rows, and therefore the number of distinct values, each chunk must absorb, so the same column fits under a smaller ceiling — at the cost of more row groups, larger footers, and the pruning trade-offs weighed in Benchmarking Row-Group Size Against Spatial Predicate Pushdown.

For spatial archives the row-group lever is usually the better first move, because spatial data is rarely uniform. A tile_key column in a dense metropolitan partition carries far more distinct values per thousand rows than the same column over open ocean or unpopulated terrain. Sizing the row group so the densest realistic chunk stays under the ceiling protects every partition without inflating the limit globally and paying that memory cost on sparse chunks that never needed it. When the densest chunk still overflows even a small row group, only then raise the ceiling, and raise it to a measured multiple of the estimate rather than an arbitrary large value — an oversized limit wastes memory on every column, not just the one that needed it.

There is one case where you want the opposite: a closed-vocabulary column whose full dictionary is small and fixed. Pin a limit comfortably above its known dictionary size and let the row group grow, so the column stays dictionary-encoded across large chunks with no memory concern. The distinction is cardinality growth: bounded vocabularies tolerate big row groups, unbounded identifiers do not.

The two knobs map onto a plane, and only one region of it is a working configuration. Small row groups with a generous ceiling waste memory on chunks that never needed it; large row groups with the default ceiling fall back on the dense partitions; the workable band runs diagonally between them:

Row-group size against dictionary page ceiling A two-axis plane. The horizontal axis is row-group size from 16 to 512 megabytes; the vertical axis is the dictionary page size limit from half a mebibyte to 32 mebibytes. The lower-right triangle is labelled silent fallback, the upper-left triangle wasted writer memory, and a diagonal band between them is the workable configuration. An operating point is marked where a 64 megabyte row group meets a 4 mebibyte ceiling. silent fallback chunk cardinality > ceiling wasted writer memory pages resident per active column workable band 64 MB row group · 4 MiB ceiling holds through the densest metropolitan tile 16 MB64 MB128 MB 256 MB512 MB row_group_size 0.5 MiB1 MiB4 MiB 16 MiB32 MiB dictionary_pagesize_limit Move along the band, not off it: raise the ceiling only after the row-group lever is exhausted.

Validation & Verification

Verify that every targeted column reports RLE_DICTIONARY across all row groups, not just the first — a partial fallback shows up as some chunks dictionary-encoded and some PLAIN.

import pyarrow.parquet as pq

md = pq.ParquetFile("features_tuned.parquet").metadata
target = "h3_cell"
ci = md.schema.names.index(target)
for rg in range(md.num_row_groups):
    enc = md.row_group(rg).column(ci).encodings
    ok = "RLE_DICTIONARY" in enc
    print(f"row_group {rg}: {'DICT' if ok else 'FELL BACK'}  {tuple(enc)}")

Expected output — every row group must report DICT; a single FELL BACK line means the ceiling is still too low for that chunk:

row_group 0: DICT  ('PLAIN', 'RLE', 'RLE_DICTIONARY')
row_group 1: DICT  ('PLAIN', 'RLE', 'RLE_DICTIONARY')
row_group 2: DICT  ('PLAIN', 'RLE', 'RLE_DICTIONARY')

Troubleshooting

  • A later row group falls back while earlier ones held. That chunk’s local cardinality spiked — a dense urban tile packs more distinct h3_cell values than a rural one. Either raise dictionary_pagesize_limit further or reduce row_group_size so every chunk stays under budget.
  • Dictionary holds but the file barely shrank. The column’s values are near-unique; dictionary encoding is not the lever. Re-measure with the ratio procedure and consider leaving it PLAIN so you stop paying dictionary-build cost.
  • Raising the limit blew up writer memory. The dictionary page lives in memory per active column chunk; a 32 MiB limit across many parallel columns multiplies fast. Cap the limit and lower row_group_size together rather than pushing the ceiling alone.

Operational Execution Checklist

Frequently Asked Questions

Is there a downside to setting the ceiling very high everywhere?

Yes, and it is a write-side cost that does not show up in the output file. The dictionary page is held in memory for every column chunk currently being written, so a 32 MiB ceiling across twelve dictionary-encoded columns and eight parallel writer processes reserves headroom measured in gigabytes. On a container-sized ingest worker that manifests as an out-of-memory kill partway through a large partition, not as a warning. Size the ceiling to the densest chunk you actually have plus a margin, and revisit it when partition density changes.

Why does the footer report PLAIN when the first rows were clearly dictionary-encoded?

Because the fallback rewrites the chunk. When the dictionary hits the ceiling the writer discards the encoding for the entire column chunk, re-emitting the buffered rows as PLAIN, so the metadata describes the state the chunk ended in rather than the state it started in. That is why a partial fallback is invisible unless you inspect every row group: a file whose first ten chunks held and whose eleventh fell back reports a mix, and a file where every chunk fell back looks exactly like one that never requested a dictionary.

Does the ceiling apply per column or per row group?

Per column chunk — that is, once per column per row group. A file with five dictionary-encoded columns and twenty row groups builds a hundred dictionary pages, each independently subject to the ceiling. This is why local cardinality is what matters: a column with 200,000 distinct values across the whole file may still encode cleanly if no single row group contains more than a few thousand of them, which is exactly what happens when features are written in spatial order.

Part of the Spatial Data Archival knowledge base.

Related pages

Measuring Dictionary Encoding Ratios for GIS Attribute Columns When to Use Dictionary Encoding for Categorical GIS Fields