Spatial Data Archival · Cold Storage

Choosing H3 vs S2 vs Quadkey for Archive Partitioning

Choosing a discrete global grid as the partition key for a cold spatial archive is a one-way decision: it fixes your directory tree, your file counts, and the shape of every predicate-pushdown query for the life of the dataset. This guide compares H3 hexagons, S2 cells, and quadkey/tile schemes head-to-head for partitioning multi-terabyte GeoParquet and columnar archives, weighing cell-shape and area uniformity, resolution-to-partition mapping, neighbor-query ergonomics, library maturity, and the file-count blowup each scheme inflicts on object storage. It is written for the data engineers and GIS archivists who have to live with the layout after ingest, under the Compression Tuning & Storage Optimization framework where partition granularity and file size drive both scan cost and per-request billing. Default “just partition by lat/lon bucket” advice fails here because naive rectangular buckets vary wildly in feature density and produce partitions that are unusable for the bounded-scan queries cold retrieval depends on.

Why the Grid Choice Is Load-Bearing

A partition key is not just a folder name. It determines how many objects land in your bucket (and therefore your LIST and GET request bill), how tightly a bounding-box query can prune partitions before it touches cold storage, and whether a “give me this cell and its neighbors” query is one index lookup or a spatial join. The three dominant discrete global grids each make a different trade. H3 tiles the globe in hexagons; S2 projects the sphere onto a cube and recursively quarters each face into cells addressed by a Hilbert curve; quadkey (the Bing/slippy-map scheme) recursively quarters a Web Mercator plane into square tiles addressed by a base-4 string. The differences look academic until a query planner has to prune 40 million partitions.

Partition grid comparison matrix: H3 vs S2 vs Quadkey A three-row matrix comparing H3, S2, and quadkey partition schemes across cell shape, area uniformity, resolution nesting, neighbor-query ergonomics, library support, and file-count risk. H3 wins on area uniformity, quadkey wins on neighbor simplicity, S2 balances both, and quadkey carries the highest file-count risk at deep zoom. Scheme Cell shape Area uniformity Resolution nesting Neighbors Libraries File-count risk H3 Uber hexagon high (±10% area) 16 res · 1→7 k-ring, uniform mature (h3) moderate S2 Google quad (curved) medium 31 levels · 1→4 edge neighbors mature (s2) low–moderate Quadkey Bing / XYZ square tile low (pole skew) 24+ zooms · 1→4 prefix string ubiquitous high (deep zoom)

Reading the Matrix Dimension by Dimension

Cell shape and area uniformity. H3 hexagons hold their surface area within roughly ten percent across the globe and, critically, every neighbor sits at an identical center-to-center distance — there are no diagonal-versus-orthogonal neighbors as there are with squares. That regularity makes per-partition feature counts far more even for spatially concentrated data such as urban vector layers or AIS tracks. S2 cells are curved quadrilaterals whose area varies more than H3 but far less than a naive lat/lon grid, because the cube projection compensates for most of the Mercator distortion. Quadkey tiles are true Web Mercator squares, so their ground area collapses toward the poles: a zoom-10 tile covers vastly more ground at the equator than at 60° latitude, which skews partition sizes badly for any archive spanning a wide latitude range.

Resolution-to-partition mapping. This is where the file-count math lives. H3 nests one parent to seven children (aperture 7), so each resolution step multiplies partition count by roughly seven and cell area shrinks by the same factor. S2 and quadkey both quarter — one parent to four children — so each level multiplies partitions by four. The practical consequence: to move from “too few, oversized partitions” to “right-sized,” H3 gives you coarser control steps (×7) while S2 and quadkey let you tune in finer ×4 increments. When you are targeting a specific partition byte-size to satisfy a cold tier’s minimum-object-size rule, that granularity matters, and it interacts directly with your row-group sizing strategy.

Neighbor queries. Quadkey wins on raw simplicity: a tile’s parent is its string with the last character removed, and adjacency is cheap integer arithmetic on the X/Y at a zoom. H3’s k-ring (or grid_disk) returns all cells within k steps in one call with uniform distance semantics — ideal for “buffer this cell” retrieval. S2 exposes edge neighbors and, because cells lie on a Hilbert curve, contiguous ranges of S2 cell IDs map to contiguous regions, which is excellent for range-scan pruning but less intuitive for ring buffers.

Library support and file-count risk. All three have production libraries (h3, s2sphere/s2geometry, mercantile). The file-count risk column is the one that silently wrecks archives: quadkey at deep zoom explodes into millions of tiny tiles, each an object with its own PUT/GET/lifecycle-transition cost. Choose a resolution that keeps partition files comfortably above the tier’s minimum object size described in Hot/Warm/Cold Tier Design for Geospatial Data, and model the request-count consequences against the spatial archive cost model before committing.

Generating Partition Keys in All Three Schemes

Compute the key the same way regardless of scheme: derive a representative point (centroid for polygons, the point itself for points) in EPSG:4326, then index it. The following resolves one partition key per feature for each grid so you can compare partition cardinality before you commit.

import geopandas as gpd
import h3
import s2sphere
import mercantile

# Read a projected source; H3/S2/quadkey all index geographic coordinates.
gdf = gpd.read_file("s3://spatial-archive/vector/2024/parcels_region_north.fgb")
pts = gdf.geometry.to_crs(4326).representative_point()

H3_RES, S2_LEVEL, QK_ZOOM = 6, 10, 10  # coarse-to-comparable partition granularity

def keys(pt):
    lat, lng = pt.y, pt.x
    h3_key = h3.latlng_to_cell(lat, lng, H3_RES)
    s2_cell = s2sphere.CellId.from_lat_lng(
        s2sphere.LatLng.from_degrees(lat, lng)
    ).parent(S2_LEVEL)
    s2_key = s2_cell.to_token()              # compact hex token, e.g. "89c2594"
    qk_key = mercantile.quadkey(             # base-4 string, e.g. "0231010121"
        mercantile.tile(lng, lat, QK_ZOOM)
    )
    return h3_key, s2_key, qk_key

gdf[["h3_cell", "s2_token", "quadkey"]] = [keys(p) for p in pts]

# Partition cardinality is the number of directories each scheme would create.
for col in ("h3_cell", "s2_token", "quadkey"):
    print(f"{col:10s} -> {gdf[col].nunique():>8d} partitions")

Write the winning key into the Hive-style path exactly as the migration pipeline does — s3://spatial-archive/vector/h3_cell=8a2a1072b59ffff/part-0000.parquet — so the partition column is prunable at query time. The mechanics of that partitioned write are covered in Converting Legacy Shapefiles to GeoParquet at Scale.

Validating Partition Balance Before You Commit

The failure mode is a heavy-tailed partition distribution: a handful of cells over dense metros hold most of the features while thousands of rural cells hold a few rows each. Measure the skew directly with DuckDB against the keyed output before you write the final tree.

duckdb -c "
SELECT h3_cell,
       count(*)                    AS features,
       count(*) FILTER (WHERE true) * 1.0 /
         (SELECT avg(c) FROM (SELECT count(*) c FROM read_parquet('keyed/*.parquet') GROUP BY h3_cell)) AS skew_ratio
FROM read_parquet('keyed/*.parquet')
GROUP BY h3_cell
ORDER BY features DESC
LIMIT 5"

Expected output — the top partitions should sit within roughly an order of magnitude of the mean; a skew_ratio above ~50 means the resolution is too coarse for your dense regions and those cells will dominate scan time:

┌─────────────────┬──────────┬────────────┐
│     h3_cell     │ features │ skew_ratio │
│     varchar     │  int64   │   double   │
├─────────────────┼──────────┼────────────┤
│ 8a2a1072b59ffff │   184213 │      7.4   │
│ 8a2a1072b5b7fff │   151902 │      6.1   │
│ 8a1fb46622dffff │   142338 │      5.7   │
│ 8a2a10725d37fff │   128004 │      5.1   │
│ 8a1fb46622c7fff │   119887 │      4.8   │
└─────────────────┴──────────┴────────────┘

If the skew is unacceptable, either step to a finer resolution for the whole dataset or split only the hot cells to a deeper resolution — a mixed-resolution layout H3 and S2 both support natively because a parent cell ID unambiguously contains its children.

Common Partition-Key Failures

Symptom Cause Fix
A few partitions 100× larger than the median Resolution too coarse for dense metros; uniform grid can’t adapt to concentrated features Step one resolution finer, or split only hot cells to a deeper H3/S2 child level
Millions of sub-megabyte partition files, high LIST/GET bill Quadkey zoom (or H3/S2 level) too deep; each cell became its own object Coarsen the level; target partition files above the tier minimum object size
Bounding-box query scans the whole tree Partition column not written into the path, so the planner can’t prune Emit scheme_cell=VALUE as a real Hive path segment, not just a table column
Cells straddling the antimeridian or poles behave oddly Quadkey undefined beyond ±85.05° Mercator latitude; polar data lost Use H3 or S2, which cover the full sphere including the poles

Operational Execution Checklist

Part of the Spatial Data Archival knowledge base.

Related pages

Partitioning Cold Spatial Archives by Administrative Boundary