Spatial Data Archival · Cold Storage

Choosing H3 vs S2 vs Quadkey for Archive Partitioning

Choosing a discrete global grid as the partition key for a cold spatial archive is a one-way decision: it fixes your directory tree, your file counts, and the shape of every predicate-pushdown query for the life of the dataset. This guide compares H3 hexagons, S2 cells, and quadkey/tile schemes head-to-head for partitioning multi-terabyte GeoParquet and columnar archives, weighing cell-shape and area uniformity, resolution-to-partition mapping, neighbor-query ergonomics, library maturity, and the file-count blowup each scheme inflicts on object storage. It is written for the data engineers and GIS archivists who have to live with the layout after ingest, under the Compression Tuning & Storage Optimization framework where partition granularity and file size drive both scan cost and per-request billing. Default “just partition by lat/lon bucket” advice fails here because naive rectangular buckets vary wildly in feature density and produce partitions that are unusable for the bounded-scan queries cold retrieval depends on.

Why the Grid Choice Is Load-Bearing

A partition key is not just a folder name. It determines how many objects land in your bucket (and therefore your LIST and GET request bill), how tightly a bounding-box query can prune partitions before it touches cold storage, and whether a “give me this cell and its neighbors” query is one index lookup or a spatial join. The three dominant discrete global grids each make a different trade. H3 tiles the globe in hexagons; S2 projects the sphere onto a cube and recursively quarters each face into cells addressed by a Hilbert curve; quadkey (the Bing/slippy-map scheme) recursively quarters a Web Mercator plane into square tiles addressed by a base-4 string. The differences look academic until a query planner has to prune 40 million partitions.

Partition grid comparison matrix: H3 vs S2 vs Quadkey A three-row matrix comparing H3, S2, and quadkey partition schemes across cell shape, area uniformity, resolution nesting, neighbor-query ergonomics, library support, and file-count risk. H3 wins on area uniformity, quadkey wins on neighbor simplicity, S2 balances both, and quadkey carries the highest file-count risk at deep zoom. Scheme Cell shape Area uniformity Resolution nesting Neighbors Libraries File-count risk H3 Uber hexagon high (±10% area) 16 res · 1→7 k-ring, uniform mature (h3) moderate S2 Google quad (curved) medium 31 levels · 1→4 edge neighbors mature (s2) low–moderate Quadkey Bing / XYZ square tile low (pole skew) 24+ zooms · 1→4 prefix string ubiquitous high (deep zoom)

Reading the Matrix Dimension by Dimension

Cell shape and area uniformity. H3 hexagons hold their surface area within roughly ten percent across the globe and, critically, every neighbor sits at an identical center-to-center distance — there are no diagonal-versus-orthogonal neighbors as there are with squares. That regularity makes per-partition feature counts far more even for spatially concentrated data such as urban vector layers or AIS tracks. S2 cells are curved quadrilaterals whose area varies more than H3 but far less than a naive lat/lon grid, because the cube projection compensates for most of the Mercator distortion. Quadkey tiles are true Web Mercator squares, so their ground area collapses toward the poles: a zoom-10 tile covers vastly more ground at the equator than at 60° latitude, which skews partition sizes badly for any archive spanning a wide latitude range.

Resolution-to-partition mapping. This is where the file-count math lives. H3 nests one parent to seven children (aperture 7), so each resolution step multiplies partition count by roughly seven and cell area shrinks by the same factor. S2 and quadkey both quarter — one parent to four children — so each level multiplies partitions by four. The practical consequence: to move from “too few, oversized partitions” to “right-sized,” H3 gives you coarser control steps (×7) while S2 and quadkey let you tune in finer ×4 increments. When you are targeting a specific partition byte-size to satisfy a cold tier’s minimum-object-size rule, that granularity matters, and it interacts directly with your row-group sizing strategy.

Neighbor queries. Quadkey wins on raw simplicity: a tile’s parent is its string with the last character removed, and adjacency is cheap integer arithmetic on the X/Y at a zoom. H3’s k-ring (or grid_disk) returns all cells within k steps in one call with uniform distance semantics — ideal for “buffer this cell” retrieval. S2 exposes edge neighbors and, because cells lie on a Hilbert curve, contiguous ranges of S2 cell IDs map to contiguous regions, which is excellent for range-scan pruning but less intuitive for ring buffers.

Library support and file-count risk. All three have production libraries (h3, s2sphere/s2geometry, mercantile). The file-count risk column is the one that silently wrecks archives: quadkey at deep zoom explodes into millions of tiny tiles, each an object with its own PUT/GET/lifecycle-transition cost. Choose a resolution that keeps partition files comfortably above the tier’s minimum object size described in Hot/Warm/Cold Tier Design for Geospatial Data, and model the request-count consequences against the spatial archive cost model before committing.

How the Three Schemes Tile the Same Extent

The clearest way to see the differences is to lay the three grids over one extent. H3 tiles with hexagons of near-equal area whose neighbours are all edge-adjacent and equidistant; S2 tiles with quadrilaterals projected from a cube face, keeping cells close to equal area but with corner neighbours; quadkey tiles with the familiar Web Mercator squares, which are simple to compute and increasingly distorted away from the equator.

H3, S2 and quadkey grids over one extent Three side-by-side panels tiling the same map extent. The first uses interlocking hexagons of near-equal area. The second uses slightly irregular quadrilaterals from a cube-face projection. The third uses Web Mercator squares of equal pixel size but unequal ground area, with the northern row visibly covering less ground. H3 6 equidistant neighbours near-equal area worldwide no clean parent nesting S2 4 edge + 4 corner neighbours cube-face projection, low distortion exact parent nesting by bit shift Quadkey 4 edge + 4 corner neighbours ground area shrinks with latitude trivial parent nesting, string prefix rows: equal pixels, unequal ground

The tiling picture explains the operational differences that follow. Equal-area cells make partition sizes predictable, which is what an archive wants when object size drives cost. Clean parent nesting makes it possible to roll a fine partition up to a coarser one without recomputing keys, which is what an archive wants when the retention plan compacts old data. Only S2 offers both; H3 trades nesting for equal area and uniform neighbour distance, and quadkey trades equal area for the simplest possible key arithmetic.

Generating Partition Keys in All Three Schemes

Compute the key the same way regardless of scheme: derive a representative point (centroid for polygons, the point itself for points) in EPSG:4326, then index it. The following resolves one partition key per feature for each grid so you can compare partition cardinality before you commit.

import geopandas as gpd
import h3
import s2sphere
import mercantile

# Read a projected source; H3/S2/quadkey all index geographic coordinates.
gdf = gpd.read_file("s3://spatial-archive/vector/2024/parcels_region_north.fgb")
pts = gdf.geometry.to_crs(4326).representative_point()

H3_RES, S2_LEVEL, QK_ZOOM = 6, 10, 10  # coarse-to-comparable partition granularity

def keys(pt):
    lat, lng = pt.y, pt.x
    h3_key = h3.latlng_to_cell(lat, lng, H3_RES)
    s2_cell = s2sphere.CellId.from_lat_lng(
        s2sphere.LatLng.from_degrees(lat, lng)
    ).parent(S2_LEVEL)
    s2_key = s2_cell.to_token()              # compact hex token, e.g. "89c2594"
    qk_key = mercantile.quadkey(             # base-4 string, e.g. "0231010121"
        mercantile.tile(lng, lat, QK_ZOOM)
    )
    return h3_key, s2_key, qk_key

gdf[["h3_cell", "s2_token", "quadkey"]] = [keys(p) for p in pts]

# Partition cardinality is the number of directories each scheme would create.
for col in ("h3_cell", "s2_token", "quadkey"):
    print(f"{col:10s} -> {gdf[col].nunique():>8d} partitions")

Write the winning key into the Hive-style path exactly as the migration pipeline does — s3://spatial-archive/vector/h3_cell=8a2a1072b59ffff/part-0000.parquet — so the partition column is prunable at query time. The mechanics of that partitioned write are covered in Converting Legacy Shapefiles to GeoParquet at Scale.

Validating Partition Balance Before You Commit

The failure mode is a heavy-tailed partition distribution: a handful of cells over dense metros hold most of the features while thousands of rural cells hold a few rows each. Measure the skew directly with DuckDB against the keyed output before you write the final tree.

duckdb -c "
SELECT h3_cell,
       count(*)                    AS features,
       count(*) FILTER (WHERE true) * 1.0 /
         (SELECT avg(c) FROM (SELECT count(*) c FROM read_parquet('keyed/*.parquet') GROUP BY h3_cell)) AS skew_ratio
FROM read_parquet('keyed/*.parquet')
GROUP BY h3_cell
ORDER BY features DESC
LIMIT 5"

Expected output — the top partitions should sit within roughly an order of magnitude of the mean; a skew_ratio above ~50 means the resolution is too coarse for your dense regions and those cells will dominate scan time:

┌─────────────────┬──────────┬────────────┐
│     h3_cell     │ features │ skew_ratio │
│     varchar     │  int64   │   double   │
├─────────────────┼──────────┼────────────┤
│ 8a2a1072b59ffff │   184213 │      7.4   │
│ 8a2a1072b5b7fff │   151902 │      6.1   │
│ 8a1fb46622dffff │   142338 │      5.7   │
│ 8a2a10725d37fff │   128004 │      5.1   │
│ 8a1fb46622c7fff │   119887 │      4.8   │
└─────────────────┴──────────┴────────────┘

If the skew is unacceptable, either step to a finer resolution for the whole dataset or split only the hot cells to a deeper resolution — a mixed-resolution layout H3 and S2 both support natively because a parent cell ID unambiguously contains its children.

Key Length, Storage Cost, and Join Behaviour

The key is not free: it is a column in every row of the archive, it appears in every directory path, and it is the join column for any catalogue that indexes the partitions. Its representation therefore shows up three times in the bill and once in every query plan.

Partition key representation and its cost per row Three key schemes compared by stored width per row in string and integer form, and by the total column footprint across 480 million rows before compression. Cost of the key column itself · 480M-row archive, uncompressed H3 · hex string 15 B/row → 7.2 GB H3 · uint64 8 B/row → 3.8 GB S2 · token 16 B/row → 7.7 GB S2 · cell id uint64 8 B/row → 3.8 GB Quadkey · level 12 12 B/row → 5.4 GB · prefix-searchable Dictionary encoding recovers most of the string overhead within a partition, where the key is constant — but not in a flat catalogue table, where it varies per row.

Store the key as an unsigned 64-bit integer in the data files wherever the scheme supports it, and keep the human-readable form only in the directory path and the catalogue. The integer form halves the column width, compares in one instruction during joins, and — for S2 and quadkey — preserves the prefix relationship that makes a coarser roll-up a bit-shift rather than a lookup. Quadkey is the exception where the string is worth keeping, because prefix matching on the string is exactly how its hierarchy is queried.

Common Partition-Key Failures

Symptom Cause Fix
A few partitions 100× larger than the median Resolution too coarse for dense metros; uniform grid can’t adapt to concentrated features Step one resolution finer, or split only hot cells to a deeper H3/S2 child level
Millions of sub-megabyte partition files, high LIST/GET bill Quadkey zoom (or H3/S2 level) too deep; each cell became its own object Coarsen the level; target partition files above the tier minimum object size
Bounding-box query scans the whole tree Partition column not written into the path, so the planner can’t prune Emit scheme_cell=VALUE as a real Hive path segment, not just a table column
Cells straddling the antimeridian or poles behave oddly Quadkey undefined beyond ±85.05° Mercator latitude; polar data lost Use H3 or S2, which cover the full sphere including the poles

Operational Execution Checklist

Frequently Asked Questions

Does the choice of grid affect compression ratio?

Indirectly but measurably. Partitioning by any spatial scheme groups nearby features together, and nearby features share coordinate prefixes, which is what makes the geometry column compressible. The difference between the three schemes is small — typically under three percent — compared with the difference between partitioned and unpartitioned writes, which can exceed thirty. Choose the grid for pruning and object-size behaviour, and take the compression benefit as a side effect of spatial ordering rather than a deciding factor.

Can two schemes be used together?

Yes, and it is a common pattern: a coarse administrative or quadkey directory level for human navigation and audit scoping, with a finer H3 or S2 key stored as a column for pruning within it. The cost is a second key to maintain and validate. Avoid the reverse arrangement — a fine directory hierarchy with a coarse column key — which produces many small objects and no useful pruning inside them.

What happens at the poles and the antimeridian?

All three schemes handle the antimeridian without special cases at the key level, but query extents that cross it must be split into two ranges by the caller, because a bounding box expressed in longitude cannot wrap. The poles are where the schemes diverge: quadkey in Web Mercator does not cover latitudes beyond about 85 degrees at all, S2 covers the poles with cells of ordinary size, and H3 covers them with twelve pentagonal cells whose neighbour distances are irregular. Polar archives should not use quadkey.

Part of the Spatial Data Archival knowledge base.

Related pages

Partitioning Cold Spatial Archives by Administrative Boundary Sizing Partitions to Avoid Small-File Overhead in Cold Storage