Spatial Data Archival · Cold Storage

Building a FlatGeobuf Spatial Index for HTTP Range Reads

Building a FlatGeobuf spatial index converts a flat .fgb file into a cloud-native asset a client can query with HTTP range requests, reading only the byte ranges its bounding box intersects instead of downloading the whole file from cold object storage. The index is a packed Hilbert R-tree written between the file header and the feature payload, and it is the single structural feature that separates an archival .fgb from an inert blob. This guide is for data engineers and GIS archivists who write .fgb artifacts into warm and cold tiers and need every one of them to stay range-readable for years — it details the on-disk anatomy of the index, the exact ogr2ogr/GDAL invocations that produce a correctly packed R-tree, and the verification steps that prove the index header is present before the object is promoted. It sits inside the broader Format Conversion & Pipeline Automation workflow and applies the tuning patterns from FlatGeobuf Optimization Techniques.

Anatomy of an Indexed FlatGeobuf File

A FlatGeobuf file is four contiguous byte segments, and a range-reading client walks them in order without ever holding the full object in memory:

Indexed FlatGeobuf file layout and range-read sequence Four contiguous segments — magic marker, header with schema and CRS and bounding box, packed Hilbert R-tree index, and feature data payload. Three numbered HTTP range reads fetch the header, then the R-tree, then only the feature byte ranges the query bounding box intersects, never the whole file. byte offset 0 → end of file Magic 8 B Header schema · CRS · bbox Packed Hilbert R-tree node ranges · feature offsets Feature data geometry + attributes matched ① fetch header ② fetch R-tree ③ fetch matches A cold read touches the header and index, then only the feature byte ranges the bounding box intersects — never the whole file.

The first eight bytes are a fixed magic marker that identifies the format and its version. Immediately after comes the header: a FlatBuffers-encoded table carrying the layer name, the attribute column definitions, the geometry type, the coordinate reference system (CRS) as WKT, and the dataset’s total bounding box and feature count. Because the header is small and lives at a known offset, a client resolves the entire schema in one short request without touching a single feature.

The third segment is the packed Hilbert R-tree, and it is what this guide exists to produce. Rather than an incrementally balanced tree, FlatGeobuf sorts every feature along a Hilbert space-filling curve — which keeps spatially adjacent features adjacent on disk — then packs a static, bottom-up R-tree over that order. Each node stores a bounding box and a byte offset into the feature payload. The tree is written breadth-first with a fixed node size, so its total length is a deterministic function of the feature count, and a client can compute exactly where the index ends and the payload begins. Traversing the tree with a query rectangle yields a small set of byte ranges; those ranges are then fetched directly from the feature data segment. This is the mechanism behind cold-storage byte-range reads: the index turns “find features intersecting this box” into “read these three offset ranges”, and the practical client-side workflow for issuing those reads is covered in the sibling guide on streaming FlatGeobuf features over HTTP range requests.

Without the index, none of this holds. An unindexed .fgb forces a client to stream every feature from offset zero to find the ones it wants — on a multi-gigabyte parcel layer parked in a warm tier, that is the difference between a 30 KB read and a full-object egress charge. The index is therefore not an optimization to add later; it is the archival contract you write at serialization time.

Writing an Indexed FlatGeobuf

The GDAL FlatGeobuf driver builds the packed Hilbert R-tree when the SPATIAL_INDEX layer-creation option is enabled. The steps below take a source dataset through a clean, index-bearing export.

  1. Lock the CRS and prune the schema first. The index is built over the geometries as they will be stored, so any reprojection must happen before packing. Force an explicit source and target CRS and keep only the attributes you archive; detailed projection-registry handling belongs to CRS synchronization in pipelines.
# Reproject and prune WITHOUT an index first, so the packing step
# in phase 2 sorts the final geometries. OGR SQL has no ST_Transform,
# so -t_srs does the reprojection, not the -sql clause.
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/region_north_normalized.fgb \
  datasets/parcels/raw/region_north.gpkg \
  -s_srs EPSG:2913 -t_srs EPSG:4326 \
  -lco SPATIAL_INDEX=NO \
  -sql "SELECT parcel_id, zoning, assessed_value FROM parcels"
  1. Pack the Hilbert R-tree. Re-serialize the normalized file with the index enabled. The driver sorts features along the Hilbert curve and writes the static tree between the header and the payload; node size and tree depth are managed automatically.
# Build the packed Hilbert R-tree spatial index.
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/region_north_indexed.fgb \
  datasets/parcels/staging/region_north_normalized.fgb \
  -lco SPATIAL_INDEX=YES
  1. Size the source for in-memory packing. The driver sorts feature bounding boxes in memory during packing. For datasets past roughly 10 GB, raise the GDAL cache so the sort does not spill, and run the export on a worker with headroom rather than inside a shared conversion pool.
export GDAL_CACHEMAX=4096          # MB of block cache for the sort
export OGR_ORGANIZE_POLYGONS=SKIP  # skip ring-orientation cost on clean data
ogr2ogr -f "FlatGeobuf" \
  s3://spatial-archive/fgb/parcels/2024/region_north_indexed.fgb \
  datasets/parcels/staging/region_north_normalized.fgb \
  -lco SPATIAL_INDEX=YES \
  --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES

The /vsis3/-style write path lets GDAL stream the indexed output straight to object storage, but the index is still assembled locally first — the temp-file config keeps the random writes off the network path until the object is complete.

How the Packed R-Tree Is Laid Out

The index is a packed Hilbert R-tree serialised breadth-first, which is why a reader can compute the byte offset of any node without following a pointer. Nodes at each level are contiguous and fixed-size, so the reader derives the position of a level from the node count alone — and fetches an entire level in one ranged request.

Breadth-first packed R-tree layout in a FlatGeobuf file Four levels of a packed R-tree laid out contiguously: a single root node, sixteen nodes at level one, 256 at level two, and 4,096 leaf entries. Byte offsets are computed from node counts rather than followed through pointers, so each level is one ranged read. Index section · node size 40 B · branching factor 16 root 1 node · bytes 0–39 level 1 16 nodes · bytes 40–679 level 2 256 nodes · bytes 680–10,919 leaves 4,096 entries · each holds a bbox and the byte offset of its feature A query descends one node per level: 4 ranged reads to reach the feature offsets, regardless of how many features the file holds.

Two authoring consequences follow. The branching factor is fixed at write time and recorded in the header, so a reader that assumes a different value computes wrong offsets and returns wrong features — which is why the header check matters more here than in formats where the index is self-describing. And because the leaf entries carry byte offsets into the feature section, any post-write modification of that section invalidates the whole index silently: the file still parses, and the offsets point at the wrong records.

Verifying the Index Header

A successful write is not proof of a usable index. Confirm the header reports a spatial index and that the feature count survived, then inspect the raw bytes to prove the index segment is physically present.

# 1. Driver-level confirmation of index + geometry metadata.
ogrinfo -so datasets/parcels/staging/region_north_indexed.fgb

Expected output — the CRS must be the resolved EPSG code, the feature count must match the source, and the extent must be non-empty:

INFO: Open of 'region_north_indexed.fgb' using driver 'FlatGeobuf' successful.

Layer name: region_north_indexed
Geometry: Polygon
Feature Count: 482817
Extent: (-123.482, 45.230) - (-121.905, 46.388)
Layer SRS WKT: GEOGCRS["WGS 84", ... ID["EPSG",4326]]

FlatGeobuf does not print an “index present” flag through ogrinfo, so confirm the index physically by checking that the header’s index_node_size is non-zero. Read it directly from the FlatBuffers header:

# Confirm the packed R-tree exists by reading index_node_size from the header.
# A value of 0 means SPATIAL_INDEX=NO was used and no tree was written.
from flatgeobuf import HeaderReader  # pip install flatgeobuf

with open("datasets/parcels/staging/region_north_indexed.fgb", "rb") as f:
    header = HeaderReader.read(f)

assert header.index_node_size > 0, "no spatial index — file is not range-readable"
assert header.features_count == 482817, "feature count drift"
assert "4326" in (header.crs.code_string or str(header.crs.code))
print(f"index_node_size={header.index_node_size} features={header.features_count}")
# index_node_size=16 features=482817

An index_node_size of 0 means the file was written with SPATIAL_INDEX=NO; the file is valid but not range-readable, and every client will fall back to a full scan. Treat this check as a hard gate before any object is promoted into a tier that bills for egress.

Troubleshooting

Symptom Root cause Fix
ogrinfo reports the correct feature count but clients still download the whole file File written with SPATIAL_INDEX=NO; index_node_size is 0 and no R-tree exists Re-run the export with -lco SPATIAL_INDEX=YES and confirm index_node_size > 0 before upload
Cannot allocate memory or a spill to disk during the index build Hilbert sort of feature bounding boxes exceeds the GDAL cache on a large layer Raise GDAL_CACHEMAX (4096+), split the layer by region, and index each partition independently
Range reads return features far outside the query box Index was built before the final reprojection, so node bounding boxes are in the wrong CRS Rebuild: normalize and reproject in phase 1, then pack the index over the reprojected geometries in phase 2

Consult the GDAL FlatGeobuf driver documentation for the exact creation-option matrix and the FlatGeobuf specification for the header and packed R-tree binary encoding.

Why Write Order Decides Query Cost

The index tells a reader which features to fetch; the write order decides whether those features sit next to each other. Two files can carry identical, correct indexes and differ by two orders of magnitude in requests per query, purely because one was written in Hilbert order and the other in insertion order.

Matching-feature placement under insertion order and Hilbert order Two strips representing the feature section of a file. Under insertion order the features matching one bounding-box query are scattered as many thin marks across the whole strip. Under Hilbert order the same features form three contiguous blocks. Same query, same index, same 4,096 features — different write order insertion order 214 separate ranges · 214 requests · 38 MB fetched to return 1.9 MB of features Hilbert order 3 contiguous ranges · 3 requests · 2.1 MB fetched — the index did the same work in both cases

This is the reason the writing procedure sorts before it indexes rather than relying on the index alone. It is also why appending to a sorted file is worse than it looks: the appended features are correct and indexed, but they sit outside the spatial ordering, so every query whose extent covers them gains an extra range. A file that has been appended to a dozen times degrades toward the insertion-order case one generation at a time.

Operational Execution Checklist

Frequently Asked Questions

Can the index be rebuilt without rewriting the features?

Only if the feature section is left byte-identical, which in practice means rebuilding the index from the existing features and rewriting the file with the new index in front of them. Since the index precedes the features and the leaf entries store absolute offsets, changing the index size shifts every feature — so the write is a full file rewrite regardless. Plan on regenerating the file rather than patching it.

What branching factor should be used?

The default of 16 suits object storage well, and there is little reason to move from it. A larger factor produces fewer levels and therefore fewer round trips, at the cost of a bigger node fetch at each level; a smaller factor does the reverse. Because every level after the first is a single ranged read of a few tens of kilobytes, the round-trip count dominates, and the default already puts a multi-million-feature file within four or five levels.

Does the index help attribute queries at all?

No. The R-tree is purely spatial: it resolves bounding-box intersection and nothing else. An attribute filter is applied by the reader after the features are fetched, so a query that is highly selective on attributes but broad in space still transfers everything in that extent. Where attribute selectivity matters, a columnar archive copy with statistics on those columns is the right tool, and the format comparison covers when to keep both.

Part of the Spatial Data Archival knowledge base.

Related pages

Converting GeoPackage to FlatGeobuf for Web Archives Optimizing FlatGeobuf for Web Mapping Archives: Pipeline Configuration and Cold Storage Validation Streaming FlatGeobuf Features Over HTTP Range Requests