Spatial Data Archival · Cold Storage

Building a FlatGeobuf Spatial Index for HTTP Range Reads

Building a FlatGeobuf spatial index converts a flat .fgb file into a cloud-native asset a client can query with HTTP range requests, reading only the byte ranges its bounding box intersects instead of downloading the whole file from cold object storage. The index is a packed Hilbert R-tree written between the file header and the feature payload, and it is the single structural feature that separates an archival .fgb from an inert blob. This guide is for data engineers and GIS archivists who write .fgb artifacts into warm and cold tiers and need every one of them to stay range-readable for years — it details the on-disk anatomy of the index, the exact ogr2ogr/GDAL invocations that produce a correctly packed R-tree, and the verification steps that prove the index header is present before the object is promoted. It sits inside the broader Format Conversion & Pipeline Automation workflow and applies the tuning patterns from FlatGeobuf Optimization Techniques.

Anatomy of an Indexed FlatGeobuf File

A FlatGeobuf file is four contiguous byte segments, and a range-reading client walks them in order without ever holding the full object in memory:

Indexed FlatGeobuf file layout and range-read sequence Four contiguous segments — magic marker, header with schema and CRS and bounding box, packed Hilbert R-tree index, and feature data payload. Three numbered HTTP range reads fetch the header, then the R-tree, then only the feature byte ranges the query bounding box intersects, never the whole file. byte offset 0 → end of file Magic 8 B Header schema · CRS · bbox Packed Hilbert R-tree node ranges · feature offsets Feature data geometry + attributes matched ① fetch header ② fetch R-tree ③ fetch matches A cold read touches the header and index, then only the feature byte ranges the bounding box intersects — never the whole file.

The first eight bytes are a fixed magic marker that identifies the format and its version. Immediately after comes the header: a FlatBuffers-encoded table carrying the layer name, the attribute column definitions, the geometry type, the coordinate reference system (CRS) as WKT, and the dataset’s total bounding box and feature count. Because the header is small and lives at a known offset, a client resolves the entire schema in one short request without touching a single feature.

The third segment is the packed Hilbert R-tree, and it is what this guide exists to produce. Rather than an incrementally balanced tree, FlatGeobuf sorts every feature along a Hilbert space-filling curve — which keeps spatially adjacent features adjacent on disk — then packs a static, bottom-up R-tree over that order. Each node stores a bounding box and a byte offset into the feature payload. The tree is written breadth-first with a fixed node size, so its total length is a deterministic function of the feature count, and a client can compute exactly where the index ends and the payload begins. Traversing the tree with a query rectangle yields a small set of byte ranges; those ranges are then fetched directly from the feature data segment. This is the mechanism behind cold-storage byte-range reads: the index turns “find features intersecting this box” into “read these three offset ranges”, and the practical client-side workflow for issuing those reads is covered in the sibling guide on streaming FlatGeobuf features over HTTP range requests.

Without the index, none of this holds. An unindexed .fgb forces a client to stream every feature from offset zero to find the ones it wants — on a multi-gigabyte parcel layer parked in a warm tier, that is the difference between a 30 KB read and a full-object egress charge. The index is therefore not an optimization to add later; it is the archival contract you write at serialization time.

Writing an Indexed FlatGeobuf

The GDAL FlatGeobuf driver builds the packed Hilbert R-tree when the SPATIAL_INDEX layer-creation option is enabled. The steps below take a source dataset through a clean, index-bearing export.

  1. Lock the CRS and prune the schema first. The index is built over the geometries as they will be stored, so any reprojection must happen before packing. Force an explicit source and target CRS and keep only the attributes you archive; detailed projection-registry handling belongs to CRS synchronization in pipelines.
# Reproject and prune WITHOUT an index first, so the packing step
# in phase 2 sorts the final geometries. OGR SQL has no ST_Transform,
# so -t_srs does the reprojection, not the -sql clause.
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/region_north_normalized.fgb \
  datasets/parcels/raw/region_north.gpkg \
  -s_srs EPSG:2913 -t_srs EPSG:4326 \
  -lco SPATIAL_INDEX=NO \
  -sql "SELECT parcel_id, zoning, assessed_value FROM parcels"
  1. Pack the Hilbert R-tree. Re-serialize the normalized file with the index enabled. The driver sorts features along the Hilbert curve and writes the static tree between the header and the payload; node size and tree depth are managed automatically.
# Build the packed Hilbert R-tree spatial index.
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/region_north_indexed.fgb \
  datasets/parcels/staging/region_north_normalized.fgb \
  -lco SPATIAL_INDEX=YES
  1. Size the source for in-memory packing. The driver sorts feature bounding boxes in memory during packing. For datasets past roughly 10 GB, raise the GDAL cache so the sort does not spill, and run the export on a worker with headroom rather than inside a shared conversion pool.
export GDAL_CACHEMAX=4096          # MB of block cache for the sort
export OGR_ORGANIZE_POLYGONS=SKIP  # skip ring-orientation cost on clean data
ogr2ogr -f "FlatGeobuf" \
  s3://spatial-archive/fgb/parcels/2024/region_north_indexed.fgb \
  datasets/parcels/staging/region_north_normalized.fgb \
  -lco SPATIAL_INDEX=YES \
  --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES

The /vsis3/-style write path lets GDAL stream the indexed output straight to object storage, but the index is still assembled locally first — the temp-file config keeps the random writes off the network path until the object is complete.

Verifying the Index Header

A successful write is not proof of a usable index. Confirm the header reports a spatial index and that the feature count survived, then inspect the raw bytes to prove the index segment is physically present.

# 1. Driver-level confirmation of index + geometry metadata.
ogrinfo -so datasets/parcels/staging/region_north_indexed.fgb

Expected output — the CRS must be the resolved EPSG code, the feature count must match the source, and the extent must be non-empty:

INFO: Open of 'region_north_indexed.fgb' using driver 'FlatGeobuf' successful.

Layer name: region_north_indexed
Geometry: Polygon
Feature Count: 482817
Extent: (-123.482, 45.230) - (-121.905, 46.388)
Layer SRS WKT: GEOGCRS["WGS 84", ... ID["EPSG",4326]]

FlatGeobuf does not print an “index present” flag through ogrinfo, so confirm the index physically by checking that the header’s index_node_size is non-zero. Read it directly from the FlatBuffers header:

# Confirm the packed R-tree exists by reading index_node_size from the header.
# A value of 0 means SPATIAL_INDEX=NO was used and no tree was written.
from flatgeobuf import HeaderReader  # pip install flatgeobuf

with open("datasets/parcels/staging/region_north_indexed.fgb", "rb") as f:
    header = HeaderReader.read(f)

assert header.index_node_size > 0, "no spatial index — file is not range-readable"
assert header.features_count == 482817, "feature count drift"
assert "4326" in (header.crs.code_string or str(header.crs.code))
print(f"index_node_size={header.index_node_size} features={header.features_count}")
# index_node_size=16 features=482817

An index_node_size of 0 means the file was written with SPATIAL_INDEX=NO; the file is valid but not range-readable, and every client will fall back to a full scan. Treat this check as a hard gate before any object is promoted into a tier that bills for egress.

Troubleshooting

Symptom Root cause Fix
ogrinfo reports the correct feature count but clients still download the whole file File written with SPATIAL_INDEX=NO; index_node_size is 0 and no R-tree exists Re-run the export with -lco SPATIAL_INDEX=YES and confirm index_node_size > 0 before upload
Cannot allocate memory or a spill to disk during the index build Hilbert sort of feature bounding boxes exceeds the GDAL cache on a large layer Raise GDAL_CACHEMAX (4096+), split the layer by region, and index each partition independently
Range reads return features far outside the query box Index was built before the final reprojection, so node bounding boxes are in the wrong CRS Rebuild: normalize and reproject in phase 1, then pack the index over the reprojected geometries in phase 2

Consult the GDAL FlatGeobuf driver documentation for the exact creation-option matrix and the FlatGeobuf specification for the header and packed R-tree binary encoding.

Operational Execution Checklist

Part of the Spatial Data Archival knowledge base.

Related pages

Converting GeoPackage to FlatGeobuf for Web Archives Optimizing FlatGeobuf for Web Mapping Archives: Pipeline Configuration and Cold Storage Validation Streaming FlatGeobuf Features Over HTTP Range Requests