Spatial Data Archival · Cold Storage

Streaming FlatGeobuf Features Over HTTP Range Requests

Streaming FlatGeobuf features over HTTP range requests lets a client read only the features inside a query bounding box directly from an .fgb object on S3 or Azure Blob, issuing a short sequence of 206 Partial Content reads against the packed spatial index instead of downloading the entire archive. This guide is for data engineers and application teams serving interactive maps or extract jobs from warm-tier object storage who need predictable latency and predictable egress as archives grow into the tens of gigabytes. It explains how a bounding-box query becomes a set of byte ranges, the exact requests to issue against cloud stores, the CDN behaviors that silently break range support, and the latency crossover where a full download beats streaming. It builds directly on the packed Hilbert R-tree produced in building a FlatGeobuf spatial index for HTTP range reads and sits within the broader Format Conversion & Pipeline Automation workflow.

The Range-Request Round Trip

A bounding-box read is not one request; it is a short, ordered conversation between the client and the object store:

FlatGeobuf range-request round trip Three ordered request-and-response pairs between a map client and an object store behind a CDN: fetch the header and index root, fetch the R-tree node ranges intersecting the bounding box, then fetch only the matched feature byte ranges. Each response is 206 Partial Content. Map client Object store via CDN edge ① Range: bytes=0–65535 header + index root ① 206 Partial Content schema · CRS · root node ② Range: bytes=… traverse R-tree for bbox ② 206 Partial Content matched leaf offsets ③ Range: bytes=… matched feature ranges ③ 206 Partial Content features ready to render Bytes fetched scale with the bounding box, not the file size — a small window against a 40 GB layer is still a few KB.

The first read pulls the header and the top of the index in one shot, resolving the schema, CRS, and the R-tree’s root node. The client then walks the tree: it descends only the nodes whose bounding boxes intersect the query rectangle, issuing follow-up range reads for deeper node blocks as needed. Traversal terminates at a set of leaf offsets, each pointing at a contiguous run of features in the payload. The final reads fetch exactly those feature byte ranges. Because Hilbert ordering keeps spatially adjacent features adjacent on disk, the matched features usually coalesce into a handful of contiguous ranges rather than hundreds of scattered ones — which is what keeps the request count, and therefore the latency, bounded. The write-side guarantee that makes this hold is covered under FlatGeobuf Optimization Techniques.

Each read is a network round trip, so the total latency of a query is roughly the number of dependent range requests multiplied by the store’s time-to-first-byte. Against a warm tier that is typically tens of milliseconds per request, and a tight bounding box resolves in three to five reads. That is the entire economic argument: you pay for a few kilobytes of transfer and a few round trips instead of egressing a whole object, which is why FlatGeobuf owns the range-read web-delivery tier while analytical extracts belong to the columnar GeoParquet Migration Workflows path.

Issuing Range Requests Against Object Storage

The steps below take a bounding-box query from a raw HTTP probe to a working GDAL client that reads only the intersecting features.

  1. Confirm the store and its edge honor byte ranges. Object stores support ranges natively, but a fronting CDN may not. Probe with an explicit range and inspect the status line — you need 206, not 200.
# A GET (not HEAD) reveals whether ranges are actually served end to end.
curl -s -r 0-65535 -o /dev/null -D - \
  https://spatial-archive.s3.us-west-2.amazonaws.com/fgb/parcels/2024/region_north_indexed.fgb
# Expected:
#   HTTP/1.1 206 Partial Content
#   Accept-Ranges: bytes
#   Content-Range: bytes 0-65535/1974330112
#   Content-Length: 65536
  1. Run the bounding-box query through GDAL’s range-reading virtual filesystem. GDAL reads .fgb over /vsis3/ (or /vsicurl/ for a public URL) using the spatial index and HTTP ranges automatically; pyogrio exposes this with a bbox filter. Only the intersecting features are materialized.
import pyogrio

# bbox is (minx, miny, maxx, maxy) in the file's CRS (EPSG:4326 here).
# GDAL walks the R-tree and issues range reads under the hood — the whole
# 1.9 GB object is never downloaded.
gdf = pyogrio.read_dataframe(
    "/vsis3/spatial-archive/fgb/parcels/2024/region_north_indexed.fgb",
    bbox=(-122.72, 45.44, -122.55, 45.60),  # Portland west-side window
    columns=["parcel_id", "zoning", "assessed_value"],
)
print(len(gdf), "features fetched inside the window")
  1. Account for the bytes actually transferred. Enable GDAL’s cURL tracing to see each range request and prove the read footprint matches the bounding box rather than the file size.
# CPL_CURL_VERBOSE logs every Range request GDAL issues for one query.
export CPL_CURL_VERBOSE=YES
export CPL_DEBUG=ON
export AWS_REGION=us-west-2
python query_bbox.py 2>&1 | grep -E "Range: bytes|Content-Range" | head
# Each line is one 206 read; sum the Content-Length values for total bytes moved.

Validation & Verification

Prove three things before trusting a range-streaming endpoint in production: the store returns 206, the query returns only in-window features, and the transferred bytes are a small fraction of the object.

# 1. Range support and object size in one probe.
curl -s -r 0-1023 -o /dev/null -D - \
  https://spatial-archive.s3.us-west-2.amazonaws.com/fgb/parcels/2024/region_north_indexed.fgb \
  | grep -E "206|Content-Range"
# Expected: HTTP/1.1 206 Partial Content
#           Content-Range: bytes 0-1023/1974330112
# 2. Spatial correctness + byte-footprint sanity check.
import pyogrio, shapely.geometry as sg

bbox = (-122.72, 45.44, -122.55, 45.60)
gdf = pyogrio.read_dataframe(
    "/vsis3/spatial-archive/fgb/parcels/2024/region_north_indexed.fgb", bbox=bbox
)
window = sg.box(*bbox)
# Every returned geometry must intersect the query window (the index is exact
# at the leaf, so no far-away features should appear).
assert gdf.geometry.intersects(window).all(), "index returned out-of-window features"
print(f"{len(gdf)} features; expected a few thousand, not all 482,817")

Expected result: a query over a neighborhood-scale window against a ~1.9 GB layer returns a few thousand features and moves on the order of a few hundred kilobytes — three to four orders of magnitude less than a full download. If the returned count approaches the full feature count, the index was bypassed and the client fell back to a scan.

CDN and Range-Support Caveats

Range streaming lives or dies on the transport layer between the client and the object bytes. The failure modes are specific and quiet.

Symptom Root cause Fix
First probe returns 200 OK with the full Content-Length, and every query downloads the whole file A fronting CDN or proxy strips the Range header or is configured to collapse ranges into a full-object fetch Enable range/origin passthrough and range-based caching at the edge; verify Accept-Ranges: bytes survives to the client
HTTP 416 Range Not Satisfiable on a valid offset The object was replaced with a shorter version but a stale edge cache still advertises the old length Purge the edge cache on republish and key the cache on the object’s ETag, not just the path
Query latency spikes to seconds and the request log shows dozens of tiny reads Attribute-heavy features fragment the payload, or the file predates the index and clients scan linearly Prune attributes at write time and rebuild with the packed index; confirm contiguous matched ranges in the cURL trace

There is a real crossover point. Each range read is a dependent round trip, so a query that must return most of the file — a country-wide extract, or a layer small enough that the whole object fits in one read — is faster and cheaper as a single full GET than as many ranged reads, because it avoids per-request overhead and index-traversal latency. Use range streaming for selective, bounding-box-scoped reads against large objects in warm and cold tiers; fall back to a full download when the selection ratio is high. The request-count and egress economics that decide this crossover for a given archive are modeled in the spatial archive cost model, and the store’s per-request pricing depends on the object storage selection for GIS archives you standardized on.

For the authoritative range semantics on each platform, see the AWS S3 GetObject Range documentation and the Azure Blob range GET documentation, and confirm your client against the FlatGeobuf specification for HTTP reader behavior.

Operational Execution Checklist

Part of the Spatial Data Archival knowledge base.

Related pages

Building a FlatGeobuf Spatial Index for HTTP Range Reads Converting GeoPackage to FlatGeobuf for Web Archives Optimizing FlatGeobuf for Web Mapping Archives: Pipeline Configuration and Cold Storage Validation