Spatial Data Archival · Cold Storage

Streaming FlatGeobuf Features Over HTTP Range Requests

Streaming FlatGeobuf features over HTTP range requests lets a client read only the features inside a query bounding box directly from an .fgb object on S3 or Azure Blob, issuing a short sequence of 206 Partial Content reads against the packed spatial index instead of downloading the entire archive. This guide is for data engineers and application teams serving interactive maps or extract jobs from warm-tier object storage who need predictable latency and predictable egress as archives grow into the tens of gigabytes. It explains how a bounding-box query becomes a set of byte ranges, the exact requests to issue against cloud stores, the CDN behaviors that silently break range support, and the latency crossover where a full download beats streaming. It builds directly on the packed Hilbert R-tree produced in building a FlatGeobuf spatial index for HTTP range reads and sits within the broader Format Conversion & Pipeline Automation workflow.

The Range-Request Round Trip

A bounding-box read is not one request; it is a short, ordered conversation between the client and the object store:

FlatGeobuf range-request round trip Three ordered request-and-response pairs between a map client and an object store behind a CDN: fetch the header and index root, fetch the R-tree node ranges intersecting the bounding box, then fetch only the matched feature byte ranges. Each response is 206 Partial Content. Map client Object store via CDN edge ① Range: bytes=0–65535 header + index root ① 206 Partial Content schema · CRS · root node ② Range: bytes=… traverse R-tree for bbox ② 206 Partial Content matched leaf offsets ③ Range: bytes=… matched feature ranges ③ 206 Partial Content features ready to render Bytes fetched scale with the bounding box, not the file size — a small window against a 40 GB layer is still a few KB.

The first read pulls the header and the top of the index in one shot, resolving the schema, CRS, and the R-tree’s root node. The client then walks the tree: it descends only the nodes whose bounding boxes intersect the query rectangle, issuing follow-up range reads for deeper node blocks as needed. Traversal terminates at a set of leaf offsets, each pointing at a contiguous run of features in the payload. The final reads fetch exactly those feature byte ranges. Because Hilbert ordering keeps spatially adjacent features adjacent on disk, the matched features usually coalesce into a handful of contiguous ranges rather than hundreds of scattered ones — which is what keeps the request count, and therefore the latency, bounded. The write-side guarantee that makes this hold is covered under FlatGeobuf Optimization Techniques.

Each read is a network round trip, so the total latency of a query is roughly the number of dependent range requests multiplied by the store’s time-to-first-byte. Against a warm tier that is typically tens of milliseconds per request, and a tight bounding box resolves in three to five reads. That is the entire economic argument: you pay for a few kilobytes of transfer and a few round trips instead of egressing a whole object, which is why FlatGeobuf owns the range-read web-delivery tier while analytical extracts belong to the columnar GeoParquet Migration Workflows path.

Issuing Range Requests Against Object Storage

The steps below take a bounding-box query from a raw HTTP probe to a working GDAL client that reads only the intersecting features.

  1. Confirm the store and its edge honor byte ranges. Object stores support ranges natively, but a fronting CDN may not. Probe with an explicit range and inspect the status line — you need 206, not 200.
# A GET (not HEAD) reveals whether ranges are actually served end to end.
curl -s -r 0-65535 -o /dev/null -D - \
  https://spatial-archive.s3.us-west-2.amazonaws.com/fgb/parcels/2024/region_north_indexed.fgb
# Expected:
#   HTTP/1.1 206 Partial Content
#   Accept-Ranges: bytes
#   Content-Range: bytes 0-65535/1974330112
#   Content-Length: 65536
  1. Run the bounding-box query through GDAL’s range-reading virtual filesystem. GDAL reads .fgb over /vsis3/ (or /vsicurl/ for a public URL) using the spatial index and HTTP ranges automatically; pyogrio exposes this with a bbox filter. Only the intersecting features are materialized.
import pyogrio

# bbox is (minx, miny, maxx, maxy) in the file's CRS (EPSG:4326 here).
# GDAL walks the R-tree and issues range reads under the hood — the whole
# 1.9 GB object is never downloaded.
gdf = pyogrio.read_dataframe(
    "/vsis3/spatial-archive/fgb/parcels/2024/region_north_indexed.fgb",
    bbox=(-122.72, 45.44, -122.55, 45.60),  # Portland west-side window
    columns=["parcel_id", "zoning", "assessed_value"],
)
print(len(gdf), "features fetched inside the window")
  1. Account for the bytes actually transferred. Enable GDAL’s cURL tracing to see each range request and prove the read footprint matches the bounding box rather than the file size.
# CPL_CURL_VERBOSE logs every Range request GDAL issues for one query.
export CPL_CURL_VERBOSE=YES
export CPL_DEBUG=ON
export AWS_REGION=us-west-2
python query_bbox.py 2>&1 | grep -E "Range: bytes|Content-Range" | head
# Each line is one 206 read; sum the Content-Length values for total bytes moved.

Streaming Versus Buffering the Response

A range request returns a byte range, not a feature boundary, so a streaming reader has to reassemble records that straddle the edges of the ranges it asked for. How that is handled decides whether the client renders progressively or waits for everything.

Buffering versus streaming a ranged feature read Two timelines for the same 1.9 megabyte range. The buffering reader produces a single draw at 1.1 seconds. The streaming reader parses each arriving chunk, drawing at 0.3, 0.6, 0.9 and 1.1 seconds, carrying a partial record across each chunk boundary. Same request, same bytes, same finish time — different experience buffering reader accumulate 1.9 MB — nothing on screen one draw at 1.1 s streaming reader draws at 0.3 s 0.6 s 0.9 s complete at 1.1 s The boundary problem Each chunk usually ends mid-record. Keep the trailing bytes in a carry buffer and prepend them to the next chunk; parse only whole records, and treat a carry buffer that never empties as a corrupt-offset signal rather than a slow network.

The practical guidance is to size the chunks the reader requests to the transport rather than to the data: 256 KB to 1 MB per chunk keeps round trips amortised while giving the renderer something to draw several times a second. Requesting the whole matched range in one call is simpler and costs the progressive rendering that makes a large extent feel responsive.

Validation & Verification

Prove three things before trusting a range-streaming endpoint in production: the store returns 206, the query returns only in-window features, and the transferred bytes are a small fraction of the object.

# 1. Range support and object size in one probe.
curl -s -r 0-1023 -o /dev/null -D - \
  https://spatial-archive.s3.us-west-2.amazonaws.com/fgb/parcels/2024/region_north_indexed.fgb \
  | grep -E "206|Content-Range"
# Expected: HTTP/1.1 206 Partial Content
#           Content-Range: bytes 0-1023/1974330112
# 2. Spatial correctness + byte-footprint sanity check.
import pyogrio, shapely.geometry as sg

bbox = (-122.72, 45.44, -122.55, 45.60)
gdf = pyogrio.read_dataframe(
    "/vsis3/spatial-archive/fgb/parcels/2024/region_north_indexed.fgb", bbox=bbox
)
window = sg.box(*bbox)
# Every returned geometry must intersect the query window (the index is exact
# at the leaf, so no far-away features should appear).
assert gdf.geometry.intersects(window).all(), "index returned out-of-window features"
print(f"{len(gdf)} features; expected a few thousand, not all 482,817")

Expected result: a query over a neighborhood-scale window against a ~1.9 GB layer returns a few thousand features and moves on the order of a few hundred kilobytes — three to four orders of magnitude less than a full download. If the returned count approaches the full feature count, the index was bypassed and the client fell back to a scan.

CDN and Range-Support Caveats

Range streaming lives or dies on the transport layer between the client and the object bytes. The failure modes are specific and quiet.

Symptom Root cause Fix
First probe returns 200 OK with the full Content-Length, and every query downloads the whole file A fronting CDN or proxy strips the Range header or is configured to collapse ranges into a full-object fetch Enable range/origin passthrough and range-based caching at the edge; verify Accept-Ranges: bytes survives to the client
HTTP 416 Range Not Satisfiable on a valid offset The object was replaced with a shorter version but a stale edge cache still advertises the old length Purge the edge cache on republish and key the cache on the object’s ETag, not just the path
Query latency spikes to seconds and the request log shows dozens of tiny reads Attribute-heavy features fragment the payload, or the file predates the index and clients scan linearly Prune attributes at write time and rebuild with the packed index; confirm contiguous matched ranges in the cURL trace

There is a real crossover point. Each range read is a dependent round trip, so a query that must return most of the file — a country-wide extract, or a layer small enough that the whole object fits in one read — is faster and cheaper as a single full GET than as many ranged reads, because it avoids per-request overhead and index-traversal latency. Use range streaming for selective, bounding-box-scoped reads against large objects in warm and cold tiers; fall back to a full download when the selection ratio is high. The request-count and egress economics that decide this crossover for a given archive are modeled in the spatial archive cost model, and the store’s per-request pricing depends on the object storage selection for GIS archives you standardized on.

For the authoritative range semantics on each platform, see the AWS S3 GetObject Range documentation and the Azure Blob range GET documentation, and confirm your client against the FlatGeobuf specification for HTTP reader behavior.

Failure Modes Unique to Ranged Reads

Ranged access introduces failure modes that whole-file downloads do not have, and most of them return data rather than an error — which is why they need explicit checks rather than reliance on exception handling.

Four ranged-read failure modes and how to detect each A table of four failure modes: truncated range, dropped Range header on redirect, stale cached range after republication, and unsatisfiable range, each with its symptom and the check that catches it. failurewhat the client seescheck that catches it short range returned proxy or storage truncates parser stops mid-record, no error compare Content-Range length to the request Range dropped on redirect 3xx to a signed URL correct features, whole file transferred assert status is 206, never 200 stale cached range layer republished in place features from the previous generation version the path; verify ETag before reuse 416 unsatisfiable offset past end of object hard error, easy to spot index built against a different file version

The stale-cache row is the one that causes the most confusion in practice, because the data is internally consistent — an old index paired with old features — and only wrong relative to what the catalogue advertises. Publishing each generation at its own path removes the failure entirely and costs nothing but a directory level, which is why archives that serve range reads should never republish in place.

Operational Execution Checklist

Frequently Asked Questions

How many concurrent range requests should a client issue?

Six to eight against a single host is the practical ceiling, matching what browsers allow per origin over HTTP/1.1 and roughly where object-storage per-connection throughput stops improving. Over HTTP/2 the connection limit disappears but the storage-side benefit does not grow much beyond a dozen streams. More important than the number is ordering: request the ranges covering the centre of the viewport first, so the progressive draw fills the part of the map the reader is looking at.

Does a failed range request need a full retry?

No — retry only the failed range. Because each request is independent and addressed by byte offset, a transient failure costs one range, not the session. Implement retries with jitter on 5xx and on truncated responses, and treat a 416 as fatal rather than retryable, since it means the index and the object disagree and no number of attempts will fix that.

Can range reads work against an archive-tier object?

Only after a restore, and only against the restored temporary copy. Objects in instant-retrieval archive classes support ranges directly and are the right home for a delivery copy that must stay reachable; objects in Flexible Retrieval or Deep Archive do not, so the pattern for those is to keep the delivery copy in an instant class and the preservation copy in the deepest tier the retention policy allows.

Part of the Spatial Data Archival knowledge base.

Related pages

Building a FlatGeobuf Spatial Index for HTTP Range Reads Converting GeoPackage to FlatGeobuf for Web Archives Optimizing FlatGeobuf for Web Mapping Archives: Pipeline Configuration and Cold Storage Validation