Spatial Data Archival · Cold Storage

Optimizing FlatGeobuf for Web Mapping Archives: Pipeline Configuration and Cold Storage Validation

This guide is for data engineers and GIS archivists who serve interactive web maps directly from object storage and need each archived .fgb artifact to stay small, deterministic, and cheap to range-read across a multi-year retention horizon. FlatGeobuf (.fgb) gives a browser deterministic HTTP range-request access — a client reads only the bytes its bounding box touches — but a default ogr2ogr export silently breaks that contract: the packed spatial index misaligns with cloud block sizes, attribute schemas expand unboundedly, and coordinate reference systems (CRS) drift during cold-storage transitions. The procedures below sit inside the broader Format Conversion & Pipeline Automation workflow and apply the tuning patterns from FlatGeobuf Optimization Techniques to enforce strict byte-alignment, deterministic index generation, and schema validation so retrieval latency stays flat instead of growing with archive size.

Web Archive Pipeline

Web-mapping archives normalize, index, validate, then verify on upload:

FlatGeobuf Web Archive Pipeline Four boxes left to right joined by arrows: Stage one normalizes schema and locks CRS (Phase 1); stage two builds the packed Hilbert index and pads to the 4 KB grid (Phase 2); stage three validates range reads return 206 Partial Content; stage four uploads to cold storage and verifies the integrity checksum (Phase 3). Each stage carries a short subtitle describing its byte-level guarantee. Normalize schema + lock CRS one explicit transform Build index pack Hilbert R-tree pad to 4 KB grid Validate range reads expect 206 Partial Upload + verify cold tier sha256 integrity PHASE 1 PHASE 2 VALIDATION PHASE 3 retrieval latency stays flat as the archive grows — bytes read scale with the bounding box, not the file

This page assumes you have already selected a target object store and storage class, that a retention policy framework governs how long .fgb artifacts are held, and that a hot/warm/cold tier design decides which tier serves live maps versus deep archive. FlatGeobuf owns the range-read web-delivery tier; if your access pattern is analytical, the GeoParquet Migration Workflows pipeline owns the columnar tier and the two formats coexist behind one manifest.

Step-by-Step Procedure

Phase 1 — Normalize Schema and Lock CRS Before Serialization

Implicit CRS declarations and unbounded attribute types are the primary drivers of archive bloat and client-side rendering failures. Lock the coordinate transformation and prune the schema in a single deterministic pass before the file is ever serialized. Detailed projection-registry handling belongs to CRS synchronization in pipelines; here you only enforce one explicit transform at the ingestion gateway.

# Force EPSG:4326 and prune attributes in one pass. ogr2ogr reprojects via
# -t_srs, so the geometry must NOT be reprojected again in -sql (OGR SQL has
# no ST_Transform). Build the index in Phase 2, not here.
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/archive_normalized.fgb \
  datasets/parcels/raw/source_parcels.gpkg \
  -s_srs EPSG:2913 -t_srs EPSG:4326 \
  -lco SPATIAL_INDEX=NO \
  -sql "SELECT id, name FROM parcels"

Constrain attribute types by casting them in the -sql/-select step; FlatGeobuf stores variable-length strings and IEEE doubles, so there is no field-width environment variable to set. Truncating attributes at the source is the same discipline that prevents silent attribute loss during format conversion.

Phase 2 — Build the Packed Hilbert Index and Align to Storage Blocks

The FlatGeobuf spatial index uses a Hilbert curve to order features. Misalignment between the index structure and cloud storage block boundaries forces excessive 206 Partial Content requests, inflating retrieval cost and latency.

# Build the packed Hilbert R-tree spatial index (depth is managed automatically).
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/archive_indexed.fgb \
  datasets/parcels/staging/archive_normalized.fgb \
  -lco SPATIAL_INDEX=YES

For datasets larger than 10 GB, bypass in-memory sorting: extract the Hilbert keys, run an external merge sort, then reassemble with the flatgeobuf CLI bindings. Cloud object storage optimizes range requests at 4 KB or 8 KB boundaries, so the transition from the spatial index to the geometry payload must be padded to prevent cross-boundary fetches.

# Pad the index-to-geometry boundary to the 4 KB cloud range-read grid.
import os

path = "datasets/parcels/staging/archive_indexed.fgb"
with open(path, "r+b") as f:
    f.seek(0, os.SEEK_END)
    size = f.tell()
    padding = (4096 - (size % 4096)) % 4096
    f.write(b"\x00" * padding)
print(f"padded {padding} bytes -> {size + padding} total")

Phase 3 — Upload to Cold Storage With Integrity-Preserving Settings

Multipart uploads and tiered-storage transitions frequently corrupt FlatGeobuf headers or fragment the spatial index. Capture a pre-upload checksum and force an opaque content type so no transfer-layer transform touches the first 4 KB block. FlatGeobuf has no internal codec, so any space savings come from the storage or transport layer — pair this tier with ZSTD level configuration for spatial files only where the client can transparently decompress.

# 1. Pre-upload checksum.
sha256sum datasets/parcels/staging/archive_indexed.fgb \
  > datasets/parcels/staging/archive_indexed.sha256

# 2. Upload to a retrieval-friendly cold tier; never let the client recompress.
aws s3 cp datasets/parcels/staging/archive_indexed.fgb \
  s3://geo-archive-prod/fgb/parcels/archive_indexed.fgb \
  --storage-class GLACIER_IR \
  --metadata-directive REPLACE \
  --content-type "application/octet-stream"

The Request Path a Map Client Actually Takes

Optimisation decisions for a web-facing archive only make sense against the full request path, which includes a CDN the archive team may not control. Between the browser and the object there are three places a range request can be answered, and two of them can quietly break it.

Where a range request can fail between browser and object storage A three-hop request path: browser, CDN edge, object storage. Two annotated failure points at the edge — stripping the Range header, and answering with the full object and a 200 status — turn a small ranged read into a full-file download. Browser Range: bytes=680-10919 CDN edge cache · forward · or collapse Object storage 206 Partial Content One ranged read, three hops — the middle hop decides whether it stays ranged Failure 1 · Range header stripped Edge forwards a plain GET; storage returns the whole 3.2 GB object. Symptom: correct data, catastrophic transfer — visible only in the bill. Failure 2 · edge answers 200, not 206 Full object cached and replayed; the client discards all but its range. Symptom: first request slow, later ones fast — looks like a warm cache.

Test the deployed path rather than the origin. A curl -r against the object storage endpoint proves the storage supports ranges, which was never in doubt; the same request against the public URL proves that the edge in front of it does too. Assert on the 206 status and the Content-Range header in a synthetic check, because both failure modes above return data that looks correct to a client and to a casual test.

Validation & Verification

Confirm feature counts, range-read behavior, and byte-for-byte integrity before declaring the artifact archival-ready.

# Spatial index + feature count present after Phase 2.
ogrinfo -so datasets/parcels/staging/archive_indexed.fgb
# Expected: "Feature Count: 482817" and a non-empty Extent line.

# Simulate a cold range request with a GET (HEAD/-I will not show 206).
curl -s -r 0-4095 -o /dev/null -D - \
  https://geo-archive-prod.s3.amazonaws.com/fgb/parcels/archive_indexed.fgb
# Expected: HTTP/1.1 206 Partial Content
#           Content-Range: bytes 0-4095/...
#           Content-Length: 4096

# Post-transfer: hash the full restored object and compare to the pre-upload sum.
downloaded=$(aws s3 cp s3://geo-archive-prod/fgb/parcels/archive_indexed.fgb - \
  | sha256sum | awk '{print $1}')
[ "$downloaded" = "$(awk '{print $1}' \
  datasets/parcels/staging/archive_indexed.sha256)" ] \
  && echo "INTEGRITY OK" || echo "INTEGRITY FAIL"
# Expected: INTEGRITY OK

Verify the schema and CRS survived the round trip by inspecting the restored object in place, then compare against the pre-upload manifest:

import pyogrio

meta = pyogrio.read_info("/vsis3/geo-archive-prod/fgb/parcels/archive_indexed.fgb")
assert meta["crs"] == "EPSG:4326", meta["crs"]
assert meta["geometry_type"] in ("Polygon", "MultiPolygon", "Point")
# pyogrio returns "dtypes" parallel to "fields"; iterate it directly.
assert all(dt in ("int32", "int64", "float32", "float64", "object")
           for dt in meta["dtypes"])
print("schema + CRS verified")

Budgeting Bytes Against the First Paint

For a web archive the meaningful measure is not total transfer but what has to arrive before a map draws something. That budget is small, fixed by user expectation rather than by bandwidth, and it constrains the archive layout more tightly than any storage consideration.

What must arrive before the map draws A timeline from zero to two seconds showing the manifest, header, index nodes and first feature range arriving inside a 1.5 second first-paint target, and a poorly sorted alternative that overruns it. 1.5 s first-paint target manifest 12 KB header 2 KB index 46 KB first feature range · 1.9 MB draws at 1.1 s 00.5 s1.0 s1.5 s2.0 s Unsorted file, same query: 214 ranges × ~40 ms round trip → 8.6 s before anything draws. The bytes are similar; the round trips are not. Write order is a front-end performance decision.

Two implications for the archive layout. Keep the manifest small enough to fetch unconditionally — a few tens of kilobytes covering footprints and file names, not a full catalogue — because it sits on the critical path for every session. And treat the number of ranges a typical viewport resolves to as a published property of each layer, measured after every republication, since it is the number that predicts perceived performance far better than file size does.

Troubleshooting

Symptom Root cause Fix
Client-side geometry jitter or NaN coordinates on render Implicit CRS drift during multi-stage pipeline staging Force -s_srs/-t_srs at ingestion and strip every source .prj; apply one deterministic transform before serialization (Phase 1).
Cold-tier retrieval latency >2 s for a <10 MB tile, 206 request count >50 per tile Unpadded index-to-geometry boundary, or the spatial index was never built Rebuild with -lco SPATIAL_INDEX=YES, pad to the 4 KB boundary, then re-run the curl -r 0-4095 range test (Phase 2).
OGR: FlatGeobuf: Invalid header or Geometry collection not supported after restore Multipart-upload chunk misalignment or cold-tier decompression altered the first 4096 bytes Disable client-side compression, force --content-type application/octet-stream, and re-validate the first 4 KB block immediately after transfer (Phase 3).
HTTP 416 Range Not Satisfiable on a known-good offset Index header exceeds the declared size after an incomplete re-serialization Re-serialize cleanly with SPATIAL_INDEX=YES and re-pad to the 4 KB boundary before upload.

Consult the GDAL FlatGeobuf driver documentation for version-specific header-parsing edge cases, the FlatGeobuf specification for strict CRS header encoding, and the AWS S3 GetObject Range header reference for storage-tier range compatibility.

Operational Execution Checklist

Frequently Asked Questions

What cache headers suit an archived FlatGeobuf?

Long and immutable. Archive generations do not change after publication, so Cache-Control: public, max-age=31536000, immutable is honest and lets the edge keep both the object and its ranges indefinitely. Where a layer is republished, publish it under a new versioned path rather than invalidating the old one — that keeps caches coherent and preserves the ability to reproduce an older map exactly.

Should the file be served compressed?

Not for range-read delivery. Content-encoding compression is applied to the response body, so a compressed response breaks the byte offsets the client computed from the index — most edges handle this by refusing to compress range responses, but the ones that do not return unusable data. Serve the file uncompressed and let the archive-tier copy carry the compression, or accept whole-file downloads and drop the index-based access pattern entirely.

How do clients discover which file covers an extent?

From a small index document published alongside the files — a STAC collection, a GeoJSON footprint index, or a plain JSON manifest of file names and bounding boxes. Keep it small enough to fetch on page load, since it is the one request that cannot be avoided, and generate it from the files themselves so a published layer cannot be missing from it.

Related pages

Building a FlatGeobuf Spatial Index for HTTP Range Reads Converting GeoPackage to FlatGeobuf for Web Archives Streaming FlatGeobuf Features Over HTTP Range Requests