Optimizing FlatGeobuf for Web Mapping Archives: Pipeline Configuration and Cold Storage Validation

This guide is for data engineers and GIS archivists who serve interactive web maps directly from object storage and need each archived .fgb artifact to stay small, deterministic, and cheap to range-read across a multi-year retention horizon. FlatGeobuf (.fgb) gives a browser deterministic HTTP range-request access — a client reads only the bytes its bounding box touches — but a default ogr2ogr export silently breaks that contract: the packed spatial index misaligns with cloud block sizes, attribute schemas expand unboundedly, and coordinate reference systems (CRS) drift during cold-storage transitions. The procedures below sit inside the broader Format Conversion & Pipeline Automation workflow and apply the tuning patterns from FlatGeobuf Optimization Techniques to enforce strict byte-alignment, deterministic index generation, and schema validation so retrieval latency stays flat instead of growing with archive size.

Web Archive Pipeline #

Web-mapping archives normalize, index, validate, then verify on upload:

This page assumes you have already selected a target object store and storage class, that a retention policy framework governs how long .fgb artifacts are held, and that a hot/warm/cold tier design decides which tier serves live maps versus deep archive. FlatGeobuf owns the range-read web-delivery tier; if your access pattern is analytical, the GeoParquet Migration Workflows pipeline owns the columnar tier and the two formats coexist behind one manifest.

Step-by-Step Procedure #

Phase 1 — Normalize Schema and Lock CRS Before Serialization #

Implicit CRS declarations and unbounded attribute types are the primary drivers of archive bloat and client-side rendering failures. Lock the coordinate transformation and prune the schema in a single deterministic pass before the file is ever serialized. Detailed projection-registry handling belongs to CRS synchronization in pipelines; here you only enforce one explicit transform at the ingestion gateway.

# Force EPSG:4326 and prune attributes in one pass. ogr2ogr reprojects via
# -t_srs, so the geometry must NOT be reprojected again in -sql (OGR SQL has
# no ST_Transform). Build the index in Phase 2, not here.
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/archive_normalized.fgb \
  datasets/parcels/raw/source_parcels.gpkg \
  -s_srs EPSG:2913 -t_srs EPSG:4326 \
  -lco SPATIAL_INDEX=NO \
  -sql "SELECT id, name FROM parcels"

Constrain attribute types by casting them in the -sql/-select step; FlatGeobuf stores variable-length strings and IEEE doubles, so there is no field-width environment variable to set. Truncating attributes at the source is the same discipline that prevents silent attribute loss during format conversion.

Phase 2 — Build the Packed Hilbert Index and Align to Storage Blocks #

The FlatGeobuf spatial index uses a Hilbert curve to order features. Misalignment between the index structure and cloud storage block boundaries forces excessive 206 Partial Content requests, inflating retrieval cost and latency.

# Build the packed Hilbert R-tree spatial index (depth is managed automatically).
ogr2ogr -f "FlatGeobuf" \
  datasets/parcels/staging/archive_indexed.fgb \
  datasets/parcels/staging/archive_normalized.fgb \
  -lco SPATIAL_INDEX=YES

For datasets larger than 10 GB, bypass in-memory sorting: extract the Hilbert keys, run an external merge sort, then reassemble with the flatgeobuf CLI bindings. Cloud object storage optimizes range requests at 4 KB or 8 KB boundaries, so the transition from the spatial index to the geometry payload must be padded to prevent cross-boundary fetches.

# Pad the index-to-geometry boundary to the 4 KB cloud range-read grid.
import os

path = "datasets/parcels/staging/archive_indexed.fgb"
with open(path, "r+b") as f:
    f.seek(0, os.SEEK_END)
    size = f.tell()
    padding = (4096 - (size % 4096)) % 4096
    f.write(b"\x00" * padding)
print(f"padded {padding} bytes -> {size + padding} total")

Phase 3 — Upload to Cold Storage With Integrity-Preserving Settings #

Multipart uploads and tiered-storage transitions frequently corrupt FlatGeobuf headers or fragment the spatial index. Capture a pre-upload checksum and force an opaque content type so no transfer-layer transform touches the first 4 KB block. FlatGeobuf has no internal codec, so any space savings come from the storage or transport layer — pair this tier with ZSTD level configuration for spatial files only where the client can transparently decompress.

# 1. Pre-upload checksum.
sha256sum datasets/parcels/staging/archive_indexed.fgb \
  > datasets/parcels/staging/archive_indexed.sha256

# 2. Upload to a retrieval-friendly cold tier; never let the client recompress.
aws s3 cp datasets/parcels/staging/archive_indexed.fgb \
  s3://geo-archive-prod/fgb/parcels/archive_indexed.fgb \
  --storage-class GLACIER_IR \
  --metadata-directive REPLACE \
  --content-type "application/octet-stream"

Validation & Verification #

Confirm feature counts, range-read behavior, and byte-for-byte integrity before declaring the artifact archival-ready.

# Spatial index + feature count present after Phase 2.
ogrinfo -so datasets/parcels/staging/archive_indexed.fgb
# Expected: "Feature Count: 482817" and a non-empty Extent line.

# Simulate a cold range request with a GET (HEAD/-I will not show 206).
curl -s -r 0-4095 -o /dev/null -D - \
  https://geo-archive-prod.s3.amazonaws.com/fgb/parcels/archive_indexed.fgb
# Expected: HTTP/1.1 206 Partial Content
#           Content-Range: bytes 0-4095/...
#           Content-Length: 4096

# Post-transfer: hash the full restored object and compare to the pre-upload sum.
downloaded=$(aws s3 cp s3://geo-archive-prod/fgb/parcels/archive_indexed.fgb - \
  | sha256sum | awk '{print $1}')
[ "$downloaded" = "$(awk '{print $1}' \
  datasets/parcels/staging/archive_indexed.sha256)" ] \
  && echo "INTEGRITY OK" || echo "INTEGRITY FAIL"
# Expected: INTEGRITY OK

Verify the schema and CRS survived the round trip by inspecting the restored object in place, then compare against the pre-upload manifest:

import pyogrio

meta = pyogrio.read_info("/vsis3/geo-archive-prod/fgb/parcels/archive_indexed.fgb")
assert meta["crs"] == "EPSG:4326", meta["crs"]
assert meta["geometry_type"] in ("Polygon", "MultiPolygon", "Point")
# pyogrio returns "dtypes" parallel to "fields"; iterate it directly.
assert all(dt in ("int32", "int64", "float32", "float64", "object")
           for dt in meta["dtypes"])
print("schema + CRS verified")

Troubleshooting #

Consult the GDAL FlatGeobuf driver documentation for version-specific header-parsing edge cases, the FlatGeobuf specification for strict CRS header encoding, and the AWS S3 GetObject Range header reference for storage-tier range compatibility.

Operational Execution Checklist #

• Source CRS resolved and a single explicit -s_srs/-t_srs transform applied; all .prj artifacts stripped
• Attribute schema pruned and cast in the -sql/-select step; serialized payload per feature kept lean
• Packed Hilbert spatial index built with -lco SPATIAL_INDEX=YES
• Index-to-geometry boundary padded to the 4 KB cloud range-read grid
• ogrinfo -so confirms feature count and non-empty extent
• curl -r 0-4095 returns 206 Partial Content with Content-Length: 4096
• Pre-upload sha256sum captured and re-verified against the restored object (INTEGRITY OK)
• Restored object uploaded with --content-type application/octet-stream to the chosen cold tier
• pyogrio.read_info() on the /vsis3/ object matches the pre-upload schema + CRS manifest

Related #

• Up to the parent topic: FlatGeobuf Optimization Techniques covers indexing, compression, schema, and CRS tuning across the whole .fgb lifecycle.
• Sibling procedure: Automating CRS Transformations in ETL Pipelines standardizes the projection step this page depends on.
• Cross-topic: How to Design a 3-Tier Spatial Storage Architecture decides which tier serves live .fgb web maps versus deep archive.
• Parent framework: Format Conversion & Pipeline Automation for the end-to-end conversion architecture these phases plug into.