Streaming COPC Octree Nodes Over HTTP Range Requests

A COPC file answers a spatial query the same way an indexed FlatGeobuf answers a bounding-box query: the client reads a small index, works out which byte ranges it needs, and fetches only those. This walkthrough is for the engineer implementing or debugging that read path against an archive — where the difference between a correct implementation and a naive one is two orders of magnitude in transferred bytes, and where the failure modes are the quiet kind that return correct-looking data.

The Read Path, Node by Node #

A COPC’s octree is stored as a hierarchy of pages, each listing child nodes with their byte offsets and point counts. A client descends only the branches that intersect the query, and stops descending when the resolution is sufficient.

Step-by-Step Procedure #

Step 1 — Read the header and the COPC info block #

The info block gives the octree’s root offset, the spacing at each level, and the extent — everything needed to plan the descent.

import requests, struct

URL = "https://spatial-archive.s3.eu-west-1.amazonaws.com/archive/lidar/2023/region_north.copc.laz"

def ranged(url, start, length):
    r = requests.get(url, headers={"Range": f"bytes={start}-{start + length - 1}"})
    assert r.status_code == 206, f"expected 206, got {r.status_code}"   # never accept 200 here
    return r.content

header = ranged(URL, 0, 375 + 160)      # LAS header plus the COPC info VLR

Step 2 — Descend the hierarchy, fetching one page per level #

def select_nodes(url, root_offset, root_size, query_bbox, max_depth):
    """Fetch hierarchy pages and return the node entries intersecting the query."""
    selected, pending = [], [(root_offset, root_size)]
    while pending:
        offset, size = pending.pop()
        page = ranged(url, offset, size)
        for i in range(0, len(page), 32):                     # 32-byte entries
            level, x, y, z, node_off, node_size, count = struct.unpack("<iiiiqii", page[i:i+32])
            if not intersects(level, x, y, z, query_bbox):
                continue                                       # prune the whole branch
            if count < 0:                                      # negative count = child page
                pending.append((node_off, node_size))
            elif level <= max_depth:
                selected.append((node_off, node_size, count))
    return selected

Pruning at the branch rather than at the leaf is the whole optimisation: a branch that does not intersect the query is never fetched, and neither are any of its descendants.

Step 3 — Fetch the selected nodes, coalescing adjacent ranges #

Nodes that are adjacent in the file should be fetched in one request. Because COPC writes nodes in octree order, spatially adjacent nodes are usually adjacent in the file too.

def coalesce(ranges, gap=1 << 20):     # merge ranges separated by less than 1 MB
    ranges.sort()
    out = [list(ranges[0])]
    for off, size, _ in ranges[1:]:
        last = out[-1]
        if off - (last[0] + last[1]) <= gap:
            last[1] = off + size - last[0]
        else:
            out.append([off, size])
    return out

Validation & Verification #

Verify that the client is reading a small fraction of the object and that every response was a partial one.

# Measure with a real client: bytes transferred for a bounded query
pdal info --stats --bounds "([503000,504000],[5431000,5432000])" \
  /vsis3/spatial-archive/archive/lidar/2023/region_north.copc.laz \
  2>&1 | grep -E 'points|bbox'

# Confirm the storage endpoint answers 206 and not 200
curl -sI -r 0-374 "$URL" | head -3

Expected output is a point count far smaller than the object’s total, and an HTTP status of 206 Partial Content with a Content-Range header. A 200 means something in the path is serving whole objects, and the query is silently downloading 8.6 GB.

Choosing a Depth Limit for the Query #

The octree lets a client stop descending at any level, and choosing where to stop is the single largest lever on how much a query transfers. The right depth follows from what the result is for: a map view needs enough points to look right at the current zoom, while an analysis needs every point in the extent.

Expose the limit as a parameter of the read rather than a property of the client, so the same reader can serve a map view and an analysis without two code paths. The COPC info block carries the spacing at each level, which is what lets a client convert a target point spacing on screen into a depth.

Troubleshooting #

Operational Execution Checklist #

• Assert 206 Partial Content on every ranged response
• Fetch the header and COPC info block in one request
• Prune at the hierarchy entry before fetching any child page
• Stop descending once the requested resolution is met
• Coalesce adjacent node ranges before issuing point-data requests
• Filter decoded points against the exact query geometry client-side
• Never cache byte offsets across object versions
• Publish each archive generation at its own path so caches stay coherent

Caching Node Ranges Between Queries #

Interactive clients issue many overlapping queries as a user pans and zooms, and the hierarchy pages they need are almost always the same ones. Caching those separately from the point data turns the second and subsequent queries into a single point-data fetch.

The last row is the constraint that makes the rest safe. Because a COPC’s byte offsets are meaningful only within one file, a client that caches offsets across a republication reads the wrong points without any error. Versioned paths remove the possibility entirely.

Frequently Asked Questions

Related #

• Point Cloud & COPC Conversion for Spatial Archives — the parent topic, including why consolidation and indexing are one decision.
• Converting LAS to COPC for Cold Archives — producing the objects this read path consumes.
• Streaming FlatGeobuf Features Over HTTP Range Requests — the same mechanics for vector features, including the CDN failure modes.
• Tuning COG Internal Tile Size for Range Requests — the over-fetch trade-off in the raster equivalent.

Up one level: Point Cloud & COPC Conversion.