Spatial Data Archival · Cold Storage

Converting LAS to COPC for Cold Archives

A LiDAR delivery arrives as a directory of tiles and a README; an archive needs consolidated, indexed, self-describing objects with a reference system that will still mean something in twenty years. This walkthrough is for the data engineer running that conversion across a full survey — typically tens of thousands of tiles and a few terabytes — where the job must be restartable, must not silently change coordinates, and must produce objects sized for the storage class they will live in.

Plan the Consolidation Before Converting Anything

The conversion is also a re-tiling, and the grouping decision is the one that determines both retrieval behaviour and the archive’s object count. Group by acquisition area rather than by an arbitrary tile count, so that one archived object corresponds to something a user would request.

# Inspect the delivery: extent, point count and header consistency across tiles
pdal info --summary datasets/lidar/2023/region_north/*.laz \
  | jq -s '{tiles: length,
            points: (map(.summary.num_points) | add),
            scales: (map(.summary.scale_x) | unique),
            offsets: (map(.summary.offset_x) | unique),
            srs: (map(.summary.srs.compoundwkt) | unique | length)}'

A scales or offsets array with more than one entry means the delivery is not internally consistent and the archive must impose a single frame. A srs count above one means the same, for the reference system — and both need a recorded decision before any conversion runs.

Three ways to group 30,000 delivered tiles Grouping by flight line, by fixed tile count, and by administrative area, with the resulting object counts, sizes and how well each matches user requests. groupingobjectssize rangematches how it is asked for? by acquisition flight line natural unit of the survey 2104–12 GB yes — one flight, one object by fixed tile count (100) simple to implement 3005–7 GB no — cuts across flight lines by administrative area matches reporting 460.2–90 GB yes, but sizes are unusable Where the natural unit produces unusable sizes, split it — do not abandon it for an arbitrary one.

Step-by-Step Procedure

Step 1 — Pin the frame for the whole survey

Decide one scale, offset and reference system, and record them where the conversion job reads them rather than in each pipeline file.

cat > survey_frame.json <<'JSON'
{ "scale":  {"x": 0.001, "y": 0.001, "z": 0.001},
  "offset": {"x": 500000, "y": 5430000, "z": 0},
  "srs": "EPSG:27700+5701",
  "point_format": 6,
  "decided_by": "archive-team", "decided_on": "2026-08-11" }
JSON

Step 2 — Build one COPC object per group 

# convert_group.py — one flight line; the fleet runs these concurrently
import json, subprocess, boto3, os

s3 = boto3.client("s3")
FRAME = json.load(open("survey_frame.json"))
BUCKET = "spatial-archive"

def convert(group_name: str, tiles: list[str]) -> dict:
    key = f"archive/lidar/2023/{group_name}.copc.laz"
    try:
        s3.head_object(Bucket=BUCKET, Key=key + ".done")
        return {"group": group_name, "status": "skipped"}
    except s3.exceptions.ClientError:
        pass

    pipeline = {"pipeline": [
        *[{"type": "readers.las", "filename": t, "nosrs": False} for t in tiles],
        {"type": "filters.merge"},
        {"type": "writers.copc",
         "filename": f"/vsis3/{BUCKET}/{key}.tmp",
         "scale_x": FRAME["scale"]["x"], "scale_y": FRAME["scale"]["y"],
         "scale_z": FRAME["scale"]["z"],
         "offset_x": FRAME["offset"]["x"], "offset_y": FRAME["offset"]["y"],
         "offset_z": FRAME["offset"]["z"],
         "minor_version": 4, "dataformat_id": FRAME["point_format"],
         "forward": "header,vlr", "a_srs": FRAME["srs"]}]}

    with open(f"/tmp/{group_name}.json", "w") as fh:
        json.dump(pipeline, fh)
    subprocess.run(["pdal", "pipeline", f"/tmp/{group_name}.json", "--nostream"], check=True)

    s3.copy_object(Bucket=BUCKET, Key=key,
                   CopySource={"Bucket": BUCKET, "Key": key + ".tmp"},
                   ChecksumAlgorithm="SHA256")
    s3.delete_object(Bucket=BUCKET, Key=key + ".tmp")
    s3.put_object(Bucket=BUCKET, Key=key + ".done",
                  Body=json.dumps({"tiles": len(tiles), "frame": FRAME}).encode())
    return {"group": group_name, "key": key, "status": "converted"}

Step 3 — Reconcile point counts group by group 

SRC=$(pdal info --summary datasets/lidar/2023/region_north/*.laz \
      | jq -s 'map(.summary.num_points) | add')
OUT=$(pdal info --metadata \
      /vsis3/spatial-archive/archive/lidar/2023/region_north.copc.laz \
      | jq '.metadata.count')
echo "source=$SRC output=$OUT delta=$((SRC - OUT))"

Expected output is a delta of zero. Any positive delta means tiles were dropped — usually one unreadable file that the merge filter skipped without failing the pipeline.

Where the Conversion Time Goes

Point-cloud conversion is dominated by decompression and recompression, and knowing the split lets a fleet be sized rather than guessed at. The octree construction that makes it COPC rather than plain LAZ is a surprisingly small share.

Where the time goes converting one COPC group Five phases of a conversion with their share of elapsed time, dominated by decompression and recompression rather than by octree construction. One 8.6 GB group · 3.1 billion points · 41 min on 16 cores read + decompress · 46% sort · 21% recompress + write · 27% read + decompress46% scales with source size; parallel across tiles sort into octree order21% memory-bound; the reason streaming is unavailable build hierarchy pages4% the COPC-specific work — nearly free recompress + write27% scales with output size validate2% headers and a sampled coordinate comparison

The four percent spent building the hierarchy is the whole reason to prefer COPC over plain LAZ, and it is the cheapest part of the job. The expensive parts happen whether or not the output is indexed, which makes converting to COPC rather than to LAZ close to a free upgrade whenever a conversion is happening anyway.

Validation & Verification

Beyond the count, check that the frame survived and that the octree exists and is usable.

pdal info --metadata /vsis3/spatial-archive/archive/lidar/2023/region_north.copc.laz \
  | jq '.metadata | {count, scale_x, offset_x, offset_y,
                     copc: (has("copc")), srs: .srs.compoundwkt[0:48]}'

# A spatial query should read a fraction of the object, not all of it
pdal info --stats --bounds "([503000,504000],[5431000,5432000])" \
  /vsis3/spatial-archive/archive/lidar/2023/region_north.copc.laz | jq '.stats.bbox'

Expected output confirms the pinned scale and offset, a copc block, a compound reference system, and a bounded query returning statistics without downloading the object.

Troubleshooting

Symptom Root cause Fix
Point count short by a few thousand One source tile unreadable; merge continued Fail the pipeline on read errors; quarantine the bad tile
Offsets differ between output objects Frame not pinned; writer derived per group Pass explicit offset_* from the survey frame file
Elevations off by tens of metres Vertical datum dropped from the reference system Use a compound SRS such as EPSG:27700+5701
Conversion runs out of memory Streaming disabled and the group is very large Split the group; COPC needs the point set in hand
Classification values shifted Point data record format changed by the writer Pin dataformat_id; compare classification histograms

Operational Execution Checklist

Handling Overlap Between Flight Lines

Adjacent flight lines overlap by design — typically twenty to thirty percent — so a naive merge of two lines double-counts the points in the overlap. Whether that matters depends on what the archive is for, and the decision has to be recorded either way.

Three ways to handle overlap between flight lines Keeping all points, thinning by scan angle, or keeping all points with a source flight-line tag, compared on what each preserves and what it costs. keep everything raw record preserved density doubles in overlaps biases density analysis honest, awkward thin by scan angle uniform density discards ~15% of points irreversible convenient, lossy keep + tag source line nothing discarded reader can thin on demand costs ~1 byte per point the archival choice The point source identifier is a standard LAS field, so tagging costs nothing beyond the byte it already occupies in most record formats.

For a preservation copy, keep everything and rely on the point source identifier to let readers thin as their analysis requires. Thinning is a derivation, and derivations belong downstream of the archive rather than baked into it.

Frequently Asked Questions

Can COPC files be merged later without re-reading the points?

No. The octree spans the whole point set, so merging two COPC objects means rebuilding the index from all their points — effectively a fresh conversion. That is why the grouping decision deserves attention before the job runs rather than after.

What point data record format should an archive use?

Format 6 or 7 for anything new: they use the 64-bit point structure, carry GPS time as standard, and support the extended classification range that later specifications rely on. Converting an older format upward is safe; converting downward silently drops fields.

How long does a full survey conversion take?

Dominated by decompress-and-recompress throughput, which runs at roughly 8–15 million points per second per core for LAZ. A 1.8 TB survey of about 60 billion points therefore needs somewhere near 1,200 core-hours — a few hours on a modest fleet, with the same transfer-versus-CPU balance point that applies to raster conversion.

Should the conversion also reproject the points?

Only if the archive has decided to standardise on a different reference system, and then as an explicit, recorded step rather than a side effect. Reprojection changes every coordinate, so it must be paired with a new frame decision and validated as a transformation rather than as a copy. Where the delivery is already in the archive’s system, pass it through untouched.

How is a partially failed conversion detected?

By the point-count reconciliation rather than by the exit status. PDAL’s merge filter will skip an unreadable input and complete successfully, so a group whose output is short by one tile’s worth of points looks like a successful run. Comparing counts per group is the only reliable check, which is why it belongs in the job rather than in a later audit.

What should be kept from the delivery besides the points?

The reports, the control-point data, the flight logs and the classification scheme documentation. These are small, they are the only record of how the survey was flown and classified, and they are routinely discarded because they are not point clouds. Archive them alongside the COPC objects with the same retention.

What should be done with tiles that fail to read?

Quarantine them with the error and the original bytes, and continue. Legacy point-cloud deliveries reliably contain a few damaged files, and halting a multi-day conversion for each is not workable. What is not acceptable is skipping them silently, which is what the count reconciliation is there to prevent.

Up one level: Point Cloud & COPC Conversion.

Related pages

Compressing LiDAR with LASzip for Deep Archive Streaming COPC Octree Nodes Over HTTP Range Requests Validating Point Cloud CRS and Scaling After Conversion