Spatial Data Archival · Cold Storage

Rebuilding a Spatial Archive After Regional Bucket Loss

This is the runbook the replication design exists to make possible: the primary region’s bucket is gone or unreachable, the replica in the second region is intact, and users are waiting. The order of operations matters more than any single command, because a recovery that restores bytes before it restores discovery produces an archive nobody can find anything in, and a recovery that promotes the replica before establishing what was lost makes the data loss unmeasurable. This procedure is for the on-call engineer and archive operator running the cutover, and it assumes the cross-account replica and rebuildable catalogue described in the parent topic.

Establish the Loss Boundary Before Touching Anything

The first question is not “how do we restore?” but “what did the replica not receive?” Replication has a lag, so the objects written in the minutes before the incident may exist only in the lost region. Establishing that boundary early gives the recovery a defensible statement of what was lost, and it becomes impossible to establish once the replica starts accepting new writes.

Recovery sequence after regional bucket loss Six ordered recovery steps: freeze writes, establish the loss boundary, rebuild the catalogue, restore the working set, redirect readers, and resume ingest. 1 · freeze stop ingest to the replica 2 · boundary what did not replicate 3 · catalogue rebuild against the replica 4 · restore the working set, not everything 5 · redirect readers to region B 6 · resume ingest, with a new replica plan Order is the runbook — discovery before bytes, bytes before traffic Steps 1–2 are irreversible once writes resume: the loss boundary cannot be reconstructed after the replica starts changing. Step 3 before step 4 is deliberate: a catalogue that lists everything lets users see what exists while restores are still running. Nothing here requires the lost region to come back. If it does, reconciliation is a separate, later exercise.

Step-by-Step Procedure

Step 1 — Freeze ingest and record the boundary

Stop the pipeline writing to either region, and capture the last object the replica received. The replication metrics give this directly, and the number becomes the recovery’s statement of data loss.

# Latest object in the replica, by last-modified
aws s3api list-objects-v2 --bucket spatial-archive-dr-euw1 \
  --prefix archive/ --profile dr-account \
  --query 'sort_by(Contents,&LastModified)[-1].[Key,LastModified]' --output text
# archive/imagery/2026/scene_08110342.tif   2026-08-11T03:47:12+00:00

# Cross-check against the pipeline's own write log for the same window
aws logs filter-log-events --log-group-name /spatial/ingest \
  --start-time 1786000000000 --filter-pattern '"object_written"' \
  --query 'events[-5:].message' --output text

Anything the ingest log records after the replica’s last-modified timestamp is presumed lost, and — critically — is usually still available upstream, because the ingest source is rarely in the same failure domain.

Step 2 — Rebuild the catalogue against the replica

Discovery comes before data. The catalogue build job already exists; point it at the replica’s bucket and region so that every asset href resolves to the surviving copy.

# Regenerate STAC items and collections from the replica, not from a stored copy
python -m archive.catalog build \
  --source-bucket spatial-archive-dr-euw1 \
  --source-region eu-west-1 \
  --href-base https://spatial-archive-dr-euw1.s3.eu-west-1.amazonaws.com \
  --collections imagery,lidar,parcels \
  --output s3://spatial-archive-dr-euw1/catalog/ \
  --profile dr-account

Users can now search the archive and see exactly what it holds, including objects that are still in an archive class and will need a restore — which is a far better experience than an empty catalogue while a multi-hour restore runs.

Step 3 — Restore the working set, not the archive

Restoring everything is the instinct and the wrong move: it costs a full retrieval of the archive and delays the objects people actually need. Derive the working set from recent access logs and restore that first.

# The 5% of objects that served 80% of reads in the last 90 days
aws athena start-query-execution --query-string "
  SELECT key, COUNT(*) AS reads
  FROM spatial_access_logs
  WHERE dt >= date_add('day', -90, current_date) AND operation = 'REST.GET.OBJECT'
  GROUP BY key ORDER BY reads DESC LIMIT 20000" \
  --result-configuration OutputLocation=s3://spatial-archive-logs/athena/

# Restore that manifest at the standard tier — hours, not the bulk tier's half-day
aws s3control create-job --account-id 210987654321 \
  --operation '{"S3InitiateRestoreObject":{"ExpirationInDays":30,"GlacierJobTier":"STANDARD"}}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},
               "Location":{"ObjectArn":"arn:aws:s3:::spatial-archive-logs/working-set.csv",
                           "ETag":"9f2c…"}}' \
  --report '{"Bucket":"arn:aws:s3:::spatial-archive-logs","Format":"Report_CSV_20180820",
             "Enabled":true,"Prefix":"dr-restore","ReportScope":"AllTasks"}' \
  --priority 20 --role-arn arn:aws:iam::210987654321:role/dr-batch --region eu-west-1

Step 4 — Redirect readers

Point the archive’s public endpoint at the surviving region. Where clients hold hard-coded URLs, this is the step that reveals it — which is why the base URL should be a configuration value in the catalogue rather than a literal in each item.

Recovery progress against the objective A timeline showing the catalogue searchable at 40 minutes, the working set restored by hour four, and the recovery-time objective of six hours met, with the remainder restored on demand afterwards. RTO · 6 h catalogue searchable · 40 min working set restoring readers redirected remainder restored on demand over days 02 h4 h6 h12 h+ Restoring the whole 240 TB first would have pushed the first usable read past hour 11 and cost roughly 20× as much. The objective is met by sequencing, not by speed: discovery first, hot subset next, long tail on demand.

Validation & Verification

A recovery is verified by a user-level check, not by an object count. Run a real query end to end and compare its result against a pre-incident baseline.

# Search the rebuilt catalogue and open the first asset it returns
python -m archive.catalog search \
  --collection imagery --bbox "-3.25,54.10,-2.90,54.35" --datetime 2024-06-01/2024-06-30 \
  --catalog s3://spatial-archive-dr-euw1/catalog/ | head -3

gdalinfo /vsis3/spatial-archive-dr-euw1/archive/imagery/2024/scene_0612.tif | head -8

Expected output is a non-empty result set whose count matches the pre-incident baseline for the same query, and a gdalinfo block reporting the same size, band count and reference system as the source did. Keep a small set of such baseline queries with expected counts alongside the runbook — during an incident, nobody has time to work out what “correct” looks like.

Troubleshooting

Symptom Root cause Fix
Catalogue rebuilds but assets 404 Href base still points at the lost region Re-run the build with the correct --href-base; never store absolute URLs in items
gdalinfo fails with an access error Reader credentials scoped to the primary account only Issue read credentials in the DR account; this is why they belong in the deployment, not in a person’s profile
Restores rejected on locked objects Compliance-mode lock replicated with a retain-until date Locks do not block reads or restores; check the request’s tier and quota instead
Object counts match, sizes differ Multipart objects replicated with different part sizes Compare content checksums rather than ETags, which differ for multipart uploads
Some collections missing entirely They were never tagged for replication Record the gap in the incident report; recover from upstream sources if they exist

The last row is the one that turns an incident into a policy change. A collection that was never in scope is not a recovery failure — it is a scoping decision that was made, perhaps implicitly, and the incident is when its consequences become visible.

Operational Execution Checklist

What to Communicate, and When

A recovery is also a communication exercise, and the archive’s users judge it as much on what they were told as on how long it took. Three messages matter, and they map onto the phases of the runbook rather than onto a fixed schedule.

The first, within the detection window, states that the archive is unavailable, that the incident is understood, and when the next update will come — not an estimate of restoration, which nobody has yet. The second, once the loss boundary is established, states what was lost: the window between the last replicated object and the incident, expressed in the terms users think in (“data ingested after 03:47 on 11 August may need to be re-supplied”). The third, once the catalogue is rebuilt, states what is discoverable now and what needs a restore request — which is the point at which users can resume planning.

What to tell users, and at which phase Four communications during a recovery, each tied to a runbook phase completing: unavailability notice, loss statement, catalogue availability, and service restored. 0:35 unavailability notice next update at 1:30 1:10 loss boundary stated “after 03:47, re-supply” 1:50 catalogue searchable restore guidance published 5:35 service restored measured figures recorded Each message follows a phase completing, so it always contains new information — and the promised next-update time is always met. Drafting the templates in advance removes the worst writing task from the worst possible moment.

Draft all four messages as templates before the incident, with the variable parts marked. Nobody writes clearly at hour two of an outage, and a template that only needs timestamps and a boundary filled in is the difference between a clear notice and a vague one.

Frequently Asked Questions

Should the replica become the new primary, or should data be copied back?

Promote the replica. Copying 240 TB back to a recovered region costs a full transfer, takes days, and leaves the archive unprotected in the meantime — whereas promoting the replica and standing up a new replica elsewhere achieves the same end state with one transfer instead of two. The exception is a residency constraint that requires the data to live in the original jurisdiction, which is a legal rather than an operational consideration.

What if the lost region comes back with data the replica never received?

Treat it as a source to reconcile from, not as a competing primary. Freeze it read-only, enumerate the objects newer than the loss boundary, and copy just those into the promoted archive. Resuming writes to a recovered region that has diverged from the promoted one is how an archive ends up with two partial truths.

How often should this runbook be exercised?

Annually end to end, and quarterly for the parts that can be rehearsed cheaply — the catalogue rebuild against the replica and a restore of a small sample. The full drill is what validates the credentials, the automation deployment and the baseline queries, all of which decay silently between incidents. Record the measured timings each time; the trend matters more than any single result.

Should the recovery restore data that nobody has asked for yet?

Only after the working set is served. Restoring on demand keeps the recovery’s cost proportional to actual need and spreads it over weeks rather than concentrating it in one bill. The exception is material with a legal or contractual availability requirement, which should be restored on the same schedule as the working set regardless of whether anyone has asked.

How is the incident’s data loss communicated to downstream systems?

By publishing the loss boundary as data rather than as prose. A machine-readable record naming the affected time window and collections lets downstream pipelines decide for themselves whether their inputs are complete, which is far more reliable than expecting every consumer to have read an email. Where the upstream sources still hold the lost window, re-ingesting it is usually faster than reasoning about who was affected.

What should be changed after the incident is closed?

Whatever the drill did not predict. Every real recovery reveals a step the runbook glossed over — a credential that had to be created, a permission that was missing, a baseline query nobody had recorded. Capture those in the runbook while the memory is fresh, and re-run the drill within a quarter to confirm the changes work.

Up one level: Multi-Region Replication & Disaster Recovery.

Related pages

Configuring Cross-Region Replication for Spatial Archives Measuring RPO and RTO for Geospatial Archives Replicating Glacier-Class Spatial Objects Across Regions