Spatial Data Archival · Cold Storage

Measuring RPO and RTO for Geospatial Archives

Recovery objectives are usually asserted and rarely measured, which is how an archive ends up promising a four-hour recovery that has never been attempted and cannot be achieved from a Deep Archive replica. This walkthrough is for the archive operator or platform engineer who needs to replace two numbers in a policy document with two numbers derived from an actual drill — and to keep them honest as the archive grows. The measurement is not difficult; what makes it useful is measuring the whole path, including the discovery and validation steps that a bytes-only test skips.

What the Two Numbers Actually Cover

Recovery point objective is a property of the replication configuration: how much recent work can be lost. Recovery time objective is a property of everything else — restore latency, catalogue rebuild, credential availability, and the human decision to declare an incident. For a spatial archive the second number is dominated by parts that have nothing to do with storage, which is why measuring only the restore produces an optimistic figure that fails on the day.

What the measured recovery time is made of Six components of a measured recovery: detection and decision, credential setup, catalogue rebuild, working-set restore, validation, and redirect, with the restore accounting for just over half the total. Measured RTO: 5 h 35 m · from incident start to first verified user query restore working set · 3 h 30 m detect + decide35 min shrinks with alerting, not with storage credentials + access15 min zero if pre-deployed in the DR account catalogue rebuild40 min parallelisable; scales with object count restore working set3 h 30 min floor is the storage class's restore latency validate + redirect35 min baseline queries prepared in advance

Step-by-Step Procedure

Step 1 — Measure the recovery point from replication metrics

RPO is observable continuously and does not need a drill. The replication latency metric gives the current exposure, and its distribution over a month gives the number worth publishing.

# 30 days of replication latency, p50 / p95 / max, in seconds
aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 --metric-name ReplicationLatency \
  --dimensions Name=SourceBucket,Value=spatial-archive \
                Name=DestinationBucket,Value=spatial-archive-dr-euw1 \
                Name=RuleId,Value=archive-of-record-to-eu-west-1 \
  --start-time 2026-07-11T00:00:00Z --end-time 2026-08-11T00:00:00Z \
  --period 86400 --statistics Average Maximum \
  --query 'Datapoints[].[Timestamp,Average,Maximum]' --output table

Publish the 95th percentile rather than the average: the average describes a quiet afternoon, and the incident will not happen during one. An archive whose nightly ingest pushes p95 to eleven minutes has an eleven-minute RPO, not a four-minute one.

Step 2 — Run the drill against a real subset

The drill restores a defined subset in the DR region and takes it all the way to a verified query. Timestamp each phase — the phase boundaries are what make the result actionable.

#!/usr/bin/env bash
set -euo pipefail
mark() { printf '%s\t%s\n' "$(date -u +%FT%TZ)" "$1" >> drill.log; }

mark "start"
python -m archive.catalog build --source-bucket spatial-archive-dr-euw1 \
  --source-region eu-west-1 --collections imagery --output s3://spatial-archive-dr-euw1/catalog-drill/
mark "catalogue-rebuilt"

aws s3control create-job --account-id 210987654321 \
  --operation '{"S3InitiateRestoreObject":{"ExpirationInDays":2,"GlacierJobTier":"STANDARD"}}' \
  --manifest file://drill-manifest.json \
  --report file://drill-report.json --priority 20 \
  --role-arn arn:aws:iam::210987654321:role/dr-batch --region eu-west-1
mark "restore-requested"

until aws s3api head-object --bucket spatial-archive-dr-euw1 \
        --key archive/imagery/2024/scene_0612.tif --profile dr-account \
        --query 'Restore' --output text 2>/dev/null | grep -q 'ongoing-request="false"'; do
  sleep 300
done
mark "restore-complete"

gdalinfo /vsis3/spatial-archive-dr-euw1/archive/imagery/2024/scene_0612.tif > /dev/null
mark "verified"

Step 3 — Convert the drill into objectives

Take the measured total, add the detection and decision time your alerting actually achieves, and round outward. An objective set at the measured figure will be missed half the time by definition; one set a margin above it is a commitment that can be kept.

Deriving a published objective from three drills Three measured drill durations of 5h35m, 6h10m and 5h05m, with a published recovery objective of eight hours set above the slowest, leaving margin for detection variance and partial failures. drill 1 · Feb 5 h 35 m drill 2 · May 6 h 10 m — bulk tier used by mistake drill 3 · Aug 5 h 05 m published objective 8 h 03 h6 h9 h The margin is not padding: it covers a restore that partially fails, an on-call engineer who is not already at a keyboard, and the drill's own optimism. Re-derive after any change to storage class, object sizing, or the catalogue build.

Validation & Verification

The objectives are validated by the same drill that produced them, run again after any material change. What makes the validation meaningful is the baseline: a set of queries whose correct results were recorded before the drill, so “verified” means “returned what it should” rather than “returned something”.

# Baseline comparison — expected counts recorded during normal operation
python -m archive.drill verify \
  --baseline baselines/2026-Q2.json \
  --catalog s3://spatial-archive-dr-euw1/catalog-drill/ \
  --tolerance 0

Expected output lists each baseline query with match and exits non-zero on any divergence. A count that is lower than baseline points at objects that never replicated; a count that is higher usually means the baseline is stale rather than that the drill found extra data.

Troubleshooting

Symptom Root cause Fix
RPO metric absent Replication time control not enabled Enable it on the rule; without it, replication lag is unobservable
Drill time dominated by credential setup DR-account roles created by hand during the drill Deploy them from the same pipeline as the primary’s
Restore phase far slower than expected Bulk tier selected, or per-object quota throttling Use the standard tier for drills; consolidate small objects
Catalogue rebuild slower each drill Build is serial over a growing object count Parallelise by collection; it is embarrassingly parallel
Verified query returns fewer features Objects outside the replication filter Widen the scope or record the exclusion as accepted risk

Operational Execution Checklist

Publishing the Objectives So They Are Usable

An objective that lives in an architecture document is a number nobody plans around. The people who need it are the analysts deciding whether to depend on the archive for a deadline, the incident responder deciding whether to invoke the runbook, and the funder asking whether the holding is safe. Each needs it expressed differently.

For analysts, the useful form is a per-collection statement attached to the catalogue: “this collection is available within eight hours of a regional incident; this one within thirty minutes.” For responders, it is the trigger threshold in the runbook. For governance, it is the measured figure with the date of the drill that produced it — an objective without a measurement date is an aspiration.

One measurement, three audiences The measured recovery objective expressed as a catalogue statement for analysts, a trigger threshold for responders, and a measured figure with a drill date for governance. one drill measured 5 h 35 m analysts · in the catalogue “imagery: available within 8 h of a regional incident” responders · in the runbook “promote when unavailability exceeds 8 h with no ETA” governance · in the record “measured 5 h 35 m on 2026-08-11; published objective 8 h” The three must not drift apart: publish them from one source so a re-measurement updates all of them together. An objective quoted in three places with three values is worse than no objective at all.

Generate all three from the drill’s output rather than transcribing them. The catalogue statement in particular should be a property of the collection, set by the same job that records the drill result, so a collection whose tiering changed since the last drill is visibly stale rather than quietly wrong.

Frequently Asked Questions

Can the RTO be shortened without moving the replica to a hotter class?

Yes, and usually more cheaply. Roughly half the measured time in a well-run drill sits outside the restore: detection, credentials, catalogue rebuild and validation. Pre-deploying DR credentials removes fifteen minutes, parallelising the catalogue build removes most of forty, and better alerting removes a large part of the detection window. Only after those are exhausted does moving the replica to Glacier Instant Retrieval — which roughly triples its storage cost — become the next lever.

Should different collections have different objectives?

Yes, and stating one objective for a heterogeneous archive is usually how the number becomes untrue. Reference layers that other work depends on justify a short objective and a hotter replica; a statutory archive read once a decade does not. Publishing two or three tiers of objective, each with the collections it covers, is more honest and cheaper than one aggressive figure applied to everything.

What is a reasonable RPO for an archive that ingests nightly?

The ingest interval itself, in most cases. If data arrives once a night from a source that retains it, the practical exposure is one batch — and that batch can usually be re-ingested rather than recovered, which makes an aggressive replication objective unnecessary. Spend the effort on the archive’s ability to re-run the previous night’s load instead.

Does the drill have to use production data?

It has to use production infrastructure; the data can be a subset. What the drill validates is the path — credentials, permissions, restore quotas, catalogue build, network routes — and a subset exercises all of them. What a subset cannot validate is throughput at full scale, so the extrapolation from drill to full recovery should be stated explicitly rather than assumed to be linear.

How do the objectives change when the archive grows?

The recovery point stays roughly constant, because replication latency depends on write rate rather than on archive size. The recovery time grows with the catalogue rebuild and with the working set, both of which scale with object count. That is why the drill should be re-run annually even when nothing has changed: the archive itself has changed, and the number that was true last year is quietly no longer true.

Is it worth measuring recovery from the loss of a single collection?

Yes, and it is the more common incident by a wide margin. A collection accidentally deleted or corrupted needs a targeted recovery that shares almost none of the regional runbook’s steps, and its objective is usually much tighter because someone is waiting. Measure both, publish both, and be explicit that they are different scenarios.

Up one level: Multi-Region Replication & Disaster Recovery.

Related pages

Configuring Cross-Region Replication for Spatial Archives Rebuilding a Spatial Archive After Regional Bucket Loss Replicating Glacier-Class Spatial Objects Across Regions