Spatial Data Archival · Cold Storage

Configuring Cross-Region Replication for Spatial Archives

Enabling replication takes five minutes; enabling it so that a spatial archive is genuinely recoverable takes an afternoon and a checklist. This walkthrough is for the cloud architect or platform engineer standing up cross-region, cross-account replication for a bucket that already holds imagery, vector archives and point clouds under a retention policy — where the defaults will replicate the wrong subset, into the wrong storage class, without the locks, and report success. The procedure below provisions the destination correctly the first time, scopes replication by tag so derived products stay put, and ends with a verification step that distinguishes a rule that is working from one that is silently matching nothing.

Why the Defaults Fail an Archive

Three defaults are wrong for archival use specifically. The destination storage class defaults to the source’s class at the time of write, which for a hot-tier ingest means the replica sits in Standard forever, unaffected by the lifecycle rules that later move the source to Glacier. Delete-marker replication defaults in a way that propagates deletions, which turns the replica into a mirror of your mistakes rather than a defence against them. And replication applies only to objects written after the rule is enabled, so a bucket with ten years of history gets a replica of last Tuesday.

Three replication defaults that misfire on an archive Storage class inheritance, delete-marker propagation and new-objects-only scope, each with the archival consequence it produces and the setting that corrects it. defaultconsequence for an archivecorrected by class follows the source at write time not the source's eventual class replica stays in Standard — about 5× the cost storage_class = GLACIER delete markers replicate when enabled on a filtered rule an accidental delete removes both copies delete_marker_replication off new objects only history is out of scope ten years of archive left unreplicated S3 Batch Replication

Step-by-Step Procedure

Step 1 — Create the destination bucket with the flags that cannot be added later

Versioning and Object Lock must both be present at creation. Retrofitting either means a new bucket and a full re-copy, which for a spatial archive is measured in days and dollars rather than minutes.

# In the DR account, in the destination region
aws s3api create-bucket \
  --bucket spatial-archive-dr-euw1 \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket \
  --profile dr-account

aws s3api put-bucket-versioning \
  --bucket spatial-archive-dr-euw1 \
  --versioning-configuration Status=Enabled \
  --profile dr-account

Step 2 — Grant the replication role exactly what it needs, in the destination account

The destination bucket policy is what allows the source account’s replication role to write. Scope it to the replication actions only: this role must never be able to delete, and confining it is what makes the replica resistant to a compromise of the primary account.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowReplicationWritesOnly",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::123456789012:role/spatial-archive-replication" },
    "Action": [
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags",
      "s3:ObjectOwnerOverrideToBucketOwner",
      "s3:GetObjectVersionTagging"
    ],
    "Resource": "arn:aws:s3:::spatial-archive-dr-euw1/*"
  }]
}

Step 3 — Tag authoritative objects so scope is explicit

Replication scope should be a property the ingest pipeline sets deliberately, not a prefix pattern someone has to keep in step with the archive’s layout. Tag at write time, and the rule becomes self-maintaining.

# Applied by the ingest job at the moment of write, not retrospectively
aws s3api put-object \
  --bucket spatial-archive \
  --key archive/lidar/2024/region_north_copc.laz \
  --body region_north_copc.laz \
  --tagging "copy_class=authoritative&retention_class=statutory-25y" \
  --storage-class STANDARD_IA

Step 4 — Create the replication rule

The rule below replicates only tagged authoritative objects, writes them straight into Glacier, refuses to propagate deletions, and enables replication time control so lag becomes measurable.

cat > replication.json <<'JSON'
{
  "Role": "arn:aws:iam::123456789012:role/spatial-archive-replication",
  "Rules": [{
    "ID": "archive-of-record-to-eu-west-1",
    "Priority": 10,
    "Status": "Enabled",
    "Filter": { "And": {
      "Prefix": "archive/",
      "Tags": [{ "Key": "copy_class", "Value": "authoritative" }]
    }},
    "DeleteMarkerReplication": { "Status": "Disabled" },
    "Destination": {
      "Bucket": "arn:aws:s3:::spatial-archive-dr-euw1",
      "Account": "210987654321",
      "StorageClass": "GLACIER",
      "AccessControlTranslation": { "Owner": "Destination" },
      "ReplicationTime": { "Status": "Enabled", "Time": { "Minutes": 15 } },
      "Metrics": { "Status": "Enabled", "EventThreshold": { "Minutes": 15 } }
    },
    "SourceSelectionCriteria": {
      "ReplicaModifications": { "Status": "Enabled" }
    }
  }]
}
JSON

aws s3api put-bucket-replication \
  --bucket spatial-archive \
  --replication-configuration file://replication.json

Step 5 — Backfill the existing archive

Everything written before Step 4 is out of scope. A batch replication job walks the existing inventory and copies it under the same rule, and it is the step most often skipped because the rule already reports as enabled.

# Generate the manifest from the daily inventory report, then submit the batch job
aws s3control create-job \
  --account-id 123456789012 \
  --operation '{"S3ReplicateObject":{}}' \
  --report '{"Bucket":"arn:aws:s3:::spatial-archive-logs","Format":"Report_CSV_20180820",
             "Enabled":true,"Prefix":"batch-replication","ReportScope":"AllTasks"}' \
  --manifest-generator '{"S3JobManifestGenerator":{
      "SourceBucket":"arn:aws:s3:::spatial-archive",
      "EnableManifestOutput":false,
      "Filter":{"EligibleForReplication":true,"ObjectReplicationStatuses":["NONE","FAILED"]}}}' \
  --priority 10 \
  --role-arn arn:aws:iam::123456789012:role/spatial-archive-batch \
  --region us-east-1

Validation & Verification

Confirm three things: that new objects replicate, that their class is right, and that the counts agree across the whole prefix.

# 1. A freshly written object reaches COMPLETED within the replication-time objective
aws s3api head-object --bucket spatial-archive \
  --key archive/lidar/2024/region_north_copc.laz \
  --query 'ReplicationStatus'

# 2. The replica landed in the intended class, not the source's
aws s3api head-object --bucket spatial-archive-dr-euw1 \
  --key archive/lidar/2024/region_north_copc.laz \
  --query 'StorageClass' --profile dr-account

# 3. Whole-prefix reconciliation from both inventory reports
diff <(cut -d, -f2 source-inventory.csv | sort) \
     <(cut -d, -f2 replica-inventory.csv | sort) | head

Expected output is "COMPLETED", "GLACIER", and no differing lines. An empty diff on a rule that has never matched anything looks identical to a healthy one, which is why the object count should also be asserted to be non-zero and to track the source within the replication window.

Replication lag against the 15-minute objective Two lag series over a day. A healthy rule peaks at eleven minutes during the nightly ingest and returns to under four. A failing rule climbs steadily past the objective and does not recover, indicating a permissions error rather than load. 15-minute objective nightly ingest · 11 min failing rule — never recovers 00:0006:0012:0018:0024:00 A lag that rises and returns is load; a lag that rises and stays is a permissions or encryption failure. Alert on the shape, not just the threshold.

What the Two Accounts Can Do to Each Other

The permission boundary is the part of this configuration that does real security work, and it is worth stating explicitly which actions each side retains after the rule is in place.

What each account can do after replication is configured A permission boundary showing that the source account writes to the replica only through a narrowly scoped replication role, and that neither account's ordinary credentials can delete the other's objects. source account read + write + delete its own objects write to the replica — only via the role cannot delete replica objects cannot change the replica's lock config destination account read + lifecycle + delete the replica no credentials into the source account separate administrators by policy holds the compliance-mode locks replicate A compromise of either account's ordinary credentials reaches one copy. That containment is the reason to accept the extra account rather than replicate in place.

Keep the administrators of the two accounts distinct where the organisation’s size allows it. The isolation is a policy boundary, and a single administrator holding both sets of credentials collapses it back to the same blast radius the configuration was built to avoid.

Troubleshooting

Symptom Root cause Fix
ReplicationStatus: FAILED on encrypted objects The replication role cannot use the destination-region KMS key Grant the role kms:Encrypt on the destination key and add SourceSelectionCriteria.SseKmsEncryptedObjects
Rule enabled, nothing replicates Filter tag never applied by the ingest job Assert the copy_class tag in the write gate; backfill tags with a batch operation
Replica objects in STANDARD StorageClass omitted from the destination block Set it explicitly, then re-replicate affected objects with a batch job
Deletions appear in the replica DeleteMarkerReplication left enabled Disable it; restore deleted replica versions from the destination’s version history
Counts diverge only for old objects Batch backfill never run Run the batch replication job filtered to ObjectReplicationStatuses: NONE

The encryption row is worth rehearsing before an incident rather than during one: an archive whose replicas cannot be decrypted in the destination region has a copy and not a recovery, and the failure is invisible until someone tries to read it.

Operational Execution Checklist

Frequently Asked Questions

Can one rule replicate to two destinations?

A replication configuration can hold several rules, each with its own destination, and they run independently — which is how an archive serves both a disaster-recovery replica and a residency-constrained copy in a specific jurisdiction. Give each rule a distinct priority and non-overlapping filters, because overlapping filters resolve by priority and the loser silently does nothing.

Does replication preserve object tags and metadata?

Tags replicate when the role has s3:ReplicateTags, and user metadata travels with the object. Tags applied after the object replicated only reach the replica if replica modifications are enabled, which is why that flag matters for archives where retention class is assigned in a later step than the write.

How long does a batch backfill take?

It is priced and paced per object rather than per byte, so a 400,000-object archive completes in hours while a 4-million-object one takes days. Run it with a lowered priority so it does not compete with live replication, and expect the destination’s request charges rather than the transfer to dominate the cost.

What happens to replication when the source bucket’s lifecycle transitions an object?

Nothing, because replication acts on writes rather than on transitions. An object replicated while it was in Standard-IA stays in whatever class the destination rule specified, regardless of what the source does afterwards — the two buckets age independently. That independence is usually what you want, and it is also why the destination needs its own lifecycle configuration rather than inheriting anything.

Should the replica bucket have its own lifecycle rules?

Yes, and they should be simpler than the source’s. The replica exists to be recovered from, not to be queried, so the useful rules are ones that expire noncurrent versions after a defined window and clean up incomplete multipart uploads. Transition rules are usually unnecessary because the replication rule already writes into the intended class.

How is replication monitored day to day?

Two signals cover almost everything: the replication latency metric with an alarm at the objective, and a daily count comparison between the source and destination inventories. The first catches a rule that has started failing; the second catches a rule that has silently stopped matching, which produces no latency at all because nothing is pending. Alerting on latency alone is the common gap.

Up one level: Multi-Region Replication & Disaster Recovery.

Related pages

Measuring RPO and RTO for Geospatial Archives Rebuilding a Spatial Archive After Regional Bucket Loss Replicating Glacier-Class Spatial Objects Across Regions