Replicating Glacier-Class Spatial Objects Across Regions
Replication rules are written against buckets, but the objects in a mature spatial archive are not all in the same storage class — and the ones most worth protecting are usually the coldest. This walkthrough is for the archive operator who enabled replication after the lifecycle rules had already moved years of imagery and point clouds into Glacier and Deep Archive, and who now needs those objects in a second region without restoring the entire archive to get them there. Default behaviour will not do it: replication reads the source object, and an object in an archive class cannot be read until it is restored.
Why Cold Sources Behave Differently
A replication rule copies an object by reading it. For Standard, Standard-IA and Glacier Instant Retrieval that read is immediate. For Glacier Flexible Retrieval and Deep Archive it is not possible at all without a restore, so the platform simply does not replicate those objects — a batch replication job filtered on “eligible for replication” skips them silently, and the inventory reconciliation that follows shows a gap nobody caused.
Step-by-Step Procedure
Step 1 — Quantify the cold backlog from the inventory
Do not discover the size of this job by running it. The daily inventory report already carries a storage-class column, so the exact object count and volume per class is one query away.
# Athena over the S3 inventory table: how much of the archive is unreplicatable today?
aws athena start-query-execution --query-string "
SELECT storage_class,
COUNT(*) AS objects,
ROUND(SUM(size)/1e12, 2) AS tb
FROM spatial_archive_inventory
WHERE dt = '2026-08-01'
AND key LIKE 'archive/%'
AND replication_status IS NULL
GROUP BY storage_class
ORDER BY tb DESC" \
--result-configuration OutputLocation=s3://spatial-archive-logs/athena/A typical result for a mature archive shows the majority of objects in GLACIER and DEEP_ARCHIVE — the exact population that the replication rule quietly skipped.
Step 2 — Restore the backlog in controlled waves
Restoring the whole cold archive at once is both expensive and operationally hostile: restores expire, and an expired temporary copy that has not yet been replicated has to be restored again. Wave the job so that each batch’s restore window comfortably exceeds the time replication needs to drain it.
# Restore one wave: bulk tier, 10-day availability window
aws s3control create-job \
--account-id 123456789012 \
--operation '{"S3InitiateRestoreObject":{"ExpirationInDays":10,"GlacierJobTier":"BULK"}}' \
--manifest-generator '{"S3JobManifestGenerator":{
"SourceBucket":"arn:aws:s3:::spatial-archive",
"Filter":{"KeyNameConstraint":{"MatchAnyPrefix":["archive/imagery/2019/"]},
"ObjectSizeGreaterThanBytes":0}}}' \
--report '{"Bucket":"arn:aws:s3:::spatial-archive-logs","Format":"Report_CSV_20180820",
"Enabled":true,"Prefix":"restore-wave-2019","ReportScope":"AllTasks"}' \
--priority 5 \
--role-arn arn:aws:iam::123456789012:role/spatial-archive-batch \
--region us-east-1Ten days is deliberate. A bulk restore takes up to twelve hours, batch replication of a large wave takes hours more, and the margin absorbs a retry without a second restore charge.
Step 3 — Replicate the restored wave
Once the temporary copies exist, batch replication can read them. Filter on the same prefix as the restore wave so the two stay in step.
aws s3control create-job \
--account-id 123456789012 \
--operation '{"S3ReplicateObject":{}}' \
--manifest-generator '{"S3JobManifestGenerator":{
"SourceBucket":"arn:aws:s3:::spatial-archive",
"Filter":{"EligibleForReplication":true,
"ObjectReplicationStatuses":["NONE","FAILED"],
"KeyNameConstraint":{"MatchAnyPrefix":["archive/imagery/2019/"]}}}}' \
--report '{"Bucket":"arn:aws:s3:::spatial-archive-logs","Format":"Report_CSV_20180820",
"Enabled":true,"Prefix":"replicate-wave-2019","ReportScope":"AllTasks"}' \
--priority 5 \
--role-arn arn:aws:iam::123456789012:role/spatial-archive-batch \
--region us-east-1Note that the destination class is still whatever the replication rule specifies, not the source’s class — a restored Deep Archive object replicates into the destination class you configured, which is exactly what you want. The source object returns to Deep Archive when the temporary copy expires, untouched.
Validation & Verification
Reconcile per wave rather than at the end, so a failed wave is caught while its restore window is still open.
# Objects in this wave, and how many now report a completed replication
aws s3api list-objects-v2 --bucket spatial-archive \
--prefix archive/imagery/2019/ --query 'length(Contents)' --output text
aws s3api list-objects-v2 --bucket spatial-archive-dr-euw1 \
--prefix archive/imagery/2019/ --query 'length(Contents)' \
--output text --profile dr-accountBoth numbers should match before the wave is signed off. The batch job’s completion report also lists per-object outcomes, and a wave with a small number of FAILED rows is normally an encryption-key or permission problem on those specific keys rather than a systemic failure.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| Batch replication job reports zero eligible objects | Wave’s restores have not completed, or already expired | Check Restore header on a sample; re-run the restore wave with a longer window |
| Some objects replicate, deep-archive ones do not | Manifest filter caught only the restored prefix subset | Align restore and replication filters on identical prefixes |
| Restore charges far above estimate | Expedited tier selected by default in a script | Set GlacierJobTier: BULK explicitly for backlog work |
Replica objects land in STANDARD |
Rule’s destination class omitted | Fix the rule and re-replicate the affected wave |
| Source objects now billed at Standard rates | Restore was interpreted as a class change on another platform | On S3 a restore is a temporary copy and does not change the class; verify with head-object |
The last row matters when an archive spans providers: a restore on S3 leaves the object in its archive class, while a rehydration on some other platforms moves the blob to a hotter tier and leaves it there, which turns a backlog replication into an unintended and expensive re-tiering of the whole archive.
Operational Execution Checklist
Frequently Asked Questions
Does replicating from a restored copy count as reading the archive?
Yes, and it is billed accordingly: the restore retrieval charge applies exactly as it would for any other read. That is the unavoidable cost of getting cold data into a second region, and it is one-off — once the replica exists, subsequent changes replicate normally because new objects are written to instant-access classes before any lifecycle rule moves them.
Should the backlog be replicated at all, or only new data?
Replicate the backlog if the archive’s value is in its history, which for spatial holdings it usually is: a decade of imagery is not reconstructable, whereas last month’s ingest often still exists upstream. The pragmatic order is to replicate new writes first, so protection starts immediately, then work the backlog in waves from the most irreplaceable material outward.
Can the destination be Deep Archive rather than Glacier?
Yes, and it lowers the standing cost further — at the price of a twelve-hour restore during a recovery. That is a defensible choice for material held purely to satisfy a retention statute and a poor one for anything a real incident would need quickly. Split the replication into two rules with different destination classes rather than accepting one compromise for everything.
Can the backlog be replicated without restoring, by copying from a different source?
Sometimes, and it is worth checking before paying for restores. If the archive retained its ingest sources — the delivered LAZ tiles, the original imagery — those are usually in a warmer location or on-premise, and populating the replica from them costs no retrieval at all. The catch is that the replica then holds objects produced by a second conversion run, which must be proven byte-identical to the primary’s, or the two copies diverge in ways an integrity audit will later flag.
How long should a backlog replication project take?
Plan in months rather than weeks, and treat that as normal rather than as slowness. The work is rate-limited by restore quotas and by the availability windows each wave needs, and rushing it means larger waves, longer windows and more temporary-copy storage. A quarter to replicate a few hundred terabytes of cold material is a reasonable pace.
What if the archive is too large to replicate in full?
Replicate by value rather than by volume. Rank collections by whether they could be reproduced from another source, by their acquisition cost, and by whether anything else depends on them, then replicate downward until the budget is exhausted. Publishing which collections are and are not protected is more useful than a replica that covers an unstated subset.
Related
- Multi-Region Replication & Disaster Recovery for Spatial Archives — the parent topic, including why the replica’s class is a recovery-objective decision.
- Configuring Cross-Region Replication for Spatial Archives — the sibling walkthrough that provisions the rule this backlog job runs under.
- Evaluating Glacier Deep Archive for LiDAR Point Clouds — restore economics for the object class that dominates most cold backlogs.
- Spatial Archive Cost Modeling — where a one-off backlog transfer belongs in the archive’s forecast.
Up one level: Multi-Region Replication & Disaster Recovery.