Spatial Data Archival · Cold Storage

Keeping STAC Catalogs in Sync with Lifecycle Transitions

A lifecycle rule moves an object from Standard to Glacier without telling anyone. The bytes are unchanged, the key is unchanged, and every catalogue entry pointing at it is now subtly wrong: it advertises an asset that used to return in milliseconds and now takes hours. This walkthrough is for the engineer keeping a STAC catalogue honest about storage class — a field users need in order to know whether a search result is something they can open now or something they must request.

Why This Drift Is Different

Most catalogue drift is caused by writes and is fixed by subscribing to write events. Storage-class drift is caused by the platform acting on its own schedule, and there is no event to subscribe to.

How each kind of catalogue drift is detected Writes and deletes are detectable from object events within seconds; storage-class transitions are not reliably event-driven and require scheduled reconciliation against the inventory. changeevent available?detection latencymechanism object written yessecondsevent notification object deleted yessecondsevent notification storage class transitioned not reliablyup to a day inventory reconciliation restore expired noup to a day inventory reconciliation

Step-by-Step Procedure

Step 1 — Model storage class as a first-class item property

Put it where a client will look, and record when it was last confirmed so staleness is visible rather than implied.

{
  "type": "Feature",
  "stac_version": "1.0.0",
  "id": "ortho_n5432_e0871_2014",
  "properties": {
    "datetime": "2014-06-17T10:22:00Z",
    "gsd": 0.25,
    "proj:epsg": 27700,
    "storage:tier": "GLACIER",
    "storage:retrieval_hint": "restore required, 3-5 h at standard tier",
    "storage:verified_at": "2026-08-11T02:14:00Z"
  },
  "assets": {
    "data": {
      "href": "s3://spatial-archive/archive/imagery/2014/ortho_n5432_e0871.tif",
      "type": "image/tiff; application=geotiff; profile=cloud-optimized",
      "file:size": 918234112,
      "file:checksum": "1220a3f1…",
      "roles": ["data"]
    }
  }
}

Step 2 — Reconcile against the inventory on a schedule 

-- Items whose advertised tier disagrees with the storage inventory
SELECT c.item_id,
       c.storage_tier      AS advertised,
       i.storage_class     AS actual,
       c.verified_at
FROM stac_items c
JOIN spatial_archive_inventory i
  ON i.key = c.asset_key AND i.dt = date_format(current_date, '%Y-%m-%d')
WHERE c.storage_tier <> i.storage_class
   OR c.verified_at < date_add('day', -2, current_timestamp);

Step 3 — Rewrite only the items that changed 

import json, boto3
s3 = boto3.client("s3")

def refresh_tier(item_key: str, actual_class: str, verified_at: str):
    obj = s3.get_object(Bucket="spatial-archive", Key=item_key)
    item = json.loads(obj["Body"].read())
    props = item["properties"]
    if props.get("storage:tier") == actual_class:
        props["storage:verified_at"] = verified_at
    else:
        props["storage:tier"] = actual_class
        props["storage:retrieval_hint"] = HINTS[actual_class]
        props["storage:verified_at"] = verified_at
    s3.put_object(Bucket="spatial-archive", Key=item_key,
                  Body=json.dumps(item).encode(), ContentType="application/geo+json")

HINTS = {
    "STANDARD":     "immediate",
    "STANDARD_IA":  "immediate",
    "GLACIER_IR":   "immediate",
    "GLACIER":      "restore required, 3-5 h at standard tier",
    "DEEP_ARCHIVE": "restore required, up to 12 h at bulk tier",
}
Catalogue tier accuracy with and without reconciliation Two series over a month: without reconciliation accuracy decays steadily to seventy-one percent, while daily reconciliation keeps it between ninety-seven and one hundred percent. 100%85%70% with daily reconciliation — 97–100% without — decays to 71% day 1day 8day 16day 30 The decay rate follows the lifecycle schedule: every transition the rules perform is one more item the catalogue misdescribes. Reconciliation cost: one inventory scan and a rewrite of the few hundred items that actually changed.

Validation & Verification 

# Nothing advertised as immediate should actually be in an archive class
python -m archive.catalog audit --check tier --max-stale-days 2
# items checked        412,037
# tier mismatches            0
# stale (>2 days)            0
# AUDIT PASS

Expected output is zero mismatches and zero stale items. A non-zero stale count with zero mismatches means reconciliation is not running often enough, which is a warning rather than an error — but it is the state in which a mismatch will next appear unnoticed.

Troubleshooting

Symptom Root cause Fix
Mismatches appear in bulk on the same day A lifecycle rule fired across a prefix Expected; the reconciliation should absorb it that night
Items rewritten every night with no change verified_at written even when nothing changed Rewrite only when the tier differs; touch a separate freshness record otherwise
Reconciliation slower each month Full catalogue rewrite rather than a delta Join on the inventory and rewrite only differing items
Client still shows the old tier Static catalogue cached at a CDN Version catalogue paths or set a short cache lifetime on item documents
Restored objects advertised as immediate Restore expiry not tracked Treat Restore expiry from the inventory as part of the tier field

Operational Execution Checklist

What Else Drifts Alongside the Storage Tier

Storage class is the field that drifts most visibly, and it is not the only one. Three others change without a write event and are worth reconciling in the same pass, since the inventory join that detects one detects all four for the same cost.

Four item properties that drift without a write Four catalogue properties that change without an object write, and what each misleads a user about when stale. property changes when stale value misleads about storage tier a lifecycle rule fires whether the asset opens now restore state + expiry a restore lapses whether a restore is still valid file size the object is rewritten how large a download will be retention lock status a lock is applied or released whether the asset is protected One inventory join detects all four; reconciling only the first leaves three fields quietly wrong.

The restore-state row deserves particular attention in archives that serve restore-on-request workflows: a user who sees a scene marked as restored, and finds the temporary copy expired an hour ago, has been given worse information than if the field had never existed.

Frequently Asked Questions

Should the catalogue try to hide the storage tier from users?

No — it is the single most useful operational fact a search result can carry. A user planning an analysis needs to know whether the twelve scenes they matched are openable now or need a restore request and a four-hour wait. Hiding it does not remove the wait; it removes the ability to plan around it.

Can the catalogue trigger restores directly?

It can, and a request-and-notify workflow attached to the search interface is a good pattern for archives with genuinely cold holdings. Keep the trigger separate from the catalogue documents themselves: the catalogue describes state, the workflow changes it, and conflating the two makes a static catalogue impossible to publish.

How often does the tier field really need to be right?

As often as someone might plan work from it — daily is comfortable for most archives, and hourly is unnecessary because lifecycle transitions themselves are evaluated once a day. What matters more than the frequency is the recorded verification time, which lets a user judge how much to trust a value rather than assuming it is live.

Should the catalogue expose the restore mechanism as well as the state?

Where restores are self-service, yes — a link or an action alongside the tier field turns an obstacle into a workflow. Where they are not, exposing the state alone is still worth it, because a user who knows a scene is cold can plan around it or ask for it rather than waiting on a request that will never complete quickly.

How large is the reconciliation job for a big archive?

Proportional to the inventory rather than the archive: a million-object inventory is a few hundred megabytes of Parquet and joins in seconds. The write side is proportional to the number of items that actually changed, which after the first run is typically hundreds rather than millions. The job is small enough to run nightly without a scheduling conversation.

What happens to items for objects that no longer exist?

They should be removed or marked, and the reconciliation is where that is detected: an item whose asset key is absent from the inventory points at an object that was deleted or moved. Removing the item is right when the deletion was intended; investigating is right when it was not, and the reconciliation is often the only place an unintended deletion becomes visible.

Can the reconciliation run against a subset to save time?

It can, and it rarely needs to — the join is against an inventory the platform already produces, and it completes in seconds even for millions of items. Where a subset is used, rotate it so every collection is covered within a week, and be explicit in the freshness field about when each item was last checked rather than implying they all were.

What happens when an item’s asset moves to a different key?

The reconciliation reports it as an item pointing at a missing object, which is correct — a moved object is a new object as far as the catalogue is concerned. Republishing the item with the new key and retiring the old one is the clean response; rewriting the href in place is not, because it silently changes what a previously published identifier resolves to.

Should the reconciliation write to the catalogue or raise a change for review?

Write directly for the tier field, which is a fact rather than a judgement, and raise for review anything that implies an object is missing or unexpectedly modified. That split keeps the routine drift self-healing while ensuring the cases that might indicate a real problem reach a person.

Does the tier field belong on the item or on the asset?

On the asset, because an item can carry several assets in different classes — a cold source raster alongside a warm thumbnail is the normal case. Putting it on the item forces a single answer where there are several, and the thumbnail’s availability is exactly what a search interface wants to know.

Up one level: Metadata Cataloging & Discovery.

Related pages

Automating STAC Catalog Generation for Archived Imagery Extracting ISO 19115 Metadata from Legacy GeoTIFFs Indexing Archived Imagery Footprints for Fast Discovery