Setting Lifecycle Transition Thresholds from Query Telemetry

Setting lifecycle transition thresholds from query telemetry replaces arbitrary age cutoffs with data-driven rules derived from how often each spatial prefix is actually read, so imagery and vector tiles move to warm and cold storage on their own measured decay curve instead of a guessed 90-day clock. This guide is for cloud architects and data engineers who own a geospatial archive large enough that a wrong threshold costs real money — either in retrieval fees when live assets get cold-tiered too early, or in storage waste when dead assets linger on hot storage. Static lifecycle rules fail because different asset classes decay at wildly different rates: a raster mosaic backing an active basemap may be queried heavily for a year, while a one-off vector export goes silent in weeks. The remedy is to mine S3 server access logs, compute a per-prefix access-recency profile, and generate lifecycle rules whose transition days match observed behavior.

Why One Cutoff Cannot Fit Two Decay Curves #

A single static threshold is either too aggressive for slow-decaying assets or too lax for fast-decaying ones. Telemetry lets each prefix transition at its own crossing point:

Mining Access Telemetry per Prefix #

The raw signal is the archive’s own read history. AWS S3 server access logs record every GET with a timestamp, key, and operation, and they are the ground truth for how live each object really is. Enable them if they are not already flowing, then aggregate.

1. Confirm access logging targets a durable log bucket:

aws s3api get-bucket-logging --bucket spatial-archive
# Expect a LoggingEnabled block pointing at s3://spatial-archive-logs/access/

If empty, enable it — you cannot derive thresholds from telemetry you never captured, and there is no retroactive fix. Give the log accumulation at least one full decay cycle (typically 90+ days) before trusting the derived thresholds.

2. Aggregate reads per top-level prefix and asset age. DuckDB parses the space-delimited access log format directly and lets you bucket by prefix and by the days elapsed between object creation and each read:

-- duckdb: reads per prefix, bucketed by object age at read time
SELECT
  regexp_extract(key, '^([^/]+/)', 1)          AS prefix,
  date_diff('day', create_ts, request_ts)      AS age_days,
  count(*)                                      AS reads
FROM read_csv('s3://spatial-archive-logs/access/*', sep=' ', header=false,
              columns={'request_ts':'TIMESTAMP','key':'VARCHAR',
                       'operation':'VARCHAR','create_ts':'TIMESTAMP'})
WHERE operation = 'REST.GET.OBJECT'
GROUP BY prefix, age_days
ORDER BY prefix, age_days;

3. Find the age where each prefix’s read rate crosses your warm-eligibility threshold. Fit the per-prefix decay and locate the crossing day, guarding against sparse tails that produce noisy fits:

import numpy as np

def transition_day(age_days, reads, floor_frac=0.15):
    peak = reads.max()
    norm = reads / peak
    below = np.where(norm < floor_frac)[0]
    # first age at which normalized read rate stays under the floor
    return int(age_days[below[0]]) if len(below) else int(age_days.max())

vector_day = transition_day(vec_ages, vec_reads)   # e.g. 76
raster_day = transition_day(ras_ages, ras_reads)   # e.g. 266

Two adjustments keep the derived day trustworthy. The floor_frac parameter sets how quiet a prefix must go before it is warm-eligible; a value near 0.15 tolerates a long low-activity tail, while a stricter 0.05 waits for near-silence and is appropriate for assets whose late reads are expensive to re-hydrate. Seasonality is the other trap: agricultural imagery, flood-season hydrology layers, and academic-calendar datasets spike annually, so a threshold fitted over a single quiet quarter will cold-tier assets weeks before their next predictable surge. Fit over at least a full seasonal cycle and, where a prefix shows clear periodicity, add a guard that suppresses a transition inside a known busy window rather than trusting the raw decay curve.

Cold-start is the honest limitation. A brand-new prefix has no history, so there is nothing to fit — do not invent a telemetry-driven threshold from a week of data. Seed new prefixes with a conservative static rule, let the access logs accumulate, and switch to a derived threshold only once the log window covers a real decay cycle. Where per-object access logs are too voluminous to retain, S3 Storage Lens and last-access analytics give a coarser but cheaper signal that still distinguishes a live prefix from a dead one.

Reading a Prefix’s Decay Curve #

The telemetry query produces a distribution, and the threshold falls out of one property of it: the age beyond which the marginal read rate no longer justifies the higher storage price. Plotting reads per object against age makes the crossing visible and turns the threshold from an argument into a reading.

Two cautions when reading a curve like this. Aggregate reads by object rather than by request, or a single client retrying a failed download will look like sustained demand. And check the tail rather than only the crossing: a prefix whose curve flattens well above zero has a long-lived minority of objects that should be excluded from the rule by tag, because tiering them will generate restores forever.

Generating Per-Prefix Lifecycle Rules #

Each derived crossing day becomes the transition.days of a prefix-scoped lifecycle rule. Generating the Terraform from the telemetry output keeps the policy honest — the archive’s rules trace directly back to measured behavior, which is exactly the telemetry-driven tiering that a Hot/Warm/Cold Tier Design for Geospatial Data demands over guesswork.

# Terraform: per-prefix lifecycle rules derived from query telemetry
resource "aws_s3_bucket_lifecycle_configuration" "telemetry_driven" {
  bucket = aws_s3_bucket.spatial_archive.id

  rule {
    id     = "vector-fast-decay"
    status = "Enabled"
    filter { prefix = "vector/" }
    transition {
      days          = 76   # measured warm-eligibility crossing
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 210  # sustained silence past the warm window
      storage_class = "GLACIER"
    }
  }

  rule {
    id     = "raster-slow-decay"
    status = "Enabled"
    filter { prefix = "raster/" }
    transition {
      days          = 266  # raster stays hot far longer than a static rule assumes
      storage_class = "STANDARD_IA"
    }
  }
}

Set a floor on any generated threshold below the storage class minimum-duration billing window: never emit a STANDARD_IA transition earlier than 30 days or a GLACIER transition that will be undone before its 90-day minimum, or the early-deletion penalty erases the saving. Model those penalties against real read volumes with the Spatial Archive Cost Modeling framework before applying. The threshold is a policy input; the cost model is the sanity check that keeps an aggressive rule from costing more than the storage it saves.

Verifying the Rules Took Effect #

Applying Terraform proves the configuration exists, not that objects are transitioning correctly. Confirm both the rule set and the actual storage class of aged objects.

aws s3api get-bucket-lifecycle-configuration --bucket spatial-archive \
  --query "Rules[].{id:ID, prefix:Filter.Prefix, days:Transitions[0].Days}"

Expected output — each prefix carries its telemetry-derived day, not a shared constant:

[
  { "id": "vector-fast-decay", "prefix": "vector/", "days": 76 },
  { "id": "raster-slow-decay", "prefix": "raster/", "days": 266 }
]

Then spot-check that an object past its transition day actually moved, and that a still-young object did not:

aws s3api head-object --bucket spatial-archive --key vector/2024/parcels_region_north.fgb \
  --query "StorageClass"

An object created more than 76 days ago under vector/ should report STANDARD_IA; a null or STANDARD result on an aged object means the rule’s filter prefix does not match the real key layout — the most common cause of a lifecycle rule that silently does nothing.

Guardrails Around an Automated Threshold #

A threshold computed from telemetry and applied automatically is an improvement over a guess and a hazard without limits. Three guardrails keep a bad month of data from re-tiering the archive.

Transitions are one-way in cost terms: moving an object to a colder class starts its minimum-duration clock, and moving it back costs both the early-deletion charge and a retrieval. That asymmetry is why the guardrails are deliberately conservative in the direction of colder, and why a threshold that turns out to be too aggressive is more expensive than one that is too cautious.

Diagnosing Threshold Misfires #

Operational Execution Checklist #

• Confirm S3 server access logging is enabled and has captured at least one full decay cycle before deriving thresholds.
• Aggregate reads per top-level prefix bucketed by object age at read time, not by wall-clock date.
• Compute each prefix’s warm-eligibility crossing from normalized read rate, guarding against sparse-tail noise.
• Generate prefix-scoped lifecycle rules whose transition.days come straight from the telemetry output.
• Clamp every generated threshold to the storage class minimum-duration window to avoid early-deletion penalties.
• Validate applied rules with get-bucket-lifecycle-configuration and spot-check aged objects with head-object.
• Re-derive thresholds on a schedule so the policy tracks changing access patterns instead of ossifying.

Frequently Asked Questions

Related #

• Up: Hot/Warm/Cold Tier Design for Geospatial Data — the parent reference for tier definitions and retrieval SLAs these thresholds drive.
• How to Design a 3-Tier Spatial Storage Architecture — the tier model whose transition edges these telemetry-derived days populate.
• Implementing Lifecycle Rules for Shapefile Archives — the retention-side counterpart where lifecycle rules meet compliance holds.
• Tuning ZSTD Compression for GeoParquet Archives — shrinking the warm-tier footprint that these thresholds route assets into.
• Spatial Archive Cost Modeling — pricing each candidate threshold against retrieval and early-deletion penalties.

Part of the Spatial Data Archival knowledge base.