Spatial Archive Cost Modeling: Retrieval, Penalties & Compression

Modeling the true cost of a geospatial archive means combining four numbers that vendor pricing pages present separately: per-gigabyte storage, retrieval and request fees, early-deletion penalties, and the compression ratio that shrinks all three. This reference is for the data engineers, cloud architects, and compliance teams who must defend a storage budget and prove that a tiering decision saves money rather than merely relocating it. Default per-GB comparisons are misleading — a tier that looks ten times cheaper can cost more once restore fees, minimum-duration billing, and read amplification against poorly sized objects are counted. The tables and formulas below give you a single, auditable model that every section of this knowledge base can reference.

Cost is not a property of a storage class; it is a property of an access pattern applied to a storage class. The same 40 TB LiDAR collection is cheap in Deep Archive if it is read once a year and ruinous if a monthly reprocessing job restores it. Everything here is built to be plugged into that access pattern.

The Total Cost Model #

Five cost components accumulate over an object’s life. Model them together, not in isolation:

The single most consequential lever is compression, because it reduces three of the five components at once. A 3:1 ratio on a vector archive does not just cut storage by two-thirds — it cuts every future restore and every egress byte by the same factor, compounding over the retention window. That is why compression tuning is a cost decision, not merely a storage decision, and why the Compression Tuning & Storage Optimization guides are upstream of every number here.

Storage Class Pricing Reference #

The table below lists representative US-region list prices for the storage classes a spatial archive typically spans. Prices drift; treat these as the model’s default coefficients and override them with your contracted rates. What rarely changes is the shape: each step down in price buys a step up in retrieval latency and minimum-duration commitment.

The per-gigabyte spread from Standard to Deep Archive is roughly 23:1, which is why aggressive cold-tiering is tempting. But the minimum-duration column is where naive models break: an object pushed to Deep Archive and deleted after 40 days still bills all 180 days. Choosing where an asset lands is the subject of Hot/Warm/Cold Tier Design for Geospatial Data, and the substrate that carries these classes is covered in Object Storage Selection for GIS Archives.

Early-Deletion Penalty Matrix #

Every tier below Standard bills a minimum number of days regardless of how long the object actually lived. The penalty for deleting — or transitioning — an object early is the remaining unmet days at that tier’s storage rate. Model it explicitly before setting any hot-to-warm or warm-to-cold transition trigger:

The practical rule this table encodes: never transition an asset into a tier whose minimum duration exceeds the time it will actually stay there. A dataset re-queried every 60 days should sit in Standard-IA, not Glacier — a too-aggressive rule that bounces it back to a hot tier on the next query pays the full 90-day Glacier minimum each round. Deriving transition days from real access telemetry rather than a static age cutoff is exactly the discipline in Setting Lifecycle Transition Thresholds from Query Telemetry.

Compression Ratio Reference #

Compression ratio converts raw dataset size into the compressed bytes every storage, retrieval, and egress fee is actually charged against. Representative ratios for spatial data, measured against uncompressed source:

Two attribute-level techniques amplify these ratios further: collapsing repetitive categorical columns with Dictionary Encoding for GIS Attributes, and matching the codec to the tier with the trade-offs in ZSTD vs LZ4 vs Snappy: Compression Trade-offs for Spatial Files. The exact level to run is governed by ZSTD Level Configuration for Spatial Files, because higher levels trade compress-time CPU for a smaller footprint — a cost you pay once against a saving you collect for the whole retention window.

A Worked Cost Model #

The following script models the total lifetime cost of an archive under a candidate tiering plan. It uses realistic coefficients and a real access pattern, so the output is directly comparable across plans.

# spatial_archive_cost.py — model lifetime cost of one dataset under a tiering plan
RATES = {
    "STANDARD":     {"gb_month": 0.023,  "min_days": 0,   "retrieval_gb": 0.0},
    "STANDARD_IA":  {"gb_month": 0.0125, "min_days": 30,  "retrieval_gb": 0.01},
    "GLACIER_IR":   {"gb_month": 0.004,  "min_days": 90,  "retrieval_gb": 0.03},
    "DEEP_ARCHIVE": {"gb_month": 0.00099,"min_days": 180, "retrieval_gb": 0.02},
}
EGRESS_GB = 0.09  # data transfer out to internet

def object_cost(raw_gb, ratio, plan, restores, egress_restores):
    """plan = list of (storage_class, days_resident); restores = GB read from cold."""
    comp_gb = raw_gb / ratio
    storage = requests = retrieval = penalty = 0.0
    for cls, days in plan:
        r = RATES[cls]
        storage += r["gb_month"] * comp_gb * (days / 30.0)
        if days < r["min_days"]:                      # early-deletion / transition penalty
            penalty += r["gb_month"] * comp_gb * ((r["min_days"] - days) / 30.0)
    retrieval = sum(RATES[c]["retrieval_gb"] * comp_gb for c in restores)
    egress = EGRESS_GB * comp_gb * egress_restores
    return round(storage + requests + retrieval + penalty + egress, 2)

# 2 TB vector estate, 6:1 GeoParquet, 1 yr Standard-IA then 4 yr Deep Archive,
# restored from Deep Archive twice with one full egress:
plan = [("STANDARD_IA", 365), ("DEEP_ARCHIVE", 1460)]
print(object_cost(raw_gb=2048, ratio=6.0, plan=plan,
                  restores=["DEEP_ARCHIVE", "DEEP_ARCHIVE"], egress_restores=1))

Run it and compare plans side by side:

python3 spatial_archive_cost.py

Expected output — the modeled five-year cost of the 2 TB estate under this plan:

228.53

Swap ratio=6.0 for ratio=3.0 and the same command returns roughly double, making the compression-versus-cost relationship concrete: the storage, retrieval, and egress lines all scale inversely with the ratio, while the request line does not.

What Actually Appears on the Bill #

Storage price per gigabyte is the line teams model and rarely the line that surprises them. A spatial archive’s invoice has five components, and their relative sizes depend far more on object count and access pattern than on total volume.

The request line is the one most often missing from a forecast, because it scales with object count rather than with volume and therefore does not appear in a model built from terabytes. An archive that halves its object size doubles that line without changing a byte of stored data — which is why partition resolution, row-group sizing and object consolidation belong in the cost model rather than in a separate performance discussion.

Cost Trade-off Analysis #

Reading the model’s output back into decisions:

• Storage dominates for cold, rarely-read data. For a 5-year Deep Archive object read once, storage is >90% of cost; shaving it with a higher compression level is the only meaningful lever.
• Retrieval dominates for warm, frequently-read data. If a dataset is restored monthly, retrieval and egress can exceed a year of storage — keep it in Standard-IA or Glacier Instant, not Flexible or Deep Archive.
• Penalties dominate for churning data. Assets rewritten or re-tiered before their minimum duration bleed penalty cost invisibly. Incremental update patterns like Incremental GeoParquet Updates Without a Full Rewrite exist partly to avoid rewriting cold objects that would trigger these penalties.
• Object sizing changes request and read amplification. Millions of tiny partitions inflate PUT/GET counts and lifecycle-transition request fees; oversized objects amplify partial-read restore cost. Aligning file and row-group size to the retrieval tier is covered in Sizing Row Groups for Glacier Retrieval of GeoParquet.

Unit Economics: Cost per Terabyte-Year and per Query #

Absolute monthly figures are hard to compare across archives and impossible to compare across years. Two normalised units make a cost model portable: the fully loaded cost of holding a terabyte for a year, and the cost of answering one typical query. Both are derived from the same components as the invoice, and both are stable enough to set targets against.

Cost per terabyte-year folds storage, the amortised share of ingest and conversion compute, and the standing integrity-verification reads into one number. Cost per query folds retrieval, requests and egress, divided by the query count over the same period. Reported together they expose the trade the archive is actually making: aggressive tiering pushes the first number down and the second up, and the right balance depends entirely on how often the archive is read.

Track both numbers per collection rather than for the archive as a whole. A single blended figure hides the collection that is being read a thousand times a year from Deep Archive, which is usually where the largest available saving sits — not in shaving the storage price of everything, but in moving one mis-tiered collection to the class its access pattern actually calls for. The full restore-cost arithmetic behind the per-query column is worked through in Modeling the Cost of a Full Archive Restore.

Forecasting a Growing Archive #

An archive’s cost is not a level but a slope, and the slope has two parts that behave differently. New data arrives at some rate and is billed from the day it lands; existing data ages into cheaper classes on the schedule the lifecycle rules set. A forecast that models only the first produces a straight line that overstates cost within two years; one that models both produces the characteristic flattening curve that lets a team commit to a budget.

The inputs are few and all measurable: the annual ingest volume, the compression ratio the pipeline achieves, the transition schedule, and the retention period after which anything is actually deleted. The output worth reporting is the year in which the cost curve flattens — the point where the volume ageing into archive classes matches the volume arriving — because that is the number a finance conversation actually turns on. Detailed multi-year projections, including how retention expiry eventually caps the curve, are built in Forecasting Multi-Year Storage Costs for Growing Imagery Archives.

Two sensitivities dominate any such forecast. Compression ratio scales the whole curve linearly, so a pipeline improvement that raises the ratio from 3:1 to 4:1 removes a quarter of every future year’s storage line — which is why compression tuning is a budget decision and not only an engineering one. Transition timing shifts the curve’s shape rather than its level: moving the hot-to-warm boundary six months earlier saves a predictable amount and pulls forward the minimum-duration clock on everything it touches, so the saving is real but smaller than the price difference suggests.

Compliance & Retention Cost Interaction #

Retention requirements set a floor under the model. When a regulatory mandate forces a 10-year hold, the cost question narrows to “the cheapest class whose minimum duration and retrieval SLA I can tolerate for 10 years.” Object Lock in COMPLIANCE mode, detailed in Configuring S3 Object Lock for Compliance Spatial Archives, removes early deletion from the table entirely for the locked window — which means the early-deletion penalty column becomes irrelevant and Deep Archive’s low storage rate wins outright, provided the ~12-hour restore is acceptable for audit response. Model the retrieval fee against the realistic number of audit or legal-discovery restores per year, not the theoretical maximum. Consult the official AWS S3 storage pricing documentation and S3 Storage Classes documentation for authoritative current coefficients before committing a budget.

Attributing Cost to the Teams That Cause It #

A single archive bill tells nobody what to do. The same total looks like a storage problem to the platform team, a query problem to the analysts, and an unavoidable cost to finance — and all three readings can be defended from the same invoice. Attribution is what turns the number into a decision, and it is a tagging exercise rather than an accounting one.

Three tag dimensions cover almost every question worth asking. A collection or dataset tag attributes storage and retrieval to the material itself, which is what identifies a mis-tiered collection. An owning-team tag attributes it to whoever decides how that material is used, which is what makes a query pattern someone’s to change. A retention-class tag separates the portion of the bill that is a business choice from the portion mandated by a statute or a contract — a distinction that matters enormously in a budget conversation, because one is negotiable and the other is not.

The mechanics are ordinary: apply the tags at ingest in the same job that writes the object, enforce their presence in the same gate that validates the reference system, and pull cost-allocation reports against them monthly. What makes the exercise worthwhile is the second-order effect. When a team can see that its habit of restoring a whole collection to answer a bounded question costs a measurable amount every month, the fix — a better partition layout, or a warm derivative — becomes something they ask for rather than something the platform team has to argue for.

Two attribution traps are worth naming. Shared reference data has no single owner and will otherwise land on whoever ingested it, so give it an explicit steward and treat its cost as an overhead line rather than distributing it. And restore costs are caused by the reader, not by the collection, so attribute them to the requesting team even though the storage line sits with the data — otherwise a well-run collection looks expensive because someone else keeps pulling it out of Deep Archive.

Where Cost Models Go Wrong #

Most archive cost models fail in one of four predictable ways, and all four are cheap to avoid once named.

The first is modelling volume and ignoring object count, which omits the request line entirely and understates the cost of a finely partitioned archive by a fifth or more. The second is using list prices without minimum billable sizes and durations, which is accurate for large objects and badly wrong for archives made of small ones — a 40 KB minimum applied to 12 KB tiles nearly triples their effective storage cost. The third is assuming a query rate rather than measuring one; access logs almost always show a distribution with a long tail of heavily read collections that the average conceals. The fourth is forecasting storage growth without modelling retention expiry, which produces a curve that rises forever and is therefore ignored by everyone who reads it.

A model that avoids those four is usually accurate to within about ten percent of the invoice, which is close enough to drive decisions. Beyond that point, the remaining error comes from things worth ignoring — regional price differences of a fraction of a cent, free-tier allowances, negotiated discounts — none of which change which lever to pull next.

Operational Execution Checklist #

• Model all five components — storage, requests, retrieval, penalties, and egress — never per-GB storage alone.
• Apply the real compression ratio measured on a representative sample, not a vendor headline number.
• Check every transition against minimum duration — never move an asset into a tier it will leave before the minimum-duration clock expires.
• Derive transition days from access telemetry, not static age cutoffs, to avoid penalty churn.
• Bound object and row-group size to the retrieval tier to control request counts and partial-read amplification.
• Re-run the model per storage class and pick the plan with the lowest modeled total, not the lowest headline rate.
• Refresh rate coefficients against contracted pricing at least annually and re-validate the chosen plan.

Frequently Asked Questions

Related #

• Compression Tuning & Storage Optimization — the ratios that scale three of the five cost components; upstream of every number here.
• Spatial Archival Architecture & Tiering Strategy — the tier model this cost reference prices out end to end.
• Format Conversion & Pipeline Automation — converting to columnar formats is what unlocks the compression ratios modeled above.
• Hot/Warm/Cold Tier Design for Geospatial Data — where the transition triggers this model prices are actually set.
• ZSTD Level Configuration for Spatial Files — tuning the compress-time CPU cost against lifetime storage savings.

Part of the Spatial Data Archival knowledge base.