Forecasting Multi-Year Storage Costs for Growing Imagery Archives

An archive’s cost is a slope, not a level, and the slope has two parts pulling in opposite directions: new data arriving and billed from day one, and existing data ageing into cheaper classes on the lifecycle schedule. A forecast that models only the first produces a straight line that overstates year three badly enough to be ignored. This walkthrough builds the forecast that finance will actually use, and identifies the year the curve flattens — the number the budget conversation turns on.

The Four Inputs #

Everything else in the model is arithmetic on these, and all four are measurable rather than assumed.

Step-by-Step Procedure #

Step 1 — Model each year’s cohort separately #

The trick that makes the forecast correct is treating each year’s ingest as a cohort that ages on its own clock, rather than modelling the archive as a single pool.

RATES = {"STANDARD": 0.023, "STANDARD_IA": 0.0125, "GLACIER": 0.0036}

def cohort_class(age_days: int) -> str:
    if age_days < 90:   return "STANDARD"
    if age_days < 395:  return "STANDARD_IA"
    return "GLACIER"

def forecast(years=10, annual_tb=62, growth=0.0, retention_years=25):
    rows = []
    for year in range(1, years + 1):
        monthly = 0.0
        for cohort in range(1, year + 1):
            age_years = year - cohort
            if age_years >= retention_years:
                continue                                   # expired and deleted
            size_gb = annual_tb * (1 + growth) ** (cohort - 1) * 1024
            monthly += size_gb * RATES[cohort_class(age_years * 365 + 180)]
        rows.append((year, monthly, monthly * 12))
    return rows

for y, m, a in forecast():
    print(f"year {y:>2}  ${m:>9,.0f}/month  ${a:>10,.0f}/year")

Step 2 — Read the shape, not just the endpoint #

Step 3 — Run the sensitivities that matter #

Two inputs dominate, and quantifying them turns the forecast into a decision tool rather than a number.

base = forecast()[-1][2]
for ratio in (3.0, 3.8, 4.5, 5.2):
    scaled = base * (3.8 / ratio)
    print(f"compression {ratio}:1 -> year-10 annual ${scaled:,.0f} "
          f"({(scaled / base - 1) * 100:+.0f}%)")

for warm_days in (30, 90, 180, 365):
    print(f"hot->warm at {warm_days} d: year-1 monthly changes by "
          f"{(90 - warm_days) / 365 * 62 * 1024 * (0.023 - 0.0125):+,.0f} $/mo")

Compression scales the entire curve linearly, so a pipeline improvement from 3.8:1 to 4.5:1 removes about a sixth of every future year. Transition timing changes the curve’s shape near the origin and matters much less by year five — worth knowing before someone proposes an aggressive re-tiering as a cost measure.

Validation & Verification #

python -m archive.cost forecast --years 10 --compare-actuals invoices/2024-2026.json
# year 1 modelled $17,520   actual $18,140   (+3.5%)
# year 2 modelled $24,960   actual $25,900   (+3.8%)
# year 3 modelled $28,080   actual $29,410   (+4.7%)
# drift is one-directional: check for an unmodelled term

Expected output is agreement within about five percent. One-directional drift, as above, indicates a missing term rather than noise — usually request charges, which scale with object count and appear in no volume-based model.

Troubleshooting #

Operational Execution Checklist #

• Measure annual ingest from load logs, not from a capacity plan
• Measure the compression ratio from written objects, not from a benchmark
• Model each year’s ingest as a cohort with its own age clock
• Charge the pre-transition period at the hot rate
• Apply retention expiry per collection
• Include a per-object request term alongside volume
• Report the year the curve flattens, not just the endpoint
• Reconcile against invoices quarterly and treat drift direction as a signal

Presenting the Forecast Without Overclaiming #

A ten-year forecast invites more confidence than it deserves, and presenting it with its uncertainty attached is what keeps it useful rather than merely reassuring. Three framings do most of that work.

Keep each year’s forecast alongside the actuals that followed it. After three years the series itself becomes the strongest evidence for or against the model, and it is far more persuasive to a finance reviewer than any argument about method.

Modelling Ingest Growth Honestly #

The forecast so far assumes a constant annual ingest, which is the right default and rarely the whole truth. Spatial archives grow for two reasons that behave differently: sensors improve, so the same survey produces more data each time it is repeated, and scope widens, so more things are surveyed. The first is a compounding multiplier on existing collections; the second adds new cohorts.

Sensor-driven growth is the larger effect and the easier to project, because it follows an acquisition programme that is usually documented. A LiDAR programme moving from 8 points per square metre to 20 over a decade triples the volume of an unchanged survey area, and that is knowable in advance from the procurement plan rather than extrapolated from history. Imagery follows the same pattern through ground sample distance: halving the GSD quadruples the pixel count.

Scope growth is harder and should be modelled as a scenario rather than a trend. Adding a new collection is a decision, not a rate, so the useful forecast presents a base case of the current programme and one or two scenarios naming specific additions with their expected volumes. That framing also makes the forecast useful in the other direction: it prices a proposed new collection before it is committed to, which is a question the archive is far more often asked than “what will next year cost?”.

Where growth genuinely is exponential — a continuously operating sensor network, for instance — model it explicitly with a compounding rate and be clear that the flattening effect of lifecycle transitions weakens as growth accelerates. At a high enough growth rate each year’s cohort is large relative to everything aged before it, and the curve stops flattening at all. Knowing where that threshold sits for your archive is more useful than any single projected figure.

Reconciling the Forecast With the Invoice #

A forecast earns its keep through the reconciliation rather than the projection. Comparing quarterly against actuals produces one of three outcomes, and each calls for a different response: noise, which is ignored; a level offset, which means a term is missing or mispriced; and a divergence that grows, which means an assumption has drifted.

The commonest missing term is request charges, which do not appear in a volume-based model and scale with object count. The commonest drifted assumption is the compression ratio, which changes quietly when a pipeline is upgraded or a new collection with different characteristics starts arriving. Both are diagnosable from the direction and shape of the gap without any additional instrumentation.

Keep each quarter’s comparison rather than only the latest. Over two or three years the series shows whether the model is systematically optimistic, which is a much stronger basis for the confidence intervals in the next forecast than any judgement about the inputs. It also, in the author’s experience, is what converts a finance reviewer from treating the forecast as an engineering estimate to treating it as a planning input — which is the point of building it.

Frequently Asked Questions

Using the Forecast to Decide, Not Just to Report #

A forecast that only answers “what will it cost?” is doing half its job. The same model answers several decision questions directly, and framing it that way is usually what gets it read.

The first is whether a proposed collection is affordable over its intended retention. Adding a cohort with its own volume, growth and retention produces a curve with and without it, and the difference is the collection’s true multi-year cost rather than its first-year storage line. For a collection with a twenty-five-year retention, that difference is often an order of magnitude larger than the number that appeared in the acquisition proposal — which is a fact better established before the commitment than after.

The second is whether a pipeline improvement pays for itself. A compression change that raises the ratio from 3.8:1 to 4.5:1 removes a sixth of every future year’s storage, and the model prices that directly against the engineering effort. In the author’s experience this comparison usually favours the improvement by a wide margin for archives past a few hundred terabytes, and the model is what makes the case concrete rather than plausible.

The third is whether the retention policy is doing what the institution believes. Modelling the curve with per-collection retention against a single global retention shows the volume that a differentiated policy sheds over time, and it is frequently the largest single lever available — larger than compression, larger than tiering, and requiring no engineering at all, only a decision about what genuinely needs keeping for how long.

The fourth is when to revisit the storage-class mix. Because the model already ages each cohort through the transition schedule, changing the schedule and re-running it prices an earlier or later transition exactly. That converts a discussion about whether ninety days is the right threshold into an arithmetic question with a stated answer and a stated sensitivity.

Present the forecast with those four questions answered rather than as a table of years. The table is the evidence; the answers are what the forecast was built to produce, and a reviewer who sees them is far more likely to act on the model than one handed a projection to interpret.

How long should each year’s forecast be kept? #

Indefinitely, alongside the actuals. The value compounds: after five years the series shows whether the model is biased and by how much, which is the only honest basis for the confidence attached to the next projection. It costs a few kilobytes a year and it is the difference between a model that is trusted and one that is merely produced.

Does the forecast need to model egress? #

Only if the archive serves data outside its region, and then as a separate line driven by the read rate rather than by volume. Egress is unusual among the cost terms in being almost entirely under the archive’s control — colocating compute removes it — so modelling it separately keeps that lever visible rather than burying it in a storage projection.

What is the smallest useful version of this forecast? #

Three inputs and one loop: annual ingest, the transition schedule, and the retention period, modelled as cohorts. That produces the flattening curve, which is the entire point, and it can be written in twenty lines. Everything else in this guide is refinement, and an archive with the simple version is far better placed than one waiting for a complete model.

How should a forecast be revised when an assumption is proven wrong? #

By re-deriving it rather than patching the output. A model whose compression assumption has moved needs that input changed and the whole curve regenerated, because the change propagates through every cohort. Adjusting the final numbers to match last quarter’s actuals produces a figure that matches the past and predicts nothing, which is the commonest way a forecast quietly stops being a model.

Does the forecast need to account for storage-class deprecations? #

Occasionally, and it is worth a sentence in the assumptions. Storage classes are added far more often than removed, but a class the archive depends on being retired would force a migration with its own cost and timeline. Naming that as an unpriced risk rather than modelling it keeps the forecast honest without inventing a number for something that has not happened.

Should the forecast be presented per collection or for the archive as a whole? #

Both, with the per-collection view leading. The whole-archive figure answers the budget question; the per-collection breakdown is what identifies the one holding whose growth or retention is driving it, which is the only actionable part of the exercise.

What if the archive’s growth rate is genuinely unknown? #

Model two bounded scenarios rather than one guess — the current programme continuing unchanged, and the most expansive commitment currently under discussion. The pair brackets the plausible range, and the gap between them is itself informative: where it is narrow, the growth question does not need resolving before a budget can be set.

Related #

• Spatial Archive Cost Modeling — the parent reference where this forecast supplies the storage curve.
• Modeling the Cost of a Full Archive Restore — the sibling model for the retrieval side.
• Setting Lifecycle Transition Thresholds from Query Telemetry — deriving the transition schedule this model takes as an input.
• Compression Tuning & Storage Optimization — the lever that scales the whole curve.

Up one level: Spatial Archive Cost Modeling.