Sizing Partitions to Avoid Small-File Overhead in Cold Storage
Partition pruning gets better as partitions get smaller, right up to the point where the archive is paying more in per-object charges than it saves in avoided reads. This walkthrough is for the engineer who has chosen a partitioning scheme and now needs to pick the resolution that minimises total cost rather than bytes scanned — a different optimum, and one that moves with the storage class the data will live in.
The Three Charges That Scale with Object Count
Volume-based reasoning misses these entirely, which is why an archive can be optimised for scan efficiency and still cost more than the version it replaced.
Step-by-Step Procedure
Step 1 — Model total cost, not bytes scanned
# Monthly cost of one partitioning resolution, all terms included
def monthly_cost(total_tb, objects, queries_per_month, partitions_per_query,
storage_per_gb=0.0036, restore_per_gb=0.01,
restore_req=0.05 / 1000, overhead_kb=40):
stored_gb = total_tb * 1024 + objects * overhead_kb / 1e6
storage = stored_gb * storage_per_gb
gb_per_partition = total_tb * 1024 / objects
restored_gb = queries_per_month * partitions_per_query * gb_per_partition
retrieval = restored_gb * restore_per_gb
requests = queries_per_month * partitions_per_query * restore_req
return {"storage": storage, "retrieval": retrieval,
"requests": requests, "total": storage + retrieval + requests}
for res, objs, ppq in [("res 4", 32, 1), ("res 5", 210, 2), ("res 6", 1_400, 4),
("res 7", 9_800, 11), ("res 8", 68_000, 34)]:
c = monthly_cost(450 / 1024 * 1024, objs, 4_000, ppq)
print(f"{res:>6} objects={objs:>6} total=${c['total']:8.2f} "
f"(storage ${c['storage']:.2f} · retrieval ${c['retrieval']:.2f} · requests ${c['requests']:.2f})")The partitions_per_query column is what makes finer resolutions worse rather than better past a point: a query covering a fixed ground area intersects more partitions as they shrink, so requests grow while each restore shrinks.
Step 2 — Find where the curve turns
Step 3 — Merge sparse partitions rather than lowering the resolution globally
Population-weighted data produces partitions differing by two orders of magnitude at one resolution. Merging the sparse ones preserves pruning where the data is dense and removes the object count where it is not.
# Merge neighbouring partitions below the floor, keeping the parent cell as the key
import h3, collections
sizes = {cell: bytes_for(cell) for cell in partitions} # from the inventory
FLOOR = 200 * 1024**2
merged, buckets = {}, collections.defaultdict(list)
for cell, size in sizes.items():
if size >= FLOOR:
merged[cell] = [cell]
else:
buckets[h3.cell_to_parent(cell, h3.get_resolution(cell) - 1)].append(cell)
for parent, children in buckets.items():
merged[parent] = children # one object per parent, named by the parent cell
print(f"{len(sizes)} partitions -> {len(merged)} objects")Validation & Verification
# Object size distribution after merging: nothing below the floor, nothing above the ceiling
aws s3api list-objects-v2 --bucket spatial-archive --prefix archive/parcels/2026/ \
--query 'Contents[].Size' --output text | tr '\t' '\n' \
| awk '{s[NR]=$1} END {n=asort(s); print "min", s[1]/1048576 "MB",
"p50", s[int(n/2)]/1048576 "MB", "max", s[n]/1048576 "MB", "count", n}'Expected output shows a minimum above the merge floor, a maximum below the object ceiling, and a count close to the modelled optimum. A minimum far below the floor means the merge step did not run on some prefix, which is the usual cause of a cost model that stops predicting the invoice.
Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| Request charges rising faster than volume | Resolution too fine for the query pattern | Model total cost; move one resolution coarser or merge sparse cells |
| A few enormous partitions | Population-weighted density at a uniform resolution | Split only the outliers; do not raise the resolution globally |
| Merged partitions break pruning | Merge key not derivable from the query’s cell | Merge to the parent cell so the hierarchy still resolves |
| Cost model diverges from the invoice | Per-object overhead or minimum billable size omitted | Add both terms; they are invisible in a volume-only model |
| Optimum shifts after a class change | Restore and request prices differ per class | Re-run the model whenever the storage class changes |
Operational Execution Checklist
Monitoring Object Size After the Decision
Partition size drifts as data grows, and an archive that sized its partitions once will find them outside the target window a few years later. Three measures, reported monthly from the inventory, catch the drift while it is still cheap to correct.
Report them per collection rather than for the archive as a whole. A blended median hides the one collection whose growth has pushed it out of the window, which is exactly the collection the monitoring exists to find.
Frequently Asked Questions
What object size should the merge floor be set to?
Between roughly 200 MB and 400 MB for archive classes, which keeps per-object overhead under a fraction of a percent and keeps restore counts manageable. Below about 50 MB the overhead and request charges start to be visible in the invoice; below the class’s minimum billable size they dominate outright.
Does merging hurt pruning?
Slightly, and much less than the object-count saving is worth. A merged object covers a parent cell rather than a child cell, so a query that would have touched one child now touches its siblings too — but those siblings were sparse by definition, so the extra bytes are small. The pruning that matters is in the dense partitions, which are never merged.
How does this interact with row-group sizing?
They are the same trade at two scales. Object size governs what a restore costs; row-group size governs what a read of the restored copy costs. Choose the object size first from the cost model, then size row groups inside it against the scan pattern — which for archive-class data means larger groups, since partial reads are not available until the restore completes.
Does the optimum shift as query patterns change?
Yes, and query rate moves it more than query shape. An archive read four thousand times a month has its floor further right — larger partitions, fewer objects — than the same archive read forty thousand times, because retrieval scales with reads while per-object overhead does not. Re-run the model when the read rate changes by more than about a factor of two.
How does compaction fit with partition sizing?
Compaction is what maintains the sizing decision over time. Partitions grow as data is added and fragment as data is corrected, so the distribution drifts away from the target window without anyone changing the resolution. A scheduled compaction that rewrites partitions outside the window keeps the archive at the size it was designed for.
Is there a lower bound below which no archive should go?
Below roughly 50 MB per object the per-request and overhead terms start to dominate in every archive class, and below the class’s minimum billable size they dominate outright. That gives a practical floor around 100 MB for archive-class storage, well above the technical minimum and well below the point where restores become coarse.
Does the model apply to the delivery copies as well as the archive?
The structure does, with different rates. A delivery copy in an instant-access class has no restore term and higher per-gigabyte storage, which moves the floor toward smaller objects because retrieval is cheap and selectivity is worth more. Running the same model with the delivery class’s rates usually produces a partition resolution one step finer than the archive’s, which is a legitimate difference rather than an inconsistency.
How is the model kept honest as prices change?
By keeping the rates in a configuration file rather than in the code, and by re-running the model annually against current published prices. Rate changes are infrequent and occasionally significant, and a model whose inputs were correct three years ago will quietly recommend the wrong resolution without ever failing.
Does the same reasoning apply to the number of files within a partition?
Yes, one level down. A partition made of many small files pays the same per-request overhead as an archive made of many small partitions, and compaction is the same remedy. Size files against the byte target and partitions against the query extent, and the two decisions stay independent.
Related
- Spatial Partitioning Techniques — the parent topic covering scheme choice and resolution.
- Partitioning Cold Spatial Archives by Administrative Boundary — the split-and-merge rule applied to boundary keys.
- Sizing Row Groups for Glacier Retrieval of GeoParquet — the same trade inside the object.
- Spatial Archive Cost Modeling — the pricing model this guide’s arithmetic comes from.
Up one level: Spatial Partitioning Techniques.