Spatial Data Archival · Cold Storage

Verifying Checksums Across Spatial Archive Tiers

Fixity is easy to run against hot objects and awkward against everything else, which is why most archives verify the newest 5% of their holdings continuously and the other 95% never. This walkthrough is for the archive operator who needs one procedure that covers every storage class an archive uses — including the Deep Archive objects that cannot be read without a restore — without turning the quarterly audit into a full retrieval of the archive. The technique is to verify from metadata wherever the platform can supply it, and to reserve object reads for the small sample where it cannot.

The Constraint That Shapes the Procedure

Checksums live in two places: recorded by the producer at ingest, and reported by the storage platform. Whether the platform can report one without reading the object depends on how the object was uploaded, not on its storage class — which is the fact that makes tier-wide verification affordable.

Which objects can be verified without being read Three upload cases and whether their checksum is available from metadata: single-part with an additional checksum, multipart with a composite checksum, and objects with only an ETag, which require a read. how the object was uploadedchecksum in the inventory?verifiable when cold? single-part + SHA-256 yes, full-object hashyes, from metadata multipart + SHA-256 yes, composite of partsyes, part-wise no additional checksum ETag only — not a content hash if multipartno — needs a restore Enabling additional checksums at ingest is what makes the whole archive verifiable later, at any temperature.

Step-by-Step Procedure

Step 1 — Record the checksum where the platform will report it

Upload with an additional checksum algorithm so the value ends up in object metadata and, from there, in the inventory report.

aws s3api put-object \
  --bucket spatial-archive \
  --key archive/imagery/2026/scene_0811.tif \
  --body scene_0811.tif \
  --checksum-algorithm SHA256 \
  --storage-class STANDARD_IA

# Record the same value in the archive's own manifest, independently of the object
sha256sum scene_0811.tif | awk '{print $1}' \
  | xargs -I{} aws dynamodb put-item --table-name archive_manifest \
      --item '{"object_key":{"S":"archive/imagery/2026/scene_0811.tif"},
               "sha256":{"S":"{}"},"ingested_at":{"S":"2026-08-11T09:14:00Z"}}'

Step 2 — Enable an inventory that carries the checksum column

The inventory is the mechanism that makes verification scale. Configure it to include the checksum and storage-class fields, and to write daily.

aws s3api put-bucket-inventory-configuration --bucket spatial-archive --id fixity \
  --inventory-configuration '{
    "Destination": {"S3BucketDestination": {
      "Bucket":"arn:aws:s3:::spatial-archive-logs","Format":"Parquet","Prefix":"inventory"}},
    "IsEnabled": true,
    "Id": "fixity",
    "IncludedObjectVersions": "Current",
    "Schedule": {"Frequency": "Daily"},
    "OptionalFields": ["Size","LastModifiedDate","StorageClass",
                       "ETag","ChecksumAlgorithm","ObjectLockRetainUntilDate"]}'

Step 3 — Join and compare, one slice at a time

The comparison is a query, not a job. Every object in the slice is checked regardless of class, and nothing is read.

SELECT m.object_key, m.sha256 AS expected, i.checksum AS observed, i.storage_class
FROM archive_manifest m
JOIN spatial_archive_inventory i ON i.key = m.object_key
WHERE m.audit_slice = 2
  AND (i.checksum IS NULL OR i.checksum <> m.sha256);

Step 4 — Read-verify only the residue

Objects the inventory cannot vouch for — legacy uploads without an additional checksum — form a shrinking backlog that must be read. Restore and hash them in waves, then write the checksum back so they never need reading again.

# Hash a restored object in place, without downloading it twice
aws s3api get-object --bucket spatial-archive \
  --key archive/legacy/2011/tile_4417.tif /dev/stdout | sha256sum
Verification coverage as the read-required backlog is retired Four quarters showing the share of objects verifiable from metadata rising from 62 to 97 percent as legacy objects are hashed and their checksums recorded, with the quarterly read cost falling accordingly. Share verifiable from metadata, by quarter Q1 $340 Q2 $196 Q3 $71 Q4 97% from metadata · 3% still read · $24 per pass Teal: verified from the inventory. Amber: required a restore and a read. Each pass permanently shrinks the amber band.

Validation & Verification

Confirm both that the pass covers what it claims and that it can fail. Coverage is the count of objects joined; failure detection is the canary.

python -m archive.audit run --slice 2 --report json | jq '{
  objects_in_slice, verified_from_metadata, read_verified, mismatches, unjoined
}'

Expected output shows unjoined: 0 — every object in the slice appeared in both the manifest and the inventory. A non-zero unjoined count is the failure that matters most and reports as a success in naive implementations, because a join simply drops the rows it cannot match.

Troubleshooting

Symptom Root cause Fix
observed is null for many objects Uploaded before additional checksums were enabled Read-verify once, write the checksum back via a copy-in-place with the algorithm set
Mismatch on every multipart object Comparing a composite checksum against a whole-file hash Compare part-wise, or store the composite value as recorded at upload
Inventory lags a day behind ingest Inventory is generated daily Verify objects older than 48 hours; do not audit today’s writes
unjoined count grows over time Objects written by a path that skips the manifest Fail the write gate when the manifest insert does not succeed
Deep Archive objects reported as unverifiable No additional checksum at upload Accept the read cost once, then record the value permanently
Whole-file versus part-wise checksum comparison A multipart object whose composite checksum cannot be reproduced by hashing the whole file, and the part-wise comparison that reproduces it correctly using the recorded part size. whole-file hash sha256(entire 4.1 GB object) compared against the composite always mismatches part-wise hash hash each 64 MB part, then hash the digests requires the recorded part size matches the platform value Record the part size in the manifest at upload — without it, a composite checksum cannot be reproduced at all.

Operational Execution Checklist

Cost of Verification Against Cost of Not Verifying

Verification budgets get cut because their benefit is invisible when nothing is wrong. Framing the comparison explicitly is what keeps the programme funded through the years in which it finds nothing.

Annual verification cost against the value it protects Three verification cost lines totalling under seven hundred dollars a year, set against the acquisition value of a single collection that could be lost undetected. 400 TB archive · annual verification cost inventory-based fixity $96 read-required backlog $340 first year, $24 thereafter structural sampling $210 total under $700 a year Against: one 8 TB aerial collection lost undetected $140,000 to acquire, and for a specific date and condition, not re-acquirable at any price.

The asymmetry is the argument, and it does not depend on corruption being likely. Verification is cheap because it is designed to read metadata rather than data; the loss it protects against is expensive because the material is unique. Present both numbers together and the programme survives its own success.

Frequently Asked Questions

Can a checksum be added to an existing object without rewriting it?

A copy-in-place with the checksum algorithm specified computes and stores one, producing a new object version whose bytes are identical. On an archive-class object that copy requires a restore first, which is why the backlog is worked in waves — but once done, the object is permanently verifiable from metadata.

Should the manifest live in the same account as the archive?

No. The manifest is the independent reference that makes verification meaningful, and keeping it in the same account under the same credentials as the data weakens that independence. A separate account, or at minimum a separate table with its own write path and its own backups, is the arrangement that survives a compromise of the archive account.

What about objects under a compliance-mode lock?

They can be read and copied but not overwritten, so a copy-in-place to add a checksum creates a new version rather than modifying the locked one — which is acceptable and leaves both versions retained. Where storage cost makes that unattractive, read-verify those objects and record the checksum in the manifest only.

How does the audit handle objects that are legitimately new since the last inventory?

By excluding them rather than failing on them. An object written after the inventory snapshot appears in the manifest and not in the inventory, which looks identical to a missing object. Filtering the slice to objects older than the inventory’s age removes the false positives without weakening the check, because those objects will be covered by the next pass.

Should the manifest record more than the checksum?

Yes — size, the multipart part size, the ingest timestamp and the producing pipeline version. Size and part size are needed to reproduce a composite checksum at all; the timestamp separates a corruption from an overwrite during triage; and the pipeline version is what lets a systematic defect be traced to the run that produced it. All four are known at write time and cost a few dozen bytes.

What is the right response to an object missing from the inventory entirely?

Investigate before assuming loss. The common causes are an object written after the snapshot, an object in a prefix the inventory configuration excludes, and a key whose encoding differs between the manifest and the report. Genuine disappearance is rare and, where versioning is enabled, usually recoverable from a noncurrent version — which is one more reason versioning belongs on every archive bucket.

How does the audit treat objects under a legal hold?

Exactly like any other object — a hold blocks deletion, not reading, and held material is often the most important to verify. The only practical difference is that a defect found in held material cannot be remediated by replacing the object, so the response is a documented correction record alongside it rather than a rewrite.

Should the audit verify object metadata as well as content?

The fields it depends on, yes: storage class, size and retention state all come from the same inventory row and cost nothing extra to check. Verifying them turns the fixity pass into a broader consistency check and catches the class of problem where the bytes are fine and the archive’s record of them is not.

Up one level: Archive Integrity Verification.

Related pages

Automating Fixity Audits with S3 Batch Operations Detecting Bit Rot in Long-Term Raster Archives Validating GeoTIFF Structure After a Cold Restore