Spatial Data Archival · Cold Storage

Automating Fixity Audits with S3 Batch Operations

The inventory join covers every object whose checksum the platform already knows; everything else has to be read, and reading a few hundred thousand archived objects by hand is not a plan. This walkthrough is for the platform engineer turning that residual work into a scheduled batch job — one that restores what it must, hashes what it reads, writes the result back so the object never needs reading again, and reports in a form an archivist can act on. It assumes the manifest, inventory and rotating-slice design from the parent topic are already in place.

What the Batch Job Is For

Batch operations are the right tool for exactly one part of the audit: applying an operation to a large, enumerated list of objects with retries, throttling and a completion report. They are the wrong tool for the comparison itself, which is a query. Keeping that boundary clear is what stops the audit from becoming a distributed system.

Which part of the audit the batch job owns The inventory join covers most objects as a query at no read cost; the batch job handles only the read-required residue by invoking a function per object; reporting comes from the query side. query side · 97% of objects join manifest against inventory no object reads, no restores produces the audit report runs in minutes, costs cents batch side · 3% residue restore if the object is cold read, hash, write the checksum back retries and throttling handled by the service runs for hours, costs per object Each pass moves objects from right to left permanently: once a checksum is recorded, the object never needs the batch side again.

Step-by-Step Procedure

Step 1 — Generate the manifest of read-required objects

The manifest is a CSV of bucket and key, produced by the same query that runs the routine audit — filtered to objects the inventory cannot vouch for.

-- Objects with no platform checksum: this quarter's batch workload
UNLOAD (
  SELECT 'spatial-archive' AS bucket, m.object_key AS key
  FROM archive_manifest m
  JOIN spatial_archive_inventory i ON i.key = m.object_key
  WHERE m.audit_slice = 2 AND i.checksum_algorithm IS NULL
)
TO 's3://spatial-archive-logs/audit/slice-2-manifest/'
WITH (format = 'TEXTFILE', field_delimiter = ',', compression = 'NONE');

Step 2 — Write the per-object function

Batch operations invoke a function once per object. It restores when needed, hashes the bytes as they stream, and writes the checksum back so this object leaves the read-required set for good.

# lambda_function.py — invoked once per object by S3 Batch Operations
import hashlib, json, boto3, urllib.parse

s3 = boto3.client("s3")
ddb = boto3.client("dynamodb")
COLD = {"GLACIER", "DEEP_ARCHIVE"}

def handler(event, context):
    task = event["tasks"][0]
    bucket, key = task["s3BucketArn"].split(":::")[-1], urllib.parse.unquote(task["s3Key"])

    head = s3.head_object(Bucket=bucket, Key=key)
    if head.get("StorageClass") in COLD and "ongoing-request=\"false\"" not in head.get("Restore", ""):
        s3.restore_object(Bucket=bucket, Key=key,
                          RestoreRequest={"Days": 7, "GlacierJobParameters": {"Tier": "Bulk"}})
        return _result(task, "TemporaryFailure", "restore requested; retry next run")

    digest = hashlib.sha256()
    body = s3.get_object(Bucket=bucket, Key=key)["Body"]
    for chunk in iter(lambda: body.read(8 * 1024 * 1024), b""):   # stream: never buffer a 4 GB scene
        digest.update(chunk)
    observed = digest.hexdigest()

    expected = ddb.get_item(TableName="archive_manifest",
                            Key={"object_key": {"S": key}})["Item"]["sha256"]["S"]
    if observed != expected:
        return _result(task, "PermanentFailure", f"MISMATCH expected={expected} observed={observed}")

    # Record the checksum on the object so it is verifiable from metadata from now on
    s3.copy_object(Bucket=bucket, Key=key, CopySource={"Bucket": bucket, "Key": key},
                   ChecksumAlgorithm="SHA256", MetadataDirective="COPY")
    return _result(task, "Succeeded", "verified and checksum recorded")

def _result(task, code, msg):
    return {"invocationSchemaVersion": "1.0", "treatMissingKeysAs": "PermanentFailure",
            "invocationId": task.get("invocationId", ""),
            "results": [{"taskId": task["taskId"], "resultCode": code, "resultString": msg}]}

Returning TemporaryFailure for objects that need a restore is what makes the job self-pacing: the service retries them on the next run, by which time the bulk restore has completed.

Step 3 — Submit and schedule the job 

aws s3control create-job --account-id 123456789012 \
  --operation '{"LambdaInvoke":{"FunctionArn":"arn:aws:lambda:us-east-1:123456789012:function:archive-fixity"}}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},
               "Location":{"ObjectArn":"arn:aws:s3:::spatial-archive-logs/audit/slice-2-manifest/part-0000",
                           "ETag":"3f9c…"}}' \
  --report '{"Bucket":"arn:aws:s3:::spatial-archive-logs","Format":"Report_CSV_20180820",
             "Enabled":true,"Prefix":"audit/slice-2","ReportScope":"AllTasks"}' \
  --priority 5 --role-arn arn:aws:iam::123456789012:role/archive-batch --region us-east-1

Run it on a schedule with the slice number rotating, so the four quarterly runs cover the archive in a year.

Task outcomes across two runs of the same audit slice Run one shows most tasks returning temporary failure while restores complete. Run two, two days later, shows nearly all succeeding, with a small number of genuine mismatches and permission errors. Slice 2 · 41,800 read-required objects run 1 · day 0 succeeded 61% restore pending 34% Temporary failures are the restore wave working as designed — not an error rate. run 2 · day 2 succeeded 99.4% 0.4% permanent failures — 167 objects reporting a genuine checksum mismatch 0.2% permission errors — a prefix the batch role cannot read, fixed in policy The 167 mismatches are the audit's actual output; everything else is mechanics.

Validation & Verification

Two assertions make the job trustworthy: that it processed every object in the manifest, and that it would have reported a mismatch had one existed.

# Every manifest line accounted for in the completion report
wc -l < slice-2-manifest.csv
aws s3 cp s3://spatial-archive-logs/audit/slice-2/job-<id>/results/ - --recursive | wc -l

# Canary: a deliberately altered object must land as a PermanentFailure
grep -c 'MISMATCH' results.csv

Expected output is equal line counts and a canary count of at least one. A completion report shorter than the manifest means tasks were dropped rather than failed, which no result code will tell you.

Troubleshooting

Symptom Root cause Fix
Function times out on large scenes Buffering the whole object in memory Stream in chunks as shown; raise the timeout to match the largest object
Job stalls at a fixed percentage Restore quota reached for the account Lower job priority and split the slice; bulk restores are rate-limited
Every task fails with access denied Batch role lacks s3:GetObject on the prefix Grant it explicitly — the batch role is separate from the pipeline role
Checksum write-back creates new versions copy_object on a versioned bucket always does Expected; the bytes are identical, and lifecycle should expire noncurrent versions
Mismatches cluster in one prefix A processing job overwrote the objects Triage before restoring: this is an overwrite, not corruption

Operational Execution Checklist

Scheduling Against the Rest of the Archive’s Work

The audit competes for the same restore quota, request budget and network capacity as ingest, replication and user restores. Left unscheduled it will occasionally collide with all three, and the collision usually manifests as someone else’s job failing.

Where the audit fits in the archive's weekly workload A weekly schedule of ingest, replication, user restores and the fixity audit, with the audit placed in the weekend window where the others are quiet. MonTueWedThu FriSatSun ingest replication user restores fixity audit fixity audit window: Sat 06:00 → Sun 18:00 · lowest job priority, yields to everything else The audit has no deadline, which is exactly what makes it the workload that should give way — and what makes a weekend window sufficient.

Give the batch job the lowest priority in the account so it defers to a user restore that arrives mid-run, and size each slice so it completes inside the window rather than spilling into Monday’s ingest. An audit that finishes late is not a problem; an audit that starves the replication of restore quota is.

Frequently Asked Questions

Why not just run this on a fleet of workers instead?

Because the hard parts are retries, throttling, and per-object reporting at scale, and the batch service provides all three. A bespoke fleet reproduces them badly, and its failure modes — a worker dying with tasks in flight, a partial manifest, an unbounded retry storm against a restore quota — are exactly what the service handles. Keep the bespoke code to the per-object logic.

How much does a full pass cost?

Almost entirely in restores and requests rather than compute. A slice of 40,000 cold objects averaging 250 MB costs roughly $25 in bulk retrieval, $10 in requests, and single-digit dollars of function time. The number falls every pass as objects gain recorded checksums and leave the read-required set.

Should the audit run against the replica too?

Yes, on its own schedule and against the same manifest values. Auditing only the primary leaves the copy you would recover from unverified, which is the wrong copy to be surprised by. Because the replica is usually colder, its pass leans more heavily on the batch side, which is another reason to enable additional checksums on the replication path.

What should the job do when the manifest is larger than the batch limit?

Split it and submit several jobs, one per shard, with the same priority. The service caps manifest size, and a shard that fails can be resubmitted without re-running the others. Sharding also makes progress legible — five completion reports arriving through the weekend is a much better signal than one job that either finishes or does not.

Can the same job verify the replica?

Not the same invocation, because the manifest names a bucket. Run a parallel job against the replica’s inventory with the same per-object function and the same expected checksums from the manifest, which is what makes the two verifications independent. Sharing the function is the point; sharing the job is not possible and would not be desirable.

How are results fed back into the audit’s own records?

Write them from the function rather than parsing the completion report. The report is a per-task record designed for retries, not an audit trail, and it expires with the job. Recording each verification result against the object — verified at, by which pass, with which outcome — is what lets the next pass skip recently verified objects and what gives the annual report its coverage figure.

Can the job’s priority be raised if a pass is running late?

It can, and it usually should not. The audit has no deadline, and raising its priority puts it in contention with work that does. A pass that slips into the following week costs nothing; one that delays a user restore costs goodwill that the programme depends on.

Up one level: Archive Integrity Verification.

Related pages

Detecting Bit Rot in Long-Term Raster Archives Validating GeoTIFF Structure After a Cold Restore Verifying Checksums Across Spatial Archive Tiers