S3 semantics
01S3 is an object ledger, not a shared folder.
An Amazon S3 route is shaped by object semantics: buckets and prefixes define scope, listings discover work, and metadata helps prove what was delivered. The design must distinguish a new object from a changed, duplicated, late, or partially processed workload.
Keys define the inventory
Define the bucket, prefix, inclusion rules, expected object metadata, and whether nested keys are part of the workload. Broad listings can create unnecessary work; narrow, explicit selection keeps intent reviewable.
A restart is object-specific
Treat each object as a recoverable unit and define what happens after a timeout, interrupted stream, or destination failure. Reconciliation should distinguish already completed objects from work that is safe to retry.
Versions change the contract
Agree on key naming, overwrite or version behavior, metadata, content type, retention, archive, quarantine, and source cleanup. These choices determine whether downstream users can interpret a successful delivery correctly.
Flower route design
02Turn object state into a controlled work plan.
The S3 adapter gives Flower the object operations needed to inspect properties, stream or range-read content, copy data, manage tiers, and walk a scoped inventory. The flow decides how those primitives become safe business work.
Inventory before movement
Listing and object properties establish the candidate set before bytes move. Prefix rules, metadata, size, and prior tracking evidence let the planner classify new, changed, frozen, duplicate, or already completed work.
Match the byte path to the object
Small objects, multi-gigabyte streams, ranged reads, and same-service copies do not need the same route. Flower can select a bounded path while transformations and validation remain in the controlled pipeline when required.
Prove delivery before cleanup
Destination properties, integrity checks, lineage, and reconciliation close the unit of work. Source deletion or archival follows that evidence; a failed validation goes to a visible recovery or quarantine path instead.
S3 production choices
03Make version, performance, and evidence policies explicit.
S3 scales easily enough to hide an imprecise contract for a long time. The production design must still state what identity means, how much parallel work is safe, and which proof authorizes destructive cleanup.
Version and delete policy
Decide whether a key, a specific version ID, or the latest visible object is the business identity. Define overwrite, delete-marker, retention, lifecycle, archive, and replay behavior before enabling source cleanup.
Concurrency with a cost ceiling
Model object-count and byte-volume distributions separately. Cap listings, transfers, multipart work, retries, and cross-region traffic; alert on age and backlog before an inexpensive object store creates an expensive recovery event.
Checksum and lineage evidence
Do not assume every ETag is a full-content MD5, especially after multipart upload. Record the checksum algorithm actually used, object size, key, version, destination identity, and final validation outcome in the delivery evidence.