Failure classification
01Decide what failed before deciding what to repeat.
Reliable automation does not send every error into the same retry loop. Flower keeps transport conditions, data-quality outcomes, and uncertain destination results as different operating states with different next actions.
Transient and retryable
Timeouts, throttling, and temporary unavailability can use bounded attempts, backoff, jitter, and a retry budget. Exhaustion becomes a visible backlog or alert, never an infinite background loop.
Permanent and actionable
Invalid credentials, missing permissions, incompatible schema, or a failed business rule do not improve through repetition. Stop the affected unit, preserve context, quarantine when appropriate, and notify the right owner.
Uncertain and reconcile-first
A connection can fail after the destination committed. Flower checks destination identity, tracking evidence, counts, or checksums before replay so an unknown outcome does not automatically become duplicated work.
Recovery control loop
02Advance only after the unit is safe and evidenced.
The same control loop follows data from source to destination. Routine transient recovery runs automatically; unsafe or uncertain outcomes retain context, follow configured policy, and raise an actionable signal only when operator judgment is needed.
Choose the restart boundary
Use an object, file, deterministic row batch, or other smallest safe unit. Store the checkpoint outside transient execution state and advance it only after destination confirmation.
Gate progress with validation
Transport completion is one signal. Structure, schema, counts, business rules, integrity, and destination reconciliation decide whether the unit may be published, retained for review, or rejected.
Keep recovery evidence attached
Record attempts, classification, source and destination identity, validation outcome, quarantine, reconciliation, and final action in one lineage path. Alerts carry enough context for an operator to act.
Shaped by production
03Reliability earned under real data pressure.
Reliability is not a slide-deck property. Flower has evolved around continuous production workloads since 2019, including very large telecommunications volumes and business-critical financial data.
Failure is explicit
Incomplete transfers, invalid records, and policy violations become visible states with defined handling instead of silent downstream defects.
Recovery is bounded
Retry budgets, backoff, restart boundaries, quarantine, destination reconciliation, and lifecycle actions adapt to the endpoint and workload rather than being improvised during an incident.
Evidence stays attached
Lineage, metadata, events, metrics, and alerts help operators reconstruct what moved, what changed, and what needs attention.
Recovery acceptance
04Specify recoverability before the first incident.
A claim of self-healing becomes operational only when the team agrees what may be repeated, what evidence closes a unit, how uncertainty is reconciled, and when automation must stop. Turn those expectations into release tests and repeat them whenever endpoints or policies change.
Write the recovery contract
For each unit, state permitted loss, duplication, ordering change, maximum delay, and required destination evidence. A precise contract prevents teams from calling a restarted process successful while the data outcome remains unknown.
Inject distinct failure classes
Test source unavailability, transport interruption, throttling, invalid credentials, schema rejection, corrupt payload, full local storage, and target timeout separately. Each should reach the intended retry, stop, quarantine, or reconciliation path.
Force an ambiguous commit
Drop the connection after the destination may have committed but before confirmation reaches the sender. Verify that destination identity, counts, checksums, or another durable signal are checked before any replay is allowed.
Restart across the state boundary
Restart the worker, host, and dependent state service at different points. Confirm that checkpoints, pending attempts, quarantine context, and lineage survive together, and that an upgrade or rollback does not reinterpret in-flight work.
Measure pressure and catch-up
Hold an endpoint unavailable long enough to build a realistic backlog, then restore it. Measure queue age, disk growth, retry rate, useful catch-up throughput, resource saturation, and whether new traffic is starved or remains protected.
Prove quarantine and replay
Send invalid and policy-violating units beside valid work. Confirm that unsafe data stops without blocking unrelated progress, keeps enough context for diagnosis, and can be corrected and replayed without bypassing validation or duplicating completed units.
Prove durable-state restoration
Remove or corrupt the active state store and restore it from the documented backup or replica. Measure the recovery-point gap, reconcile source and destination evidence, and confirm that the rebuilt checkpoint neither skips eligible work nor silently republishes completed units.
Test the operator handoff
Exhaust a retry budget and trigger a permanent failure. The alert should identify the affected unit, source and destination, last safe boundary, attempts, evidence, owner, and approved next actions without requiring reconstruction across unrelated tools.
Gate releases with recovery drills
Keep the failure matrix, expected evidence, and recovery timings as a repeatable suite. Run it after runtime, connector, schema, policy, or infrastructure changes so recoverability is a tested release property rather than historical confidence.