01Explore the recovery policy
- 01SourceAmazon S3
CSV · gzip - 02Validate, filter, aggregate and transformFlower
CSV → Parquet - 03DestinationAzure Blob
Parquet
SourceAmazon S3
DestinationAzure BlobThe sample passes validation and is converted to Parquet. The illustration shows the completed delivery path; actual acceptance checks depend on the configured endpoint and transformation.
A temporary destination failure pauses delivery. Retry-safe work follows bounded attempts and backoff. Restore the destination to explore recovery; exhausted policies require operator attention.
The sample contains an invalid record. This scenario quarantines the entire source file and blocks downstream delivery. Retrying unchanged data would not fix the validation failure.
The destination is available again and the retry-safe operation completes. Source cleanup remains a separate configured decision. This is an illustrative sequence, not a measured recovery time.
02What to verify in your environment
| Scenario | Acceptance check |
|---|---|
| Normal delivery | Check that the Parquet output is readable and matches the agreed schema and expected records after transformation. |
| Temporary outage | Make the destination temporarily unavailable. Observe bounded retries and backoff for retry-safe operations, then check escalation when the retry policy is exhausted. |
| Invalid file | Submit a file with a record that violates a validation rule. Check that the whole source file is quarantined, with its failure reason, and that invalid data is not delivered downstream. |
| Recovered within the retry policy | Restore the destination within the retry window and verify the completed output. Check source retention or cleanup against its separately configured policy. |
03From illustration to a production flow
For the S3-to-Azure scenario, agree endpoint permissions, decompression, the CSV schema, validation rules, Parquet mapping and quarantine destination. Test temporary failures independently from malformed data. Completion verification is conditional on the copy path and provider capabilities; transformed data needs an output acceptance check, not an assumed source-byte match.