Auto-retry failed Airbyte syncs before paging on-call
Every 30 minutes, quietly re-run any failed Airbyte sync on a critical connection once, and only page the on-call in PagerDuty and Slack if the retry still fails.
Integrations
Airbyte
PagerDuty
Slack Bot
Type
Categories
- Operations
- Engineering
Run this workflow on a cron every 30 minutes. Its job is to give critical Airbyte pipelines exactly one automatic retry before a human is paged, since Airbyte's own auto-disable only kicks in after 14 straight days of failures which is far too slow for pipelines the business actually depends on.
Start each run by calling Airbyte List Jobs with a status filter of failed, scoped to the last hour. For every failure returned, decide whether the underlying connection is tagged critical (only critical connections are in scope for this workflow, everything else should be ignored).
Before retrying, inspect the failure reason. If it looks like an authentication problem (expired token, invalid credentials, 401/403) or a schema mismatch (destination column missing, incompatible type, normalization error), skip the retry entirely and jump straight to the escalation path below, because those failures will not fix themselves. Also skip the retry if the same connection has already been retried by this workflow in the last hour, so we do not hammer a broken source.
Otherwise, kick off one retry with Airbyte Trigger Sync or Reset Job (jobType=sync) for that connection, wait a short interval to give the job time to complete, then call Airbyte Get Job Details on the new job id to see the outcome. If the retry succeeded, log the recovery and move on silently.
If the retry also failed (or the failure was skipped as auth/schema), escalate. Call PagerDuty Create Incident on the data-ops service, using a title that names the Airbyte connection and a body that includes the failure reason, the failing job id, and a link back to the connection in Airbyte. Then use Slack Bot Send a Message to post in the #data-ops channel, referencing the PagerDuty incident URL and the connection name so the on-call can jump straight to the right place.
A note on Airbyte Cloud auth: the cloud API's access tokens expire in roughly 3 minutes, so mint a fresh token at the start of each burst of calls rather than caching one across the whole 30-minute cycle.
Keep the output quiet on the happy path. A run that retries three failures and recovers all three should produce no Slack noise at all. The value of this workflow is that humans only hear from it when something is actually broken.
What it does
- Sweeps Airbyte every 30 minutes for syncs that failed in the last hour on connections you have tagged as critical.
- Automatically kicks off one retry per failure so transient blips fix themselves without waking anyone up.
- Waits a short interval, checks whether the retry succeeded, and moves on if it did.
- If the retry also fails, creates a PagerDuty incident naming the connection and the failure reason, then posts a Slack message in your data-ops channel with a link to the incident.
- Skips the retry entirely for failures that look like an authentication problem or a schema mismatch, because those need a human, and never retries the same connection more than once per hour so a broken source does not get hammered.
What you’ll need
- An Airbyte workspace where your critical connections are tagged so the workflow can tell them apart from lower-priority pipelines.
- A PagerDuty account with the service you want incidents created against.
- A Slack workspace with a bot installed in the channel where your data team wants the escalation posts to land.
How to customize it
- Change the sweep cadence — every 15 minutes for latency-sensitive pipelines, hourly if 30 minutes feels noisy.
- Adjust the retry-wait window and the failure-reason keywords that count as auth or schema errors, so more or fewer failures get retried automatically.
- Swap the PagerDuty service or the Slack channel that receives the escalation, or add extra recipients like a mobile-only on-call channel.
Use cases
- Notifications & Alerts
- Data Sync
