Daily Airbyte sync health digest in Slack

Every weekday at 7:30am, check whether last night's critical Airbyte syncs actually landed on time and post a single green/red data-readiness digest to a Slack channel before the analytics team logs on. Trigger: cron, Monday through Friday at 07:30 in the workspace's timezone.

Configurable inputs for the workflow: the Slack channel to post to (for example #data-alerts), and the way to identify critical connections. Support all three approaches and let the user pick one: a name convention (a prefix or substring, for example "prod_" or "critical"), an Airbyte tag name, or an explicit list of connection names. Default the low-volume warning threshold to a 40 percent drop below the recent median, and expose it as an adjustable knob.

Step 1. Call Airbyte's List Connections to enumerate every connection across the accessible workspaces. Filter to the critical set using the configured strategy (name convention, tag, or explicit list). If the filter matches zero connections, post a short Slack message noting no critical connections were found and stop.

Step 2. For each critical connection, call Airbyte's List Jobs scoped to that connection and to sync jobs from the last 24 hours. Identify the most recent overnight run. For any run that is not clearly a healthy success (failed, cancelled, still running, incomplete, or missing entirely), call Airbyte's Get Job Details on that job id to pull the failure reason, start and end timestamps, and the records-extracted count.

Step 3. For each critical connection, decide freshness, success, and volume: did an overnight sync run at all, did it succeed, how many records did it extract, and is that volume in line with recent history. Compute a rolling median of records-extracted across the prior successful runs of that connection (use List Jobs and, if needed, Get Job Details to gather the counts). A successful sync whose records-extracted is below the low-volume threshold versus that median is a volume anomaly, not a pass.

Step 4. Assign a status per connection: green check for a sync that finished successfully overnight with volume in line with history; yellow warning for a sync that succeeded but with anomalously low volume; red cross for a failed, cancelled, still-running, or entirely missing sync. Capture the failure time or missing-run note for anything red.

Step 5. Use Slack Bot's Send a Message to post a single message to the configured channel. Structure the message as: a one-line headline summary at the top (for example "3 of 4 critical pipelines healthy — Salesforce sync failed at 3:12am"), then a per-connection status line with the green/yellow/red indicator, the connection name, the finish time, the records-extracted count, and a short reason for any non-green status. End with a call-out block that lists any connection that needs immediate attention with the specific failure reason or anomaly detail. If every critical pipeline is green, keep the message short and confirm all clear.

Notes. Airbyte access tokens are short-lived (about three minutes for Cloud), so re-mint when needed during the run. Never invent connection names or job ids: only report what List Connections, List Jobs, and Get Job Details actually returned. If Airbyte itself is unreachable, still post the digest with a red banner explaining that Airbyte could not be reached rather than silently skipping the morning. Reference pattern: the freshness, volume, and schema pillars documented at https://airbyte.com/data-engineering-resources/data-observability.