Airbyte pipeline control room for stuck and failed syncs

By General Input

See every Airbyte connection across your workspaces on one screen, with failures and slow runs on top, and restart or cancel a sync in place.

Integrations

  • Airbyte
  • Slack Bot

Type

App

Categories

  • Engineering
  • Operations

Build me an Airbyte pipeline control room: an internal app I open whenever something looks off, instead of clicking through the Airbyte UI workspace by workspace. Nothing in this app runs on a schedule and nothing sends automatic alerts. It is a live console an engineer opens and acts from.

Main screen: a single table of every connection across all of my Airbyte workspaces. Load the workspaces with List Workspaces, then load connections with List Connections scoped to those workspace ids, and enrich each row with Get Connection Details for the connection name, source, destination, and configured schedule. For each connection, pull its most recent job with List Jobs filtered to that connection id and use Get Job Details for the run metrics. Each row shows: connection name, workspace, source to destination, last run status, how long that run took, rows synced, and how long ago it finished as a relative time like "14 minutes ago".

Sort so problems float to the top by default: failed and incomplete runs first, then jobs currently running that have been going noticeably longer than that connection's recent average duration, then everything else by most recently finished. Give me a filter by workspace and a filter by status, and let me sort by duration or rows synced if I want to hunt for a bottleneck instead of a failure.

Clicking a connection row opens a drawer for that connection. The drawer shows recent job history, roughly the last twenty jobs from List Jobs for that connection, one line per job with status, start time, duration, rows synced, and job type (sync or reset), using Get Job Details for the per job metrics. Put the connection's own details at the top of the drawer so I have the source, destination, and schedule in front of me.

From the drawer I can act. A Start sync button calls Trigger Sync or Reset Job with job type sync. A Reset button calls the same operation with job type reset, behind a confirmation dialog because a reset clears the destination data. A Cancel button calls Cancel a Job on any job that is currently pending or running. These three are the only writes in the app.

Important: do not build pause connection, edit schedule, or rename connection actions. There is no documented update or patch connection operation in our Airbyte catalog, so those buttons would not work. Schedule and naming changes stay in the Airbyte UI, and the app should not pretend otherwise.

Auto refresh while work is in flight: whenever any job visible on the screen is pending or running, refresh the affected rows and the open drawer every twenty to thirty seconds so I can watch a job finish. Stop refreshing once nothing is running, so the app is quiet when everything is healthy. Never make the whole page flash or lose my scroll position and filters on a refresh.

Show which person triggered each run. The Airbyte API does not tell you which of my teammates pressed a button, so the app has to record this itself: whenever someone starts, resets, or cancels from inside the app, store a row in the app's own storage with the signed in app user, the connection id, the returned job id, the action taken, and the timestamp. Then display it in the job history and on the main table as something like "Started by Priya 4 minutes ago". Jobs that came from Airbyte's own scheduler or from the Airbyte UI have no in app actor, so label those "Scheduled or started outside this app" rather than guessing or leaving it blank. Everyone on the team sees the same attribution log, which is the point: it tells you who is already working the problem before you jump in.

Share button: on the selected connection, a share button composes a short plain text status summary and posts it to a Slack channel I choose, using the Slack Bot Send a Message operation. The summary should be a handful of lines covering the connection name and workspace, the last run status, duration, rows synced, how long ago it finished, whether something is running right now, and who triggered it if the app knows. This is a manual hand off button, one message per click, so I can give the team context without retyping it. Do not post to Slack automatically, do not build a digest, and do not treat Slack as an alert channel.

Technical notes for the handlers. Airbyte Cloud access tokens are short lived at roughly three minutes, so mint a fresh token inside each handler for that burst of requests rather than caching one across requests. Connection, source, destination, and workspace ids are UUIDs, but job ids are 64 bit integers, so carry job ids as strings through the UI to avoid losing precision. List endpoints paginate with limit and offset, so page through when a workspace has more connections than one page. Handle rate limit responses with a short backoff, and if a connection's job history fails to load, show that one row as unavailable instead of failing the whole board.

Related prompts

Explore more prompts
A brand asset library your marketing team actually searchesTurn Mailjet email clicks into ranked HubSpot follow-upsClean out the Looker dashboards and Looks nobody opensLiveKit live operations console for room moderationWake up dormant Keap leads with a researched reasonLiveChat coverage board for planning next week's shiftsPhone routing control panel for LiveKit voice agentsLinkedIn Ads budget pacing dashboard for every client accountGive your team Looker numbers without buying more seatsPause marketing emails to escalated customers, then restore them