Fix broken Fivetran pipelines from one triage board

By General Input

Open one board every morning to see which data pipelines are broken, test and retry them in place, and spot the ones that keep failing.

Integrations

  • Fivetran
  • Linear
  • Slack

Type

App

Categories

  • Engineering
  • Operations

Build me an app my data team opens every morning to actually triage broken Fivetran pipelines, not just get told about them. This is a control surface, a workbench where I test, retry, pause, and ticket. It is not a digest and not a monitor, so nothing should run on a schedule or push a notification on its own. Everything happens because someone opened the app and clicked.

The main view is a board of every connection, pulled with Fivetran List Connections, grouped into four columns by health: broken, stale, paused, and healthy. Every row shows the connection name and its source service, the last sync time, the sync frequency, the destination it loads into, and the failure reason when there is one. Resolve destinations by calling Fivetran List Destinations once per load and joining connections to destinations by group so I can see the warehouse each pipeline feeds without a lookup per row. Pull the per connection detail with Fivetran Get Connection and Fivetran Get Connection State to get the real status and failure reason rather than a generic error.

Use these grouping rules. Broken means the connection reports a failed sync or a failed setup state, or its setup tests are failing. Paused means the connection is paused, and keep these separate from broken so an intentionally paused pipeline never reads as an incident. Stale means the last successful sync is older than two of the connection's own sync intervals, falling back to a 24 hour threshold when the frequency is unknown. Healthy is everything else. Sort broken first and show a count on each group so the board leads with what needs attention.

From any row I can act without leaving the app. Run Connection Setup Tests confirms whether credentials or connectivity are the problem, and I want the individual test results rendered as a readable pass or fail list, not a raw blob. Sync Connection retries an incremental sync and Re-sync Connection kicks off a full historical re-sync, with a confirmation step on the full re-sync since it is expensive. Re-sync Connection Tables covers the narrower case where only specific tables need rebuilding. Modify Connection lets me pause a noisy pipeline or stretch its sync frequency straight from the row. Show the destination warehouse and current schedule near these controls so I know what I am about to change.

Bake in an important Fivetran nuance. The sync, re-sync, and schema reload calls are asynchronous: they queue work and return immediately, so never treat a successful response as a finished sync. After triggering one, poll Get Connection and read the sync state from the connection status to drive a live progress indicator on that row, and stop polling when it settles or when a sensible timeout is reached. Setup tests are different, they run synchronously and return their results directly, so those can be shown as soon as the call returns. Fivetran also rate limits per account, with much tighter limits for sync and setup test calls, so batch the board load, respect the retry delay the API asks for, and degrade gracefully instead of hammering it.

Every broken connection gets a Diagnose button that kicks off a background agent for that connection. The agent pulls Get Connection, Get Connection State, and Get Connection Schema Config, runs Run Connection Setup Tests, and works out what is actually wrong: expired or rotated credentials, a source schema change, a permissions problem at the destination, or a transient failure that will clear on a retry. It then writes a plain English root cause note back into the app beside that connection, including what it checked and what it recommends I do next. When the failure looks real rather than transient, it files a Linear issue with Create Issue, putting the connection name, the destination, the failure reason, and its recommendation in the description, and posts a Slack heads up with Send a Message linking the connection and the new issue. When the failure looks transient it says so in the note and does not file anything, so we are not opening tickets for blips. Show the diagnosis state on the row while the agent is working and surface the note, the issue link, and the timestamp when it finishes.

Keep a visible history of which connections break repeatedly so chronic offenders stand out instead of the team re-triaging the same pipeline every week. Record a health snapshot for every connection each time the board loads, and store the diagnosis notes and any filed issue links against the connection permanently. Each row shows how many times that pipeline has broken in the last 30 and 90 days, and anything crossing a repeat offender threshold gets a clear badge. Give me a history view per connection with its past failures, past root cause notes, and linked tickets, so I can see at a glance that a pipeline has failed for the same reason six times and needs a real fix rather than another re-sync.

The audience is data engineers and analytics engineers who own the ELT stack, so density beats decoration. Make the board scannable at a glance, keep the failure reason readable without expanding a row, and make the destructive actions, full re-sync and pause, deliberate rather than easy to fire by accident.

Related prompts

Explore more prompts
Call overdue Xero customers with an AI collections agentWin back LiveChat visitors whose chats went unansweredChat quality review board for LiveChat support leadsWin back no-show and cancelled appointments every morningLive Loop returns analytics with product-level drill-downNewsletter pre-flight and approval board for Mailjet sendsTurn a prospect spreadsheet into personalized sequence enrollmentsMailjet email delivery lookup console for support teamsCatch feature flags that never got switched on in productionKajabi customer support console for member access fixes