Airbyte backfill console with a Google Sheets audit trail

By General Input

Run Airbyte resyncs and clears from one screen, with a typed reason, a confirmation step on destructive runs, and every launch logged to a sheet.

Integrations

  • Airbyte
  • Google Sheets

Type

App

Categories

  • Engineering
  • Operations

I want an internal app my data team can open to run Airbyte backfills safely, instead of clicking through the Airbyte UI where a wrong click is unrecoverable and nothing records why the backfill happened.

The main screen lists my Airbyte connections using List Connections, showing each connection's name, source, destination, and status. When I select one, load its full detail with Get Connection Details, and show the streams it covers using Get Stream Properties so I can see exactly what data a clear would affect. Also show that connection's recent runs using List Jobs, with status, job type, start time, and duration, and let me expand any run to pull its full detail with Get Job Details.

Below that, a launch panel. It has a required free text Reason field, and a choice of two modes: Resync, which starts a normal sync, and Clear and resync, which erases the data Airbyte wrote to the destination and then rebuilds it. Default the selection to Resync. I should not be able to launch anything without typing a reason.

Clear and resync is destructive and needs to feel that way. Airbyte's own documentation says a clear erases all of the data Airbyte has created in the destination, that there is no way to recover from a clear, and that if the source does not retain all of its records this may lead to data loss. Put that warning on screen, in plain language, whenever the clear option is selected. Then require me to type the connection's exact name into a confirmation box before the launch button becomes active. Use the word Clear rather than Reset throughout the interface, since Airbyte renamed the operation in version 1.0.0.

Launching calls Trigger Sync or Reset Job. For a plain resync that is a single sync job. For Clear and resync, the app runs the clear job first, waits for it to succeed, and then automatically starts a sync job, because Airbyte no longer re-syncs automatically after a clear and historical data is only backfilled once a sync runs after the clear succeeds. Present that chain as one operation with two stages rather than two separate things I have to babysit, and if the clear fails, stop and say so instead of starting the sync.

While a job runs, poll Get Job Details and show live progress in the app: current stage, status, rows synced, and elapsed time. Show a Cancel button that calls Cancel a Job on the running job, and reflect the cancelled state once it takes effect.

Every launch appends one row to a Google Sheet using Append Values, capturing the timestamp, who launched it, the connection name and id, whether it was a resync or a clear and resync, the reason typed, and the resulting job id. Write the row at launch time so nothing is lost if the job later fails or is cancelled. On the main screen, read the last five rows back with Get Values and display them as a recent backfill activity list, so whoever is about to run a backfill can see what the team already did.

Two constraints. The app is read only with respect to configuration: there is no supported operation to update an Airbyte connection, so it must never try to change stream selections, schedules, or any other connection settings. Get Stream Properties is there purely as context for the decision. And this audit sheet is only for human initiated backfills, so keep it in its own sheet and do not turn it into a general export of every Airbyte job.

Related prompts

Explore more prompts
Call overdue Xero customers with an AI collections agentLocal listing health board for every location you manageLet support send one-off Loops emails without an engineerStop cold emails to anyone with a live deal in PipedriveiMessage campaign console with pre-flight checks and delivery boardLinkedIn Ads budget pacing dashboard for every client accountFront desk appointment confirmation board for the next 3 daysGive your team Looker numbers without buying more seatsBuild audience segments from product usage and push to LoopsTurn the people who engage with your posts into Pipedrive leads