Triage Hightouch sync failures into Slack and Linear tickets
Every 15 minutes, scan Hightouch for newly failed syncs, post a triage summary in Slack, and open a Linear ticket when a sync keeps failing.
Integrations
Hightouch
Slack Bot
Linear
Type
Categories
- Operations
- Engineering
Build an agent workflow that triages Hightouch sync failures and gets them in front of our data team fast.
Trigger: cron, every 15 minutes.
On each tick, the agent should:
1. Call Hightouch List Syncs to enumerate every sync in the workspace.
2. For each sync, call Hightouch List Sync Runs and identify any runs that newly entered an error or aborted state since the previous tick. The agent must remember the last seen run id per sync between runs so we only alert on genuinely new failures, never re-alert on the same failed run twice.
3. For each newly failed run, gather a short context block: destination name, model name, rejected row count vs added/changed/removed row counts, the failure reason or error message from Hightouch, and whether the previous 5 runs of that sync were also failing.
4. Post a Slack message via Slack Bot Send a Message to a configurable channel (default #data-alerts) with that triage summary plus a direct link back to the sync in the Hightouch app. One Slack message per failed sync, formatted for quick scanning.
5. For any sync that has failed 3 or more consecutive runs (a true incident, not a flake), also escalate to Linear. Before filing, use Linear Search Issues to look for an existing open issue mentioning that sync by name in the data team. If one already exists, skip Linear and only post the Slack message. If none exists, call Linear Create Issue in the data team with a clear title like "Hightouch sync <sync name> failing repeatedly", the full diagnostic context (destination, model, recent run history, last error) in the description, and priority High.
Important behavior:
- Persist per-sync state (last seen run id, consecutive-failure count) across ticks. On the very first run, treat all currently-known runs as the baseline and do not alert retroactively.
- Never duplicate a Linear ticket. Search by sync name first, every time.
- A single-run failure should only ping Slack, never Linear. Linear only opens once we cross the 3-consecutive-failure threshold.
- Disabled or paused syncs in Hightouch should be ignored.
- If Hightouch rate-limits the run, back off and finish on the next tick rather than erroring the whole workflow.
Integrations: Hightouch (List Syncs, List Sync Runs), Slack Bot (Send a Message), Linear (Search Issues, Create Issue).
What it does
- Runs every 15 minutes and only flags syncs that just entered an error or aborted state, so you do not get pinged about the same failure twice.
- Posts a clear triage summary to your data alerts Slack channel for each new failure: destination, model, rows rejected versus changed, the failure reason, and whether the previous runs also failed.
- Opens a high-priority Linear ticket only when a sync has failed three runs in a row, so flaky one-off errors do not clutter your backlog.
- Checks Linear for an existing open ticket on the same sync before filing a new one, so you never end up with duplicate work in the queue.
What you’ll need
- A Hightouch workspace with API access. A workspace Admin can generate the key from Settings.
- A Slack workspace, with a channel you want failure alerts to land in (for example, #data-alerts).
- A Linear workspace, with a team that owns data quality where escalation tickets should be filed.
How to customize it
- Change the cadence. The default is every 15 minutes, but you can run it every 5 minutes during business hours or hourly overnight.
- Route alerts to the right place. Send all failures to one channel, split by destination (Salesforce vs HubSpot vs Iterable), or DM an on-call data engineer for the worst ones.
- Tune the escalation threshold. Three consecutive failures is the default for opening a Linear ticket, but some teams escalate sooner or only file tickets for production destinations.
Use cases
- Notifications & Alerts
