Triage firing Datadog monitors into Linear issues every morning

By General Input

Every weekday at 9am, an agent reviews your currently firing Datadog monitors, posts a prioritized Slack digest, and opens Linear issues for the ones that have been burning too long.

Integrations

  • Datadog
  • Slack Bot
  • Linear

Type

Agentic Task

Categories

  • Engineering
  • Operations

Build me an agent that runs every weekday at 9am in my local timezone and triages my currently-triggered Datadog monitors into a Slack digest plus Linear issues for the persistent ones. Trigger: cron.

Step 1 — Pull active alerts. Use Datadog Search Monitors to fetch every monitor whose current state is Alert or Warn. Capture for each: monitor id, name, link to the monitor in Datadog, severity (Alert vs Warn), tags, the time it most recently transitioned into its current state, and the message/owner field if present.

Step 2 — Group and clean. Group monitors by service or by the most meaningful tag (service:, team:, env: in that order of preference). Drop obvious duplicates that point at the same underlying check. Skip known-flapping monitors based on a configurable allow/deny list of monitor names or tags (default deny list is empty, but make it easy for me to add). Tag each monitor as either NEW today or CARRIED OVER (was already firing in yesterday's run — keep a small piece of state across runs to know this, e.g. write the previous run's monitor ids to a workflow memory store and diff).

Step 3 — Post the Slack digest. Use Slack Bot Send a Message to post one message to the channel I configure (default #alerts-triage). Format: a one-line headline with total counts ("7 firing: 3 Alert, 4 Warn — 2 new, 5 carried over from yesterday"), then top offenders first (highest severity, then longest-firing), grouped by service. Each line should show the monitor name as a link to Datadog, severity, how long it has been firing, and the NEW/CARRIED OVER tag. Use Slack mrkdwn formatting (*bold*, <url|text> for links). Keep it scannable — no walls of text.

Step 4 — File Linear issues for persistent alerts. For any monitor that has been firing for longer than a configurable window (default 4 hours), before doing anything, use Linear Search Issues to check whether an open issue already exists for that monitor. Match on the monitor name and/or monitor id (include the monitor id as a stable token in issue titles or descriptions so future runs can find them). If a matching open issue exists, skip — do not create a duplicate. If none exists, use Linear Create Issue in the team I configure, with a clear title like "[Datadog] <monitor name>", a description containing the monitor link, current severity, how long it has been firing, recent state transitions, and the monitor's tags. Suggest an assignee based on a team:/owner: tag when present, otherwise leave unassigned. Set priority based on severity (Alert → High, Warn → Normal).

Step 5 — Close the loop in Slack. At the bottom of the digest, list the Linear issues that were created this run with their identifiers and links, and the ones that were skipped because a matching open issue already existed. If nothing was filed, say so explicitly so I know dedupe is working.

Configurable inputs I want exposed: Slack channel to post to, Linear team to file in, the persistence window (default 4 hours), the monitor name/tag deny list for known flappers, and the timezone for the schedule.

Edge cases to handle gracefully: zero firing monitors (post a short "all clear" digest, file nothing), Datadog API rate limits (retry with the Retry-After header), and Linear partial errors (continue filing the remaining issues, summarize failures in the Slack digest).

Related prompts

Explore more prompts
A brand asset library your marketing team actually searchesTurn Mailjet email clicks into ranked HubSpot follow-upsClean out the Looker dashboards and Looks nobody opensLiveKit live operations console for room moderationWake up dormant Keap leads with a researched reasonLiveChat coverage board for planning next week's shiftsPhone routing control panel for LiveKit voice agentsLinkedIn Ads budget pacing dashboard for every client accountGive your team Looker numbers without buying more seatsPause marketing emails to escalated customers, then restore them