Monthly Grafana alert rule hygiene audit

By General Input

On the first Monday of every month, find stale, silent, and missing-runbook alert rules in Grafana and file owner-grouped cleanup tickets in Linear.

Integrations

  • Grafana
  • Linear
  • Slack Bot

Type

Agentic Task

Categories

  • Engineering
  • Operations

Build me a monthly Grafana alert rule hygiene auditor that runs on a cron the first Monday of every month. The reasoning about what counts as stale or misconfigured is judgment-heavy, so this should be an agent workflow.

Step 1: gather state. The agent should call Grafana's List Alert Rules to fetch every provisioned alert rule, List Contact Points to see what notification destinations exist, and Get Notification Policy Tree to see how rules route. Then call Find Annotations over the past 90 days filtered to alert-state annotations, so each rule has a recent fire history.

Step 2: flag three classes of hygiene problems. (a) Stale: rules that have never fired in the last 90 days are candidates for deletion or threshold review. (b) Silent: rules whose labels do not match any route in the notification policy tree and therefore would not reach any contact point if they fired. The silent check is the most valuable, treat it as P0. (c) Missing metadata: rules without a severity label or without a runbook annotation.

Step 3: file cleanup tickets in Linear. For each cleanup item the agent should call Create Issue in a configured Alerting Hygiene team (the team name should be configurable). Each issue should have a clear title that includes the rule's folder and name, a severity for the hygiene problem itself (silent = high, stale = medium, missing metadata = low), and a checklist in the description of what to verify or fix. Group issues by Grafana folder so folder owners can claim their cleanup as a batch. Be conservative: never auto-delete or auto-modify rules, only file tickets.

Step 4: post a single Slack summary via Send a Message to a configured channel. The summary should say how many rules were audited, how many tickets were filed in each of the three categories, link the new Linear tickets in a bundle, and call out the silent alerts explicitly because those are the ones most likely to bite during an incident.

Configurables I want to be able to set: the Slack channel, the Linear team name, the lookback window (default 90 days), and whether to skip rules in specific folders (e.g. a sandbox folder).

Related prompts

Explore more prompts
A brand asset library your marketing team actually searchesTurn Mailjet email clicks into ranked HubSpot follow-upsClean out the Looker dashboards and Looks nobody opensLiveKit live operations console for room moderationWake up dormant Keap leads with a researched reasonLiveChat coverage board for planning next week's shiftsPhone routing control panel for LiveKit voice agentsLinkedIn Ads budget pacing dashboard for every client accountGive your team Looker numbers without buying more seatsPause marketing emails to escalated customers, then restore them