Morning Grafana on-call briefing in Slack

Every weekday at 8am ET, build a short morning on-call briefing in Slack that summarizes what actually misbehaved in Grafana over the last 24 hours. Trigger: cron, Monday through Friday at 8am America/New_York.

Step 1. Pull alert activity from Grafana. Use Grafana Find Annotations filtered to type=alert and time range = the last 24 hours. These annotations carry the rule name, the state transition (firing, resolved, normal, no_data), the severity if tagged, and timestamps. Pull the full set for the window.

Step 2. Analyze and rank. Group the annotations by alert rule. For each rule, count how many state transitions happened, what fraction were firings versus resolves, the longest sustained firing duration, and the severity. Classify each rule as flapping (many short fire/resolve cycles) or sustained (one or a few long firings). Rank rules by impact, weighting severity, total fire count, and sustained duration. Take the top three.

Step 3. Find a dashboard link for each top offender. For each of the top three rules, call Grafana Search Dashboards and Folders to find the dashboard most associated with that rule, matching on the rule name, the service or component in the rule name, or shared tags. Construct a deep link to that dashboard using the Grafana instance URL. If nothing reasonable matches, omit the link rather than guess.

Step 4. Decide whether to post at all. If the last 24 hours had no firings, or only trivial transient blips with no severity and no repeats, skip the briefing entirely. Do not post an empty digest. The goal is signal, not noise.

Step 5. Post one Slack message into the on-call channel using Slack Bot Send a Message. Ask the user at setup which channel to use (default the channel name should be configurable, for example #on-call or #incidents). The message should be short and skimmable in Slack mrkdwn, structured like this:

Header: *Overnight on-call briefing* with the date range covered.

Top three to watch today: a numbered list. For each item include the rule name in bold, the fire count, the current state (firing, resolved, flapping), the severity if known, and the dashboard deep link as <url|dashboard name>. Underneath each item add one short synthesized observation in plain English, for example: "auth-api p99 latency fired twelve times overnight, all clustered around the 2am deploy window" or "checkout-worker queue-depth has been sustained-firing since 3:14am and has not recovered".

Footer: a one-line summary of total alerts in the window and how many distinct rules fired.

Keep the whole message under roughly 1500 characters. Do not paste raw JSON, do not list every alert, do not include rules that did not fire. The prioritization, grouping, and natural-language synthesis are the entire point of this being an agent workflow, so do the thinking before posting.

This is distinct from a monthly alert-rule hygiene audit. This workflow does not change any Grafana configuration, does not file tickets, and does not mute or silence anything. It only reads and reports.