CloudWatch alarm threshold tuning bench for on-call teams

Build me an alarm threshold tuning workbench for Amazon CloudWatch that my on-call team opens during alarm hygiene reviews. It is an interactive bench, not a scheduled report: I select an alarm, look at how it has actually behaved, try a candidate threshold, and commit the change with a reason. Show the connected AWS region prominently in the header, because CloudWatch alarms and metrics are region scoped and this app only ever sees the one region the credential is connected to.

Main screen: a table of every alarm from Describe Alarms, paginating through NextToken until the list is complete. Each row shows the alarm name, current state (OK, ALARM, or INSUFFICIENT_DATA), the metric it watches (namespace, metric name, and dimensions), the statistic, the comparison operator and threshold, the period, the number of evaluation periods and datapoints to alarm, and whether alarm actions are currently enabled. Let me filter by state and by name prefix, and sort by the simulated firing count described below. Also show a badge on rows that are covered by an active mute rule, using List Alarm Mute Rules to fetch current mutes for the region and matching them against alarm names.

Every row gets a "would have fired N times" count for the last 30 days. Compute it in a handler: call Get Metric Statistics for that alarm's namespace, metric name, dimensions, and statistic over the trailing 30 days at the alarm's own period (fall back to a coarser period such as 300 or 3600 seconds when 30 days at the configured period would exceed the datapoint cap, and label the chart with the period actually used), then walk the datapoints applying the alarm's comparison operator against its threshold and counting each run where the breach persists for the configured datapoints to alarm. Count transitions into the breaching state, not individual breaching datapoints, so one long incident counts once. Cache these results per alarm for the session so the table does not refetch on every render.

Selecting an alarm opens a detail panel. Chart the last 30 days of the underlying metric from Get Metric Statistics as a line, with a horizontal line at the configured threshold drawn on the same axes so I can see the headroom between normal behaviour and the trip point. Underneath the chart, render a firing history strip built from Describe Alarm History for that alarm over the same window, filtered to state update items, showing each firing as a block on the same time axis as the chart. Label the alarm with a plain classification derived from that history: a flapper (many short firings), a permanent screamer (in ALARM for a long continuous stretch), or never fired (no state updates in the window). Also list any sibling alarms watching the same metric using Describe Alarms For Metric, so I do not tune one of three overlapping alarms without noticing the others.

Beside that, a what-if panel. I type a candidate threshold and the app immediately recomputes the firing count using the metric datapoints already fetched, no extra AWS calls, and shows it side by side with the current-threshold count, for example "current 80: 47 firings, candidate 92: 3 firings". Draw the candidate threshold as a second line on the chart in a distinct colour, and highlight which past firings would disappear at the candidate value. Let me also try a different number of evaluation periods in the same panel, since the same threshold with three evaluation periods instead of one is often the actual fix.

From that panel I can commit the change with Put Metric Alarm. Important: Put Metric Alarm replaces the whole alarm definition rather than patching a field, so the handler must first read the existing alarm with Describe Alarms, merge only the changed threshold and evaluation periods into the full definition, and resend everything else unchanged (metric name, namespace, dimensions, statistic, period, comparison operator, alarm actions, OK actions, insufficient data actions, treat missing data, description, and unit). Require me to type a short reason before the save button enables.

Same panel also handles noise suppression. I can snooze an alarm for a scheduled window with Put Alarm Mute Rule, see all currently active mutes for the region from List Alarm Mute Rules (with full details from Get Alarm Mute Rule), and lift a mute early with Delete Alarm Mute Rule. Separately I can turn notifications off entirely with Disable Alarm Actions and back on with Enable Alarm Actions, shown as a toggle on the alarm row and in the detail panel.

Every committed change gets appended to an in-app change log stored in the app's own storage: timestamp, region, alarm name, what changed (threshold, evaluation periods, mute created or lifted, actions disabled or enabled), the old value, the new value, the signed-in user who made the change, and their typed reason. Give the change log its own view with newest first and a filter by alarm. Immediately after each successful write, post the same change to a configured Slack channel using the Slack Bot Send a Message handler, formatted as the alarm name, the old threshold, the new threshold, who changed it, and why. Let me pick the target channel once in a settings view populated from the Slack Bot List Channels handler. If the Slack post fails, still record the change locally and surface the failure in the app rather than silently dropping it.

Scope notes: this covers the CloudWatch monitoring API only, so do not reference CloudWatch Logs, log groups, or log queries anywhere in the app. The connection needs permission to write alarms and mute rules, not read-only access. Never imply the app spans an AWS account or multiple regions; it shows exactly the one connected region and says so.