Page on-call when a production deploy fails in Buildkite

By General Input

The moment a build fails on one of your production deploy pipelines, automatically open a PagerDuty incident so on-call gets paged right away.

Integrations

  • Buildkite
  • PagerDuty

Type

Deterministic Code

Categories

  • Engineering

When a build finishes on one of my Buildkite pipelines, run this workflow automatically. Trigger on Buildkite's build.finished webhook, and only continue when the build's state is failed and its pipeline is on my allowlist of production deploy and release pipelines. Any other pipeline, or any non-failed state, should stop the workflow immediately so routine feature-branch failures never page anyone. No AI reasoning is needed: this is a fixed, deterministic mapping from the failed-build payload to a PagerDuty incident.

First, read the full build using Buildkite's Get a Build operation for the pipeline slug and build number from the webhook payload. This gives me the build number, branch, commit hash, commit author, build message, and the web URL of the build.

Then create a PagerDuty incident using PagerDuty's Create Incident operation on my deployment service. Set the incident title to name the pipeline, the branch, and the build number, for example 'Deploy failed: <pipeline> on <branch> (build #<number>)'. Set the incident body to include the commit hash, the commit author, and a direct link to the failed build in Buildkite so on-call can jump straight to it.

Derive the incident urgency from the pipeline: release and production deploy pipelines should page at high urgency, while lower-priority deploy pipelines can page at low urgency. Keep the mapping from pipeline name to urgency explicit so it is easy to adjust. Because only allowlisted pipelines reach this step, on-call is only paged for the deployments that actually matter.

Related prompts

Explore more prompts
Rebuild an incident timeline from Grafana and PagerDutyCI control room for watching and unsticking Buildkite buildsFlaky test triage board for Buildkite, with Linear quarantineDeploy board for every environment, with one-click redeploysTurn passing Buildkite deploys into customer-ready release notesPage on-call when a production service goes down on RenderFind which feature flag caused your Sentry error spikeInvestigate New Relic alerts before waking your on-callAlert on-call the moment a Fivetran pipeline breaksAI triage for failed Buildkite builds, posted to Slack