Weekly reliability report in Notion from Railway and Sentry

Every Friday at 16:00 in our team timezone, build a weekly reliability report and write it as a single page in Notion so the on-call handoff is actually useful. This should be an agent workflow on a cron trigger.

Step 1. Gather Railway deploys. Use Railway's List Projects to find our production project(s), then for each production service use List Deployments to pull every deployment from the last 7 days. For each deployment capture: service name, status (success, failed, crashed, removed), commit SHA and message if available, the deploy URL, the staticUrl/domain, and the createdAt and finishedAt timestamps. Mark a deploy as a rollback if it was triggered as a rollback or a redeploy that immediately followed a failed deploy on the same service.

Step 2. Gather Sentry signal. Use Sentry's List Organization Issues filtered to the same 7-day window, sorted by frequency, to get the active issues. For the top ~10 issues, also call List Project Error Events to get accurate event counts and the firstSeen/lastSeen timestamps. Capture the Sentry issue URL, project slug, issue title, level (error/fatal/warning), event count for the week, firstSeen, and the service or environment the issue belongs to.

Step 3. Correlate. We do NOT use Sentry's Create a Deploy to mark deploys, so do the correlation post-hoc by timestamp. For each Sentry issue, check whether its firstSeen (or a clear spike in event count) falls within 1 hour AFTER a Railway deployment on a service whose name maps to the Sentry project (use fuzzy/substring matching on service name to project slug). If yes, flag it as a likely regression and link it to that specific deployment. Also flag: services with one or more failed deploys followed by a rollback, and services that had zero deploys AND zero new Sentry issues for the week (the "quiet" services).

Step 4. Write a single Notion page. Use Notion's Create a Page with the parent set to a configured "Engineering / On-call" page (the user will paste the parent page ID during setup). Title it something like "Weekly reliability report — week ending {Friday date}". The page body should be one concise page (not a database, no child pages) with these sections in order: Week summary (1-2 sentences plus headline numbers: total deploys, failure rate, new regressions, quiet services); Deploys by service (a simple list or table: service, deploy count, success/failure counts, failure rate); New regressions linked to deploys (each item: Sentry issue title, event count, firstSeen, the Sentry issue URL, and the Railway deployment URL it correlates to); Quiet services (just a list); Recommendations for next week (3-5 short bullets the agent infers from the data, e.g. "investigate flaky deploys on api-worker" or "add Sentry alerting for billing-service since it's been silent for 3 weeks").

Keep the page concise and skimmable. No Slack post. The user will share the Notion page link manually. If the week was quiet (no deploys or no new errors), still write the page and say so plainly in the summary.