Audit Upstash Redis backups daily and fix gaps automatically

Every morning at 6am, audit backup coverage across all of my Upstash Redis databases. Fix whatever can be fixed automatically, and escalate only the cases a human actually has to look at.

Start by calling Upstash List Redis Databases to get every database in my account. Skip any database whose name marks it as throwaway, so short lived environments do not generate noise. By default, treat a name as throwaway if it contains ephemeral, preview, tmp, temp, scratch, sandbox, pr- or -test. Keep track of what you skipped so it can be mentioned in the summary, but take no action on those.

For each remaining database, call Upstash List Backups and classify it into one of three states. Healthy means there is at least one successfully completed backup from within the last 24 hours. Stale means the most recent successful backup is older than my freshness threshold, which defaults to 24 hours. Uncovered means there are no backups at all, or none that ever completed successfully. A backup that exists but failed, or is still stuck in progress, does not count as usable.

For every stale and uncovered database, try to self heal before escalating anything. Call Upstash Create Backup to take a backup immediately. Then, if that database does not already have automatic daily backups turned on, call Upstash Enable Daily Backup so the same database does not show up as failing again tomorrow. Taking one backup fixes today; turning on daily backups fixes the pattern.

After each repair attempt, call Upstash List Backups again for that database to confirm whether a usable backup now exists. Record the outcome of every attempt, including the exact error text and status code when an attempt fails.

Escalate to Linear only when the problem survives the repair attempt. Escalate if the database still has no usable backup after the attempt, if the plan does not support backups, if the database is in a state that blocks backups, or if this database has been failing repeatedly. Before filing anything, call Linear Search Issues to look for an open issue that already names this database. If one exists, add a comment with today's findings using Add Comment to Issue instead of opening a duplicate, and treat that existing issue as evidence this is a repeat offender rather than a one off. Only call Linear Create Issue when there is no open issue for that database yet.

Whatever you write in Linear has to stand on its own as actionable work. Name the database, its region and its plan. State when the last good backup was, or state plainly that there has never been one. Then state explicitly what you already attempted and exactly what happened, including the error returned, so the assignee does not waste time repeating a backup attempt that has already failed. The issue should make it obvious what is left for a person to decide, such as upgrading the plan or retiring an abandoned database.

Finish with exactly one summary message using Slack Bot Send a Message to my infrastructure channel. Cover the healthy count, what was self healed this run and is now confirmed covered, and anything escalated to Linear with a link to the issue. Mention the databases skipped as ephemeral in a single short line. If everything was already healthy and nothing needed action, say that in one line rather than padding the message, so that the mornings it does have something to report are the ones worth reading.