Daily Databricks data health check with Slack alerts

Every weekday at 7am, check the health of my most important Databricks tables and alert me in Slack only when something looks wrong. I will give you the list of critical tables (as catalog.schema.table), the SQL warehouse to run against, the key columns for each table, and the thresholds for each check. Treat all of this as configuration so I can point the workflow at my own tables and tune what counts as a problem.

For each table, run the data-quality checks using the Databricks Execute SQL Statement operation against my warehouse, then read the outcomes with the Get SQL Statement operation (poll until the statement state is succeeded, then read the returned rows). Run these three checks:

1) Freshness: check whether the latest max(updated_at) is within the expected window for that table (for example, updated within the last N hours). 2) Row-volume anomaly: compare today's row count against the trailing average of the prior period (for example the last 7 or 14 days) and flag when it deviates beyond the configured percentage in either direction. 3) Null and duplicate rates on key columns: measure the share of null values in the key columns and check for duplicate values on columns that are supposed to be unique.

Interpret each result against the configurable thresholds and use judgement about what is genuinely worth flagging and how urgent it is. A table that has not updated in a day is more serious than a small blip in row count, and a duplicate key on a primary table matters more than a minor null-rate wobble. Do not flag noise: only surface a check when it clearly crosses its threshold, and rank the findings by urgency.

When one or more checks fail, use the Slack Bot Send a Message operation to post a concise, plain-language alert to my configured channel. For each problem, name the affected table, the specific check that failed, and the actual numbers versus the expected threshold (for example: latest data is 26 hours old, expected within 12; today's row count is 4,120 versus a 7-day average of 9,800). Lead with the most urgent items. When every check passes, either stay quiet or post a short all-clear line, depending on my preference.

Keep these configurable: the list of tables and their key columns, the SQL warehouse id, the freshness window per table, the row-volume tolerance percentage and comparison period, the acceptable null and duplicate rates, the destination Slack channel, and whether to post a daily all-clear or only speak up when something is wrong.