A warehouse alert is a state machine. Slack is only where it renders.

A client asked for alerts on lakehouse metrics: freshness, row counts, data-quality ratios, the slow drifts that only exist once data has landed in tables. The request came with a picture of what they already used for service metrics, a Slack message with Acknowledge, Resolve and Close buttons, and one complaint about it. It fired so often that the critical alerts got lost.

The obvious design is a scheduled job that runs a query, compares the result with a threshold and posts to Slack. It works until the first person clicks a button. A cron job is not running at the moment of the click. It also misses the complaint: an alert that is just a query plus a post has no memory, so it cannot tell a new problem from the same problem reported for the fortieth time.

Operating rule. An alert is a state machine with an external user interface. Design the state, its key, and the order of writes first; the query is a parameter, and Slack is a renderer. Anything that can change an alert — a scheduled run, a click, an auto-resolve hours later — goes through one transition function and lands in one row.

This is a design I reviewed with a client and did not build or operate. The decisions are worth writing down because each one was forced by a platform fact, and the facts are checkable.

The first click breaks the cron job

Slack delivers an interaction payload either to a public HTTPS Request URL or down a Socket Mode WebSocket, and the app must acknowledge it within 3 seconds. Either way, something has to be running when the click arrives. That single fact turns one job into two deployables:

DETECTOR  (scheduled, stateless)          RECEIVER  (long-lived)
  load config                               hold an outbound WebSocket to Slack
  compute the window                        on click: ack within 3 s
  query source tables  ← only reader        transition → write state
  evaluate → transition                     chat.update the original message
  write state, append history               never reads source tables
  post or update Slack
            \                                  /
             └──── one config, one state store, one lifecycle module ────┘

The split is also a permission boundary. The detector is the only component that reads source data, so it holds SELECT on the source schemas and write access only to its own state and history tables. The receiver never reads source data at all. Both import the same lifecycle module, so a status change looks identical whether a person or the scheduler caused it.

Where the receiver can live

The client’s platform was Databricks, and the obvious home for a small always-on process there is a Databricks App. The obvious way to use one does not work.

Socket Mode was the answer because it inverts the direction of the connection, not because it is newer. The constraint was “nothing may arrive inbound”, and a receiver that dials out satisfies it by construction. The App also brings its own service principal, which is already the identity it needs to write alert state, so there is no credential to copy into another cloud. The trade-offs are the ones you accept with any stateful process: WebSockets drop, redeploys interrupt the connection briefly, and if egress is restricted, Slack’s WebSocket domain has to be allowlisted. Socket Mode apps are also not allowed in the public Slack Marketplace, which does not matter for an internal tool.

The function stayed in the design as a documented fallback, for the case where the standing cost turns out to be material. Keeping the receiver thin — ack, transition, update — is what makes that swap a deployment change rather than a rewrite.

The state is the product

The lifecycle has five states, and most of the rules exist to keep two of them apart.

              breach × consecutive_breaches
     OK ─────────────────────────────────────► ACTIVE
      ▲                                         │   │
      │                             [Acknowledge]   │ [Close]
      │                                         ▼   │
      │                                  ACKNOWLEDGED
      │                                         │   │
      │        healthy × recovery_runs          │   │ [Close]
      │   ┌─────────────────────────────────────┘   │
      │   │   ([Resolve] or auto, ACTIVE or ACKED)  │
      │   ▼                                         ▼
      └── RESOLVED ──[breach again]──► ACTIVE      CLOSED
                                                    │
                                     [healthy first, then may re-fire]
                                                    ▼
                                                    OK

The rules that need to be written down, because they are where the bugs live:

• Acknowledged still evaluates. Acknowledgement suppresses notification, not measurement.
• Resolved re-fires on the next breach as a new instance with a new message.
• Closed does not re-fire until the metric has been healthy again. That is the whole difference between Resolve and Close, and the reason to have both.
• A manual transition wins over an automatic one in the same run.
• Illegal transitions are rejected and logged, never quietly coerced into the nearest legal state.

The lifecycle is a pure function, (prior_state, evaluation, config) → (new_state, action), with no I/O. That makes it exhaustively testable without a warehouse or a Slack workspace, which matters because it is the code most likely to be wrong. The state it produces lives in a small mutable table, one row per alert instance, with an append-only history beside it:

CREATE TABLE ops.alert_state (
  alert_id              STRING    NOT NULL,  -- stable id from config; never reused
  group_key             STRING    NOT NULL,  -- '' when the alert has no group_by
  status                STRING    NOT NULL,  -- OK | ACTIVE | ACKNOWLEDGED | RESOLVED | CLOSED
  state_version         BIGINT    NOT NULL,  -- bumped on every transition; carried by buttons
  consecutive_breaches  INT       NOT NULL,
  consecutive_healthy   INT       NOT NULL,
  current_value         DOUBLE,
  comparison_value      DOUBLE,
  first_breached_at     TIMESTAMP,
  last_notified_at      TIMESTAMP,              -- cooldown is computed from this
  slack_channel_id      STRING,                 -- needed to edit the message later
  slack_message_ts      STRING,
  acked_by              STRING,
  updated_at            TIMESTAMP NOT NULL
);
-- Logical key: (alert_id, group_key). Every MERGE names both columns explicitly.

group_by is what makes the key two columns: an alert grouped by region is one independently tracked instance per region, each with its own lifecycle and its own message. Every merge names both key columns, because a loose predicate turns a single-row update into a table scan and a contention source.

The state_version column is what makes a click idempotent. Each button carries alert_id|group_key|state_version. If the version on the click no longer matches the row — someone else resolved it a minute ago — the handler answers that person privately instead of overwriting a newer state with an older intention. It is the same move as deduplicating on an idempotency key when an agent’s tool call can be delivered twice: a side effect against an external party has to be safe to receive twice.

Write the state before you call Slack

The order of the last two steps in a run decides whether an outage loses alerts.

tick
 ├─ compute the window (wall clock, aligned)
 ├─ query current and comparison windows      ── timeout, row cap
 ├─ evaluate → breached | healthy | error
 ├─ transition(prior, result) → new state + action
 ├─ MERGE alert_state; INSERT alert_history    ◄── commit here
 └─ action: post | update | suppress | digest  ◄── Slack, after the commit
       └─ on post: store channel + ts in alert_state

Slack down?  state is committed; next tick finds ACTIVE with no ts → posts it.
Reversed?    message exists, process dies before MERGE → buttons point at
             a state that does not exist; the retry posts a second message.

With state first, a Slack outage delays a notification and loses nothing. With Slack first, the failure produces the worst artefact an alerting system can have: a message nobody can resolve.

The receiver has the same rule with one more constraint: the 3-second acknowledgement. It acknowledges first, then writes state, then edits the message. It never blocks the acknowledgement on a database read.

That edit is also why the message timestamp is state rather than a log detail. Slack offers two ways to change a message after a click. A response_url can be used up to five times within 30 minutes of the click; chat.update works on messages the app itself posted, given the channel and ts. Auto-resolve happens hours later, from the scheduled side, with no click to hang a response_url on. So the detector needs the stored channel and ts, and chat.update is the primary path.

Where the row lives was the one decision left deliberately reversible:

At tens of alerts, one scheduled writer and occasional clicks, contention is rare and a bounded retry absorbs it. Paying for a database to serve a handful of row updates an hour is the wrong trade. So the recommendation was a Delta table behind a store interface, with the Postgres implementation designed in and switched on only if write conflicts appear in the logs. My own research notes had worried that the cold resume would land inside the acknowledgement budget. The documented figure says it is a fraction of it — and the acknowledge-first rule makes it irrelevant either way.

Time and noise belong to the state machine too

Most false alerts in systems like this come from time, not thresholds. A scheduler is not a clock: runs slip. So a window is computed from wall-clock time and aligned to its own size, never as “now minus the schedule interval”. With a 15-minute window and a 5-minute offset, a run at 08:07 evaluates [07:45, 08:00), and a run that slips to 08:11 evaluates exactly the same window. The offset has to exceed real ingestion lag, because a partially loaded window looks identical to a collapse. All arithmetic is in UTC, so an hourly schedule is not skipped or doubled twice a year. A metric with a weekly rhythm compares with the same window last week, not the previous one, or Monday morning alarms against Sunday night forever.

The client’s stated pain was volume, so noise control is requirements, not polish, and each mechanism is a field in the state:

• Consecutive breaches before an alert fires — the cheapest defence against a transient spike.
• A cooldown inside which an active alert updates state but stays quiet.
• Acknowledgement suppresses re-notification once someone owns the problem.
• Low-severity digests: one scheduled summary instead of many messages.
• Auto-resolve after a run of healthy evaluations, so the channel reflects reality.
• Backfill before enable. The same evaluation code runs over historical windows and reports how often a proposed rule would have fired. A rule that would have paged every day of the last month does not get switched on.

Alert definitions are config, reviewed in a pull request, with structured filters rendered as parameterised SQL. An illustrative definition, with an invented table:

id: orders_per_hour_by_region      # stable; the dedupe key
severity: P3
source: { catalog: analytics, schema: sales, table: orders }
metric:
  expression: count(*)
  timestamp_column: created_at
  group_by: [region]
window: { size: 1h, offset: 15m, compare_to: same_time_last_week }
condition: { type: delta_pct, operator: lt, value: -30, consecutive_breaches: 2 }
notify: { channel: "#data-alerts", cooldown: 2h, auto_resolve: true, digest: true }

A model may help write that file at design time. It does not write SQL inside a scheduled run at three in the morning. Non-determinism is the one property an alerting path cannot have.

The chart that goes with each alert is a smaller example of checking a platform fact before designing around it. Slack’s old files.upload method lost access for new apps on 16 May 2024 and was sunset on 12 November 2025; uploads now go through files.getUploadURLExternal, a direct POST of the bytes, and files.completeUploadExternal. My research notes had an earlier date. It made no difference to a new build, but it is why every date in this essay was re-checked against the source. One detail does change behaviour: completing an upload without a channel leaves the file hosted but private, shared nowhere, which looks exactly like a broken image.

What I would check before writing code

Three answers change the build, and none of them is about code. The standing cost of an always-on receiver decides between the App and the function. The real ingestion lag on the target tables sets the window offset, and an offset chosen by guess is a false alert waiting for a slow day. And the question of who may acknowledge and close decides whether the receiver needs an authorisation check of its own on top of Slack membership.

Everything else follows from the first decision. Once the alert is a row with a key, a version and a lifecycle, the transport is replaceable, the noise rules have somewhere to live, and an outage delays a message instead of losing one.

Frequently asked questions

Why can’t a scheduled job handle Slack alert buttons on its own?

Slack delivers a button click either to a public HTTPS Request URL or down a Socket Mode WebSocket, and expects an acknowledgement within 3 seconds. A cron job is not running at the moment of the click, so an alerting system with buttons needs a second, long-lived process to receive them.

Should alert state be written before or after posting to Slack?

Before. If the state write commits and Slack is down, the next run sees an ACTIVE alert with no message and posts it. If Slack is called first and the process dies before the state write, you have a message with buttons that point at state that does not exist, and a retry that posts it twice.

How do you make Slack button clicks idempotent?

Put the alert key and a state version in the button value. The handler compares the version with the stored row before transitioning. A stale click on a message someone already actioned gets an ephemeral reply instead of overwriting the newer state.

Is Slack Socket Mode suitable for an internal alerting receiver?

Yes, when the receiver cannot expose a public endpoint. Socket Mode opens an outbound WebSocket, so nothing arrives inbound. The costs are a long-lived connection to keep healthy and a process that runs, and is billed, continuously. Socket Mode apps cannot be listed in the public Slack Marketplace, which does not matter for an internal tool.

The same discipline — state and idempotency before transport — runs through running agent tools on a user’s machine. More on what to measure and what should page someone in Observability & SLOs.