One Channel, Every Severity

There is a Slack channel called #alerts. It receives around four hundred messages a day.

At 14:06 it gets a disk usage warning from a build agent. At 14:07 it gets a notification that the payment provider integration is returning errors on every request. Both are grey. Both have the same title format. Both are addressed to nobody in particular. Neither has a severity a human can see at a glance, and the second one scrolls out of view in ninety seconds.

Six of the nine people in that channel have it muted. The three who have not are the three who feel worst about it.

This is not the alert fatigue problem. That one is about how many alerts fire, and it is a real and separate thing. You could fix it completely — tune every threshold, delete every noisy rule, get the channel down to four messages a day — and still have this problem, because this one is about where alerts go and who is expected to act.

Routing Is the Last Step, So It Never Happens

Setting up monitoring has an order to it. Deploy the agents. Collect the metrics. Build the dashboards. Write the alert rules. Connect a notification channel to prove the rules work.

That last step is where it stops, and the reason is entirely reasonable: one webhook is enough to demonstrate that alerting functions. The rest — severities that mean something, destinations per severity, an owner per service, a schedule, an escalation path — is a second project that nobody scoped, and its absence causes no visible failure on the day.

Routing needs information the tool does not have. An alert can only be sent to the right person if the system knows which service it belongs to, who owns that service, who is on call for it this week, and what to do if they do not answer. That is organisational information. It lives in people’s heads and in an org chart nobody has ever expressed as configuration, and keeping it current is an administrative commitment with no natural owner.

Severity gets set once and stops meaning anything. Every alerting tool has a severity field. On most estates we take over it has been filled in with a single value — either everything is warning or, more commonly, everything is critical, because the person writing the rule cared about it at the time. A field where every row has the same value carries no information at all.

And the failure is silent in the reassuring direction. Alerts are firing. The integration works. Somebody watching the channel would see them. Every part of the system reports success, and the only symptom is that a specific alert, on a specific night, did not reach a specific person — which nobody discovers until the morning after.

What Turns Up When You Actually Look

Ask to see the notification configuration rather than the alert rules, and the pattern repeats across estates.

• One channel for everything. Muted by most of the people in it, which is a rational response to a stream that cannot be triaged and mostly is not urgent.
• Alerts addressed to an individual. The engineer who set the monitoring up, by name, on their personal number or email. Sometimes that person has left the company and the alerts are still going to a mailbox that bounces silently.
• No escalation at all. If the first notification is not acknowledged, nothing further happens. The entire out-of-hours process depends on one person’s phone being charged, audible, and next to them.
• Escalation to the same person. Technically configured, functionally decorative.
• Routing into a channel that was archived. Created for a migration that finished eighteen months ago. Messages are still being delivered, in the sense that the API accepts them.
• No difference between working hours and the middle of the night. The same rule pages at 15:00 and 03:00, so either the daytime alerts are too aggressive or the night-time ones are not aggressive enough. Usually both, at once, for different rules.
• No distinction between degrading and down. A replica falling behind and a total outage produce identical notifications, so the recipient has to open both to find out which one they are in.
• The real P1 path exists in conversation only. Everyone knows that if something is truly broken you message a particular senior engineer directly. That is the actual escalation policy, it is undocumented, it does not survive that person taking a holiday, and it is why the 3 AM test is worth running: ask who gets the alert, and the honest answer is usually a name, not a rota.

The cost lands entirely on the alerts that mattered. Nobody is harmed by a disk warning going to the wrong place. The damage is that the one notification a year which needed an immediate response arrived in the same undifferentiated stream as everything else, and was treated accordingly.

Severity Means Customer Impact, or It Means Nothing

The fix is not a better integration. It is a set of decisions, written down, that the integration then implements.

Define severity by what the customer experiences. Not by which metric fired, not by how alarming the number looks. Critical means customers cannot use the product, or are about to be unable to. Degraded means it is worse than it should be and someone should look during working hours. Informational means it belongs in a queue and nobody should be interrupted. Three levels is usually enough, and every rule gets placed against that definition rather than against how the person writing it felt.

Give each severity a destination and a promise. Critical pages a phone, with an acknowledgement expected in minutes and an automatic escalation to a second person when it is not. Degraded goes to a team channel that is read during the day. Informational becomes a ticket. The destination is what communicates urgency, far more reliably than a word in the message body.

Attach every alert to a service, and every service to an owner and a rota. This is the administrative half, it is the part that decays, and it is the part that makes the rest work. An alert that cannot name its service cannot be routed, and the fix for that is at rule-writing time, not at 3am.

Make escalation automatic and timed. Unacknowledged in five minutes goes to the next person. Unacknowledged again goes to whoever is accountable. No step in that chain should depend on somebody noticing that nothing happened.

Then test it. Page the rotation deliberately, on a quiet afternoon, once a quarter, and watch what actually occurs. This is the only way to discover the archived channel, the bouncing mailbox and the phone with notifications disabled, and it takes twenty minutes.

The Part a Tool Can’t Do

Every paging product does this well. Schedules, escalation policies, overrides, routing rules — the mechanics have been solved for a decade and cost very little.

What no product can supply is the decision about whose sleep is worth interrupting, and for what. That is a commitment an organisation makes to itself: this class of failure is important enough to wake a named person at 3am, and this one is not. Writing it down means somebody has to agree to be woken, which is why it stalls — the configuration is trivial and the conversation behind it is not.

The mapping also rots faster than almost anything else in monitoring. People change teams, services change owners, rotas change shape, and none of those events triggers a review of the alerting configuration. Nothing fails when the mapping goes stale. It simply becomes wrong, quietly, and stays wrong until an incident tests it.

And the rota underneath it has to be humane enough that people stay. Routing every critical alert to one person works perfectly for about four months, which is roughly how long it takes to burn out the person receiving them.

IOanyT Innovations has been running client infrastructure for over ten years, across more than 150 engagements. We have almost never inherited an estate with no alerting. We have very frequently inherited one where every alert went to the same place and the real escalation policy was a person’s name.

Vigil maps every alert to a service, every service to an owner, and every severity to a destination with a timed escalation behind it — and then puts our on-call team at the front of the critical path, so the first phone that rings is ours. We test the routing on a schedule rather than waiting for an incident to test it for us.

Your engineers get the alerts that need an engineer. The rest reach us, and stop there.

See what’s included in monitoring →

Start with a free infrastructure assessment →