It’s 3:17 AM. Your phone buzzes. “CPU utilization above 80% on prod-web-03.” You open one eye, check the dashboard, see it’s already recovering. You go back to sleep. At 3:42 AM, it buzzes again. Different alert, same story. By morning, you’ve been woken up three times. None of them mattered.
At 4:15 PM the next day, an alert fires that actually matters — a database connection pool exhaustion that will cascade into a full outage in 20 minutes. But the on-call engineer has already learned to check alerts slowly, skeptically. The response is 12 minutes late. The outage happens.
Alert fatigue isn’t an inconvenience. It’s a reliability risk. The boy who cried wolf, running on your infrastructure.
The Numbers Are Worse Than You Think
The average engineering team receives 2,000+ alerts per week. Only 3% require immediate human action. 73% of organizations have experienced outages directly linked to ignored or delayed alert response.
On-call engineers report spending 30-40% of their time triaging alerts that lead nowhere.
The conventional wisdom: “You need to tune your alerts better.” Every monitoring vendor, every SRE blog post, every DevOps conference talk says the same thing. Reduce noise. Set better thresholds. Use anomaly detection. Implement escalation policies.
The problem with this advice: it assumes the team has the time, expertise, and incentive to continuously maintain alert quality. It’s correct in theory. It fails in practice for the same reason gym memberships fail — ongoing discipline with no accountability structure.
Alert tuning is maintenance. Maintenance requires ownership. In most organizations, alert quality is everyone’s responsibility — which means it’s nobody’s responsibility.
Why Tuning Alone Can’t Fix This
Three structural problems prevent “just tune your alerts” from working:
The entropy problem. Infrastructure changes constantly. New services deploy weekly. Thresholds that were correct last month are noise this month. Alert tuning is not a project — it’s a continuous function. Teams that treat it as a project will always fall behind.
The expertise gap. Good alert tuning requires deep operational experience. Knowing that “CPU at 80%” is meaningless without context — is it sustained? Is it a burst? Is it a c5.xlarge or a t3.micro? — is the difference between an actionable alert and noise. Most engineering teams don’t have an SRE. They have developers who rotate through on-call.
The incentive misalignment. The engineer who gets woken up at 3 AM is not the engineer who wrote the alert rule. The engineer who wrote the rule left the company, or moved to another team, or just copied a threshold from a blog post. There’s no feedback loop between “this alert woke someone up for nothing” and “this alert rule should change.”
Tuning is necessary but insufficient. Without structural ownership of alert quality, entropy wins. Every time.
Alert Fatigue as an Ownership Problem
Alert fatigue is not a technical problem. It’s a management problem. Specifically, it’s a problem of who owns the quality of the signal.
Questions most teams can’t answer:
- Who is accountable for the signal-to-noise ratio of your alerting system?
- Who reviews alerts that fire but don’t result in action?
- Who removes or tunes alerts that have become noise?
- Who measures whether on-call burden is improving or worsening over time?
In most organizations, alerting is configured during setup, occasionally tuned during post-mortems, and otherwise neglected. It’s owned the same way a shared kitchen is owned — everyone uses it, nobody cleans it.
When you buy a monitoring tool, you’re buying the ability to create alerts. You’re not buying someone who cares whether those alerts are good.
What Solving This Actually Looks Like
The shift: from alert tuning as a side project to alert quality as a managed function.
What a managed alerting function does:
- Continuously reviews alert firing patterns: frequency, response time, resolution rate
- Removes or modifies alerts that fire without action (the 97%)
- Correlates alerts across services to reduce duplicate noise
- Escalates only actionable signals — with context, not just a metric name and threshold
- Runs weekly signal-to-noise audits and reports improvement over time
- Owns the pager — meaning the incentive to reduce noise is intrinsic
The accountability test: if the person tuning the alerts is also the person who gets woken up by bad ones, the feedback loop closes. If those are different people — or if nobody does it at all — the loop stays open.
Why this is a service, not a feature: no monitoring platform will ever ship this. It requires judgment, context, and willingness to carry the pager. That’s not a product feature. It’s an operational commitment.
The Path Forward
Alert fatigue isn’t inevitable. It’s a symptom of a responsibility boundary drawn in the wrong place.
Vigil by IOanyT manages your monitoring, including the part nobody else wants: the pager, the 2 AM pages, the signal-to-noise ratio, and the continuous improvement of alert quality.
Your engineers build. We watch. Nobody loses sleep.
