You version-control your infrastructure. Every server, every network rule, every IAM policy lives in Terraform, reviewed in a pull request, with a history you can read. Then you configure your monitoring by clicking around a dashboard UI by hand, and it lives nowhere except that vendor’s database. This asymmetry is absurd, and it’s why your monitoring is a mess.
Here’s the tell. Ask anyone on your team why a particular alert exists, who created it, or whether it’s safe to delete. Nobody can answer. The alert has no history, no author, no review, no rationale. It just appeared one day and now everyone’s afraid to touch it in case it was important. That fear is the direct cost of click-ops monitoring.
The Problem With Click-Ops Monitoring
Monitoring configured by hand in a UI has the same failure modes that infrastructure-as-code was invented to solve. We just tolerate them for observability because “it’s just some alerts.”
- No history. You cannot see who changed a threshold, when, or why. When an alert starts misbehaving, there’s no
git blameto tell you what changed. You’re archaeology-ing through a UI’s audit log if one even exists. - No review. Alerts get created and edited with zero peer review. A threshold copied from a blog post, a dashboard built at 2 AM during an incident and never revisited — none of it was checked by a second person. The quality bar you enforce on code is completely absent for the thing that watches your code.
- No reproducibility. Your staging environment’s monitoring doesn’t match production, because they were both hand-built at different times by different people. You cannot stamp out identical monitoring for a new service, so each one gets a slightly different, slightly worse setup.
- Configuration drift. The dashboard someone tweaked live during an incident is now the source of truth, and it doesn’t match anything documented. Over months, the monitoring config drifts away from any shared understanding of it, and nobody can say what “correct” even looks like anymore.
None of these are exotic. They’re the exact problems Terraform solved for infrastructure. We just never applied the lesson to the layer watching the infrastructure.
What Monitoring as Code Actually Means
Monitoring as code means your alerts, dashboards, SLOs, and escalation rules are defined in text files, stored in version control, reviewed in pull requests, and applied through a pipeline — exactly like your infrastructure.
Concretely, that’s your alert definitions in YAML or a provider’s declarative config, your dashboards exported as JSON and committed, your SLO thresholds and burn-rate rules as code, all in a repository. A change to an alert threshold becomes a pull request that a colleague reviews before it merges. The current state of your monitoring is always readable in one place, in text, with a full history.
This isn’t a new tool you buy. It’s a discipline you apply, supported by capabilities most monitoring providers already expose through Terraform providers, APIs, or native config-as-code.
Why This Changes Everything
The benefits aren’t cosmetic. Version-controlling observability fixes the specific reasons monitoring rots.
Every alert has a rationale. When an alert is created in a pull request, the PR description explains why. Six months later, when someone asks “why does this exist,” the answer is in the commit history. The fear of deleting an alert evaporates, because you can read the reason it was added and judge whether that reason still holds.
Alert quality gets peer review. A new alert or a threshold change gets a second pair of eyes before it goes live. Bad thresholds, duplicate alerts, and noise-generating rules get caught in review instead of at 3 AM. This is a direct, structural attack on alert fatigue — you’re reviewing signal quality before it can wake anyone up.
New services get consistent monitoring for free. A new microservice inherits a monitoring template from the repository instead of being hand-instrumented from scratch. Every service gets the same baseline coverage, and the baseline improves for everyone at once when you improve the template.
Rollback is trivial. An alert change that turns out to be wrong is reverted with a single commit, and you can see exactly what the previous state was. No reconstruction from memory.
Your monitoring is documented by definition. The config is the documentation. There’s no separate wiki that drifts out of date, because the source of truth is the thing that’s actually running.
Where Teams Get Stuck
Monitoring as code is straightforward to describe and genuinely hard to sustain, and it’s worth being honest about why.
The initial migration from a hand-built setup to code is real work — exporting what exists, cleaning up the accumulated cruft, and getting it into a repository that actually reflects production. Most teams start this migration and abandon it half-done, ending up with a hybrid where some alerts are code and some are still clicked in, which is worse than either pure state because now nobody knows which source of truth to trust.
And the discipline has to hold under pressure. The entire value collapses the first time someone edits an alert live in the UI during an incident and doesn’t backport it to the repository. Now the code lies about production, and everyone learns not to trust it. Keeping monitoring-as-code honest requires enforcing that the repository is the only way changes happen — including during incidents, which is exactly when the temptation to click is strongest.
That enforcement is continuous ownership work. It’s the same pattern that shows up everywhere in reliability: the setup is a project, but keeping it true is a function that needs an owner. On a small team, the person disciplined enough to reject the live edit and demand the PR often doesn’t exist, and the practice quietly decays back to click-ops.
Treat Observability Like Production Code
The principle is simple and the payoff is large: your monitoring is production infrastructure, so treat it the way you treat production infrastructure. In version control. Reviewed. Reproducible. Owned. The alternative — a pile of hand-clicked alerts nobody understands or dares to touch — is a liability that compounds silently until an incident exposes it.
Vigil by IOanyT runs your observability as code and owns keeping it honest — alerts and SLOs in version control, changes reviewed, consistent monitoring templated across services, and the discipline enforced even during incidents so the config never lies about production.
Your engineers build. We keep the monitoring reviewed, reproducible, and true.
