Ask an engineering leader how many production resources they run. You’ll get a number. Ask where the number came from, and you’ll get a spreadsheet, a wiki page, or a Terraform state file. Ask when it was last reconciled against the actual cloud accounts, and the conversation stops.
That pause is the problem. The number is a snapshot of a decision someone made months ago. The infrastructure kept moving after the snapshot was taken.
Nobody plans for this. It happens through a series of individually reasonable actions:
- An instance is launched to absorb a traffic spike, and never terminated.
- A service moves to a second account during a migration. The old copy keeps running.
- A domain quietly expires because the person who registered it changed jobs.
- A team stands up a workload for a pilot that nobody outside that team ever hears about.
None of these are failures of discipline. They’re the normal exhaust of a system people are actively working on. And every one of them lands in the same blind spot.
Anything Not in the Inventory Is Not Monitored
That sentence is what turns an accounting problem into an outage.
No alert rule points at a resource you don’t know about. No uptime check hits it. No dashboard shows it. No runbook mentions it. When it breaks, nothing fires — because nothing was watching. You find out when a customer emails you.
Your coverage numbers won’t warn you. Coverage, uptime, mean time to detect: every one of them is measured against the things you’re aware of. If the denominator is wrong, the metrics are reassuring and meaningless at the same time. MTTR is four numbers wearing a trench coat, and the first of them — time to detect — isn’t slow for an unknown asset. It’s undefined.
Neither will your vendor. A monitoring platform reports on what you told it to watch. It has no opinion about what you left out. The 3 AM Test asks who gets the alert, who triages, who diagnoses, who remediates. All five questions assume an alert fired. Nothing in that chain starts if the thing that broke was never on the list.
Why We Built Discovery Before Dashboards
IOanyT Innovations has been running and managing infrastructure for clients for over ten years, across more than 150 engagements — most of those estates inherited rather than built by us.
At that scale the inventory problem stops being theoretical. Every promise you make to a client is downstream of knowing what they actually have. Uptime coverage. Patching. Cost control. Incident response. Evidence for an audit. All of it rests on an accurate answer to one question: what exists, in which account, owned by whom.
Get that answer wrong and every guarantee built on top of it is wrong too. You don’t find out until the thing you didn’t know about is the thing that’s down — and you’re the one who promised it wouldn’t be.
Manual inventory does not survive contact with that. We tried. Onboarding checklists. An asset register per client. Quarterly reconciliation. It held for roughly as long as it took the client’s engineers to ship something. The register was accurate the day it was written and decaying by the end of that week.
Worse, it was confidently wrong. It looked authoritative. It had owner columns and a last-updated date. So people stopped questioning it — which is the exact failure mode you don’t want in the document everything else depends on.
So the first thing Vigil had to do was not dashboards, and not alerting. It was discovery: a system that goes and looks, continuously, and reports what is actually there. We built it because we were the ones on the hook when the answer was wrong.
This is a different failure from the one in Monitoring as Code. That post is about monitoring configuration drifting away from the definition you wrote down. This is about the estate drifting away from the list of things you think you’re configuring. You can have perfect, version-controlled, peer-reviewed monitoring and still be blind, because none of it knows the resource exists.
What Turns Up When You Actually Look
Every estate we take over starts the same way. A read-only role. An enumeration of every account, region and managed service across every provider in use. Then the result laid next to whatever the team currently believes it is running.
The gap is never zero. What is striking is how little it varies. The same few categories come up again and again:
- Compute in a region nobody watches. Spun up for a load test, a migration rehearsal, or a seasonal capacity bump. The work finished. The instances didn’t.
- Accounts outside the process. Created for a proof of concept, an acquisition, or a team that needed to move quickly once. They never reached the central list, because the central list was written before they existed.
- The old copy after a move. A service migrates to a new account or cluster. The new one gets monitored. The old one is still up, still reachable, and no longer receiving updates.
- Resources with no owner. Not unused — unowned. Nobody can tell you this week who is responsible if it breaks. Establishing that is usually the slowest part of the whole exercise.
- Domains and certificates registered to a person. Set up by someone who has since changed teams or left. Nothing fails until a renewal doesn’t happen.
It is never one dramatic miss. That is what makes it hard to catch. Every item arrived through a decision that was correct at the time, made by a competent person, in a week when something else was on fire. There is no moment where somebody did something unreasonable — which is precisely why a process that depends on someone remembering will keep producing this result.
The teams this happens to are not careless. The spreadsheets we are handed are usually good spreadsheets: owner columns filled in, a last-updated date, maintained by someone who takes it seriously. That is the uncomfortable part. Care is not the constraint. The constraint is that a document maintained by hand can only ever be as current as the last time a human touched it, and the estate does not wait for that.
The cost lands twice. Once as risk: production surface with no detection path at all, where the first signal is a customer noticing. Once on the invoice, in resources nobody has reviewed since the day they were created. That second one is the same pattern behind the $700K infrastructure illusion — spend nobody adds up, because nobody can see all of it at once.
Derived, Not Maintained
The fix is not a better spreadsheet, a stricter process, or a quarterly audit with more people in the room. It is removing the human step from the inventory entirely.
Discovery runs continuously across every account and provider in the estate. Each pass enumerates what exists and compares it against the last one. A new resource appearing is an event, not something someone notices six weeks later.
Three things follow, and together they are the operating model:
Coverage is the default, not a task. When discovery finds a new instance, database or endpoint, it enters monitoring automatically — metrics through Terraform-deployed agents, uptime checks on anything with a URL. Nobody has to remember. The distance between “this exists” and “this is watched” is one cycle, instead of however long it takes someone to notice.
Ownership is settled at discovery, not audited later. Anything that appears without an owner is an exception, and exceptions get worked that week. That is the difference between an inventory with an owner column and an inventory where the owner column means something.
Removal is tracked as carefully as addition. The same comparison that catches new resources catches the ones that should be gone and aren’t. The early passes over a new estate almost always produce a decommission list the team did not have — a cost result as much as a reliability one.
The change is small to describe and large in practice. The inventory stops being a document somebody maintains and becomes a read of the estate itself.
| Maintained inventory | Derived inventory | |
|---|---|---|
| What it is | A document someone updates | A read of the accounts themselves |
| Accurate | On the day of the audit | Every cycle |
| A new resource is | Noticed, eventually | An event |
| Monitoring covers | Whatever was on the list | Whatever exists |
| It goes stale when | Someone gets busy | It doesn’t |
| Who keeps it true | Whoever has time | Whoever runs the cycle |
The Part a Tool Can’t Do
Continuous discovery is not a hard idea. Enumerate the accounts, compare against last time, surface the delta. A capable platform team could build it in a sprint, and some have.
The hard part is everything after the delta.
Discovery produces a list of things you didn’t know about, and that list is worth nothing until somebody works it — assigns the owner, decides what gets decommissioned, tunes what the new monitoring should actually alert on, and does it again next cycle when the list has changed.
That work never finishes, which makes it exactly the kind of work that loses to a product deadline. Not because anyone decided it was unimportant. Because it is always possible to do it next week, and there is always a next week.
Vigil by IOanyT runs discovery continuously across AWS, Azure, GCP and DigitalOcean, inside your own cloud account. New assets are instrumented and uptime-checked as they appear. Ownership exceptions and decommission candidates come to our team — not to a queue on your platform engineer’s desk.
Your engineers build. We keep the inventory honest, and everything in it watched.
