Green Inside, Down Outside

The certificate expired at midnight UTC.

Every instance stayed healthy. CPU, memory and disk did not move. Every health check passed, every alarm remained in OK, and the deployment pipeline was green. From inside the account, the platform had never been in better shape.

From outside, the product was unreachable for every customer in the world, and had been for nine hours by the time somebody’s tweet reached the founder.

Nothing failed that any of the monitoring was watching. That is not an oversight in the setup. It is a description of where the setup is standing.

Everything You Monitor Is Inside the Building

Infrastructure monitoring measures the things you run, from a vantage point inside your own network. That is the correct place to measure them from, and it means the entire first half of the customer’s request is out of frame.

A request to your product resolves a domain, which depends on a registrar and a DNS provider. It hits a CDN, which may terminate TLS with a certificate that has a date on it. It passes a WAF with rules somebody edited. It reaches a load balancer with a listener configuration. Only then does it arrive at the first thing your dashboard knows about.

You do not run most of that, and all of it can fail. Registrar auto-renew on an expired card. A DNS record changed during an unrelated migration. A CDN configuration deployed on a Friday. A WAF rule that starts blocking a legitimate pattern. A certificate with a ninety-day life and no automation behind it.

And /health is not designed to catch any of it. The health endpoint exists to answer the load balancer’s question — should I keep routing to this instance — and it answers it as cheaply as possible. On most estates we inherit it returns 200 without touching the database, calling a dependency, or exercising a single line of the code paths customers use. It is doing its job correctly. Its job is not to tell you the product works.

What Turns Up When You Actually Look

The first thing we do on a new estate is check it from outside, the way a customer would. The findings are consistent enough to be a checklist.

• No external checks at all. Or one, on the marketing homepage, set up years ago by whoever bought the domain. The application itself — the thing customers pay for — is unverified from outside.
• Certificates with nothing watching the date. The most preventable outage in this industry, and still one of the most common. It is worse where certificates are managed in more than one place: the load balancer’s is automated, the one on the legacy subdomain is not, and nobody knows the legacy subdomain is load-bearing.
• Domain expiry on a personal card. The registrar account belongs to someone who has changed roles, the renewal notices go to an address nobody reads, and the failure mode is total. This is the same category of finding as the unowned resources in your cloud inventory is already wrong — an asset that works perfectly until a date passes.
• Checks from a single location. So a regional CDN fault, a geo-routing mistake or a DNS propagation problem in one part of the world is invisible from the one place you happen to be looking.
• Checks that accept any 200. Following redirects and calling it success means a maintenance page, a “something went wrong” screen and a hijacked domain all pass. The check verifies that a web server answered, which was never in doubt.
• Nothing behind the login. Everything a customer actually does requires authentication, and almost nobody runs an authenticated check, because it needs a test account, credential rotation, and somebody to keep it working. So the entire product is unverified and the marketing site is thoroughly monitored.
• The API forgotten entirely. Where the integrations live, where the largest customers connect, and where a breaking change is least likely to be noticed by a human.
• No check on the dependencies. The payment provider, the identity provider, the email sender. When one of them degrades, the product degrades, and the first signal is support volume.

Every item is outside the blast radius of your infrastructure monitoring, and inside the customer’s. That is the whole point. These are not exotic failures. They are the ordinary ones, and they share a property: no metric you collect moves at all while they are happening.

Check It From Where the Customer Stands

The correction is small and unglamorous, which is part of why it stays undone — there is no project to propose.

Check from outside, from several places. Multiple regions, on a short interval, hitting the public address the way a customer does. Anything less answers a narrower question than you think.

Assert on content, not status. The check should look for something only a working page contains — an element, a string, a field in a JSON response. A status code tells you a server replied. A content assertion tells you it replied with your product.

Walk a journey, not an endpoint. Log in. Search. Add to basket. Call the API with a real token and check the shape of what comes back. This is the difference between knowing the site is up and knowing it works, and it is the same distinction that separates infrastructure metrics from the layer customers experience.

Monitor the dates as first-class facts. Certificate expiry and domain expiry are not really uptime checks — they are known future events with a countdown. Alarm at thirty days, not at zero. Every certificate, on every hostname discovery finds, including the ones nobody remembers.

Watch your dependencies’ status too. You cannot fix a third party’s outage, but knowing within a minute that it is theirs and not yours is most of the value.

The Part a Tool Can’t Do

External checks are the most commoditised thing in monitoring. Dozens of services will ping a URL from twelve countries for a few pounds a month, and configuring one takes about four minutes.

What takes judgement is deciding what to check. “Is the site up” is one question with an obvious implementation and limited value. “Does the thing customers pay us for still work” requires somebody to name the two or three journeys that matter, decide what a successful outcome looks like at each step, and accept that a failure there is worth waking a person for.

Then the journeys have to be kept alive. A synthetic check that walks a login and a search is a small piece of software coupled to your UI, and every redesign breaks it. A broken check that nobody repairs becomes a permanently red signal, which is worse than no check at all, because it trains people to ignore the thing that was supposed to be authoritative — alert fatigue arriving through the back door.

And somebody has to own the boring dates. Certificate inventories change as services come and go, and a list of hostnames assembled once is wrong within a quarter.

IOanyT Innovations has been running client infrastructure for over ten years, across more than 150 engagements. A meaningful share of the outages we have been called into after the fact were invisible to a fully functioning monitoring setup, because the thing that broke was in front of it.

Vigil runs external checks from multiple regions against every endpoint discovery finds — with content assertions, authenticated journeys where they matter, and certificate and domain expiry tracked as scheduled events rather than surprises. New endpoints enter the check set on the next discovery pass. Our team maintains the journey scripts as your product changes, and takes the page when one fails.

Your engineers build the product. We watch it from where your customers are standing.

See what’s included in monitoring →

Start with a free infrastructure assessment →