The failures that never throw

Errors are the failures that get fixed

An error is a gift. It has a stack trace, a timestamp and a place in a queue. Somebody is paged, or at minimum somebody sees a count go up. The entire apparatus of modern operations — alerting, tracing, dashboards, on-call rotations — is built to convert a failure into an interruption, and it works.

Which means the failures that survive are the ones that never generate one.

Not because they are subtle in themselves. Because every component involved in producing them believes, correctly, that it did its job. There is nothing to catch, nothing to retry, nothing to log above debug level. The system is not degraded. It is doing exactly what it was told, and what it was told was slightly wrong in a way that no participant is positioned to notice.

Three that report success

Consider what these look like from inside the pipeline.

• A build inlines a credential into the client bundle. The bundler substituted a prefixed environment variable into browser-bound code, which is documented and intended behaviour. The build log says completed. The deploy notification is green.
• A query returns rows to a stranger. The database evaluated the policies attached to the table, found nothing that denied the request, and returned what was asked for. That is what a database is for. The response is a 200.
• A refund completes and the transfer is not reversed. The payments API executed a valid instruction. The customer is credited, the support ticket closes, and no line item anywhere says who ultimately paid.

Build completed in 42s
Deployed to production

Three different stacks, three different teams, no shared technology. The same shape: a success signal that is accurate about the operation and silent about the consequence.

Nothing errored. That is the problem — errors are the failures that get fixed.

Why the monitoring stack stays quiet

This is also why observability does not help, and it is worth being fair about that rather than treating it as a gap in someone's tooling. Monitoring answers whether the system is up, fast and erroring. Here availability is fine, latency is fine, and the error rate is zero — correctly zero. Every signal a monitoring system is designed to collect will say the system is healthy, and every one of those signals is true.

The organisational consequence is the part worth dwelling on. Engineering teams allocate attention by interruption: the thing that pages someone gets fixed this week, the thing in a backlog gets fixed eventually, and the thing that generates no signal at all is not in the backlog, because nobody has anything to write down. A failure class that produces no interruption does not compete badly for attention — it does not compete at all.

That also explains why these tend to be discovered in the same three ways, none of which is your own tooling. Somebody external reports it. A bill arrives that does not match expectations. Or a person goes and looks, usually for an unrelated reason, and finds something they were not searching for. All three are slow, and two of them are somebody else noticing first.

The tell, and what to do about it

The tell is worth learning to recognise, because it generalises well past these three examples. Whenever an operation's success is reported by the same component that performed it, and no separate observation confirms the result, you have the conditions for this. Sometimes that is fine — the operation is trivial or self-evidently correct. When money, credentials or access are involved, it is worth having something else look.

The only way to see this class of failure is to go and look at the result: the file the CDN actually serves, the rows an anonymous request actually gets back, the ledger entries that actually exist. Not the intent, not the logs, not the health check. The artifact.

That is a small, dull, entirely tractable habit, and it is the one thing none of the green ticks can do for you.

See the three products