The “three pillars of observability” (logs, metrics, and traces) might be one of the most successful marketing ideas in modern software.
It gives teams a clean checklist: collect logs, collect metrics, collect traces, and suddenly your systems are supposed to become transparent.
But reality is much messier.
At scale, these pillars do not stand neatly on their own. They overlap, break down, and often create more noise than clarity. The problem is not that logs, metrics, and traces are useless. The problem is treating them as separate products to buy instead of signals that should work together to answer one question:
Why is this broken?
Here is where the traditional observability model falls short, and what works better.
1. Logs stop being the source of truth at scale
Logs are the oldest debugging tool we have. In a small monolith with low traffic, they can feel like the source of truth. You search the logs, find the error, and usually have enough context to understand what happened.
In a distributed, high-throughput system, that breaks down fast.
Where logs fail
The signal-to-noise ratio collapses. When thousands of services are emitting debug logs, finding the useful line is not like finding a needle in a haystack. It is like finding a needle in a pile of other needles.
Sampling can hide the thing you needed most. Logging everything is expensive, so teams often sample aggressively or reduce log volume. That saves money, but it also means the one log line you needed during an incident may never have been stored.
Context gets lost. A log might tell you, “Service A timed out.” It often will not tell you that the timeout started with a user clicking “Checkout,” which triggered three downstream calls, one slow database query, and a retry storm.
What works better
Treat unstructured, free-form logging as a limited tool, not the center of your observability strategy.
Use structured events instead of plain text wherever possible. Log rich, consistent JSON events with fields like request ID, user ID, service name, endpoint, region, latency, and error type.
Be intentional about what you log. Default to actionable warnings and errors, then use dynamic log levels when you need deeper detail during an active investigation. Verbose logs are useful, but they should not be permanently blasting from every service in production.
2. Metrics tell you something is wrong, but not why
Metrics are cheap, compact, and fast to query. Counters, gauges, and histograms are great for dashboards, alerts, and trend analysis.
That is also their limitation.
Metrics are compressed by design. They are excellent at showing patterns, but they often lose the context needed to explain those patterns.
Where metrics fail
The cardinality trap is real. Metrics are great for questions like, “How many 500 errors did we get?” They struggle when you ask, “Which users saw those 500 errors?” or “Which customer, region, build version, and endpoint were involved?”
Adding too many unique labels can overwhelm traditional time-series systems and make costs explode.
Dashboards create the illusion of understanding. A CPU spike tells you something changed. It does not tell you whether users were affected, what request caused it, or which dependency started the problem.
Metrics are good smoke alarms. They are not always good fire investigators.
What works better
Focus less on infrastructure charts and more on user impact.
Alert on service-level indicators: checkout latency, payment success rate, error rate by endpoint, queue processing delay, API availability. These are closer to what users actually experience.
A CPU graph still matters, but it should not be your first page at 3:00 AM. If CPU is at 99% and the service is still fast, healthy, and meeting its reliability targets, that may not be an incident.
Use metrics to detect the problem, then connect them to deeper context. For example, metric exemplars can link a spike directly to a specific trace. Instead of staring at a dashboard and guessing, you click from the spike into the request that caused it.
3. Traces are powerful, but not free
Distributed tracing is probably the closest match to how modern systems actually behave.
A single user request might touch an API gateway, three services, a database, a cache, a message queue, and a background worker. A trace shows that path. It gives you the shape of the request, where time was spent, and where things failed.
That makes traces incredibly valuable.
It also makes them expensive and difficult to manage.
Where traces fail
Instrumentation takes real work. Traces do not appear magically. Getting trace context to move cleanly across languages, frameworks, queues, background jobs, and third-party services takes engineering effort.
One broken propagation point can turn a useful trace into a disconnected set of fragments.
Storing everything gets expensive fast. Every request creates spans. A single request might generate dozens of them. Multiply that by millions or billions of requests, and full-fidelity tracing becomes hard to justify financially.
Teams often respond by sampling traces randomly, but random sampling has the same problem as log sampling: it may discard the rare, weird, high-value request you actually needed.
What works better
Do not abandon traces. Be smarter about which traces you keep.
Use tail-based sampling where possible. Instead of deciding whether to keep a trace at the beginning of a request, wait until the request finishes. Then keep the traces that matter: errors, high-latency requests, unusual paths, retries, timeouts, and edge cases.
Discard the boring, healthy traffic. Keep the evidence.
And avoid locking your instrumentation to one vendor. Use OpenTelemetry as the standard layer for collecting telemetry. That keeps the instrumentation in your code portable, so you can change storage or analysis tools later without rewriting everything.
The bottom line
Observability is not about having three pillars.
It is about answering one question quickly:
Why is this broken?
Logs, metrics, and traces are useful, but they are not equally useful in every situation, and they should not live in separate silos.
The better model is connected observability:
- Metrics tell you when something is wrong.
- Traces show you where it went wrong.
- Structured events and high-cardinality context help explain why it happened.
The goal is not to collect more data. The goal is to reduce the time between “something is broken” and “we understand what caused it.”