Your Dashboards Are Lying: Monitoring vs Observability in Practice

0 0 8 minutes read

Key takeaway: Monitoring tells you a known thing broke. Observability lets you investigate something you never predicted. The practical difference is whether you can slice your data by dimensions you did not decide on in advance.

The Question You Did Not Anticipate

Your dashboards are green. Error rate is normal. CPU is fine. Latency looks acceptable. And a customer is on the phone saying the application is unusable.

This situation is nearly universal, and it is not caused by insufficient dashboards. It is caused by what a dashboard fundamentally is: a set of answers to questions someone thought of earlier. Every panel encodes a prediction about how the system would fail.

Production systems fail in ways nobody predicted. The failure affects users of one specific plan tier, on one API version, in one region, whose accounts have more than a certain number of records. No dashboard exists for that intersection, and no dashboard could — the number of possible intersections is combinatorially larger than the number of panels anyone would build.

The distinction matters practically rather than semantically:

Monitoring answers “is the thing I predicted happening?” Predefined metrics, thresholds, dashboards. Essential and bounded by imagination.

Observability answers “what is happening, and why?” for questions formed during the incident. It requires data rich enough to slice along dimensions nobody chose in advance.

Most organisations have extensive monitoring and call it observability because they bought a tool with that word in the marketing.

Why Averages Hide Your Worst Failures

Start with the single most common measurement error: average latency.

Consider 1,000 requests. Nine hundred and ninety complete in 50 milliseconds. Ten take 30 seconds. The average is roughly 350 milliseconds — a number that looks acceptable and describes not one actual request. Nobody experienced 350 milliseconds. Some people had an instant response; ten people had a broken application.

Averages are structurally incapable of showing the failures that matter, because outliers are exactly what averages exist to suppress.

Percentiles fix this, provided you use enough of them. The p50 tells you the typical experience. The p95 and p99 tell you what your unluckiest users experience — and those users are the ones who complain, churn, and file support tickets.

Measurement	990 fast + 10 broken requests	What it tells you
Average	~350 ms	Nothing real
p50	50 ms	Typical case is fine
p95	55 ms	Most users fine
p99	~28,000 ms	Something is badly broken

One additional subtlety catches teams out: percentiles cannot be averaged. Averaging the p99 across ten servers does not produce the system’s p99. It produces a number with no meaning. Percentiles must be computed from the underlying distribution, which is why storage format matters and why some tooling gives wrong answers confidently.

High Cardinality Is the Whole Point

If there is one technical concept that separates observability from monitoring, it is cardinality — the number of distinct values a dimension can take.

Low cardinality: region (a handful of values), HTTP method (fewer), environment (two or three). Traditional metrics systems handle these well.

High cardinality: user ID, request ID, account ID, session ID, feature flag combination. Millions of distinct values. Traditional metrics systems handle these catastrophically, because each unique combination of label values creates a separate time series to store and index.

This matters because high-cardinality dimensions are what you need during a real investigation. “Latency is elevated” is not actionable. “Latency is elevated for requests from accounts created after the migration, on API v2, hitting the search endpoint” is a diagnosis. Reaching the second statement requires slicing by dimensions with many possible values.

The architectural consequence is that pre-aggregated metrics cannot support this. If you aggregate latency by endpoint at collection time, the per-account detail is gone permanently. You cannot recover a dimension you discarded.

This is the case for event-based telemetry: emit one structured record per unit of work, containing every dimension you might later want, and aggregate at query time rather than collection time. Storage is more expensive. The ability to ask unanticipated questions is what you are buying.

Structured Logs Beat Grep

Most logging is unstructured prose, which makes it unqueryable at scale.

# Unqueryable: information trapped in a sentence
log.info(f"User {user_id} checkout failed after {ms}ms: {error}")

# Queryable: dimensions available for filtering and aggregation
log.info("checkout_failed", extra={
    "user_id": user_id,
    "account_id": account_id,
    "plan_tier": plan,
    "duration_ms": ms,
    "error_type": type(error).__name__,
    "payment_provider": provider,
    "retry_count": retries,
    "trace_id": trace_id,
})

The first version supports text search. The second supports “show me p99 checkout duration grouped by payment provider for enterprise accounts over the last hour” — which is the kind of question that actually resolves incidents.

Two practices multiply the value:

Attach context automatically. Request ID, user ID, and trace ID should be injected into every log line within a request’s scope by middleware, not passed manually. Manual propagation gets forgotten precisely in the error paths where it matters most.

Log the whole unit of work once. Rather than scattering five log lines through a request, emit one rich event at completion containing everything: inputs, outcome, timings for each phase, and any errors. One wide event is easier to query than five narrow ones and cheaper to store.

Where Tracing Earns Its Cost

Distributed tracing follows a single request across service boundaries, producing a timeline of every operation with parent-child relationships.

The value is specific and largely irreplaceable: it answers “where did the time go?” in a system where a single user action touches eight services. Logs tell you each service’s view. Traces tell you the shape of the whole request — which call was slow, which ran unexpectedly in sequence rather than parallel, which was retried four times.

Two failure patterns are essentially undiagnosable without tracing. The N+1 pattern, where a service makes hundreds of small calls that individually look fine and collectively dominate latency, is invisible in per-service metrics and obvious in a trace. And cross-service retry amplification, where each layer retries three times and the innermost service receives 27 requests for one user action, only becomes visible when you can see the call tree.

The cost is real. Instrumentation must propagate context across every service boundary, including through queues and async work, and any service that drops the context breaks the trace. Storage is expensive at full volume, which is why sampling is standard. Tail-based sampling — keeping all traces that contain errors or exceed a latency threshold, sampling the rest — captures the interesting cases at a fraction of the cost.

Trace IDs should appear in every log line. Being able to move from a log entry to the full request trace, and back, is where the pieces combine into something more useful than either alone.

Alert on Symptoms, Not Causes

Alerting is where most teams accumulate the most pain, and the root cause is usually alerting on causes rather than symptoms.

Cause-based alerts — CPU above 80 percent, memory above 90 percent, disk queue depth elevated — fire constantly during normal operation and frequently do not correspond to user impact. High CPU on a batch processing node is expected. Paging someone for it trains them to ignore alerts.

Symptom-based alerts describe user-visible degradation: error rate exceeded, p99 latency above the threshold users tolerate, request throughput dropped unexpectedly, queue backing up faster than it drains. These correspond to actual harm.

The practical test for any alert: if this fires and nobody does anything, does a user suffer? If no, it is not a page. It may be a dashboard panel or a ticket, and those are fine — they are just not worth waking someone.

Alert fatigue is the failure mode that matters most, because it is self-reinforcing. Every low-value page reduces the response quality to the next real one. A team receiving thirty alerts daily is a team that no longer reads alerts. Fewer, higher-signal alerts genuinely produce faster incident response than comprehensive coverage, which is counterintuitive enough that teams resist it until they have experienced both.

Controlling the Bill

Observability data volume grows superlinearly with traffic, and observability bills exceeding infrastructure bills is common enough to be a recognised problem.

Tactics that reduce cost without destroying utility:

Sample traces, keep all errors. Head-based sampling at 1 to 10 percent for successful requests, plus tail-based rules retaining everything that errored or exceeded a latency threshold. The interesting traces are a small fraction of total volume.

Tier retention by data type. Detailed events for a week, aggregated metrics for a year. Nobody queries raw request events from four months ago; everyone queries the monthly trend.

Drop dimensions you never query. Audit actual query patterns and remove fields nobody has filtered on in months. There are usually several.

Aggregate at the edge for known questions. For metrics you always want — total request rate, error rate by endpoint — pre-aggregation is far cheaper than computing from raw events. Keep raw events for exploration, use aggregates for dashboards.

Do not log at debug level in production. Obvious, routinely violated, and frequently the single largest volume contributor.

Common Pitfalls

Buying a tool and declaring victory. Tools store and query data. If your services emit unstructured logs without context, an expensive platform stores unstructured logs expensively.

Instrumenting infrastructure but not business logic. CPU and memory rarely explain why checkout is failing. Instrument the operations that matter to users.

Unbounded cardinality in metric labels. Putting user ID in a metrics label creates millions of time series and will exhaust your metrics backend. High cardinality belongs in events, not in pre-aggregated metrics.

Dashboards nobody looks at. Panels accumulate and are never deleted. A dashboard with eighty graphs conveys less than one with eight.

No correlation between signals. Logs, metrics, and traces in separate systems with no shared identifiers means manual correlation during incidents, which is exactly when nobody has time.

Conclusion

The difference between monitoring and observability is not tooling. It is whether your telemetry preserves enough dimensionality to answer questions you had not thought of when you wrote the instrumentation.

Practically, that means emitting structured events with many attributes rather than pre-aggregated counters, measuring percentiles rather than averages, propagating trace context so requests can be followed across services, and alerting on user-visible symptoms rather than machine-level causes.

Start with structured logging and correlation IDs — the cheapest change with the largest immediate return. Add percentiles wherever an average currently sits. Introduce tracing when service count makes request paths genuinely hard to reason about. And prune alerts ruthlessly, because an alert nobody trusts is worse than no alert at all.

Frequently Asked Questions

Do I need distributed tracing for a monolith? Usually not for cross-service visibility, since there are no service boundaries. In-process timing spans can still help identify slow phases, but the return is much lower than in a distributed system.

What sampling rate is appropriate for traces? 1 to 10 percent of successful requests, plus tail-based rules keeping all errors and slow requests. Low-traffic services can afford higher rates; high-traffic services usually cannot afford much.

Should I run my own observability stack or buy one? Self-hosting trades a large recurring bill for engineering time and operational risk — the observability stack becomes a production system requiring on-call coverage. Vendors cost more and remove that burden. The crossover point depends on data volume and available engineering capacity.

How long should telemetry be retained? Detailed events for 7 to 30 days covers nearly all incident investigation. Aggregated metrics for 12 months supports capacity planning and trend analysis. Retaining raw events for a year is expensive and rarely queried.

Is OpenTelemetry worth adopting? Yes, primarily for vendor portability. Instrumenting with a vendor-specific agent means re-instrumenting to switch vendors. A vendor-neutral standard makes that a configuration change rather than a project.

How do I convince management to fund this? Frame it as incident duration. Time spent investigating is the cost, and it is measurable. If incidents routinely include hours of “we cannot tell what is happening,” that time has a number attached that usually exceeds the tooling cost.

Can I add observability to a system I did not build? Partially. Automatic instrumentation captures HTTP calls, database queries, and framework operations without code changes, which is a reasonable start. Business-level events require touching the code, and that is where the highest-value signal usually lives.

Your Dashboards Are Lying: Monitoring vs Observability in Practice

Table of Contents