Key takeaway: Every distinct label value combination creates a separate stored time series. One thoughtless label can multiply storage by five orders of magnitude.
The Multiplication Nobody Sees Coming
A counter with labels for method, endpoint and status is well behaved: five methods, forty endpoints and six status codes yield at most 1,200 series.
Add user_id and each of those 1,200 multiplies by your active user count. Ten thousand users produce twelve million series. The instrumentation change was one line and looked harmless in review.
Metrics systems store and index per series, not per data point written. Cost, memory and query latency all scale with series count. This is why monitoring bills sometimes grow tenfold after a release that changed no traffic volume whatsoever.
Labels That Reliably Cause Damage
| Label | Cardinality | Verdict |
|---|---|---|
user_id, session_id, request_id |
Unbounded | Never |
url with path parameters |
Unbounded | Template it |
error_message |
Unbounded | Use an error code |
timestamp |
Unbounded | Never |
pod_name |
Grows with churn | Careful |
status_code, method, region |
Bounded | Fine |
Raw URLs are the most common accident. /orders/8f2a91c4 becomes a distinct series per order. The fix is templating at instrumentation time so the label reads /orders/{id}, which is what you actually want to aggregate on anyway.
Container names deserve attention in dynamic environments. Every deploy creates new pod names, so series accumulate continuously even though the number of running pods is constant.
Choosing the Right Signal
The underlying error is asking metrics to answer questions metrics cannot answer. “Which requests failed?” is not a metrics question — it is a traces or logs question.
Metrics answer aggregate questions cheaply at any retention: how many, how fast, what proportion. Traces answer per-request questions with full context, sampled to control cost. Logs answer detailed narrative questions.
Putting a user ID in a metric label attempts to make metrics do a trace’s job at a metric’s storage model, which is the worst combination of both. Put the user ID in the trace and the log, where per-item storage is expected and sampling controls the cost.
Controlling It Operationally
Query series count by metric name to find your top consumers — the distribution is usually extremely skewed, with a handful of metrics dominating. Set a cardinality limit at the ingestion layer so a bad deploy is rejected rather than absorbed. Add a review checklist item asking whether any new label can take unbounded values, since this is one of the rare problems that is trivial to prevent and painful to remediate.
The Bottom Line
Keep metric labels to bounded, low-cardinality dimensions. Route per-request identifiers to traces and logs. Then monitor your own series count as a first-class metric, because it predicts both cost and query performance.
