Observability in Practice: The Sampling Decisions You Are Making Without Knowing

Nobody stores everything. At any real volume, full-fidelity retention of every trace, every log line and every metric series at full cardinality is unaffordable, and the vendors who sell you observability know this better than you do.

So you’re sampling. The only question is whether you chose how, or inherited it from a default, a budget conversation you weren’t in, or a limit you’ll discover at 3am.

Most teams can’t answer basic questions about their own sampling posture. That’s not negligence — it’s that these decisions are distributed across six places, none of them labelled “sampling.”

Head-Based Sampling Throws Away the Interesting Part

Trace sampling comes in two shapes, and the cheap one has a structural flaw.

Head-based sampling decides at the start of a trace, before anything has happened. Keep 1%, discard the rest. It’s cheap, it’s stateless, and every component can make the decision independently as long as they propagate it.

It also means you discard errors at exactly the same rate as successes.

Head-based sampling throws away failures and slow requests at the same rate as everything else, which is precisely backwards.

The traces you want are the rare ones. A 1% head-based sample of a service doing a million requests a day keeps ten thousand traces, of which the overwhelming majority are the boring successful path you already understand. The 40 requests that hit the pathological timeout become, on average, fewer than one retained trace.

Tail-based sampling buffers the whole trace and decides after it completes, so you can keep every error, every request over 2 seconds, every trace touching a particular customer, and 0.1% of the rest. It is strictly better at the job and considerably more expensive — the collector has to hold spans in memory until the trace finishes, and every span of a given trace has to reach the same collector instance, which is a real constraint once you scale collectors horizontally.

Most teams run head-based because it’s what the default configuration does.

Metrics Sample Too, in Two Dimensions

Traces get all the sampling discussion. Metrics are sampled just as aggressively and almost nobody frames it that way.

In time. A scrape interval is a sampling rate. At 15 seconds, a CPU spike lasting three seconds may never be observed at all. Not smoothed — absent. Your graph is a series of instants connected by lines you’re interpreting as continuity, and the gaps are wide enough to hide the thing that woke you up.

In value. Your p99 is not measured. For a histogram, it’s interpolated from bucket boundaries somebody picked — frequently the client library’s defaults, which were chosen for a generic HTTP service and not for yours. If your buckets jump 100ms → 500ms and your real p99 sits at 180ms, the number on your dashboard is a guess between two boundaries, and it will stay wrong consistently enough to look credible.

Your p99 is not measured. It is interpolated between two numbers someone chose as defaults.

Check your bucket boundaries against your actual latency distribution once. Most teams find the resolution is in the wrong place entirely — fine-grained where nothing happens, coarse exactly where the interesting tail lives.

The Silent Drops

Then there’s the category that isn’t configured as sampling at all, and behaves like it.

Log rate limits. Logging libraries drop under pressure. Agents have buffers that overflow. Ingestion endpoints throttle. All of these shed load precisely when volume spikes — which is during the incident, about the logs you are about to need.

Cardinality caps. Every vendor has one. Past some number of unique series, your per-customer or per-endpoint dimension gets dropped, collapsed into an “other” bucket, or silently stops recording new values. The metric still exists. It just no longer means what you think.

Retention rollups. Full resolution for seven days, then five-minute averages. A spike that lasted ninety seconds and defined your worst outage of the quarter is, by the time of the post-incident review, a barely visible bump in an average.

None of these announce themselves. They don’t fire an alert. The data is simply not there later, and you find out while looking for it.

Why This Bites Specifically During Incidents

Normal operation is the condition your sampling was tuned for, and normal operation is when you don’t need the data.

Incidents invert every assumption. Volume spikes, so rate limits engage. Error rates climb, so if you’re tail-sampling on errors your collector memory balloons at the worst time. New error strings appear, so cardinality jumps and caps trigger. Everything is anomalous, which is exactly the population your 1% uniform sample is worst at representing.

And the investigation you want to run is almost always specific — this request, this customer, this one slow call at 02:47. Aggregate sampling is fundamentally bad at specific. You can have statistically valid percentiles and still be unable to answer “what happened to this order,” because that trace was one of the 99%.

Writing the Policy Down

The fix isn’t more data. It’s knowing what you kept.

Write down, per signal, what your sampling posture actually is. Trace sampling rate and strategy. Scrape interval. Histogram buckets. Log rate limits at every hop, including the agent and the ingest endpoint. Cardinality limits for each vendor. Retention tiers and what each rollup discards. Most teams cannot produce this document today, and assembling it the first time reliably turns up two surprises.

Then bias toward the interesting rather than sampling uniformly. Keep every error. Keep everything over your latency objective. Keep 100% for a handful of named high-value customers. Sample the boring successful path hard — that’s where the volume is and the least information per trace.

And run one test. Pick a request that happened three days ago and try to reconstruct it end to end. If you can’t, you now know your real retention policy, as opposed to the one in the contract.

The goal was never to keep everything. It’s to make sure the thing you threw away was the thing you chose to throw away.