What is sampling in tracing and what are the trade-offs? | LetsGit.IT

Observabilitymedium

What is sampling in tracing and what are the trade-offs?

Tags

#tracing#sampling#cost

Back to category Practice quiz

Answer

Sampling keeps only a subset of traces to control cost. It reduces storage and overhead but can hide rare failures or edge cases, so sampling strategy matters.

Advanced answer

Deep dive

Common strategies:

Head-based sampling: decide at request start (cheap, may miss errors).
Tail-based sampling: decide after completion (captures slow/error traces).
Rate-limited sampling: keep N traces/sec for baseline visibility.

Trade-offs:

Cost vs coverage.
Debuggability of rare issues.
Consistency across services.

Examples

Tail-based policy:

Keep all traces with error=true or duration > 2s
Sample 1% of the rest

Common pitfalls

Sampling too aggressively and losing error traces.
Mixed sampling decisions across services (broken traces).
No sampling for burst traffic, leading to cost spikes.

Interview follow-ups

When would you prefer tail-based sampling?
How do you ensure error traces are retained?
How do you sample for low-volume services?

Related questions

How do you investigate a latency regression in production?

#latency#incident#tracing

What is distributed tracing and how do you propagate context?

#tracing#context#distributed-systems

Logs vs metrics vs traces — when do you use each?

#observability#logs