You Probably Don’t Need Distributed Tracing. Until You Do.
Distributed tracing is one of those tools that feels essential once you’ve solved metrics and logs. And then you realize you’re paying infrastructure costs to watch a single request bounce between three services, and you think, “I could’ve just added a log statement.”
But there’s a sweet spot, especially in self-hosted setups where you’ve got multiple services talking to each other, and something occasionally breaks in weird ways. That’s where Tempo + Grafana shines. It’s not Jaeger. It’s not complicated. It’s the observability equivalent of a socket wrench instead of a full toolbox.
What Is Distributed Tracing, Actually?
Imagine a request hits your API, which calls your database, which triggers a cache invalidation. In a monolith, you read the logs. In a multi-service setup, the request ID gets logged in five places across four machines, and you’re stitching them together manually.
Distributed tracing does that stitching for you. A trace is a request’s journey through your system, broken into spansindividual operations. Service A starts a span, calls Service B, which starts its own span and logs context back to A. Every span tracks timing, errors, and metadata.
Without it: “API seems slow” means you’re grepping logs by request ID.
With it: “Here’s the timeline. Database query took 800ms. Cache miss. Here’s why.” Grafana shows you the whole picture.
The Home Lab Reality Check
Let’s be honest: if you run one application on one server, tracing is like hiring a forklift to move a couch. It works, but your infrastructure will judge you.
Tracing starts making sense when:
- Multi-service stacks. An app talking to an API, cache, and database. Multiple instances. Async jobs. Message queues.
- Occasional mysteries. “Why did the report job hang?” “Why does the payment webhook sometimes fail?” That’s a 5-minute trace read instead of a 30-minute log grep.
- Integration debugging. You’ve got services from different teams, different languages, different log formats. A single trace view is worth the overhead.
If you’re running Nextcloud, a database, and one custom app, skip this. Use better logging and metrics. You don’t need the complexity.
Tempo vs. Jaeger vs. Zipkin
Quick rundown for people tired of comparing tools:
Jaeger is the industry standard. Full-featured. Has its own index, storage backends, collectors. It’s powerful and heavyweight. Overkill for home scale.
Zipkin is older. More database-centric. Solid, but you’ll spend time tuning it.
Tempo is the new kid. Grafana built it. Here’s the key insight: Tempo is intentionally dumb about indexing. It doesn’t try to be clever. It pushes your trace data to cheap object storage (S3, Minio, local filesystem). To find a trace, you either know the trace ID (from logs or metrics) or you search by tag with Grafana’s TraceQL. No full-text index. Simpler. Cheaper. Perfect for single-digit terabytes of trace data.
For self-hosting, Tempo is the right choice. One binary. No database. Point it at a filesystem or Minio bucket. Done.
OpenTelemetry: How Traces Get Created
Your application doesn’t magically create traces. You instrument it using the OpenTelemetry SDK. This is how:
Auto-Instrumentation Libraries
If you’re using a popular framework, OTel has you covered:
- Python:
opentelemetry-instrumentation-flask,opentelemetry-instrumentation-django - Go:
go.opentelemetry.io/contrib/instrumentation/... - Node:
@opentelemetry/auto-instrumentations-node
These intercept HTTP requests, database calls, and message queue operations automatically. You don’t add manual span code everywhere.
Manual Spans When You Need Them
For custom business logic, you add spans manually:
from opentelemetry import tracefrom opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporterfrom opentelemetry.sdk.trace.export import BatchSpanProcessor
# Set up exporter to send spans to Tempootlp_exporter = OTLPSpanExporter(endpoint="localhost:4317")trace.set_tracer_provider(TracerProvider())trace.get_tracer_provider().add_span_processor( BatchSpanProcessor(otlp_exporter))
tracer = trace.get_tracer(__name__)
def process_order(order_id): with tracer.start_as_current_span("process_order") as span: span.set_attribute("order.id", order_id)
# Nested span for database call with tracer.start_as_current_span("fetch_inventory") as db_span: db_span.set_attribute("db.operation", "query") # ... actual code ...
# Another span for external API with tracer.start_as_current_span("call_payment_service") as api_span: api_span.set_attribute("service.name", "payment-api") # ... actual code ...The trace automatically propagates to downstream services via headers. Service B sees the trace ID and parent span ID, chains its spans under the same trace. Grafana connects the dots.
Sampling: The Dumb vs. Smart Debate
Here’s a loaded question: do you log every request? No. You sample.
Traces cost I/O and storage. You can’t afford to trace everything in production. So you sample.
Head-Based Sampling (Simple, Dumb)
At the source (your app), you decide: “Keep 1 in 100 traces.” The decision is made at the start of the request. All downstream spans are kept because they’re part of a kept trace.
Easy to implement. Consistent overhead. But you might sample away the trace of the request that actually failed, because you made the decision before you knew it would fail.
# 10% of traces, picked randomlysampler: type: probability probability: 0.1Tail-Based Sampling (Smart, Complex)
You capture all traces locally, then decide after the fact: “Keep this one because it errored. Keep that one because it was slow.” The OTel Collector acts as a decision point.
This is better. You catch the errors. But it requires shipping all traces to a central place first, deciding there, then storing only the sampled ones. More infrastructure.
For self-hosting: Start with head-based sampling at 5-10%. You’ll notice if you’re missing important traces, and you can adjust.
The Magic Loop: Traces, Metrics, and Logs
Here’s where Tempo + Grafana becomes more than a toy:
- Metrics alert → “API latency spiked to 500ms”
- Click exemplar → Links from the metric graph to a specific trace
- Trace shows → Database query took 450ms
- Logs are there → Slow query logs from PostgreSQL at that timestamp
All three systems are connected. You don’t jump between three dashboards. Grafana stitches it together via trace IDs and timestamps.
In Grafana, set up:
- Exemplars on metrics. When you graph request latency, annotations link to actual traces.
- Trace-to-logs links. From a span, jump to logs filtered by service and timestamp.
- Trace-to-metrics links. From a span, see metrics from that time window.
This is why people call it “observability.” It’s integrated.
A Minimal Tempo Setup
Here’s a tempo.yaml that works on a home lab server:
server: http_listen_port: 3200
distributor: ring: kvstore: store: inmemory rate_limit_enabled: false
ingester: lifecycler: ring: kvstore: store: inmemory replication_factor: 1
storage: trace: backend: local local: path: /var/lib/tempo/traces
querier: frontend_worker: frontend_address: localhost:3200
overrides: defaults: metrics_generator: processors: [service-graphs, span-metrics] ingestion: max_traces_per_user: 10000Point it at /var/lib/tempo/traces on disk. Run it in a Docker container or systemd. Send traces via gRPC on port 4317 or HTTP on port 4318.
docker run -d \ -v /var/lib/tempo/traces:/var/lib/tempo/traces \ -v /path/to/tempo.yaml:/etc/tempo.yaml \ -p 3200:3200 \ -p 4317:4317 \ -p 4318:4318 \ grafana/tempo:latest \ -config.file=/etc/tempo.yamlAdd Tempo as a data source in Grafana (URL: http://localhost:3200), and you’re tracing.
The OTel Collector: Do You Need It?
The OpenTelemetry Collector is a separate service that receives telemetry, transforms it, and exports it. Think of it as a router + transformer for observability data.
You need it when:
- Multiple apps sending spans, and you want to centralize sampling decisions (tail-based)
- You want to enrich spans (add environment tags, redact PII)
- You’re sending to multiple backends (Tempo AND Jaeger AND something else)
You don’t need it when:
- Apps send directly to Tempo
- Head-based sampling is enough
For a home lab, skip the Collector at first. Add it later if you’re drowning in trace data.
If you do add it:
receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318
processors: tail_sampling: policies: - name: error_traces type: status_code status_code: status_codes: [ERROR] - name: slow_traces type: latency latency: threshold_ms: 1000
exporters: otlp: endpoint: tempo:4317 tls: insecure: true
service: pipelines: traces: receivers: [otlp] processors: [tail_sampling] exporters: [otlp]Collectors are optional. Don’t overthink it.
Common Gotchas
Clock Sync: If your services have different system clocks, spans will appear out of order. NTP exists for a reason.
Trace ID Propagation: Services must pass the trace ID in request headers. Most OTel libraries do this automatically. Check your HTTP middleware.
Cardinality Explosion: If you tag spans with high-cardinality data (user IDs, transaction amounts), your storage blows up. Tag wisely.
Trace ID Correlation: Make sure your application logs include the trace ID. Then you can jump from Grafana to your log aggregator by trace ID.
# Include trace ID in logstrace_id = trace.get_current_span().get_span_context().trace_idlogger.info("Order processed", extra={"trace_id": trace_id})Retention: Traces live on disk. A week of data at 10% sampling from three services is maybe 10 to 50 GB. Plan accordingly.
When Tracing Earns Its Keep
You’ve built a real system. Multiple services. Background jobs. Caching. The occasional mysterious slowdown or failed request. You want to understand what happened without spending thirty minutes grepping logs across four machines.
That’s when you set up Tempo. Not because it’s trendy. Because it saves you time. And if you’re self-hosting, it costs almost nothing, a few gigs of disk, minimal CPU, and the peace of mind that your 2 AM on-call wake-up will be a ten-minute investigation, not a two-hour treasure hunt.
Distributed tracing isn’t magic. It’s just structured storytelling about your requests. Tempo tells the story. Grafana shows you the plot. And you sleep better knowing where the bugs are hiding.
