Why Self-Host SigNoz Instead of Just Paying Datadog
Datadog is great. It’s also priced like you’re running a FAANG data center. If you’ve ever gotten a bill mid-month and done a double-take, you already know why we’re here.
SigNoz is a full-stack observability platform (traces, metrics, logs, dashboards, alerts) built on OpenTelemetry and ClickHouse. It’s open source, it runs on your hardware, and it doesn’t charge you per seat or per GB of logs like it’s printing money.
If you’re still deciding whether SigNoz is the right tool, check out SigNoz vs Uptrace or SigNoz vs Grafana LGTM first. If you know you want SigNoz and you just want to get the thing running, you’re in the right place. We’ll go from nothing to a working instance with real telemetry flowing in under an hour.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/observability/self-host-signoz
Also worth reading alongside this guide: OpenTelemetry for Self-Hosters and ClickHouse for Self-Hosted Observability for deeper context on the underlying pieces.
Prerequisites
Before you spin anything up, make sure your host can handle it. SigNoz is not a lightweight install; ClickHouse is doing real work behind the scenes.
Minimum sizing:
- 4 CPU cores (ClickHouse is happier with 6+)
- 8 GB RAM (16 GB recommended if you’re ingesting more than a few services)
- 50 to 100 GB of disk for a few weeks of retention at moderate volume, with SSDs preferred
- Linux (Ubuntu 22.04 / Debian 12 / Rocky 8+ all work fine)
- Docker 24+ and Docker Compose v2 (
docker compose, notdocker-compose)
Check your versions:
docker --versiondocker compose versionIf docker compose version fails, you’re on the old standalone binary. On Ubuntu:
sudo apt install docker-compose-pluginStep 1: Deploy Options (and Why We’re Doing It Our Way)
SigNoz ships an official deploy script that does a git clone of their full repo and runs a setup wizard. It works, and it’s fine, but you’re pulling down their entire mono-repo including changelogs, frontend source, and Helm charts just to get a Compose file. That feels excessive for a home lab or a small team.
Instead, we’ll use a self-contained docker-compose.yml that you actually control. Same services, no mystery scripts.
The three containers you need:
- ClickHouse: stores all your traces, metrics, and logs
- signoz-otel-collector: receives OTLP data from your apps and writes to ClickHouse
- signoz (the query service + frontend): the UI and API layer
Create a directory and drop in the Compose file:
mkdir -p ~/signoz && cd ~/signozservices: clickhouse: image: clickhouse/clickhouse-server:24.1.5-alpine container_name: signoz-clickhouse restart: unless-stopped environment: CLICKHOUSE_USER: admin CLICKHOUSE_PASSWORD: changeme CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1" volumes: - clickhouse-data:/var/lib/clickhouse - clickhouse-logs:/var/log/clickhouse-server ulimits: nofile: soft: 262144 hard: 262144 healthcheck: test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8123/ping"] interval: 10s timeout: 5s retries: 10 start_period: 30s
otel-collector: image: signoz/signoz-otel-collector:0.88.21 container_name: signoz-otel-collector restart: unless-stopped command: ["--config=/etc/otel/config.yaml"] environment: CLICKHOUSE_URL: tcp://admin:changeme@clickhouse:9000 volumes: - ./otel-collector-config.yaml:/etc/otel/config.yaml:ro ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP depends_on: clickhouse: condition: service_healthy
signoz: image: signoz/signoz:0.50.1 container_name: signoz-frontend restart: unless-stopped environment: CLICKHOUSE_URL: tcp://admin:changeme@clickhouse:9000 ALERTMANAGER_API_PREFIX: http://signoz-alertmanager:9093/api/ ports: - "3301:3301" # SigNoz UI - "8080:8080" # Internal API depends_on: clickhouse: condition: service_healthy otel-collector: condition: service_started
alertmanager: image: signoz/alertmanager:0.23.4 container_name: signoz-alertmanager restart: unless-stopped volumes: - alertmanager-data:/data command: - "--config.file=/etc/alertmanager/alertmanager.yml" - "--storage.path=/data"
volumes: clickhouse-data: clickhouse-logs: alertmanager-data:You’ll also need a minimal collector config. SigNoz’s collector is a customised OpenTelemetry Collector build that needs to know how to forward to ClickHouse:
receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318
processors: batch: send_batch_size: 10000 send_batch_max_size: 11000 timeout: 10s memory_limiter: limit_mib: 1500 spike_limit_mib: 512 check_interval: 5s
exporters: clickhousetraces: datasource: tcp://admin:changeme@clickhouse:9000/signoz_traces clickhousemetricswrite: endpoint: tcp://admin:changeme@clickhouse:9000/signoz_metrics resource_to_telemetry_conversion: enabled: true clickhouselogsexporter: datasource: tcp://admin:changeme@clickhouse:9000/signoz_logs
service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [clickhousetraces] metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [clickhousemetricswrite] logs: receivers: [otlp] processors: [memory_limiter, batch] exporters: [clickhouselogsexporter]Change the password.
changemeis fine for a home lab behind a firewall. Not fine if port 9000 is exposed to the internet.
Now bring it up:
docker compose up -ddocker compose logs -f --tail=50Give it 60 to 90 seconds. ClickHouse needs to initialize its schema on first boot, and the other services will restart-loop until it’s ready. That’s normal, and it doesn’t mean anything is broken.
Step 2: First Login
Once the containers are stable, open your browser to http://your-host-ip:3301.
You’ll land on the SigNoz setup wizard asking you to create an admin account. Fill in a name, email, and password. This is local auth; nothing phones home.
After login you’ll see the main dashboard. It’s empty. That’s expected. You haven’t sent it any data yet.
Before wiring up your app, confirm the collector is listening:
curl -v http://localhost:4318/v1/traces# Expect: 400 or 405 — the endpoint exists but rejected your empty GET. That's fine.Step 3: Wire Up a Real App via OTLP
This is where it gets interesting. We’ll instrument two apps: one Python/Flask and one Node.js, both pointing at the collector on localhost:4317 (gRPC) or localhost:4318 (HTTP).
Python / Flask
OpenTelemetry has an auto-instrumentation path for Python that instruments Flask, SQLAlchemy, requests, and a bunch of other libraries without you touching a line of application code.
pip install flask \ opentelemetry-distro \ opentelemetry-exporter-otlp-proto-grpc
opentelemetry-bootstrap -a installfrom flask import Flask, jsonifyimport time, random
app = Flask(__name__)
@app.route("/")def index(): # Simulate variable latency so you have something interesting to alert on time.sleep(random.uniform(0.01, 0.5)) return jsonify({"status": "ok"})
@app.route("/slow")def slow(): time.sleep(random.uniform(0.8, 2.5)) return jsonify({"status": "done, eventually"})
@app.route("/error")def error(): raise ValueError("This error is intentional. Probably.")
if __name__ == "__main__": app.run(host="0.0.0.0", port=5000, debug=False)Run it with auto-instrumentation pointing at SigNoz:
export OTEL_RESOURCE_ATTRIBUTES="service.name=flask-demo"export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
opentelemetry-instrument python app.pyHit the endpoints a few times to generate traces:
curl http://localhost:5000/curl http://localhost:5000/slowcurl http://localhost:5000/error || trueIn 15 to 30 seconds you should see flask-demo appear in the SigNoz Services list.
Node.js
Install deps in your project:
npm install @opentelemetry/sdk-node \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/exporter-trace-otlp-grpc \ @opentelemetry/exporter-metrics-otlp-grpcCreate a tracing bootstrap file. This must be required before anything else:
const { NodeSDK } = require("@opentelemetry/sdk-node");const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const sdk = new NodeSDK({ serviceName: "node-demo", traceExporter: new OTLPTraceExporter({ url: "grpc://localhost:4317", }), metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: "grpc://localhost:4317", }), exportIntervalMillis: 15000, }), instrumentations: [getNodeAutoInstrumentations()],});
sdk.start();
process.on("SIGTERM", () => { sdk.shutdown().then(() => process.exit(0));});Start your app with the tracing file loaded first:
node -r ./tracing.js server.jsBoth services will appear in SigNoz within a minute of generating traffic. Traces, spans, service maps, latency distributions: the whole deal.
Step 4: Configure Retention and TTL
By default SigNoz keeps data “forever”, or until you run out of disk, whichever comes first. ClickHouse handles retention via TTL rules on the underlying tables.
In the SigNoz UI:
- Go to Settings → Retention Period
- Set your desired TTL for Traces, Metrics, and Logs independently
- Hit Save
SigNoz translates this into ALTER TABLE ... MODIFY TTL statements against ClickHouse.
The important caveat about ClickHouse TTL: data doesn’t disappear the instant it expires. ClickHouse deletes expired rows during background merge operations, which happen on their own schedule. In practice, data might stick around 1 to 3 days longer than your TTL setting. Don’t rely on TTL for compliance or billing accuracy; it’s a storage management tool, not a scalpel.
If you need to force an immediate merge (for testing or emergency disk recovery):
docker exec -it signoz-clickhouse \ clickhouse-client --user admin --password changeme \ --query "OPTIMIZE TABLE signoz_traces.signoz_index_v2 FINAL"For a home lab, 14 days of traces and 30 days of metrics is a reasonable starting point. Scale up when you know your actual volume.
Step 5: Set Up an Alert
An observability platform without alerts is just a dashboard you never look at. Let’s fix that.
Create a metrics-based alert:
- Go to Alerts → New Alert Rule
- Choose Metrics Based Alert
- Select your metric. Start with something easy:
signoz_calls_totalwith a filter onhttp_status_code = 5xx - Set the condition:
Sum > 5 per 5 minutes - Give it a name like “High error rate - flask-demo”
- Set severity to Warning
Add a notification channel (Slack/webhook):
- Go to Settings → Alert Channels → New Channel
- Choose Slack or Webhook
- For Slack, paste your incoming webhook URL:
https://hooks.slack.com/services/T.../B.../... - Test the channel. SigNoz sends a test notification immediately
- Go back to your alert rule and assign the channel
For a latency alert (the more interesting one):
- New Alert Rule → Metrics Based Alert
- Metric:
signoz_latency_bucket, use the p99 aggregation - Condition: p99 > 1000ms for 5 minutes
- Assign to your Slack channel
Alerts fire through the Alertmanager container you already have running. If you want to customize deduplication, inhibition rules, or routing, edit alertmanager.yml and docker compose restart alertmanager.
Step 6: Reverse Proxy and Backups (Optional but Smart)
Reverse Proxy
Running SigNoz on port 3301 directly is fine for internal use. If you want HTTPS or a real domain, put it behind Caddy or Traefik. The short version for Caddy:
signoz.your-domain.com { reverse_proxy localhost:3301}That’s it. Caddy handles the cert. For Traefik, add the usual labels to the signoz service in your Compose file. Either way, SigNoz doesn’t need any special configuration on its end.
Backing Up ClickHouse
ClickHouse is where all your data lives. If that volume dies, you lose everything. The simplest backup approach:
# Freeze the data (creates a hard-link snapshot, not a service outage)docker exec signoz-clickhouse \ clickhouse-client --user admin --password changeme \ --query "ALTER TABLE signoz_traces.signoz_index_v2 FREEZE"
# Copy the frozen snapshot to your backup destinationdocker exec signoz-clickhouse \ tar czf /tmp/clickhouse-backup.tar.gz /var/lib/clickhouse/shadow/
docker cp signoz-clickhouse:/tmp/clickhouse-backup.tar.gz ./backups/For serious production use, look at clickhouse-backup (the third-party tool, not the built-in freeze). It handles incremental backups, S3 uploads, and restore properly. That’s a whole separate article. For a home lab, a weekly snapshot to a NAS is probably enough.
Troubleshooting
No data showing up in SigNoz
Check the collector is reachable:
# From your app hostcurl -v http://signoz-host:4318/v1/tracesnc -zv signoz-host 4317If the connection is refused, check that ports 4317 and 4318 are actually bound:
docker compose ps# Look for 0.0.0.0:4317->4317/tcp in the otel-collector rowIf the container is up but the port isn’t bound, your Compose file has a typo in the ports: section. Double-check spacing and the "4317:4317" format.
Check collector logs for export errors:
docker logs signoz-otel-collector --tail=100 2>&1 | grep -i errorCommon culprits: connection refused to ClickHouse (it’s still starting), or authentication errors (password mismatch between the collector config and ClickHouse).
ClickHouse OOM (killed by OOM killer)
ClickHouse will cheerfully consume all available RAM if you let it. Add memory limits:
environment: CLICKHOUSE_USER: admin CLICKHOUSE_PASSWORD: changeme CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1" ulimits: nofile: soft: 262144 hard: 262144 # Add this: mem_limit: 6g memswap_limit: 6gAlso add a users.xml override to cap ClickHouse’s internal memory limit at the OS level; the container memory limit is a hard stop but ClickHouse’s own max_memory_usage setting gives it a softer ceiling with graceful query rejection instead of a crash.
Disk filling up faster than expected
SigNoz’s retention UI gives you the intent, but ClickHouse’s actual on-disk footprint depends heavily on your data volume and compression ratio. Check what’s actually taking space:
docker exec signoz-clickhouse \ clickhouse-client --user admin --password changeme \ --query " SELECT database, table, formatReadableSize(sum(bytes_on_disk)) AS size FROM system.parts WHERE active = 1 GROUP BY database, table ORDER BY sum(bytes_on_disk) DESC LIMIT 20 "If traces are the culprit (they usually are), drop your trace retention to 7 days and see how much that recovers. Also force a merge as shown in the retention section above. Unmerged parts take extra space.
Services keep restarting
Check the full logs, not just the last few lines:
docker compose logs clickhouse 2>&1 | tail -200The most common cause is ClickHouse failing to start because /var/lib/clickhouse is owned by the wrong user (can happen if you moved the volume). Fix:
docker run --rm -v signoz_clickhouse-data:/data alpine chown -R 101:101 /dataThe SumGuy Take
SigNoz is genuinely good software that hits a sweet spot most tools miss: it’s OpenTelemetry-native (so your instrumentation isn’t vendor-locked), the UI is actually usable without a training course, and ClickHouse as the storage backend means you get real compression and fast queries even on a mid-range home lab box.
The trade-off is that ClickHouse is not small. Eight gigs of RAM is the honest minimum, and if you’re running this on the same box as your app stack you’ll feel it. On dedicated hardware or a VM set aside for observability, it’s fine.
The install we walked through here (self-contained Compose file, no mystery deploy scripts) is what I’d use for a small team or home lab. For anything bigger, the official Helm chart or a managed SigNoz Cloud instance starts making sense around the point where you’re ingesting gigabytes of traces per day.
But for the “I want to actually understand what my services are doing without signing up for a Datadog trial” use case? SigNoz is exactly what it needs to be.
Your 2 AM self will appreciate having a working dashboard instead of grep-ing through raw logs.