What Actually Fills Your Loki Storage

The Disk Is Full and Nobody Admits to Anything

You set up Loki a few months ago. It was going to be the tidy, cheap alternative to Elasticsearch. Now /var/lib/docker/volumes is 40 GB heavier and you have no idea who is responsible.

Most Loki bloat comes from a handful of chatty streams. One container logging at debug level, one reverse proxy writing an access log line for every health check, one pair of labels that splits a single app into 400 tiny streams. A bigger disk does not fix that. You find the offenders with a few queries, then you cut the noise at ingest in Grafana Alloy, before Loki ever sees it. The rest is retention settings that are actually switched on.

Loki handles the storage side and Grafana Alloy handles the ingest side. If you still run Promtail, note that Grafana’s docs list it as end of life as of March 2, 2026, with future development in Alloy. Everything below uses Alloy. Versions checked in October 2026: Loki 3.7.8 and Alloy v1.20.1.

What Loki Actually Stores

Loki keeps two things on disk. Chunks hold the compressed log lines. The index maps label sets (streams) to chunks. Loki does not index line content, which is why it is cheap, and also why you cannot ask “which word is taking the most space.”

Three things drive size:

1. Raw bytes ingested. More lines, longer lines, more disk. Chunks are compressed, but the compressed size still tracks what you send.
2. Stream count. Every unique combination of label values is a stream, and each stream gets its own chunks. Thousands of low-traffic streams make thousands of tiny chunks and a fat index.
3. Retention. Out of the box, Loki keeps logs forever. The docs say so directly: the compactor’s retention flag is not set by default, so logs live forever.

That third one is the usual reason the disk grows without a ceiling. Fix the first two to slow the growth, and the third to cap it.

Step One: Find Who Is Writing

Do not guess. Loki can tell you. You have two tools: logcli and plain LogQL in Grafana Explore.

Volume by container with logcli

Loki has a volume endpoint, /loki/api/v1/index/volume, controlled by volume_enabled, which defaults to true. It needs the TSDB index, which the config below uses. logcli wraps it:

export LOKI_ADDR=http://localhost:3100

# volume per container over the last 24 hours
logcli volume --since 24h --targetLabels=container '{container=~".+"}'

# the same thing as a time series, one point per hour
logcli volume_range --since 24h --step=1h --targetLabels=container '{container=~".+"}'

The output is JSON in Prometheus vector format, with one entry per container. The number is a byte count. In a test stack it landed within a few percent of what bytes_over_time reports, so read it as roughly raw log bytes, not compressed disk usage. Sort by value and start at the top.

Volume by container with LogQL

If you prefer Grafana, bytes_over_time sums the size of log lines in a window:

topk(10, sum by (container) (bytes_over_time({container=~".+"}[24h])))

Run it as an instant query in Explore. Then compare against line counts, which separate “many small lines” from “few huge lines”:

topk(10, sum by (container) (count_over_time({container=~".+"}[24h])))

A container with a huge byte count and a modest line count is writing stack traces or JSON blobs. A container with a huge line count is writing chatter. The fixes differ.

Stream count and label cardinality

Now check the other cost, the stream count:

logcli series '{}' --since=1h --analyze-labels

The --analyze-labels flag prints each label name with its number of unique values and how many streams it appears in. Loki’s own cardinality docs point you at this exact command. Any label with hundreds of values (a pod name, an IP address, a request ID, a status code combined with a path) is a stream factory. A label set like {container, status, path} turns one proxy into hundreds of streams.

The Usual Suspects

The same names show up in most home labs.

• Debug-level apps. Someone set LOG_LEVEL=debug during a 2 AM troubleshooting session and never set it back. One container now writes more than the other 30 combined.
• Health check access logs. Uptime Kuma or some other monitor hits /health through your reverse proxy every 10 seconds. Traefik or Caddy writes an access log line for each hit. Per monitored service, that is 8,640 lines a day of pure nothing. (Docker’s own HEALTHCHECK runs inside the container and never touches the proxy.)
• Reverse proxy access logs in general. Useful for ten minutes during an incident. Then they sit there at full volume for a year.
• Labels that create tiny streams. Putting client_ip or request_id in a label is the classic mistake. You get a stream per value and a chunk per stream.
• Double ingest. Alloy reads Docker container logs, and also reads the journal, where Docker’s journald driver or the host’s systemd units log the same container output a second time. Compare the two sources before you ship both.

Now the fixes. Fix the biggest offender first, then re-run the volume query to see if it moved.

Fix It at Ingest with Alloy

Dropping a line in Alloy costs nothing downstream. The line never hits Loki’s disk, index, or ingestion limits. Alloy’s loki.process component runs a pipeline of stages on each line, and several of them cut volume.

The Compose file

Here is a working Loki plus Alloy setup. Pin the tags. Do not use latest on a logging stack you want to survive an upgrade.

services:
  loki:
    image: grafana/loki:3.7.8
    command: -config.file=/etc/loki/config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - loki-data:/loki
    ports:
      - "127.0.0.1:3100:3100"
    restart: unless-stopped

  alloy:
    image: grafana/alloy:v1.20.1
    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config.alloy
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - alloy-data:/var/lib/alloy/data
    ports:
      - "127.0.0.1:12345:12345"
    depends_on:
      - loki
    restart: unless-stopped

volumes:
  loki-data:
  alloy-data:

The Docker socket mount gives Alloy read access to your container list. Treat that as the root-level access it is. Port 12345 is Alloy’s UI and metrics, bound to localhost only.

On the first start, Alloy reads each container’s existing log history, and Loki rejects anything older than a week with timestamp too old errors. That comes from reject_old_samples, which is on by default with a one week limit. The errors stop once Alloy catches up. They are not a broken config.

The Alloy config

This reads every container, labels it by name, and then runs a loki.process pipeline that drops the noise:

discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = []

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "containers" {
  host          = "unix:///var/run/docker.sock"
  targets       = discovery.docker.containers.targets
  relabel_rules = discovery.relabel.containers.rules
  forward_to    = [loki.process.trim.receiver]
}

loki.process "trim" {
  // 1. Health checks and proxy ping noise: drop the line entirely.
  stage.match {
    selector = "{container=~\"traefik|caddy\"} |~ \"/(healthz?|ping)\\b\""
    action   = "drop"
    drop_counter_reason = "healthcheck"
  }

  // 2. Debug lines from any container.
  stage.drop {
    expression          = "(?i)(\\blevel=debug\\b|\"level\":\"debug\")"
    drop_counter_reason = "debug"
  }

  // 3. Absurdly long lines (dumped payloads, base64 blobs).
  stage.drop {
    longer_than         = "16KB"
    drop_counter_reason = "too_long"
  }

  // 4. Keep only 1 in 10 lines from one very chatty container.
  stage.match {
    selector = "{container=\"chatty-app\"}"

    stage.sampling {
      rate                = 0.1
      drop_counter_reason = "sampled"
    }
  }

  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

A few details, all from Alloy’s loki.process reference:

• The health check filter matches on the path only, so it catches both Traefik’s text access log (GET /health HTTP/1.1) and Caddy’s JSON one ("uri":"/health").
• The debug filter covers logfmt (level=debug) and JSON ("level":"debug"). An app that prints [DEBUG] needs its own pattern.
• stage.drop with several options treats them as AND. To drop on A or B, write two stage.drop blocks.
• stage.match takes a LogQL stream selector plus line filters. Its action is keep (the default) or drop. Label filter expressions such as | label == "x" are not supported there.
• stage.sampling takes a rate between 0 and 1. A rate of 0.1 keeps 10% of lines and drops the rest. Use it for streams where you want a pulse, not a record.
• longer_than on stage.drop drops lines above a size, for example "16KB".
• Every dropped line increments loki_process_dropped_lines_total, tagged with your drop_counter_reason. That gives you a counter proving the filters work, and you can scrape it from Alloy’s port 12345.

Sampling error logs is a bad idea. Sample the chatter, keep the errors. Your 2 AM self will thank you.

A rate limit instead of a drop

Some containers spam in bursts: a crash loop that writes a stack trace every 200 ms. A rate limit caps the damage without needing a regex:

stage.limit {
  rate  = 20
  burst = 50
  drop  = true
}

With drop = true, lines above 20 per second (with a burst of 50) are discarded. Without it, Alloy queues them and applies backpressure instead. For a log shipper that is rarely what you want, so set it explicitly.

Trim labels instead of lines

If logcli series --analyze-labels showed a high-cardinality label, strip it. stage.label_drop removes labels from entries:

stage.label_drop {
  values = ["client_ip", "request_id"]
}

If you still want the data to be searchable, move it to structured metadata. That stores the value next to the line without making it part of the stream identity:

stage.structured_metadata {
  values = {
    request_id = "",
  }
}

An empty value means “use the extracted value with the same name.” Structured metadata needs an earlier stage such as stage.json or stage.logfmt to extract the field first. If you move a label this way, put stage.structured_metadata before any stage.label_drop that removes the same field. Loki’s allow_structured_metadata limit defaults to true in the current config reference. Metadata still costs bytes, so use it for fields you will query, not for everything.

Fix It in Loki: Retention That Actually Runs

Alloy slows the intake. Retention stops the pile from growing forever. This is the Loki config for a single-container home lab, with TSDB, schema v13, and retention on:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules

schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /loki/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

limits_config:
  retention_period: 720h
  max_query_lookback: 720h
  retention_stream:
    - selector: '{container=~"traefik|caddy"}'
      priority: 1
      period: 168h
    - selector: '{container="chatty-app"}'
      priority: 2
      period: 48h
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 6
  per_stream_rate_limit: 3MB
  max_global_streams_per_user: 5000
  max_line_size: 256KB

The retention rules, straight from Loki’s retention docs:

• retention_enabled: true and delete_request_store are both required. Without the first, the compactor only compacts and nothing is deleted. The docs say delete_request_store is required when retention is enabled. For local storage, set it to filesystem, the same name as the object store.
• The index period must be 24h. Retention is only available with a 24h index period, and single-store TSDB requires it anyway.
• retention_period is the global default. Its default is 0s, which means keep forever. Here it is 720h, 30 days.
• retention_stream overrides it per stream. Access logs get 7 days, the chatty container gets 2 days. The selector takes label matchers only, not full LogQL. Each period must be at least 24h, or Loki refuses the config.
• Higher priority wins. If two stream rules match, the one with the larger priority value applies. If priorities tie, the shorter period applies.
• max_query_lookback should match retention. It stops queries from reaching past what you keep.

The ingestion limits above are the documented defaults, written out so you can see them. ingestion_rate_mb of 4 is per tenant, per_stream_rate_limit of 3MB catches a single runaway stream, and max_global_streams_per_user of 5000 caps cardinality damage. When a limit trips, Loki rejects pushes with a 429. That is a loud failure, which beats a quiet disk fill. Tune the numbers to your volume, but do not set them to unlimited.

Compression and the schedule of a delete

Chunk compression is set by chunk_encoding in the ingester block. The config reference lists none, gzip, lz4-64k, snappy, lz4-256k, lz4-1M, lz4, flate, zstd, with a default of gzip. Changing it affects new chunks only. Existing chunks keep their old encoding. Gzip already compresses text well, so moving off it mostly trades size for CPU. Cutting the bytes in Alloy beats any codec choice.

Retention also does not delete instantly. The compactor marks expired chunks, then waits retention_delete_delay (2h above) before removing them. Disk space comes back hours after the data crosses the retention line, not at the moment you change the config.

Verify the Fix

Re-run the volume queries after 24 hours:

logcli volume --since 24h --targetLabels=container '{container=~".+"}'

Then check Alloy’s own counter. The Compose file publishes port 12345 on localhost, so run this on the Docker host (the Alloy image ships without curl, so docker compose exec will not help):

curl -s http://localhost:12345/metrics | grep loki_process_dropped_lines_total

Each reason label matches one of the drop_counter_reason names above. If healthcheck is zero, your selector does not match your real access log format. Look at one raw proxy line and fix the regex.

The SumGuy Take

Treat log volume like any other budget. Measure first, cut the top three, set a retention number, and move on. A 12 GB Loki volume of mostly health checks is not a storage problem, it is a filter you forgot to write. Buying a bigger disk for it is like hiring a forklift to move a couch: it works, but your neighbors will have questions.

Run the volume query once a month. Put it in your calendar next to the backup restore test you also skip.

Common Questions

How do I see which labels use the most space in Loki?

Run logcli series '{}' --since=1h --analyze-labels to list each label with its count of unique values. For bytes, run logcli volume --targetLabels=container or the LogQL sum by (container) (bytes_over_time(...)). Labels with many unique values create many streams, and each stream costs index space and small chunks.

Does deleting logs in Loki free disk space immediately?

No. The Loki compactor marks expired chunks first, then deletes them after retention_delete_delay, which defaults to 2h. Retention runs on the compactor’s own schedule, not on demand. Expect disk space to return a few hours after logs pass the retention period, and only if retention_enabled is true.

What is the minimum retention I can set in Loki?

Each retention_stream rule must have a period of at least 24h, and Loki rejects the config otherwise. The global retention_period has no such minimum. The default of 0s means keep logs forever, so a value of 0 does not mean delete everything.

Do I need Alloy instead of Promtail for Loki 3?

Yes, you should move. Grafana’s Loki docs state that Promtail reached end of life on March 2, 2026, with no further updates and all new development in Alloy. The docs include a migration tool that converts a Promtail config to an Alloy config with one command.

Related Reading

• Promtail to Alloy Migration: A Practical Diff
• Loki vs Vector for Log Shipping
• Loki vs ELK: Centralized Logging Without the RAM Tax
• Why Your Docker Logs Are Eating Your Disk
• Log Rotation for Self-Hosted Apps