Your 8 GB mini PC just OOM-killed your DNS
It’s 2 AM. Pi-hole is down, Home Assistant is restarting, and a Jellyfin transcode ate the RAM that three other things were counting on. You never set a single resource field, so Kubernetes had no idea any of this was a problem.
Requests and limits look like one setting with two numbers. They are two different mechanisms. A request is a promise the scheduler uses to place your pod, and it also sets your CPU priority. A limit is a ceiling the kernel enforces, and what happens at the ceiling differs completely between CPU and memory. My opinion for a home lab: always set a memory request, set the memory limit equal to it for anything that matters, set a CPU request, and skip CPU limits on most things. The rest of this post is the reasoning, with output from a real k3s run.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/devops/k8s-requests-vs-limits-home-lab
Everything below ran on k3s v1.37.1+k3s1 (Kubernetes v1.37.1) in a Docker container on a cgroup v2 host. The host had 16 CPUs and about 27 GiB of RAM, so the absolute numbers are bigger than your mini PC. The behavior is the same.
What a request does, and what a limit does
A request does two jobs. The scheduler adds up the requests of pods already on a node and only places a new pod if its request fits in what’s left. Then the kubelet translates the CPU request into a cgroup cpu.weight, which only matters when the CPU is contested. In the test, a pod with cpu: 100m got cpu.weight 17 and a pod with no request got 1. Idle CPU is free for anyone. Busy CPU is divided by weight.
A limit is a wall. A memory limit becomes memory.max (the Guaranteed test pod showed 67108864 for 64Mi). A CPU limit becomes a CFS quota in cpu.max: the same pod showed 10000 100000, meaning 10 ms of CPU time per 100 ms period.
The two resources fail differently at the wall:
- CPU is compressible. Hit the quota and the kernel pauses your threads until the next period. Slow, but alive.
- Memory is incompressible. You can’t pause a byte. Hit
memory.maxand the kernel’s OOM killer ends the process.
Picture the forklift and the couch: CPU throttling is the forklift driver waiting for the lane to clear. An OOMKill is the couch going through the wall.
QoS classes: the label Kubernetes hands out for free
You never set a QoS class. Kubernetes derives it from what you wrote:
- Guaranteed: every container has a CPU and memory request and limit, and requests equal limits.
- Burstable: at least one request or limit exists, but the pod isn’t Guaranteed.
- BestEffort: nothing set at all.
Three pods from the kit show it:
apiVersion: v1kind: Podmetadata: {name: qos-guaranteed}spec: containers: - name: app image: busybox:1.37 command: ["sleep", "3600"] resources: requests: {cpu: 100m, memory: 64Mi} limits: {cpu: 100m, memory: 64Mi}kubectl get pod -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass,PHASE:.status.phaseNAME QOS PHASEcpu-throttle-demo Burstable Runningoom-demo Burstable Failedqos-besteffort BestEffort Runningqos-burstable Burstable Runningqos-guaranteed Guaranteed RunningNote that oom-demo and cpu-throttle-demo are Burstable. Each sets only one resource type, so neither meets the Guaranteed rules. This is a common reason a pod you meant to protect isn’t Guaranteed: you set the memory pair and forgot CPU.
What QoS really controls
This part is often misread. The kubelet’s node-pressure eviction does not rank pods by QoS class. The Kubernetes docs say it ranks by whether usage exceeds requests, then by pod priority, then by how far usage exceeds the request. QoS only gives you an estimate of the likely order: a BestEffort pod has a request of zero, so any usage exceeds it, and it goes first.
On a stock k3s node this ranking never runs for memory, because k3s sets no memory.available threshold (see the node allocatable section below). Until you add one, the kernel OOM killer and oom_score_adj decide.
QoS does directly control oom_score_adj, which the kernel OOM killer reads when the kubelet is too slow to evict anything. From inside the three pods:
qos-besteffort oom_score_adj=1000qos-burstable oom_score_adj=998qos-guaranteed oom_score_adj=-997The Burstable value follows the documented formula, min(max(2, 1000 - (1000 × memoryRequestBytes) / machineMemoryCapacityBytes), 999). A 64Mi request on a 27.2 GiB host rounds to 998. A bigger memory request pulls the number down, which makes the pod a less attractive victim. That is the practical reason to set memory requests honestly.
Memory: OOMKilled, exit code 137
Memory is where a limit bites hardest. This pod allocates 200 MiB under a 64Mi limit:
apiVersion: v1kind: Podmetadata: {name: oom-demo}spec: restartPolicy: Never containers: - name: hog image: python:3.13-alpine command: ["python", "-c", "b = bytearray(200 * 1024 * 1024); b[::4096] = b'x' * len(b[::4096]); print('allocated')"] resources: requests: {memory: 64Mi} limits: {memory: 64Mi}kubectl get pod oom-demo -o jsonpath='{.status.containerStatuses[0].state.terminated.reason} {.status.containerStatuses[0].state.terminated.exitCode}{"\n"}'OOMKilled 137Exit code 137 is 128 plus signal 9. On a pod with restartPolicy: Always you’d see the reason under lastState.terminated.reason after the restart, which is the field to check when a pod has a restart count and no useful logs.
My rule: set the memory limit equal to the memory request for anything you care about. Memory you’ve promised is memory you can’t be surprised by. A Burstable pod with a request of 256Mi and no limit can grow to eat the node, and when the node runs dry, the kernel picks a victim by score, which may not be the hog.
CPU: throttling and the limits debate
Now the CPU side. A busy loop with a 100m limit:
apiVersion: v1kind: Podmetadata: {name: cpu-throttle-demo}spec: containers: - name: burn image: busybox:1.37 command: ["sh", "-c", "while true; do :; done"] resources: requests: {cpu: 100m} limits: {cpu: 100m}kubectl exec cpu-throttle-demo -- cat /sys/fs/cgroup/cpu.statFive seconds apart, the interesting lines were:
nr_periods 492nr_throttled 492throttled_usec 34160756...nr_periods 545nr_throttled 545throttled_usec 38176554Every period was throttled (nr_throttled equals nr_periods), and throttled_usec grew by about 4 seconds in 5. That is what a loop that wants a whole core under a tenth of a core looks like.
The debate is about bursty apps. A web service with a 200m limit that idles at 5% and spikes to a full core for 30 ms per request gets throttled on every spike: 200m is 20 ms of CPU per 100 ms period, so the last 10 ms of the spike waits up to 80 ms for the next period. Multi-threaded apps burn the quota even faster. The request latency gets a nasty tail and the CPU graph looks calm. That mismatch has eaten many afternoons.
My opinion, and it is an opinion: on a home lab, set CPU requests and skip CPU limits for most workloads. The requests already give each pod its fair share when the CPU is busy, through cpu.weight. A limit only adds a ceiling on a node that would otherwise sit idle. Keep a CPU limit for the few things that misbehave on purpose: a transcoder, a build runner, anything you’d rather run slowly than let it starve Home Assistant. What protects Home Assistant from a hungry neighbor is its own CPU request, so make sure the important pods have one.
Node allocatable: the reservation your 8 GB box needs
The scheduler doesn’t hand out the node’s full capacity. It hands out Allocatable, which is Capacity minus reservations. In k3s, the defaults are thin. I checked the kubelet’s effective config (via the configz endpoint) on v1.37.1:
evictionHard {'imagefs.available': '5%', 'nodefs.available': '5%'}evictionMinimumReclaim {'imagefs.available': '10%', 'nodefs.available': '10%'}kubeReserved NonesystemReserved NoneenforceNodeAllocatable ['pods']That matches the k3s source (pkg/daemons/agent/agent.go), which sets only those two eviction thresholds and no reservations. Two gotchas follow:
- Upstream Kubernetes defaults include
memory.available<100Mi. The docs say those defaults are only set if you change none of the eviction parameters. k3s sets two, so on a stock k3s node there is no memory eviction threshold. Memory pressure goes straight to the kernel OOM killer. - Nothing is reserved for k3s itself, containerd, or the OS. On an 8 GB mini PC running etcd or SQLite, a few pods, and your own SSH session, pods can request every byte and leave the node nothing.
Fix both in /etc/rancher/k3s/config.yaml. Setting any eviction-hard key drops the upstream defaults for the other keys, so list every threshold you want:
kubelet-arg: - "kube-reserved=cpu=200m,memory=512Mi" - "system-reserved=cpu=200m,memory=512Mi" - "eviction-hard=memory.available<256Mi,nodefs.available<5%,imagefs.available<5%"Restart k3s after editing (sudo systemctl restart k3s). Before and after, from kubectl describe node:
Capacity: cpu: 16 memory: 28506216KiAllocatable (before): cpu: 16 memory: 28506216KiAllocatable (after): cpu: 15600m memory: 27195496KiThe memory arithmetic: 28506216Ki minus 1 GiB of reservations minus the 256Mi eviction threshold equals 27195496Ki. The 400m CPU came from the two CPU reservations. Allocatable subtracts the eviction threshold too, which surprises people.
One more thing I confirmed in the same output: enforceNodeAllocatable stays ['pods']. The kubelet caps the parent kubepods cgroup at Capacity minus the reservations (memory.max was 28116623360 bytes, exactly 1 GiB under Capacity), so pods as a group can’t eat the reserved memory. The system daemons themselves get no cgroup cap. That’s fine for a home lab.
LimitRange and ResourceQuota
BestEffort pods sneak in whenever someone writes a manifest without resources. A LimitRange fixes that per namespace by supplying defaults at admission time:
apiVersion: v1kind: LimitRangemetadata: {name: defaults, namespace: quota-demo}spec: limits: - type: Container defaultRequest: {cpu: 50m, memory: 64Mi} default: {memory: 128Mi}defaultRequest fills in requests and default fills in limits. Here the default has only a memory limit, matching the opinion above: no CPU limit.
A ResourceQuota caps the namespace total:
apiVersion: v1kind: ResourceQuotametadata: {name: caps, namespace: quota-demo}spec: hard: requests.cpu: "1" requests.memory: 1Gi limits.memory: 2GiThe gotcha is the order. A quota on requests.memory makes the field mandatory. Apply a bare pod to that namespace before the LimitRange exists and the API server says no:
Error from server (Forbidden): error when creating "/kit/no-resources-pod.yaml": pods "bare" is forbidden: failed quota: caps: must specify limits.memory for: app; requests.cpu for: app; requests.memory for: appApply the LimitRange and the same manifest goes through. The defaults land in the pod spec:
Burstable{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}Resource Used Hard-------- ---- ----limits.memory 128Mi 2Girequests.cpu 50m 1requests.memory 64Mi 1GiIf a Deployment applies fine but its pods never show up, the quota is the usual suspect. Run kubectl describe replicaset and look in Events for the same “failed quota” message. On the Deployment it only shows up as a ReplicaFailure condition, which is easy to miss.
Sizing for a Pi or a mini PC
Real numbers beat guesses, so start from kubectl top pod and round up. A quiet service like Pi-hole or a small Go API fits in a 64Mi to 128Mi request. A database wants whatever its cache setting says plus headroom. Do the sum before you deploy: on an 8 GB node with the reservations above, Allocatable memory is roughly 6.5 GiB (the kernel usually reports about 7.7 GiB, minus 1 GiB of reservations and the 256Mi threshold), and the requests of all your pods have to fit in that.
Overcommit memory limits if you like, but never overcommit requests. Limits summing past 100% is a bet that pods don’t spike together, and you can survive losing that bet if the important pods sit at request equals limit. Requests summing past 100% is impossible, because the scheduler refuses. That refusal is the feature. A pod stuck in Pending with Insufficient memory is the cluster telling you the truth before the OOM killer does.
How to see what’s going on
A short list for 2 AM:
# what the scheduler has handed out, and the Allocated resources tablekubectl describe node <node>
# live usage (needs metrics-server; k3s ships it by default)kubectl top pod -A --sort-by=memory
# QoS class of every podkubectl get pod -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass
# why did it restart?kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}'
# is it being throttled? (cgroup v2)kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.statThe Allocated resources section of kubectl describe node shows requests and limits as a percentage of Allocatable. Requests should stay below 100%, because the scheduler enforces that. Limits are allowed to exceed 100% and the header says so: “Total limits may be over 100 percent, i.e., overcommitted.” Memory limits well over 100% means you’re betting no two pods spike together.
In-place resize is stable now
You used to need a pod restart to change resources. The Kubernetes docs list in-place pod resize as stable since v1.35, so on the k3s release I tested it just works. I patched the Guaranteed pod’s CPU from 100m to 200m through the resize subresource:
kubectl patch pod qos-guaranteed --subresource resize --patch \ '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"200m"},"limits":{"cpu":"200m"}}}]}}'The pod reported restarts 0, stayed Guaranteed, and cpu.max inside it changed to 20000 100000. Handy for a one-off adjustment. For a managed workload, the Deployment template is still the source of truth, so change it there too. For recommendations on what numbers to pick, a tool like Goldilocks reads real usage and suggests them. That’s a separate post.
The SumGuy Take
Set these and go to bed:
- Reserve memory for the node in
config.yamland add amemory.availableeviction threshold. A stock k3s node has none. - A LimitRange in every namespace, with a memory request default and a memory limit default.
- Memory request equals memory limit for databases, DNS, and anything that hurts when it restarts.
- CPU requests everywhere, CPU limits only on the noisy neighbors.
- Check
lastState.terminated.reasonbefore blaming the app.
Your 2 AM self will thank you.
Common Questions
Should I set CPU limits on Kubernetes pods?
No, not by default. A CPU request already gives a pod its fair share through cgroup cpu.weight when the node is busy. A CPU limit adds a CFS quota that throttles bursty apps even when average usage is low. Keep CPU limits only for noisy workloads like transcoders or build runners.
What happens if a pod exceeds its memory limit?
The kernel OOM killer ends the container, and Kubernetes records OOMKilled with exit code 137. The container then restarts according to the pod’s restartPolicy. A memory limit is enforced through the cgroup memory.max file, so the process gets no warning and no grace period.
Do Guaranteed pods get evicted last?
Usually, but QoS class is not the ranking key. The kubelet evicts pods whose usage exceeds their requests first, then ranks by priority. A Guaranteed pod never exceeds its request, so it lands near the end. Memory eviction only runs once a memory.available threshold is set, and stock k3s sets none.
How much RAM should I reserve for k3s on a mini PC?
Reserve about 1 GiB in total on an 8 to 16 GB node: 512Mi for kube-reserved and 512Mi for system-reserved. Add a memory.available<256Mi hard eviction threshold. That combination is what I used in the test, and it removed roughly 1.25 GiB from Allocatable.
