Linux Real-Time Kernel: When PREEMPT_RT Actually Earns Its Keep

Your Kernel Is Lying to You About Time

Here’s the thing about a standard Linux kernel: it’s fast on average and occasionally terrible in the worst case. That’s a perfectly fine trade-off for a web server. It is a disaster if you’re running a CNC machine, a MIDI sequencer, or any system where “I’ll get to it in a moment” means a missed deadline, a blown note, or a robot that punches itself in the face.

In 2024, after roughly two decades of carrying around as an out-of-tree patch set, PREEMPT_RT was finally merged into the Linux mainline kernel. Real-time Linux is no longer something you cobble together from kernel.org patches on a Tuesday night. The point release was 6.12 — stable, shipping, usable today.

So what does this actually change, and more importantly — should you care?

”Real-Time” Doesn’t Mean Fast. It Means Predictable.

This trips people up constantly. Real-time is not about throughput. A stock 6.x kernel can shuffle gigabytes of data per second. An RT kernel won’t beat it at that. What an RT kernel does is guarantee a worst-case latency — a hard ceiling on how long any piece of code can be delayed by something else.

Think of it like this: a stock kernel is a busy restaurant kitchen. Most orders come out in 8 minutes. Sometimes a big table rolls in and your order takes 45 minutes. The average is fine. The outliers are brutal.

An RT kernel is the same kitchen with a rule: no order waits more than 12 minutes, full stop. Throughput might drop slightly to make that guarantee hold. But you’ll never wait 45 minutes again.

In technical terms: the RT patchset makes most kernel code paths fully preemptible, converts spinlocks to sleeping mutexes where safe, and forces interrupt handlers to run in kernel threads that can themselves be preempted by higher-priority tasks. The result is that user-space tasks running SCHED_FIFO or SCHED_RR actually get to run within a bounded time window — measured in microseconds, not milliseconds.

Where RT Kernels Actually Belong

Pro Audio (JACK / PipeWire)

This is the canonical use case. If you’ve ever set up a low-latency audio workstation on Linux, you’ve fought with this. JACK or PipeWire running at 64-sample buffer sizes needs the audio callback to fire within something like 1.3 ms at 48 kHz. Stock kernels can hit that most of the time. Xruns happen when they don’t.

With an RT kernel and audio group membership in /etc/security/limits.conf, you can push buffer sizes that would be laughable on a stock kernel — 32 samples, sub-millisecond latency — and actually sustain it under load. The improvement is real and measurable.

Industrial Control and Robotics

Hardware-in-the-loop (HIL) systems, PLC replacements, motor controllers — anything where your control loop needs to execute at exactly 1 kHz and you cannot tolerate a 50ms jitter spike because the servo went to lunch. Stock kernels have no business in these environments. RT kernels exist precisely for this.

MIDI Performance and Synthesis

Same category as pro audio but worth calling out: software synths and MIDI sequencers where timing drift of a few milliseconds turns a groove into a mess. Real-time priority gives those threads the scheduling guarantee they need.

What Doesn’t Belong Here

General-purpose servers: databases, web servers, container hosts, Kubernetes nodes. An RT kernel on a K8s worker will actually hurt you — you sacrifice throughput and CPU efficiency for determinism guarantees that no HTTP request ever asked for. Your p99 latency might get slightly better but your p50 gets worse, and your overall ops/sec drops. Not worth it.

Docker hosts: same story. Don’t do it.

Getting an RT Kernel Without Compiling One

You don’t need to build from source anymore. Distributions ship RT kernels:

Ubuntu (via Ubuntu Pro / linux-image-lowlatency-hwe):

# Ubuntu Pro gives access to linux-image-realtime
sudo pro attach <token>
sudo apt install linux-image-realtime

For home lab use without Pro, linux-image-lowlatency is the next best thing — not full RT, but significantly better scheduling behavior than the generic kernel.

Debian:

sudo apt install linux-image-rt-amd64

That’s it. Reboot, select the RT kernel in GRUB, done.

Fedora / RHEL / CentOS Stream:

# Fedora
sudo dnf install kernel-rt kernel-rt-devel

# For RHEL/CentOS you need the RT repository enabled
sudo subscription-manager repos --enable=rhel-8-for-x86_64-rt-rpms
sudo dnf install kernel-rt

Arch Linux:

# AUR
yay -S linux-rt linux-rt-headers

Verify you’re running it after reboot:

uname -r
# Should contain 'rt' in the string, e.g.:
# 6.12.0-1015-realtime

Measuring What You Actually Got: cyclictest

Don’t assume the RT kernel helped. Measure it. cyclictest is the standard tool — it spawns a high-priority thread that wakes on a timer, measures how late it actually woke up, and reports min/avg/max latency over thousands of samples.

# Install
sudo apt install rt-tests   # Debian/Ubuntu
sudo dnf install rt-tests   # Fedora

# Run a basic test: 60 seconds, 1000 Hz loop, all CPUs
sudo cyclictest --mlockall --smp --priority=80 --interval=1000 --distance=0 -D 60

What you’ll typically see:

Stock kernel, idle system:

T: 0 ( 1234) P:80 I:1000 C: 60000 Min:    3 Act:    8 Avg:    9 Max:   312
T: 1 ( 1235) P:80 I:1000 C: 60000 Min:    4 Act:   11 Avg:   10 Max:   428

Stock kernel under load (run stress-ng --cpu 0 --io 4 in another terminal):

T: 0 ( 1234) P:80 I:1000 C: 60000 Min:    4 Act:   22 Avg:   18 Max: 11842
T: 1 ( 1235) P:80 I:1000 C: 60000 Min:    5 Act:   19 Avg:   17 Max:  9305

There’s your 10ms spike. Under load, the stock kernel will absolutely abandon your real-time thread for several milliseconds. That’s a missed audio callback, a control loop skip, or a jitter event in your MIDI timeline.

RT kernel under the same load:

T: 0 ( 1234) P:80 I:1000 C: 60000 Min:    4 Act:    7 Avg:    8 Max:    87
T: 1 ( 1235) P:80 I:1000 C: 60000 Min:    5 Act:    8 Avg:    8 Max:    92

Under 100 microseconds. Consistent. That’s the whole point.

Giving Your Process Real-Time Priority with chrt

Booting an RT kernel is step one. Step two is actually running your process at real-time priority. Without this, you’re running a Ferrari on a go-kart track but still in third gear.

# Run a command at SCHED_FIFO priority 80
chrt -f 80 your-audio-app

# Change priority of a running process
chrt -f -p 80 <PID>

# Check what scheduler policy a process is using
chrt -p <PID>

SCHED_FIFO means: once this thread is runnable, it runs until it voluntarily yields or a higher-priority RT thread preempts it. SCHED_RR adds a time quantum. For audio callbacks and control loops, SCHED_FIFO at priority 80–90 is the standard setup.

For audio specifically, you also want to set memory lock limits so the audio thread can’t page fault mid-callback:

@audio - rtprio 95
@audio - memlock unlimited
@audio - nice -19

Add your user to the audio group, log out and back in, and JACK/PipeWire can self-elevate to RT priority without needing root.

The Pitfalls That Will Still Get You

Getting sub-100µs latency is achievable. Staying there requires fighting a few more things:

CPU Frequency Scaling

The default schedutil or ondemand governor will downclock your CPU during low-load intervals, then ramp back up when you need it. That ramp introduces latency. For RT workloads, pin it:

# Set performance governor on all CPUs
sudo cpupower frequency-set -g performance

Or permanently in /etc/default/cpupower:

CPUPOWER_START_OPTS="frequency-set -g performance"

SMM Interrupts (System Management Mode)

Here’s the dirty secret: certain latency spikes have nothing to do with your kernel. The motherboard firmware (SMM/BIOS) can fire interrupts that are invisible to the OS and pause all CPUs for tens to hundreds of microseconds. cyclictest will show these as random spikes even on a perfectly tuned RT kernel.

You can detect them with hwlatdetect:

sudo hwlatdetect --duration=60s --threshold=10

If it reports hardware latency events above your target, no amount of kernel tuning will fix it. You need to disable SMM-triggering features in BIOS (C-states, BIOS power management, sometimes even Secure Boot on certain platforms).

NMIs and Hyperthreading

Non-maskable interrupts are exactly what they sound like — they can’t be blocked. SMM falls into this category. Additionally, if you’re running Hyper-Threading, a sibling thread’s cache activity can blow your RT thread’s cache state and add latency. CPU isolation (isolcpus=, nohz_full=) is the next level of tuning if you’re chasing sub-50µs guarantees.

The Short Version

If you’re running JACK, PipeWire in low-latency mode, a hardware control loop, a software synth, or any system where timing jitter causes real problems — get the RT kernel. It’s in mainline now, distros ship it, and the improvement is not subtle. Your cyclictest max will drop from 10ms-class spikes to under 100µs under load. That’s a 100x improvement in worst-case latency.

If you’re running a NAS, a web server, a Kubernetes cluster, a database, or anything throughput-oriented — leave the RT kernel alone. You’d be trading ops/sec for determinism guarantees nobody asked for.

Real-time Linux isn’t magic and it isn’t for everyone. But for the workloads it was built for, it’s one of the most impactful kernel config changes you can make — and now that it’s in mainline, there’s no excuse not to try it.

Your MIDI sequencer will thank you. Your 2 AM recording session definitely will.

Related Reading

• io_uring: Linux’s Async I/O Future Is Already Here
• The Linux OOM Killer: Why It’s Killing Your App
• Linux PSI: The Pressure Metrics Load Average Wishes It Were
• Sysctl Tuning: The Linux Kernel Knobs That Actually Matter
• Tmpfs vs Ramfs: When Your Disk Is Too Slow and Your RAM Is Just Sitting There