Skip to content
Go back

When Your NVMe Falls Off the PCIe Bus

By KingPin 16 min read
When Your NVMe Falls Off the PCIe Bus
Contents

The Blue Screen Pointed at the Wrong Suspect

My mini PC started dying with a blue screen full of corruption. Linux has a BSOD now: the kernel’s DRM panic screen (CONFIG_DRM_PANIC=y in the CachyOS kernel) paints one when the machine goes down. I read the last lines of the journal, saw a GPU warning, and blamed the GPU.

The GPU was innocent. The last log line lied. The real failure was the NVMe drive falling off the PCIe bus when the kernel tried to wake it from a deep power state. Once the root filesystem vanishes, the journal has nowhere to write the actual error, so you only see whatever got logged just before the lights went out.

Three kernel flags have kept the machine stable since. A replacement drive with a heatsink is the real fix, and it is on the way. Here is the trail, dead ends included.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/tree/main/linux/nvme-falls-off-pcie-bus

The Setup

The machine is a small AMD Ryzen 8000-series mini PC, about 135 mm on each side, running CachyOS with systemd-boot. The boot drive was a budget DRAM-less 500 GB Gen4 NVMe drive with a Silicon Motion controller, no heatsink, running at PCIe Gen4 x4. The case has a black plastic bottom cover full of vent holes. The CPU blower sits on the opposite side of the board and does not touch the SSDs. Remember that cover, because it matters later.

The first symptom showed up before the first crash. A heavy gaming session lagged badly, with swap thrashing and the drive working flat out. About twelve minutes after I closed the game, the machine died. Most likely the SSD was thermal throttling under that load. I can’t prove it, so I’m calling it likely and moving on.

Over the next few hours I got at least four or five panics. The boot timeline looked like this:

boot 1 ~35 seconds crash
boot 2 54 minutes crash (panic)
boot 3 17 seconds crash (panic), new kernel
boot 4 2h43m clean poweroff, first boot with the fix

Before all this, the previous boot had run for more than 17 hours without a hiccup. Keep that in mind, because it kills one of the suspects.

Suspect One: The GPU Driver

The last kernel lines before the first crash were amdgpu warnings:

amdgpu 0000:c5:00.0: Mode Validation Warning: Viewport size failed validation

They landed right as KDE’s Wayland session was setting display modes, and then the journal just stopped. It looked like a confession. It wasn’t.

The same warning appeared in the 17-hour boot, during out-of-memory events, and that boot never crashed. The journal also held no panic, oops, or watchdog message for the crashing boot. A warning that shows up in healthy boots is noise, not evidence. The ACPI BIOS errors that print on every boot are the same kind of noise.

Suspect Two: A Bad Kernel

Next I blamed the kernel. I ran a full system upgrade, which moved the kernel from 7.3.0 to 7.3.4 along with a Mesa and Plasma bump. Gotcha number one: the boot after the upgrade was still on the old kernel, because the bootloader had not picked up the new one yet. That boot panicked.

The next boot used the new kernel and panicked 17 seconds in, right as the login screen started. Same crash, two kernel versions, same point in boot. That is useful data. A bug introduced in one kernel version would not show up on the other, so this pointed at hardware or firmware.

Turn On pstore Before You Need It

I had a crash and no panic message. The journal could not record the death because the thing it writes to was dying.

This is what pstore is for. It stashes panic logs somewhere that survives a reboot, such as EFI firmware variable storage, and systemd-pstore.service copies them into /var/lib/systemd/pstore/ on the next boot. The CachyOS kernel ships with CONFIG_EFI_VARS_PSTORE=y and CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y, so EFI pstore is off by default. My /var/lib/systemd/pstore/ directory existed and was empty, which is what a disabled feature looks like.

The fix is one kernel parameter:

efi_pstore.pstore_disable=0

Then enable the service:

Terminal window
sudo systemctl enable systemd-pstore.service
ls /var/lib/systemd/pstore/

Do this today, on every machine you care about. Your 2 AM self will thank you. I did it after the first few crashes, which is the wrong order.

The Clue in the Journal

One boot started with this from systemd-journald:

system.journal corrupted or uncleanly shut down, Device or resource busy

The previous boot had reached the shutdown target normally. A clean shutdown that still corrupts the journal points at storage, not the GPU. I also saw two NVMe lines in the kernel log:

block nvme0n1: No UUID available providing old NGUID
Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)

Both are harmless quirks of this drive. The NGUID line prints on every boot, healthy ones included. journalctl with -b -1 and -b 0 side by side is the fastest way to tell the difference.

The Real Cause: D3cold and 0xffffffff

I dropped into the emergency shell and read the kernel ring buffer. There it was:

Unable to change power state from D3cold to D0
CSTS=0xffffffff, PCI_STATUS=0xffff

Here is what those lines mean. D3cold is the deepest PCIe device power state. The kernel puts idle devices there to save power. D0 is fully awake. The kernel asked the NVMe controller to wake up and the controller never answered.

CSTS is the NVMe controller status register. When a PCIe read to a device that is not there comes back, the bus returns all ones. A register that reads 0xffffffff and a PCI status of 0xffff mean one thing: the controller is not answering PCIe reads. The device has dropped off the bus. Not slow, not erroring, gone. It happened about 35 seconds into boot, which matches the “dies near the login screen” pattern. The GPU warnings were just the last thing logged before the root filesystem disappeared.

Another symptom made the story confusing. fsck.ext4 crashed with SIGSEGV, which looks exactly like filesystem corruption. It was not corruption. The drive was dropping off the bus in the middle of reads, and fsck choked on the failures. Once the drive stayed online, the filesystem checked out fine.

So the firm finding is the wake-up failure. The link to heat is a hypothesis: the controller appears to lock up during the power-state change when it is hot. That fits a DRAM-less budget drive with no heatsink in a small case that had just been through a gaming session with heavy swap. It is the likely trigger, not a proven one.

The Fix: Three Kernel Flags

If the drive dies when it changes power states, stop it from changing power states. Three kernel parameters do that:

nvme_core.default_ps_max_latency_us=0
pcie_aspm=off
pcie_port_pm=off

What each one does:

The price is higher idle power draw and a little more heat. This is the opposite of the advice in cutting idle watts, where you turn ASPM on to save power. Both are correct. Power saving is the right default. Turning it off is the right call only when a drive is actively falling off the bus. A forklift is a bad commute vehicle, but it is the right tool when you have to move the couch.

The Gotcha That Would Have Brought the Crashes Back

I applied the flags by editing the systemd-boot entries by hand. That works until the next kernel update, because sdboot-manage (from the systemd-boot-manager package) regenerates the entries every time a kernel package updates. It builds them from /etc/sdboot-manage.conf, and mine still held the old options:

LINUX_OPTIONS="nowatchdog quiet splash"

The next kernel update would have rewritten the entries and silently dropped the fix. The crashes would have returned with no obvious reason. I would have blamed the new kernel. The correct line puts everything in the config:

LINUX_OPTIONS="nowatchdog quiet splash nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off efi_pstore.pstore_disable=0"

Then regenerate the entries:

Terminal window
sudo sdboot-manage gen

On a distro that uses GRUB, the same flags go in GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, followed by sudo grub-mkconfig -o /boot/grub/grub.cfg. The config file differs, the trap is the same: put the fix where the regeneration step reads from.

Verify It Stuck

Do not trust a config file. Trust the running kernel:

Terminal window
cat /proc/cmdline
cat /sys/module/nvme_core/parameters/default_ps_max_latency_us

The second command should print 0. To check APST from the drive’s side, nvme get-feature /dev/nvme0 -f 0x0c from nvme-cli reads the APST feature (feature ID 0x0c). Then hunt for errors:

Terminal window
journalctl -b 0 | grep -ciE 'nvme.*(timeout|reset|error)'

I wanted 0, and I got it. The first boot with the flags ran almost three hours and ended in a clean power-off. The kernel log showed the controller initializing normally, ext4 mounted read-write, and no D3cold errors.

What SMART Said

The drive passed its overall health check with zero media errors. That is the part that makes this failure mode so annoying: the flash was fine, and the controller or firmware was the problem. This failure mode can look exactly like a dead drive.

The interesting numbers were in the temperature counters. The “Warning Comp. Temperature Time” and “Critical Comp. Temperature Time” fields showed six-digit numbers. NVMe reports those counters in minutes, not hours, so divide by 60 before you panic. Even divided by 60, that came to thousands of hours above the drive’s thermal thresholds, which is an ugly history for a drive without a heatsink. See NVMe SMART for how to read the rest of that output.

Terminal window
sudo smartctl -a /dev/nvme0 | grep -i 'temperature time'

Picking the Replacement

Flags are a workaround. I ordered a 2 TB Gen4 drive with a heatsink pre-attached. Two things about the listing are worth passing on.

First, the spec sheet said PCI Express Gen3 next to a rating of 7000 MB/s sequential read. Those two numbers cannot both be true, because Gen3 x4 tops out around 3500 MB/s in practice. The 7000 MB/s rating says Gen4, and I will confirm the real link speed after install. Always check the controller and the real interface, not the copy-pasted line item.

Second, its rated operating maximum was 70°C, which is lower than the old drive’s warning threshold of 82.8°C. A heatsink gives you better cooling and less headroom at the same time, so you still need to watch temperatures. See NVMe heatsinks for what actually cools. A bare drive in a tiny case runs hot under sustained load.

The heatsink adds about 10 mm of height, so measure the clearance to the cover before you order. A ball of modeling clay on the slot area, a gently screwed-on cover, and a ruler give you the number. I’ll cover the migration from the old drive in a separate post.

Where Does the Second M.2 Slot Live?

My first assumption, based on lspci, was one SSD slot. Wrong. lspci lists only devices that are present, and an empty M.2 slot does not appear. Review photos answered what software could not. With the board flipped over, only the CPU cooler sat on the far side, so every slot had to be on the RAM side. Screw posts about 80 mm from each connector confirmed two 2280 slots: one flat on the board, one stacked above the Wi-Fi card.

The heatsinked drive goes in the flat slot, because the stacked one would trap heat against the Wi-Fi card. After install, check the link you got:

Terminal window
cat /sys/class/nvme/nvme*/device/current_link_speed
cat /sys/class/nvme/nvme*/device/current_link_width

You want 16.0 GT/s PCIe and a width of 4 for a Gen4 x4 link.

Cooling a Mini PC With a Plastic Lid

Most mini-PC cooling advice says to stick a thermal pad between the SSD and the metal bottom plate, so the case becomes a heatsink. My case has a plastic bottom cover. Plastic does not move heat, so the pad does nothing.

The CPU blower is on the other side of the board, so the SSDs get only passive convection through the vents. A drive with its own heatsink is the right answer for this case. The vents face down, so keep the machine off carpet and beds.

An interim fix was a USB fan beside the case, aimed at the vents. It does not blow into the case, but it moves the warm air away. The drive read 40 to 41°C afterward, though I measured under a different load, so that is not a fair before-and-after.

The longer plan is a 120 mm USB fan with rubber feet, laid flat under the case, blowing up through the vents. Push beats pull here. Pushing sends cool room air straight onto the SSDs and RAM and feeds the CPU blower’s intake. Pulling draws air from every gap in the case, and most of it bypasses the drives. Pushing blows dust in, so put a thin filter on the intake. A 120 mm fan covers the vent area and stays inside the case edges. Pick a model with speed control. If you want a number instead of my opinion, run the same heavy disk load in both directions and compare temperatures after ten minutes.

A Temperature Alert You Can Copy

A hot drive that nobody notices is how this story started. This script sends a desktop notification when any NVMe drive reaches a limit. It reads sensors straight from /sys/class/hwmon, finds NVMe devices by name instead of by hwmonN number (the number can change between boots), and needs no root.

nvme-temp-alert
#!/usr/bin/env bash
# Desktop alert when any NVMe composite temp reaches LIMIT (°C).
# Re-alerts every REPEAT seconds while it stays hot.
LIMIT=${LIMIT:-70}
INTERVAL=${INTERVAL:-10}
REPEAT=${REPEAT:-300}
declare -A last
while :; do
for h in /sys/class/hwmon/hwmon*; do
[[ $(<"$h/name") == nvme ]] || continue
t=$(( $(<"$h/temp1_input") / 1000 ))
dev=$(basename "$(readlink -f "$h/device")")
now=$(date +%s)
if (( t >= LIMIT )); then
if (( now - ${last[$dev]:-0} >= REPEAT )); then
echo "$dev at ${t}°C (limit ${LIMIT}°C)"
notify-send -u critical -a "NVMe temp" "$dev at ${t}°C" "Limit is ${LIMIT}°C"
last[$dev]=$now
fi
else
unset "last[$dev]"
fi
done
[[ $ONCE ]] && exit 0
sleep "$INTERVAL"
done

Save it as ~/.local/bin/nvme-temp-alert and make it executable. Then add a systemd user unit:

nvme-temp-alert.service
[Unit]
Description=Desktop alert when NVMe temperature is high
After=graphical-session.target
PartOf=graphical-session.target
[Service]
ExecStart=%h/.local/bin/nvme-temp-alert
Restart=on-failure
[Install]
WantedBy=graphical-session.target

Put the unit at ~/.config/systemd/user/nvme-temp-alert.service, then enable it:

Terminal window
systemctl --user daemon-reload
systemctl --user enable --now nvme-temp-alert.service

I tested it with LIMIT=30, which fired a notification at 41°C, and with the default 70, which stayed silent. The alert is critical priority, repeats every five minutes while the drive stays hot, and resets when it cools. To change the limit, run systemctl --user edit nvme-temp-alert and add Environment=LIMIT=65 under [Service]. For a quick look without any service, watch -n2 sensors works if you have lm_sensors installed.

Lessons From a Very Bad Weekend

  1. The last log line lies. If storage dies, the journal cannot record the real error. Distrust whatever came last.
  2. The same crash on two kernel versions rules out a bug specific to one version. It points at hardware, firmware, or a long-standing driver issue.
  3. A crashing fsck can be a symptom of a vanishing drive, not a corrupt filesystem. Check dmesg before you reformat anything.
  4. CSTS=0xffffffff means the device is gone from the bus. Treat it as a hardware or firmware event.
  5. Power management is a classic trigger for budget NVMe drives on Linux. APST-off and ASPM-off workarounds are widely cited for budget NVMe drives on Linux, and they are still workarounds.
  6. Workarounds can vanish silently. If your boot entries are generated, fix the generator’s config, not the output.
  7. Turn on pstore before you need it. A disabled feature does not log anything.
  8. Spec sheets are sometimes wrong. A Gen3 label next to a 7000 MB/s rating is almost certainly a typo, not a miracle.
  9. lspci cannot see an empty M.2 slot. Photos and screw-post spacing can.
  10. A plastic bottom cover means thermal pads to the lid are pointless. Airflow and a heatsink are what work.

The darkly funny part: that drive had logged 82.6 TB of writes on a 500 GB disk and a pile of unsafe shutdowns, and it was blamed last. The GPU took the heat for a crime the SSD committed.

Common Questions

Does disabling NVMe APST hurt SSD lifespan?

No. Disabling APST with nvme_core.default_ps_max_latency_us=0 keeps the drive in its full-power state, which raises idle power and heat slightly. It adds no writes. Sustained heat is a reliability risk for any drive, so pair the flag with a heatsink or airflow on small cases.

How do I know if my NVMe is dropping off the PCIe bus?

Look for Unable to change power state from D3cold to D0 or CSTS=0xffffffff in dmesg or the journal. Both mean the controller stopped answering. Other signs are a filesystem that goes read-only, fsck crashing, and a journal that reports corruption after a clean shutdown. If the journal is gone, enable pstore to capture the panic.

Do NVMe power-saving kernel flags increase power usage?

Yes. pcie_port_pm=off and nvme_core.default_ps_max_latency_us=0 keep the PCIe ports and the drive awake when they would otherwise idle, and pcie_aspm=off stops the kernel from enabling extra link power states, so idle watts go up. Measure the difference with a wall meter instead of guessing. Use these flags only on a machine with a drive that is actually dropping off the bus.

Can I remove pcie_aspm=off after replacing the SSD?

Yes, and you should test it. Wait for a week of stable running on the new drive, then remove one flag at a time, a few days apart. Do not reuse the old drive in the machine, because the flags stay needed while it is installed. If crashes return, put the last flag back.


Share this post on:

Send a Webmention

Written about this post on your own site? Send a webmention and it'll show up above once verified.


Next Post
Lutris vs Heroic vs Bottles on Linux

Discussion

Powered by Garrul . Sign in with GitHub or Google, or post anonymously.

Related Posts