Stop Reading the Restart Log
It’s 2 AM. Your service has died eleven times since midnight, and systemctl status says status=11/SEGV over and over. You scroll the journal, find the same “Scheduled restart job” line, and start guessing. Maybe it’s the config. Maybe it’s the new library. Maybe it’s you.
Stop guessing. When a process dies of a signal, the kernel hands its memory to systemd-coredump, which stores it and logs a stack trace. The answer to “what was it doing when it died” is already on disk. coredumpctl finds it, and gdb reads it. The restart loop is just noise sitting on top of the evidence.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/linux/coredumpctl-crash-loop
Everything below ran on systemd 262 against a small C program that crashes on purpose. The output is trimmed, and hostnames, users, and paths are anonymized.
The Crime Scene: What systemctl Tells You
Meet the victim. It waits two seconds, then dereferences a NULL pointer inside a function called parse_config().
static int parse_config(struct config *cfg){ /* Bug on purpose: the caller passes NULL, so this write segfaults. */ cfg->port = 8080; strcpy(cfg->name, "crashy"); return 0;}
static int load_settings(void){ struct config *cfg = NULL; return parse_config(cfg);}The unit restarts it on failure, three seconds apart, and gives up after three starts in 60 seconds:
[Unit]Description=Crashy demo service (segfaults 2 seconds after start)StartLimitIntervalSec=60StartLimitBurst=3
[Service]ExecStart=/home/you/bin/crashyRestart=on-failureRestartSec=3LimitCORE=infinityAfter a few cycles, systemctl status shows this:
× crashy-demo.service - Crashy demo service (segfaults 2 seconds after start) Active: failed (Result: start-limit-hit) since Thu 2026-10-29 15:02:32 EDT; 4s ago Process: 2897343 ExecStart=/home/you/bin/crashy (code=dumped, signal=SEGV) Main PID: 2897343 (code=dumped, signal=SEGV)
crashy-demo.service: Failed with result 'core-dump'.crashy-demo.service: Scheduled restart job, restart counter is at 3.crashy-demo.service: Start request repeated too quickly.crashy-demo.service: Failed with result 'start-limit-hit'.Three clues matter here. code=dumped means the process died and a core dump was produced. signal=SEGV means it touched memory it does not own. Result: 'core-dump' is the unit’s failure reason. If you want a refresher on reading this block, the systemctl status guide covers every line.
Also note the last line. After three failures in 60 seconds, systemd stops restarting the unit and reports start-limit-hit. That is the loop giving up, and it’s why you end up staring at a dead service instead of a flapping one.
Don’t Let the Loop Hide the Evidence
A tight restart loop can bury your crash under a pile of identical journal lines, and a start limit can lock the unit out entirely. Three settings control this:
RestartSec=is the pause between a crash and the next start. Set it to 5 to 10 seconds while you debug so the journal stays readable.StartLimitIntervalSec=is the window systemd counts starts in.StartLimitBurst=is how many starts fit in that window before the unit is refused.
If the unit is stuck in start-limit-hit, clear it before you try again:
systemctl reset-failed myapp.servicesystemctl start myapp.serviceDon’t disable the limit with StartLimitIntervalSec=0 and walk away. A service that crashes every three seconds forever will happily fill your disk with cores. Keep the limit, raise the RestartSec=, and move on to the real evidence.
coredumpctl list: Find the Core
coredumpctl list prints one row per crash:
coredumpctl listTIME PID UID GID SIG COREFILE EXE SIZEThu 2026-10-29 15:02:18 EDT 2897117 1000 1000 SIGSEGV present /home/you/bin/crashy 19.4KThu 2026-10-29 15:02:23 EDT 2897244 1000 1000 SIGSEGV present /home/you/bin/crashy 19.3KThu 2026-10-29 15:02:28 EDT 2897343 1000 1000 SIGSEGV present /home/you/bin/crashy 19.3KThe COREFILE column is the one to watch. The man page lists these values: present (stored and readable), journal (stored in the journal), truncated (too big to store whole), none (not stored), missing (stored once, since deleted), error (not accessible), and - (no core was available). The missing and none cases get their own section below.
There is no -u or --unit flag. coredumpctl filters with positional matches instead, and it accepts four kinds:
coredumpctl list 2897343 # a PIDcoredumpctl list crashy # a command name (no slashes)coredumpctl list /home/you/bin/crashy # an executable path (needs a slash)coredumpctl list COREDUMP_USER_UNIT=crashy-demo.service # any journal fieldThat last form is the “by unit” filter, and it has a trap. For a system service, you match COREDUMP_UNIT=myapp.service. For a user service like my demo, COREDUMP_UNIT= does not match, because the system-level unit recorded for the crash is [email protected]. Use COREDUMP_USER_UNIT= instead. Not sure what values exist? Ask:
coredumpctl -F COREDUMP_USER_UNITFor time and count filters:
coredumpctl list --since="-10min" # -S is the short formcoredumpctl list --until="-1h" # -Ucoredumpctl list -r -n 5 # newest first, five rowscoredumpctl -1 list # newest only (same as -r -n 1)Any command that finds no match exits non-zero, which is handy in scripts.
coredumpctl info: The Free Backtrace
Before you open gdb, run info. systemd already logged a stack trace when the crash happened:
coredumpctl info 2897343 PID: 2897343 (crashy) Signal: 11 (SEGV) si_code: SEGV_MAPERR Control Group: /user.slice/user-1000.slice/[email protected]/app.slice/crashy-demo.service Unit: [email protected] User Unit: crashy-demo.service Storage: /var/lib/systemd/coredump/core.crashy.1000.<boot-id>.2897343.1793300548000000.zst (present) Size on Disk: 19.3K Message: Process 2897343 (crashy) of user 1000 dumped core.
Stack trace of thread 2897343: #0 0x00005564f0b5f155 n/a (/home/you/bin/crashy + 0x1155) #1 0x00005564f0b5f193 n/a (/home/you/bin/crashy + 0x1193) #2 0x00005564f0b5f1c8 n/a (/home/you/bin/crashy + 0x11c8) #3 0x00007f2e6ea27d0e n/a (libc.so.6 + 0x27d0e)SEGV_MAPERR means the address was not mapped at all, the classic NULL dereference signature. But look at the frames: n/a where the function names should be. The journal trace had no symbol names for my binary, so it only shows addresses. That’s the cue to bring in gdb.
coredumpctl debug: Drop Into gdb
debug finds the matching core, decompresses it, and starts gdb with the right executable already loaded:
coredumpctl debug crashyIt picks the newest match by default. Inside gdb, five commands cover most crashes:
(gdb) bt # backtrace of the crashing thread(gdb) bt full # backtrace plus local variables in each frame(gdb) info threads # list every thread(gdb) thread apply all bt # backtrace of every thread(gdb) frame 1 # jump to frame 1, then "info locals" works thereFor a one-shot answer you can paste into a ticket, skip the interactive session. The -A (--debugger-arguments) option passes arguments straight to gdb:
coredumpctl debug crashy --debugger-arguments="-batch -ex bt"Core was generated by `/home/you/bin/crashy'.Program terminated with signal SIGSEGV, Segmentation fault.#0 0x00005564f0b5f155 in parse_config (cfg=0x0) at crashy.c:1313 cfg->port = 8080;#0 0x00005564f0b5f155 in parse_config (cfg=0x0) at crashy.c:13#1 0x00005564f0b5f193 in load_settings () at crashy.c:21#2 0x00005564f0b5f1c8 in main () at crashy.c:28There it is. parse_config (cfg=0x0): the function got a NULL pointer, and it was called from load_settings() on line 21. That is a one-line fix, found in about ten seconds without reading a single restart log. Here the symbols resolved because I compiled with gcc -g. Your distro binaries need more help, which is the debuginfod section below.
Copy the Core Somewhere Safe
If the box is about to be reimaged, or you want to debug on your laptop, extract the core with dump:
coredumpctl dump crashy -o /tmp/coregdb /home/you/bin/crashy /tmp/coreWithout -o, dump writes the core to stdout. It refuses when stdout is a terminal, so redirect it to a file. The gdb command needs the same executable the service ran, built from the same version. A core plus the wrong binary gives you nonsense frames.
Why Storage Says “none” or “missing”
Two columns of sadness, two different causes.
missing means the core file was stored, and has since been deleted. By default, systemd-tmpfiles cleans /var/lib/systemd/coredump on a timer. On my machine the rule in /usr/lib/tmpfiles.d/systemd.conf is:
d /var/lib/systemd/coredump 0755 root root 2wThat is a two-week retention on this system (check yours, vendors differ). The journal entry with the stack trace outlives the file, so you still see the row in coredumpctl list. If you want the mechanics of that cleanup, the systemd-tmpfiles post explains the age field.
none means the core was never stored. The usual cause is Storage=none in coredump.conf, which keeps the log line and trace but drops the memory image. A core larger than ExternalSizeMax= is the other suspect: it can end up truncated or not stored at all.
Check what you actually have, with all drop-ins merged:
systemd-analyze cat-config systemd/coredump.confTuning coredump.conf
Never edit the vendor file. Drop your overrides into /etc/systemd/coredump.conf.d/, which the man page recommends:
[Coredump]Storage=externalCompress=yesProcessSizeMax=2GExternalSizeMax=2GMaxUse=2GKeepFree=2GWhat each key does, per man coredump.conf:
Storage=isnone,external(the default, files in/var/lib/systemd/coredump/), orjournal.Compress=compresses external cores. It defaults to yes.ProcessSizeMax=is the largest core systemd will process into a stack trace. The default is 32G on 64-bit systems.ExternalSizeMax=is the largest core saved to disk. The default is also 32G on 64-bit.MaxUse=deletes the oldest cores once the total passes this size. The default is 10% of the disk, capped at 4 GiB.KeepFree=keeps at least this much disk free. The default is 15% of the disk, capped at 4 GiB.
On a small SSD, two or three crashes of a 6 GB process will eat the disk before the two-week cleanup runs, so set MaxUse= to something you can afford.
Which service handles a crash depends on your kernel and systemd version. Many systems start a fresh [email protected] instance for every crash, and that instance reads the config each time, so the change applies to the next crash. Others run the long-lived systemd-coredumpd.service, which reads the config only once at startup. Check, and reload it if it’s running:
systemctl is-active systemd-coredumpd.servicesudo systemctl reload systemd-coredumpd.service # only if it printed "active"When No Core Appears At All
If coredumpctl list shows nothing for a crash you know happened, check three things in order.
- Is systemd-coredump the handler? The kernel decides who gets the core. Run
cat /proc/sys/kernel/core_pattern. If it starts with|/usr/lib/systemd/systemd-coredump, you’re covered. If it sayscoreor a path, something else is in charge andcoredumpctlwill never see it. - Is the core size limit zero? The man page says the core is only processed when
RLIMIT_COREis large enough. Read the limit of a running service straight from/proc:
grep -i 'core' /proc/$(systemctl show -p MainPID --value myapp.service)/limits- Is it a signal crash? A process that calls
exit(1)cleanly leaves no core. Only the core-dumping signals do, such as SEGV, ABRT, BUS, and FPE. SIGKILL and SIGTERM never produce one.
On my machine, systemd-system.conf(5) says PID 1 sets RLIMIT_CORE to infinity and children inherit it, so services get cores with no extra setup. Your distro or a PAM limits.conf might set it to zero for login sessions. Put LimitCORE=infinity in the unit, as in the example above, and the question goes away. It’s the unit-file equivalent of ulimit -c unlimited in a shell.
Debug Symbols and debuginfod
My own binary has symbols because I compiled it with -g. The binaries your package manager ships are stripped, so the backtrace shows addresses and n/a instead of function names and line numbers.
debuginfod fixes that. It’s a server that hands out debug symbols on request, keyed to the exact build. Distros run public ones, and gdb can query them. On this machine the DEBUGINFOD_URLS variable was already set to the Arch and CachyOS servers. The first time gdb sees it, it asks:
This GDB supports auto-downloading debuginfo from the following URLs: <https://debuginfod.archlinux.org> <https://debuginfod.cachyos.org>Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]Debuginfod has been disabled.To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.Interactively, answer y. In batch mode, as above, it answers no for you and the download never happens. To turn it on without a prompt:
coredumpctl debug myapp --debugger-arguments="-batch -ex 'set debuginfod enabled on' -ex bt"If DEBUGINFOD_URLS is empty on your system, set it to your distro’s server first. The first lookup downloads symbols for every library in the trace, so expect it to be slow once and fast afterwards.
Cores From Docker Containers
A crash inside a container still gets caught. The kernel’s core_pattern is global to the host, not per container, so the pipe to systemd-coredump fires on the host. I tested it by running a short-lived process in a container and killing it with SIGSEGV. The row appeared in coredumpctl list on the host:
TIME PID UID GID SIG COREFILE EXE SIZEThu 2026-10-29 15:02:54 EDT 2898407 472 0 SIGSEGV inaccessible /bin/busybox -And coredumpctl info showed what the host knows about it:
PID: 2898407 (sleep) Signal: 11 (SEGV) si_code: SI_USER Control Group: /system.slice/docker-<container-id>.scope Unit: docker-<container-id>.scope Slice: system.slice Hostname: <container-id-prefix> Storage: /var/lib/systemd/coredump/core.sleep.472.<boot-id>.2898407.1793300574000000.zst (inaccessible) Message: Process 2898407 (sleep) of user 472 dumped core.
Stack trace of thread 7: #0 0x00007fe662728185 n/a (/lib/ld-musl-x86_64.so.1 + 0x6c185)Three observations from that run. First, the unit is the Docker scope, so coredumpctl list COREDUMP_UNIT=docker-<id>.scope is how you filter to one container. Second, EXE is the path inside the container (/bin/busybox), and the Hostname is the container’s. Third, COREFILE said inaccessible because my user could not read a core owned by root, so rerun with sudo to get at it. The inaccessible value is not in the man page’s list, so treat it as “your user lacks permission”.
The man page adds a caveat. The host-side stack trace is built from the host’s copy of libraries, which differ from the container’s, so symbol resolution in the trace is often poor. The EnterNamespace= option in coredump.conf (added in version 257) makes systemd-coredump use the container’s own filesystem for symbols. It defaults to off for security reasons. For gdb, the practical rule is the one from the dump section: you need the container’s binary, so extract the core and run gdb inside a container built from the same image.
What to Do With the Backtrace
You have the crash location. Now what.
- Read frame 0 and its caller. The crash is usually one or two frames up from where it looks like it happened. In my case, the NULL came from
load_settings(). - Check the arguments.
cfg=0x0is a NULL. A huge or negative size points to an integer bug. Usebt fullto see them. - Search the exact function name plus the signal. Crashes in libraries have usually been reported already. Add the library version.
- Look for a version mismatch. A crash right after an upgrade often means a plugin or module was built against the old library.
- File a bug with the backtrace. Paste the output of
bt fullandthread apply all bt. Maintainers will ask for it, so save them the round trip.
For shell-script crashes, which have no core, the Bash debugging post covers the right tools instead. For tracing what happened around the crash, these journalctl queries get you the context lines.
Privacy and Disk Warnings
A core dump is a copy of the process’s memory. That includes whatever the process was holding: passwords, API tokens, session cookies, decrypted data. Treat a core like a secrets file.
- Don’t paste or attach a raw core to a public bug report. Send the backtrace text instead, or scrub it first.
- Check the file permissions. The cores under
/var/lib/systemd/coredump/are root-owned, and that is what you want. - Cap the disk use. Set
MaxUse=andKeepFree=as shown above, and letsystemd-tmpfilesdo its two-week cleanup. - Clean up by hand when you’re done.
coredumpctlhas no delete command, so remove the file from/var/lib/systemd/coredump/as root. The journal entry will remain until journal rotation removes it.
Debugging a crash is one of the few times a service’s secrets are lying around in plain files. Don’t leave them there.
Common Questions
Where does systemd store core dumps?
systemd stores core dumps in /var/lib/systemd/coredump/ by default, as compressed files named core.<command>.<uid>.<boot-id>.<pid>.<timestamp>.zst. Setting Storage=journal in coredump.conf moves the core into the journal instead. The Storage: line in coredumpctl info shows the exact path for each crash, and the default tmpfiles rule removes old files after two weeks.
How do I enable core dumps for a systemd service?
Check that kernel.core_pattern points at systemd-coredump, then set LimitCORE=infinity in the unit under [Service] and restart it. Many systems already inherit an unlimited core limit from PID 1, but setting the directive explicitly removes any doubt. Verify the result in /proc/<pid>/limits.
Why does coredumpctl say the core file is missing?
The core file was stored, then deleted. systemd-tmpfiles removes files in /var/lib/systemd/coredump/ after a retention period, which is two weeks in the default rule on a current systemd, and MaxUse= can also delete older files early. The journal entry and its stack trace usually remain after the file is gone.
Do I need debug symbols to read a core dump?
Yes, for readable function names and line numbers. Without symbols, coredumpctl info and gdb show raw addresses and n/a. Build your own code with gcc -g. For distro packages, enable debuginfod by setting DEBUGINFOD_URLS and answering y when gdb offers to download symbols.
Can coredumpctl filter by service name?
Yes, coredumpctl filters by service through a journal match, because it has no --unit flag. Pass COREDUMP_UNIT=myapp.service for system services, or COREDUMP_USER_UNIT=myapp.service for user services. You can also match by PID, command name, or executable path, and coredumpctl -F COREDUMP_UNIT lists the unit values that exist.
