You Don’t Have Four Racks of Logs
Somewhere on Reddit right now, someone with three Raspberry Pis and a NAS is asking how to set up Elasticsearch, Logstash, and Kibana to “centralize their logs.” That’s like hiring a forklift to move a couch. Technically it works, but your neighbors will have questions, and now you’re also the guy who maintains a JVM heap size on a home server.
If your boxes run systemd, they are already storing structured logs. Every entry in the journal is a set of FIELD=value pairs, not a flat line of text. You can add your own fields to that structure, query by them, and when you’re ready to stop SSHing into four machines to check on last night’s backup job, you can ship every entry to one central box with two systemd units that ship in the box: systemd-journal-remote and systemd-journal-upload. No agents to install, no index to tune, no Java.
This gets you real structured logging and one place to read it, for a handful of hosts, before you need a query language, a retention policy UI, or a second server just to run the logging stack. It also has a ceiling. No web dashboard, no full-text index spanning every host at once, no alerting, and it gets weaker fast the moment a host isn’t running systemd. We’ll get to exactly where that ceiling sits.
Every Entry Is Already a Key/Value Blob
Run any command through journalctl -o verbose and you’ll see it: that “log line” you thought was one string is actually a bag of fields.
journalctl -u sshd -n 1 -o verbose --no-pagerMon 2026-09-28 09:03:10 EDT [s=74d4087f...] _UID=0 _GID=0 _BOOT_ID=3f9a1c0e5b7d4e2a8c6f0b1d2e3a4b5c _MACHINE_ID=a1b2c3d4e5f60718293a4b5c6d7e8f90 _HOSTNAME=web01 _TRANSPORT=journal _SYSTEMD_UNIT=sshd.service MESSAGE=Accepted publickey for deploy from 10.0.0.5 port 51234 SYSLOG_IDENTIFIER=sshd _PID=1722724Every field falls into one of two buckets. Fields prefixed with an underscore, _PID, _UID, _SYSTEMD_UNIT, _HOSTNAME, _MACHINE_ID, are trusted fields. The journal daemon adds these itself from kernel and process credentials, and no client can forge them. Fields without the underscore, MESSAGE, PRIORITY, SYSLOG_IDENTIFIER, are user fields: whatever the logging client sent. man systemd.journal-fields lists the well-known ones, but nothing stops you from inventing your own. The rest of this article does exactly that.
There are rules for custom field names, straight from man sd_journal_print: uppercase letters, digits, and underscores only, and the name cannot start with an underscore (that prefix is reserved for the trusted fields journald controls). BACKUP_STATUS is valid. backup_status and _BACKUP_STATUS both get silently dropped.
Want the same data as JSON instead of the verbose block? Same query, different -o:
journalctl -u sshd -n 1 -o json-pretty --no-pager{ "MESSAGE" : "Accepted publickey for deploy from 10.0.0.5 port 51234", "SYSLOG_IDENTIFIER" : "sshd", "_SYSTEMD_UNIT" : "sshd.service", "_HOSTNAME" : "web01", "_PID" : "1722724"}journald was structured before “structured logging” became a marketing term for a $40/month SaaS tier.
Writing Your Own Fields
Reading the fields journald already gives you is nice. Writing your own is where this becomes useful for anything you actually care about, cron jobs, backup scripts, deploy hooks, that one Python thing that runs at 3 AM and nobody watches.
logger --journald reads FIELD=value lines from stdin (or a file) and writes them straight into the journal, verified in man logger:
logger --journald <<'EOF'MESSAGE=nightly backup completedBACKUP_TARGET=nas01.lanBACKUP_STATUS=okSYSLOG_IDENTIFIER=backup-jobEOFThe logger man page also recommends a MESSAGE_ID= line (a 128-bit ID; systemd-id128 new generates one) so you can find an event type later without matching on message text. Priority also has to travel as a field here (PRIORITY=3), since --journald ignores logger’s other command-line flags entirely.
systemd-cat takes the opposite approach: point it at a command or a pipe, and it wraps stdout/stderr into the journal with a tag and a priority you choose:
rsync -av /data/ nas01.lan:/backup/ | systemd-cat -t rsync-backup -p infoAnything that script prints now lands in the journal tagged rsync-backup, queryable with journalctl -t rsync-backup. It won’t let you attach arbitrary custom fields the way logger --journald does, it’s built for wrapping a whole command’s output, not hand-crafting individual fields.
Python, via the systemd-python package on PyPI (pip install systemd-python, or your distro’s python-systemd package), gives you the same field-writing power as a function call:
from systemd import journal
journal.send( "nightly backup finished", SYSLOG_IDENTIFIER="backup-job", BACKUP_JOB="nightly", BACKUP_STATUS="ok", BYTES="48213912",)journal.send() takes the message as the first positional argument, and any keyword argument becomes a custom field, capitalized exactly as you write it. The same uppercase-letters-digits-underscore rule applies, and here the library enforces it: backup_status="ok" raises ValueError: Invalid field name before anything reaches the journal. That beats the silent drop you get from logger --journald.
Put those together and a real backup script looks like this:
#!/usr/bin/env bashset -euo pipefail
TARGET="nas01.lan"SRC="/data"
if rsync -a "$SRC/" "rsync://${TARGET}/backup/"; then STATUS="ok" MSG="nightly backup to ${TARGET} succeeded"else STATUS="failed" MSG="nightly backup to ${TARGET} failed"fi
logger --journald <<EOFMESSAGE=${MSG}BACKUP_TARGET=${TARGET}BACKUP_STATUS=${STATUS}SYSLOG_IDENTIFIER=backup-jobEOF
[ "$STATUS" = "ok" ]Now every run of that cron job leaves a queryable, structured entry behind instead of a line of text you’d have to regex apart later.
Querying What You Wrote
If you haven’t read the journalctl queries cheat sheet, start there for -u, -p, --since, and the basics. This part is just about your custom fields specifically.
Query directly by field:
journalctl BACKUP_STATUS=failedField logic follows one rule: repeating the same field is OR, different fields are AND.
# OR: either status, still constrained to backup-job entries belowjournalctl SYSLOG_IDENTIFIER=backup-job BACKUP_STATUS=ok BACKUP_STATUS=failed
# AND: only backup-job entries where status is failedjournalctl SYSLOG_IDENTIFIER=backup-job BACKUP_STATUS=failedTo OR across an entire match group instead of one field, use +:
# every failed backup, OR every entry tagged webapp-test, regardless of statusjournalctl BACKUP_STATUS=failed + SYSLOG_IDENTIFIER=webapp-testNot sure what values a field actually takes across your history? -F lists them:
journalctl -F BACKUP_STATUS# ok# failedAnd for scripting, JSON plus jq beats parsing verbose output by hand:
journalctl BACKUP_STATUS=failed -o json --no-pager | jq -r '.MESSAGE'If you only want a couple of fields back instead of the full entry, --output-fields= trims the verbose or JSON output down to just what you name:
journalctl SYSLOG_IDENTIFIER=backup-job -o verbose --output-fields=MESSAGE,BACKUP_STATUSThat’s a database query, running against a log store you didn’t have to install.
Keeping It Around: Persistence and Retention
Older systemd releases default to Storage=auto, and on a box without a /var/log/journal directory that means the journal lives in /run/log/journal: memory-backed and gone on reboot. Current systemd (the 261 man page) defaults to persistent, but distro builds and older LTS releases vary, so check for /var/log/journal before you trust it. Pin it explicitly:
[Journal]Storage=persistentThat alone moves the journal into /var/log/journal, which survives reboots. Storage= also accepts volatile (memory only), auto (persistent only if /var/log/journal already exists), and none, but persistent is the one you want on a home lab box.
Uncapped retention on a box that’s now permanently logging will eventually bite you, so set limits in the same drop-in:
[Journal]Storage=persistentSystemMaxUse=2GMaxRetentionSec=30daySystemMaxUse= caps total disk space the journal is allowed to consume on persistent storage. MaxRetentionSec= deletes anything older than the window regardless of size. Drop-ins over editing /etc/systemd/journald.conf directly are the right call here: they survive a distro upgrade that ships a new default config.
Check what you’re actually using and prune on demand:
journalctl --disk-usage# Archived and active journals take up 2.1G in the file system.
journalctl --vacuum-size=1G# or by age:journalctl --vacuum-time=14dRestart the daemon to pick up the drop-in: systemctl restart systemd-journald.
Shipping It to One Box
This is the part that replaces the log shipper you were about to install. Two units, both shipped with systemd itself: systemd-journal-remote receives, systemd-journal-upload sends.
Server side. On Debian and Ubuntu, install the systemd-journal-remote package (other distros ship it under a similar name; check your package manager). It brings a socket unit, systemd-journal-remote.socket, listening on port 19532 by default, and a matching service that writes incoming entries into journal files under /var/log/journal/remote/.
The stock service is wired for HTTPS out of the box; its ExecStart= runs with --listen-https=-3. For a trusted LAN or a tailnet you already control, that’s more setup than you need on day one. Override it to plain HTTP with a drop-in:
sudo systemctl edit systemd-journal-remote.service[Service]ExecStart=ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/The empty ExecStart= line first is required. It clears the vendor’s command before you set your own. Without it, the service ends up with two ExecStart= lines, which systemd refuses to load for anything other than Type=oneshot. Then enable the socket:
sudo systemctl enable --now systemd-journal-remote.socketClient side. Every box you want to ship from needs systemd-journal-upload pointed at the server:
[Upload]URL=http://logbox.lan:19532sudo systemctl enable --now systemd-journal-upload.serviceClient defaults to HTTPS on port 19532 too, so an explicit http:// in the URL matters, or the client will try to negotiate TLS against a server that isn’t listening for it.
When you’re ready to stop trusting the LAN and lock this down, both sides take real certificates: ServerKeyFile= and ServerCertificateFile= on the receiving end, TrustedCertificateFile= on both, so each side can verify the other. man systemd-journal-upload walks through generating a private CA with openssl if you don’t already have one lying around.
Query what landed on the server the same way you’d query local logs, just point at the remote directory:
journalctl -D /var/log/journal/remote -u nginx --since todayor a specific file if SplitMode=host gave you one journal per source host:
journalctl --file=/var/log/journal/remote/remote-web01.journalOne more detail worth knowing before your first restart scare: the upload client tracks its position with a cursor saved at /var/lib/systemd/journal-upload/state. Restart the service and it resumes from there instead of replaying everything or dropping entries in the gap.
Where This Falls Over
Be straight with yourself about what you just built. It is not a replacement for Loki, Vector, or an ELK stack, it’s what you do before you need one.
No web UI. Reading logs means SSH and journalctl, there’s no dashboard to hand a coworker or check from your phone. No alerting: nothing here pages you when BACKUP_STATUS=failed shows up, you’re still the alerting layer. No index built for scale: journalctl -D merges every host’s file at query time, which is fine for five boxes and slow once you have months of logs from dozens, where Loki’s label index earns its keep. And it leans entirely on systemd: an Alpine container, a BSD box, or anything running plain syslog doesn’t get any of this for free, you’d need a separate forwarder just to get its logs into a journal in the first place.
For four or five systemd boxes and cron jobs you actually want visibility into, this covers it without another service eating RAM. The moment you want a graph of error rates over the last month, or an alert that fires at 2 AM instead of a query you remember to run, that’s your graduation signal. Stand up Loki with Grafana Alloy or Vector as the shipper, and mean it that time.
Common Questions
Does the Docker journald log driver write structured fields too?
Yes. The Docker journald log driver writes each container output line as MESSAGE= and adds CONTAINER_NAME, CONTAINER_ID, CONTAINER_ID_FULL, CONTAINER_TAG, and IMAGE_NAME as custom fields. The trusted _ fields describe dockerd, not the container. Query one container with journalctl CONTAINER_NAME=myapp, and those container logs ship to the central box with everything else.
Does systemd-journal-upload encrypt logs in transit?
Yes, with an https:// URL and certificates on both ends. systemd-journal-upload uses HTTPS by default and needs ServerKeyFile=, ServerCertificateFile=, and TrustedCertificateFile= to make the TLS connection. An http:// URL sends log entries in cleartext, which is acceptable on a tailnet that already encrypts traffic and nowhere else.
Does this work on non-systemd hosts like Alpine?
No, not directly. systemd-journal-upload and the journal fields covered here are systemd-specific; Alpine’s default busybox/OpenRC init has no journal to read from. Ship an Alpine box’s logs by pointing its syslog output at a remote collector instead, or run a separate forwarder like Vector on that host specifically.
Can rsyslog and journald run side by side?
Yes, and many distros ship both. rsyslog usually reads journal files directly through its imjournal module, which is why ForwardToSyslog= defaults to off in current systemd. Existing rsyslog rules and remote destinations keep working while you build the journal-based setup next to them. The cost is storing every message twice on disk.
How much disk space does a year of persistent journald logging cost?
At most the SystemMaxUse= cap. Without a cap, journald defaults to 10% of the filesystem, limited to 4G. Once the journal reaches the cap, journald deletes the oldest files first, so a year of logs on a SystemMaxUse=2G box costs 2G. Size the cap, not the time window, and check real usage with journalctl --disk-usage.
