The Day Your Backup Finally Has to Do Its Job
The day your backup script finally has to do its job is not the day you want to discover you broke it three months ago.
You know the story. You set up BorgBackup, wrote a little bash script, threw it in cron, and felt briefly like a responsible adult. Then you chmod +x something you shouldn’t have, the script silently failed, cron kept running it and reporting nothing because nobody set up stderr handling, and for ninety days you had zero backups while your monitoring dashboard showed green because it was measuring “did the cron job run” instead of “did the backup actually work.”
Then your ZFS pool died.
Borgmatic exists precisely to prevent that story. It’s the wrapper that turns BorgBackup from a sharp, capable CLI tool into a complete homelab-grade backup system — one with retention policies, hooks, healthcheck pings, and the kind of noise-free alerting that only talks to you when something’s actually wrong.
Let’s actually set it up.
What BorgBackup Actually Is (Quick Recap)
If you haven’t used Borg before, here’s the thirty-second version.
BorgBackup is a deduplicated, encrypted, compressed backup tool. It stores backups as a repository of chunks, where identical data across backups is only stored once. A 200 GB dataset with small daily changes? Your first backup is 200 GB, each subsequent one is only the delta. Practically speaking, daily backups for a year might cost you 240 GB total instead of 73 TB.
The encryption is client-side. Your backup server never sees your data in plaintext. You pick a passphrase, Borg derives a key, done. Even if your backup provider gets breached, your data is unreadable without the passphrase you hold.
Append-only mode is the ransomware defense story. When you configure a repo in append-only mode on the remote, the backup client can create new archives but can’t delete old ones or overwrite data. Ransomware encrypts your local files and your backup client obediently backs up the encrypted garbage — but it can’t reach back and destroy your old archives. You restore from before the attack.
Borg is genuinely excellent. The problem isn’t Borg.
Why Borg Alone Isn’t Enough
The problem is the operational wrapper around Borg.
A production-worthy backup system needs to:
- Run on a schedule, reliably, even after reboots
- Apply retention policies (keep 7 daily, 4 weekly, 12 monthly) without manual math
- Run
borg compactso the repo doesn’t bloat from deleted archives - Run
borg checkperiodically to verify data integrity - Run pre/post hooks (stop a Docker container, dump a database, restart it)
- Report success and failure to something that’ll page you
- Not spam your inbox every time it runs successfully
Doing all of this in a bash script is totally possible. It’s also the kind of thing that accretes cruft, breaks silently on edge cases, and becomes a personal archaeology project six months later when you’re trying to figure out why your retention isn’t working.
Borgmatic handles all of this with a single YAML file.
Enter Borgmatic
Borgmatic is a Python wrapper around BorgBackup. You describe what you want in a config file, and it orchestrates borg create, borg prune, borg compact, and borg check in sequence, with hooks before and after each phase. Install it once, write one config, and you have a backup system that’s auditable, version-controllable, and maintainable by future-you who has forgotten everything.
Install it:
pip install borgmatic# or, on Debian/Ubuntu:sudo apt install borgmaticInitialize your first repository:
# Local repoborg init --encryption=repokey-blake2 /mnt/backup/myrepo
# Remote repo over SSHborg init --encryption=repokey-blake2 user@backupbox:/backup/myrepoNow write your config.
Annotated Config
# Where your data livessource_directories: - /home - /etc - /var/lib/docker/volumes - /opt/services
# Where your backups gorepositories: label: homelab-primary - path: /mnt/external/myrepo label: local-mirror
# Encryption passphrase — see notes on storing this safelyencryption_passphrase: "your-very-long-passphrase-here"
# Compression: lz4 is fast, zstd is good, auto picks per-filecompression: auto,zstd
# Exclude noisy directories that aren't worth backing upexclude_patterns: - "*.pyc" - "*/.cache" - "*/.venv" - "*/node_modules" - "*/__pycache__" - "/home/*/.local/share/Trash" - "/var/lib/docker/overlay2"
# Retention policy — this replaces your bash mathkeep_daily: 7keep_weekly: 4keep_monthly: 12keep_yearly: 2
# Run consistency checks (can be slow — tune to taste)checks: - name: repository frequency: 2 weeks - name: archives frequency: 4 weeks
# Hooks — run before/after the entire backup jobbefore_backup: # Dump PostgreSQL before backing up - pg_dump -U postgres mydb > /var/backups/mydb.sql # Stop a container that writes to disk - docker stop paperless-ngx || true
after_backup: # Restart the container - docker start paperless-ngx || true # Ping healthchecks.io to signal success - curl -fsS --retry 3 https://hc-ping.com/your-uuid-here > /dev/null
on_error: # Ping healthchecks.io failure endpoint - curl -fsS --retry 3 https://hc-ping.com/your-uuid-here/fail > /dev/null # Push a notification via ntfy.sh - curl -d "Borgmatic backup FAILED on $(hostname)" https://ntfy.sh/your-backup-topic
# Logging verbosityverbosity: 1A few things worth calling out:
The || true on Docker stop/start. If the container isn’t running, the hook still succeeds. Without this, a stopped container causes your entire backup to abort. Defensive programming in config files.
Healthchecks.io. This is the “did the backup actually succeed” monitor. You create a check, set a schedule (e.g., daily), and if Borgmatic doesn’t ping the success URL within that window, Healthchecks.io emails or Telegrams or Slacks you. It’s the difference between “the cron job ran” and “the backup succeeded.”
ntfy.sh on error. Push notifications to your phone when something breaks. Entirely optional, entirely worth it.
systemd Timer Setup (Preferred Over Cron)
Cron is fine. systemd timers are better — they handle missed runs, log to journald, and support flock for preventing overlapping jobs.
[Unit]Description=BorgBackup via BorgmaticAfter=network-online.targetWants=network-online.target
[Service]Type=oneshotExecStart=/usr/bin/flock -n /tmp/borgmatic.lock /usr/bin/borgmatic --verbosity 1 --log-file /var/log/borgmatic.log# Run as root so it can read everythingUser=root
# Security hardening (optional but good)PrivateTmp=trueProtectSystem=strictReadWritePaths=/etc /home /opt /var /mnt /tmp[Unit]Description=Run borgmatic daily
[Timer]# Run at 2:30 AM daily, with a 30-minute random jitter so your 10 homelabs# don't all hammer the backup server simultaneouslyOnCalendar=dailyRandomizedDelaySec=1800Persistent=true
[Install]WantedBy=timers.targetEnable it:
systemctl daemon-reloadsystemctl enable --now borgmatic.timersystemctl list-timers borgmatic.timerThe flock -n prevents two borgmatic instances from running simultaneously. If a backup is already running when the timer fires, the new one exits immediately. The Persistent=true catches up on missed runs after reboots.
Verifying Your Backups
Setting up backups and never testing restores is just anxiety with extra steps.
Check data integrity:
borgmatic check --verbosity 1This verifies the repository structure and, depending on your config, extracts and checksums random archive chunks. Run it manually after setup, then let the config’s checks: block handle it on schedule.
List archives:
borgmatic listborgmatic list --archive latestRestore a specific file:
borgmatic extract --archive latest --path /home/you/important-file.txtTest restore a full directory (to a temp location):
borgmatic extract --archive latest --path /etc --destination /tmp/restore-test --dry-run# Remove --dry-run when you're confidentborgmatic extract --archive latest --path /etc --destination /tmp/restore-testDo this quarterly. Set a calendar reminder. Your future self who’s restoring from backup at 2 AM is counting on past-you to have verified this actually works.
Repository Targets
Borgmatic works with any Borg-compatible repository. Your options:
SSH to a remote machine is the simplest. Another server you control, a Raspberry Pi in a different physical location, a VPS. Append-only mode is a one-line flag during borg init --append-only.
rsync.net has native Borg support, reasonable pricing, and they know what they’re doing with backup infrastructure. Good option if you don’t want to operate your own backup server.
BorgBase is purpose-built for Borgmatic/Borg, has a web UI for managing repos and viewing archive history, and integrates directly with Healthchecks.io. If you want zero-ops backup infrastructure, this is the move.
Hetzner StorageBox is cheap, German (GDPR-friendly), supports SSH/SFTP, and works well with Borg. Mount it, point borgmatic at it, done.
For homelab setups, the recommended pattern is two repositories: one local (NAS, external drive) for fast restores, one remote (BorgBase, rsync.net) for disaster recovery. Both defined in your config, both get every backup, borgmatic handles the split.
Borgmatic vs. The Alternatives
Honest comparison, because no tool wins everything:
Restic is the closest alternative. Similar deduplication approach, simpler CLI, backends for S3/B2/GCS/Azure out of the box. No native append-only mode (though Backblaze B2 has object lock). If you’re backing up to cloud storage natively, Restic is often the better fit. If you’re backing up to SSH remotes and want append-only, Borg/Borgmatic wins.
Kopia is newer, has a web UI, excellent compression (zstd), and a more modern design. Great for people who want a dashboard. The config-as-code story is less mature than borgmatic’s single YAML. Still maturing.
Duplicacy has a clever lock-free design that lets multiple machines share a repo without conflicts. The CLI is free; the GUI is paid for personal use. Niche but genuinely solves a real problem if you have many clients.
rsnapshot / rdiff-backup / rsync are fine for simple use cases and have been battle-tested for decades. They don’t do client-side encryption, deduplication is cruder, and you’re back to writing wrapper scripts. Good for local LAN backups where encryption isn’t a requirement.
Use Borgmatic when: SSH remotes, client-side encryption is non-negotiable, you want config-as-code, append-only ransomware protection matters.
Homelab Tips That’ll Save You
Backup your encryption passphrase separately from the backup. This is obvious in retrospect and catastrophic when you forget it. Write it down, put it in a password manager, print it and put it in a fireproof envelope. If you lose the passphrase, your entire backup archive is gone. The encryption is that good.
Exclude caches and ephemeral data. /home/*/.cache, /home/*/.venv, node_modules, Docker’s overlay filesystem — none of this needs to be in your backup. It bloats the repo and wastes bandwidth. The exclusion patterns in the config above are a reasonable starting point.
Snapshot before backup for consistency. If you’re on LVM or ZFS, snapshot the filesystem before Borgmatic runs so you’re backing up a consistent point-in-time view. Put the snapshot creation in before_backup hooks and cleanup in after_backup. For databases, do a dump — the pg_dump and mysqldump patterns in the config above handle this.
Don’t ignore the compact step. After pruning old archives, disk space isn’t reclaimed immediately. borg compact does that. Borgmatic runs it automatically. If you’re using raw Borg commands, remember to compact.
Test your healthcheck setup before you need it. After initial setup, deliberately cause a failure (borgmatic --dry-run won’t trigger on_error hooks, so temporarily misconfigure something). Verify you actually get the alert. A healthcheck that never pages you isn’t a healthcheck.
The Backup That Pages You When It’s Broken
There’s a particular flavor of homelab hubris where you spend three hours setting up a backup system and then never think about it again until disaster strikes. Borgmatic is designed to fight that tendency.
The Healthchecks.io integration means your backup system actively proves it’s working every time it runs. Not “did the cron job fire” — “did borgmatic complete successfully, prune old archives, and check data integrity.” If it stops pinging, you hear about it.
The hooks mean your databases are dumped before backup so you’re not restoring corrupted mid-transaction state. The retention policy means you have a month of weekly snapshots without managing that manually. The systemd timer means a server reboot doesn’t silently break your backup schedule.
Your 2 AM self — the one who just lost a drive or fat-fingered a rm -rf — deserves a backup system that’s been quietly working in the background, paging you every time it fails, and sitting ready with a clean restore point. Not a bash script that last worked in March.
Set it up. Test the restore. Set the healthcheck. Then forget about it until it tells you something’s wrong.
That’s the whole idea.