Avoiding absolute desaster – automatic backup check for ZFS

Having backups of your servers data is essential. But what if somethings goes wrong with the backup process? An automated backup check can be an incredibly important tool. I wrote a small script to check for new backups on my ZFS based server and notify me if any are missing.

Always backup your data!

If you save any important data yourself you should always plan for backups! Not only can this prevent you from loosing the data but also help against accidental changes to your data or even external threads like encryption viruses. Make sure to have regular automatic backups including some kind of offsite storage that can not easily be infected by threads in your own network.

As I described in earlier articles I built a home server based on the ZFS filesystem. This awesome filesystem makes it relatively simple to backup your data using snapshots and the send/receive feature. You can read about the full file system and backup setup here. This leaves me with very frequently updated versions of all my important data that I can use to undo changes as well a rotating set of drives that store a copy of the data in some other place.

Make sure that your backup actually works

This backup setup worked great … until it didn’t. Something happened that stopped the USB connection to the drives from working properly and when I next manually checked the logs I saw that I was missing several days of backup. Fixing the problem was quite easy: pulling the USB cable out and pushing it back into the slot. Afterwards my backups continued but this left me quite unsatisfied about the backups.

Having backups only really makes sense if you can rely on them. After this incident I decided to create a monitoring system that informs me if any backups are missing. This will work together with an already existing external system that notifies me if the whole server becomes unreachable (hardware error, Internet problems, …).

In the end I created a little bash script that runs regularly and sends pushover notifications (like I used for my other smarthome alerts) to my phone if anything goes wrong.

Building a backup check script

The idea for this script is pretty straight forward:

• check which backup disk is currently connected
• find the latest snapshot on that disk
• check if that snapshot is from today
• otherwise send an alert to my phone

And here is the code I came up with. As always you can also find the up to date version in my GitHub repository.

#!/bin/bash

# Backup-Pools to check the status of
BACKUPPOOLS=("backupPool" "backupPool2" "backupPool3" "backupPool4")

# needed paths
LOGFILE="/var/log/backupCheck.log"
ZFS="/sbin/zfs"

# pushover data
PO_TOKEN=abc123
PO_UK=def456

# -------------- program, don't change ---------------
hasRecentBackups=false

for BACKUPPOOL in ${BACKUPPOOLS[@]}
do
	isOnline=$(/sbin/zpool status $BACKUPPOOL | grep -i 'state: ONLINE' | wc -l)

	if [ $isOnline -ge 1 ]
	then
		echo "$(date) - Found online pool $BACKUPPOOL" >> $LOGFILE
		# find and compare latest snapshot to today
		lastestSnapshotDate=$($ZFS list -t snapshot -o name -s creation -r $BACKUPPOOL | tail -1 | egrep -o '[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}-[[:digit:]]{4}')
		timeAtStartOfTheDay=$(date +"%Y-%m-%d"-0000)
		echo "$(date) - Lastest date from the pool $BACKUPPOOL are from $lastestSnapshotDate and today started at $timeAtStartOfTheDay" >> $LOGFILE

		if [[ "$lastestSnapshotDate" > "$timeAtStartOfTheDay" ]] ;
		then
		    echo "Backups for $BACKUPPOOL are up to date" >> $LOGFILE
		    hasRecentBackups=true
		else
			echo "Backups for $BACKUPPOOL are not up to date. Last one is from $lastestSnapshotDate" >> $LOGFILE
		fi
	fi
done

if [ "$hasRecentBackups" = true ]; then
  	echo "$(date) - Found recent backups. Run finished" >> $LOGFILE
else
	echo "$(date) - Alarm - no recent backups found!!" >> $LOGFILE
	curl -s -F "token=$PO_TOKEN" \
    -F "user=$PO_UK" \
    -F "title=No recent backups!" \
    -F "message=Unable to find recent backups on the server. Last one is from $lastestSnapshotDate" https://api.pushover.net/1/messages.json
fi

All you need to do to use this is to update the variables of the start of the script. Set your list of backup ZFS pools, point the script to the ZFS installation and a log file you want to use and provide a token and user key for pushover (I described how to get those in the smarthome alert article linked above).

Then simply run ./backupCheck.sh and check the logs, or your phone for the alert. I attempted to log every descision including the timestamp to make this easy to debug if needed.

Start watching – add a crontab job

Now the idea of this whole backup process is to automate everything and not have to run this script manually. Therefore I used the same mechanism as for the actual backup to automate this: crontab. After downloading the script just run sudo crontab -e and edit it like this:

# copy backups and clean up
0,30 * * * * bash /usr/local/bin/backup.sh

# check backups daily
0 22 * * * bash /usr/local/bin/backupCheck.sh

The first 2 lines are from the backup article and runs the actual backup process every 30 minutes. The lines below are added for this backup check script. It runs every day at 10pm / 22:00 by just running the script once. The script will then go through the process described above and send a notification if no backup from the same day is found.#

After adding this just save and close the editor (CTRL*O and CTRL+X if you are using nano) and sleep well knowing that you will be notified if the server stops sending backups for some reason.