The point of having a backup is to restore a service when it dies catastrophically. Virtualisation helps insulate the service from the underlying hardware, but that's not sufficient: what happens when somebody fat-fingers rm -Rf /tmp/* into rm -Rf / tmp/*?
Having established that the point of backing up is to restore a service, it seems sensible to consider how we restore a service. Commercial backup solutions make much ado about being a 'silver bullet' - restore your machine without installing an OS first. Running an OS without installing one is rather simple - live linux CD or a pxe-booted client can do that. Then we might partition our disks and restore from our backup.
What to backup?
Assuming that we restore by booting into some live Linux distribution, what do we need to backup to restore the functionality of the machine? There's very little point in backing up the contents of the distributed debian packages: these contain files which are already backed up on Debian's web servers. Omitting these files leads to a big reduction in the space required to back up a machine - in effect, we're compressing a package of, say, 200kB down to ~30 characters (its name and version).
The algorithm is quite simple: back up everything which isn't contained in a package. Well, that's not quite true: packages also contain configuration files which get altered - we ought to back them up too.
How to backup?
rsnapshot is a jolly clever backup tool and maintains multiple snapshots of the data at different times. Each snapshot appears to be complete but files which are unchanged use hard-links to appear in multiple places at the same time. This means that one can notice that something broke between, say, Thursday and Monday, and see which files changed by comparing the snapshots.
rsnapshot can take a list of files to ignore, and we generate that list by running a script on splot which SSHs into the node to backup and finds which files are installed from which packages: this script is called prepare.
Yep, that script is quick-and-dirty but it does the job. From time to time we find it necessary to update things - like dumping Postgres data to a file - so adjust if there's something which isn't easily recovered from a dump of files.
To call this script, /etc/rsnapshot.conf might have the following entry:
backup_script /etc/rsnapshot/prepare host.domain.tld rsnapshot/host.domain.tld backup firstname.lastname@example.org:/ host.domain.tld/ exclude_file=/etc/rsnapshot/host.domain.tld/exclude,exclude_file=/etc/rsnapshot/global_ignore
We ignore some files across all hosts as determined by global_ignore.
And to restore?
Using a debian CD (or otherwise!) be running debian. Create any partitions and filesystems required and mount them somewhere (e.g. /mnt. Move the restore script onto the machine from the network and run it to restore the machine from the most recent backup on the backup server (backup-server is the name in the script - you might need to change that!): ./debi-restore.sh hostname.domain.tld /mnt. You will also need fix-uids-and-gids.pl to adjust UIDs and GIDs if these have changed.
As the script says, you will want to configure /etc/fstab and /etc/network/interfaces before trying to boot. The latter might be just copied from /etc/network/interfaces.manual but /etc/fstab is likely to require re-writing - /etc/fstab.manual exists to remind you what the old machine had.