Remote Secure Incremental Backups

Here is my solution for a remote, incremental backup system. It uses rsnapshot which uses rsync and hard links to create snapshots of a file system. The nice part about using hard links, is you don't end up with multiple copies of files - just hard links to those files. Plus, with hard links, if you delete a specific snapshot, files that appear in other snapshots are not deleted. The hard link count is just decremented for those files, but the file itself remains. Only files where the hard link count moves to 0 when decremented are deleted (files that _only_ appear in that particular snapshot or series of snapshots). I've heard rdiff is more efficient (it uses file diffs instead of hard links) but rsnapshot fits my needs perfectly. YMMV.

For this, I have stolen/borrowed liberally from the rsnapshot howto, the tutorial at http://troy.jdmz.net/rsnapshot/, and this posting on BlogFish.

I'm posting this mostly so I remember how to do it - use this at your own risk. While it limits the risk, we are _still_ using passwordless keys and allowing a limited user to perform tasks as root, so there are security concerns. It is a trade-off that you have to consider.

Here is what I did:

Let's assume you have two servers - one production (production.server.com) and one for storing your incremental backups of that production server (backup.server.com).

ON backup.server.com

First, we'll need rsnapshot installed.

yum install rsnapshot

Now, we'll set up the ssh keys for connecting to the main server.
We'll be doing this as root since I want to secure the passwordless key as much as possible.

cd /root/.ssh
ssh-keygen -t rsa -b 4096 -f /root/.ssh/production.server-key
if [ ! -f config ]; then touch config ; chmod 600 config ; fi
echo Host production.server-rsnapshot >> config
echo Hostname production.server.com >> config
echo IdentityFile /root/.ssh/production.server-key >> config


At this time, I'd probably throw my config file into RCS... but that is a different howto for a later date. In general, any file I'm editing in this howto, I'm putting into RCS.

Better secure the backup server a little:
edit /etc/ssh/sshd_config to have:

PermitRootLogin no

and restart the sshd process

/etc/init.d/sshd restart

Now transfer the public key over to your production server.

scp /root/.ssh/production.server-key.pub root@production.server.com:

Next, we move over to the production.server

ON production.server.com

Again, log in and open a root shell.

Create a new user 'backup' on the production server using your preferred method (gui/cli). You can use apg (yum install apg) to come up with a good strong password as shouldn't need the password of the new account.

apg -m 12 -s -M SNCL

APG will provide you with a few good passwords to use.

Next up, we'll move the public key over that we copied from the backup server.


cd /home/backup
mkdir .ssh
chmod 700 .ssh
mv /root/production.server-key.pub .ssh/
cat .ssh/production.server-key.pub \>\> .ssh/authorized_keys
chmod 400 .ssh/authorized_keys
chown -R backup:backup .ssh/

Before the next step, you may want to do a sanity check to ensure the key is working properly by logging in to the production.server from the backup.server. From root on the backup.server you should be able to log in without a password via 'ssh backup@production.server-rsnapshot' (since that is what you added in your ssh config file).

If the test worked, we'll tighten down the public key on the production.server.

su - backup
chmod 600 ~/.ssh/authorized_keys

You can tighten the access of the key by limiting what the key is allowed to do and where it is allowed to log in from. At the beginning of the key, you'll see 'ssh-rsa'. Before that, add a 'from' entry and a 'command' entry. Edit the key so the key looks like:

from="IP.OF.BACKUP.SERVER.COM",command="/home/backup/validate-rsync" ssh-rsa...

We are validating what commands it is capable of sending in the validate-rsync file. Here is a sample validate-rsync file:


#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac


Make sure the validate-rsync is file is located in the location specified in the pub key (in the example /home/backup/) and chmod 500 it for good measure.

Finally, fix the authorized_keys permissions:

chmod 400 ~/.ssh/authorized_keys


Next up, since we want the 'backup' user to be able to access all of the files on the production server, we'll need to hijack the rsync command for the backup user and call it with a sudo command to allow this access. To do this, we'll have to reset the backup user's path to first look in ~/bin

Edit ~/.bashrc to include the following line:

PATH=$HOME/bin:$PATH

Next, we'll create an rsync file in ~/bin. if ~/bin doesn't exist, create it. Then, create an ~/bin/rsync file that looks like:

#!/bin/bash
/usr/bin/sudo /usr/bin/rsync "$@"


**Remember this is based on Fedora 7 - your paths may be different. Next,

chmod 500 ~/bin/rsync

Now, the final step on the production server is to fix the sudo stuff so the backup user can run only rsync with full privileges.

As root on the production.server.com, run visudo and comment out the requiretty line, and add a line for the backup user in the appropriate section, like so:

#Defaults requiretty
backup ALL=NOPASSWD: /usr/bin/rsync


That allows the backup user to (from an ssh connection) run (only) rsync as root. Definitely still a security risk, but at least we are limiting it.

While we are root on the production.server, remember to change your /etc/ssh/sshd_config to deny root ssh login on this box too..

Finally, on the production.server, if you happen to be running a mysql server, you'll want a daily copy of that also. Edit /root/.my.cnf to include

[client]
user = root
password = my_mysql.password
host = localhost


Then crontab -e and add:

8 0 * * * /usr/bin/mysqldump --all-databases --complete-insert > /mysql.backup

Now your mysql database will be backed up to /mysql.backup nightly at 12:08am.


Finally, back to backup.server...

ON backup.server.com

As root, edit /etc/rsnapshot.conf (you _are_ using RCS.... right??). This is not a complete listing of rsnapshot.conf - just the settings I changed and/or confirmed:

snapshot_root /.snapshots/

cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du

interval daily 7
interval weekly 4
interval monthly 6
interval yearly 1

ssh_args -o BatchMode=yes

exclude_file /root/rsync-exclude

backup backup@production.server-rsnapshot:/ production.server.com/

**NOTE: Remember that tabs must separate all elements, and that there must be a trailing slash on the end of every directory. Spaces are not used to separate elements! This will screw things up!

See 'man rsnapshot' for details on the 'backup' lines. That is where you tell rsnapshot what you would like backed up. In this instance, I am taking the entire / directory. You can have multiple backup lines, listing specific directories. There are examples in the file. The ending 'production.server.com/' entry tells rsnapshot where under snapshot_root (in this case /.snapshots/) to store the backup of this server. The theory is, you may be backing up multiple production servers to the backup server, so each server will have it's own directory under /.snapshots/

Adjust the intervals as you want also - remember that it must go from smallest interval to biggest interval.
Once you've edited rsnapshot.conf, test the config file for errors:

rsnapshot configtest

Errors? Remember... TABS NOT SPACES

Did you notice the 'exclude_file /root/rsync-exclude' line we added in the rsnapshot.conf file? That means, we'll use the file /root/rsync-exclude to list folders that we do _not_ want backed up from the production server. Here is a sample /root/rsync-exclude file:

- /.snapshots/
- /sys/
- /proc/


See 'man rsync' for details on exclude file syntax. In general, I'm not backing up /sys or /proc. The /.snapshots/ entry is in case I happen to be backing anything up to the production server, I don't want to make backups of other backups.

OK... finally, ready to test it all. A couple things you may want to think about first...
You may want to increase the verbosity level in /etc/rsnapshot.conf to give hints at why things are not working as planned.
You probably don't want to back up your entire / directory on a test run. I'd limit it to like... /var/log/ or something.

You can run a test to see what commands will be run:

rsnapshot -t daily

The -t ensures it is just a test. If the test looks ok, then run it and hope for the best:

rsnapshot daily

That will perform the actual backup. Check /.snapshots/ to make sure the backup worked. You should see all your files, etc. You can run it again, that will perform all the normal copying etc. so you end up with a second snapshot. Finally, if you are happy, you can probably remove the test /.snapshots directory so that it will be created during the actual backup. Reset any rsnapshot.conf changes (back to the full / directory if you like, etc), run the final 'rsnapshot daily' command to get the first 'true' backup.

You'll now want crontab -e something like:

28 3 * * * /usr/bin/rsnapshot daily
18 2 * * 0 /usr/bin/rsnapshot weekly
18 1 24 * * /usr/bin/rsnapshot monthly
18 0 22 1 * /usr/bin/rsnapshot yearly


So that you get nice daily/monthly/yearly backup snapshots.