#rsnapshot

[michaelkueper]

Sicherung meines Docker Servers

Sicherungskonzept

Der Datenbereich des Servers wird von einem cron job zyklisch gesichert. Um die Verbindung zum Sicherungs-NAS herzustellen, wird beim Start der Sicherung das Netzlaufwerk mit autofs eingehängt und 5 Minuten nach Ende der Sicherung wieder ausgehängt.

Die Sicherung wird mit rsnapshot durchgeführt. Es werden nur die relevanten Verzeichnisse gesichert. Das ist zum einen das Verzeichnis etc und zum anderen das Verzeichnis, in dem die Docker Container (hier /docker) ihre Daten ablegen.

Notwendige Software

rsync

rsnapshot

autofs

anacron oder einen anderer cron

Sicherungs Zyklus

Die Sicherungen laufen cron gesteuer (Beispieleinträge):

täglich

0 22 * * * /usr/bin/rsnapshot daily

wöchentlich

0 23 * * 1 /usr/bin/rsnapshot weekly

monatlich

59 23 1 * * /usr/bin/rsnapshot monthly

Beispielkonfiguration für autofs

Die Konfiguration besteht im Wesentlichen aus den Dateien *auto.master und auto.backup.

Beispiel für die Konfigurationsdatei auto.master

Vorasusetzungen:

Das Verzeichnis /Backup existiert im root dateisystem

Durch diesen Eintrag wird festgelegt, dass die, in der Datei auto.backup definierten, externen Laufwerke unterhalb des Verzeichnis /Backup eingebunden werden.

Durch den Parameter --ghost werden die einzubindenden Verzeichnisse beim Start von autofs automatisch unterhalb von /Backup angelegt und binden beim Zugriff auf diese Ghost Freigabe das externe Laufwerke ein.

/Backup /etc/auto.backup --ghost +dir:/etc/auto.master.d +auto.master

Der hier verwendete Name auto.backup ist nicht vorgegeben und kann frei gewählt werden.

Beispiel für die Konfigurationsdateiauto.backup

In der Datei auto.backup werden die einzubindenden Laufwerke mit Dateisystem, Zugriffsberechtigungen und Mountpoint (unterhalb von /Backup) beschrieben.

Backup -fstype=nfs,rw,soft,tcp,nolock xxx.xxx.xxx.xxx:/Backups

Nach dem Starten von autofs ist dann die Verzeichnisstruktur /Backup/Backup vorhanden. Beim Betreten des Verzeichnisses /Backup/Backup wird diese Freigabe eingebunden und 5 Minuten (default) nach dem Verlassen wieder ausgehängt.

Eine Beschreibung zu autofs kann hier gefunden werden.

Beispielkonfiguration für rsnapshot

Die folgende Konfiguration wird in die Datei rsnapshot.conf eingetragen.

Für das Beispiel gelten folgende Annahmen:

Das Verzeichis /Backup enthält die durch auto.backup eingebunden Verzeichnisse

Auf dem unter /Backup/backup eingebundenen Laufwerk befindet sich das Verzeichnis, hier im Beispiel, dockerhost

Inhalt der Datei rsnapshot.conf

config_version 1.2 snapshot_root /Backup/Backup/dockerhost/ no_create_root 1 cmd_cp /bin/cp cmd_rm /bin/rm cmd_rsync /usr/bin/rsync cmd_logger /usr/bin/logger retain daily 7 retain weekly 5 retain monthly 3 verbose 2 loglevel 3 lockfile /var/run/rsnapshot.pid exclude lost+found/ exclude var/ link_dest 1 ############################### ### BACKUP POINTS / SCRIPTS ### ############################### # LOCALHOST backup /etc/ ./ backup /docker/ ./

Eine Erklärung zu rsnapshot ist hier verfügbar

Hinweise

Kann das Verzeichnis während des Sichewrungstarts nicht gefunden werden weil z.B. das NAS down ist wird die Sicherung abgebrochen, da rsnapshot das notwendige root Verzeichnis nicht anlegen darf.

[techsakh-blog]

Rsnapshot - Rsync based utility to take backup in Linux and Unix

Rsnapshot - Rsync based utility to take backup - IntroductionRsnapshot is an rsync based untility

[sevennet-blog]

Fix: What Folders Should I Not Backup on CentOS? #fix #programming #answer

Fix: What Folders Should I Not Backup on CentOS? #fix #programming #answer

What Folders Should I Not Backup on CentOS?

I’m using rsnapshot to start backing up a CentOS 5.5 install and I need a list of folders that I should probably exclude from the backups. The server is primarily a LAMP web server, and will be serving at the time of the backup, though should be relatively low volume. Is backing up /var/lib/mysql a bad idea?

I’m assuming I should not worry about…

View On WordPress

[goodanswerbiz]

How-to: Doing an rm -rf on a massive directory tree takes hours #fix #dev #it

How-to: Doing an rm -rf on a massive directory tree takes hours #fix #dev #it

Doing an rm -rf on a massive directory tree takes hours

We’re using rsnapshot for backups. It keeps lots of snapshots of the backuped up file, but it does delete old ones. This is good. However it’s taking about 7 hours to do a rm -rf on a massive directory tree. The filesystem is XFS. I’m not sure how many files are there, but it numbers in the millions probably.

Is there anyway to speed it up?…

View On WordPress

[sevennet-blog]

How to: Doing an rm -rf on a massive directory tree takes hours #dev #answer #computers

How to: Doing an rm -rf on a massive directory tree takes hours #dev #answer #computers

Doing an rm -rf on a massive directory tree takes hours

We’re using rsnapshot for backups. It keeps lots of snapshots of the backuped up file, but it does delete old ones. This is good. However it’s taking about 7 hours to do a rm -rf on a massive directory tree. The filesystem is XFS. I’m not sure how many files are there, but it numbers in the millions probably.

Is there anyway to speed it up?…

View On WordPress

[amonkeysden-blog]

Sending rsnapshot reports to InfluxDB

https://github.com/thompsonson/scripts/blob/master/rsnapshot-influxdb.pl

Some simple config info https://github.com/thompsonson/scripts/blob/master/README.md

[adimahajan]

Automatic backups using rsnapshots

More than once I have hit myself on the foot by running

git --init

on an existing git repository and loosing my local changes. So, it was about time that I started using a backup system. I tried rsnapshot and it works like a charm. I am currently using the default configuration and syncing everything on my local machine (not a good idea for backup). Check out the rsnapshot page for more details

[viraptor-blog]

Constructing a (linux) home backup system

I've got a couple of Linux machines that need a transparent backup solution at home. Transparent as in - they should happen all the time, without asking, without notification, without any interaction. Ideally it shouldn't run on the client either just to avoid issues with system updates. Making it all server-side allows me also to backup my girlfriend's machine without explaining the setup - which is great when even periodic upgrades are a pain to enforce.

Another couple of requirements I had were:

backing up in local network (I don't have enough bandwidth to backup remotely all the time)

saving storage space where files don't change

able to run on a small device (I've got a TonidoPlug ready)

directory-based rather than fs-based (I've got a number of VMs I really don't want to waste time backing up)

Unfortunately there aren't many good, ready solutions and there aren't many choices in general either. Most solutions fall into one of the following groups: trivial copy, archiving solutions, rsync-like and full-blown systems.

To be honest, maybe a big system like bacula can do what I need... but with it having more components that I have local hosts, I didn't want to invest that much time in learning the configuration.

At small companies where I set up backups, I was happy with DAR. Full backups split into volumes of a given size and stored on removable disks weekly, with daily diffs stored locally - perfect. Here it won't work - I can't saturate the network or disk io every couple of hours. I really don't want to deal with replacing removable disks. Some more intelligent solution is required.

Enter rsnapshot

Rsnapshot is an interesting mix between systems that understand backup schedules and a minimal backup utility. Rsnapshot can copy files from a remote host using rsync (and ssh for example). Then, it knows how to hard-link files that didn't change to avoid wasting both bandwidth and storage. It also knows how to rotate backup directories when new backups come in and how to promote backups between levels.

The schedule and promotion is done in a very simple way too. Since rsnapshot needs to be run from some scheduler like cron, it takes the name of the backup as a parameter. This can be for example "daily", "weekly", "purple", or whatever you want to call it. What's important is the order in rsnapshot config file - only the first entry will get copied from source, others will get promoted from the "upper" level locally.

This is pretty much exactly what was needed. For the remote copy, ssh can be configured to use a password-less key to authenticate on the remote hosts. In case it's possible to access the backup server from the internet, making sure it can run only rsync is a good idea too. For paranoid people, this can be also locked down with apparmor or selinux to deny anything apart from read-only access.

Sample configuration

So what does this look like in practice?

The server which collects the backups has rsnapshot and rsync installed. It also has a password-less ssh key created: just run ssh-keygen and copy the resulting id_rsa.pub into client machine's ~/.ssh/authorized_keys.

It also has a separate partition available for each backup. In my case they're mounted under /media/disk1partX.

Now, each machine which needs backing up requires a configuration file. I put them all in my plug's root directory: /root/rsnapshot_configs/xxx. Each machine looks almost the same. For example host walker has /root/rsnapshot_configs/walker:

config_version    1.2 snapshot_root    /media/disk1part2/rsnapshot cmd_rsync    /usr/bin/rsync cmd_ssh    /usr/bin/ssh cmd_du    /usr/bin/du exclude_file    /root/rsnapshot_configs/walker.exclude retain    hourly    5 retain    daily    7 retain    weekly    5 backup    [email protected]:/home/viraptor    viraptor

This configuration says that /media/disk1part2/rsnapshot/viraptor will keep copies of /home/viraptor (minus some excludes like .cache, .steam, etc.) - 5 hourly ones, then 7 daily ones, then 5 weekly ones. All files which have not changed will be hard-linked and no space will be wasted on exact duplicates. The backup is taken from the host walker.local (thanks Avahi!).

But that's only the definition of what... what about when? Crontab entries are what actually triggers the rsnapshot actions and they live in /etc/cron.d. The content is:

55 */2 * * * root (/bin/ping -c1 walker.local >/dev/null 2>/dev/null) && /usr/bin/rsnapshot -c /root/rsnapshot_configs/walker hourly 20 3 * * * root /usr/bin/rsnapshot -c /root/rsnapshot_configs/walker daily 20 4 * * 1 root /usr/bin/rsnapshot -c /root/rsnapshot_configs/walker weekly

This means that every 2 hours if the host is on (it most likely isn't), rsnapshot will take another hourly copy. Then every day in the morning it will promote an old hourly snapshot to daily and once a week it will promote daily to weekly.

Summary

All simple and easy. There's no specific recovery procedure involved, because all backups are available as normal files on the server. It's actually easy to browse the old entries in case only one or two files need recovering. And in case of serious issues - just rsync everything back from the last snapshot. So far I'm very happy with it.

One thing to watch out for is the exclusions file in case you don't have a massive harddrive available. If you happen to install Steam with some games you may start getting emails about failed backups... but there's only so many copies of portal or tf2 that you'll need in your life - you probably don't need them in the backups unless you have lots of space. VM images which change all the time are also very bad candidates for backing up with rsnapshot. Every modification there will need a separate, complete copy.

With a "normal" developer-style usage the size of old copies is close to none. Currently, my last copy is 22GB, but all previous snapshots and modifications add only 2GB to that.