Small blog about system administration.

Ubuntu/Debian - Encrypted incremental backups with duplicity on Amazon S3

An example on how to use duplicity to perform encrypted incremental backups on Amazon S3.

Getting started

If you've never heard about duplicity before, you should check the documentation.

Install duplicity

First, you need to install duplicity, I always install it from source since the duplicity package is not often updated.
$ sudo apt-get install python-dev librsync-dev
$ cd /opt
$ sudo wget https://code.launchpad.net/duplicity/0.6-series/0.6.20/+download/duplicity-0.6.20.tar.gz
$ sudo tar xvzf duplicity-0.6.20.tar.gz
$ cd duplicity-0.6.20
$ python sudo setup.py install
But you can install it with apt-get
$ sudo apt-get install duplicity
Next you can also install s3cmd from S3 Tools, it's a command line tool for managing your S3 buckets, but it's not required.
$ sudo apt-get install s3cmd
$ s3cmd --configure

Encrypted Backups

Before backing up the data, you need to think about encryption, duplicity makes use of gpg and handles both private/public key pair (a gpg key) and symmetric encryption (a passphrase).
I use passsphrases since I'll never lose it and I don't have to backup a gpg key.

My backup script

Since you need to specify many args to perform the differents actions, I crafted a bash script that make working with duplicity easier, duptools.

Features

  • Backup multiple directories
  • Send email report on backup
  • Quickly list file and show bucket status
  • Restore your files easily

Duptools

#!/bin/bash
export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export PASSPHRASE=YOU_PASSHRASE

# directories, space separated
SOURCE="/home/yegorg/backup /home/yegorg/bin /home/yegorg/documents"
BUCKET=s3+http://mybucket
LOGFILE=/home/yegorg/tmp/duplicity.log
# set email to receive a backup report
EMAIL=""

backup() {
  INCLUDE=""
  for CDIR in $SOURCE
  do
    TMP=" --include  ${CDIR}"
    INCLUDE=${INCLUDE}${TMP}
  done
  # perform an incremental backup to root, include directories, exclude everything else, / as reference.
  duplicity --full-if-older-than 30D $INCLUDE --exclude '**' / $BUCKET > $LOGFILE
  if [ -n "$EMAIL" ]; then
    mail -s "backup report" $EMAIL < $LOGFILE
  fi
}

list() {
  duplicity list-current-files $BUCKET
}

restore() {
  if [ $# = 2 ]; then
    duplicity restore --file-to-restore $1 $BUCKET $2
  else
    duplicity restore --file-to-restore $1 --time $2 $BUCKET $3
  fi
}

status() {
  duplicity collection-status $BUCKET
}

if [ "$1" = "backup" ]; then
  backup
elif [ "$1" = "list" ]; then
  list
elif [ "$1" = "restore" ]; then
  if [ $# = 3 ]; then
    restore $2 $3
  else
    restore $2 $3 $4
  fi
elif [ "$1" = "status" ]; then
  status
else
  echo "
  duptools - manage duplicity backup

  USAGE:

  ./duptools.sh backup 
  ./duptools.sh list
  ./duptools.sh status
  ./duptools.sh restore file [time] dest
  "
fi

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=

Installation

Set up config vars at the top of the script and make the script executable.

Backup

$ ./duptools.sh backup

List/Status

$ ./duptools.sh list
$ ./duptools.sh status

Restore

Be careful while restoring not to preprend a slash to the path.
Restoring a single file to tmp
$ ./duptools.sh restore home/yegorg/bin/setupscreen tmp/setupscreen
Restoring an older version of a directory to tmp (interval or full date)
$ ./duptools.sh  restore home/yegorg/bin 1D3h5s tmp/bin
$ ./duptools.sh  restore home/yegorg/bin 2012/7/5 tmp/bin

No comments :

Post a Comment