Creating Encrypted FTP Backups With duplicity And duply On Debian Squeeze - Page 2

Want to support HowtoForge? Become a subscriber!
 
Submitted by falko (Contact Author) (Forums) on Thu, 2012-07-05 18:07. ::

We can now create three other files, although that is totally optional:

  • /root/.duply/exampleuser/exclude: contains a list of directories to be excluded from the backup (one directory per line);
  • /root/.duply/exampleuser/pre: contains command(s) to be executed prior to the backup (e.g. create a MySQL database dump);
  • /root/.duply/exampleuser/post: contains command(s) to be executed after the backup.

Here's a sample /root/.duply/exampleuser/exclude and /root/.duply/exampleuser/pre file (the syntax of /root/.duply/exampleuser/post is the same as in /root/.duply/exampleuser/pre):

vi /root/.duply/exampleuser/exclude

/home/exampleuser/news
/home/exampleuser/messages
/home/exampleuser/tmp

chmod 600 /root/.duply/exampleuser/exclude

vi /root/.duply/exampleuser/pre

/usr/bin/mysqldump --all-databases -u root -pyourrootsqlpassword > /home/exampleuser/db.sql

(It should be noted that the database gets locked during the creation of SQL dump; this might not be an issue for a small web site, but can be a problem for high-traffic web sites - your visitors won't be able to access database-driven pages during mysqldump. Here's a link to an interruption-free MySQL backup method: How To Back Up MySQL Databases Without Interrupting MySQL)

/root/.duply/exampleuser/pre must be executable (as well as /root/.duply/exampleuser/post if you choose to create one):

chmod 700 /root/.duply/exampleuser/pre

As I said before, you only need /root/.duply/exampleuser/conf; the other files are optional.

Now duply is ready to be used; to create our backup, we simply run

duply exampleuser backup

If all goes well, you won't be prompted for a password:

root@server1:~# duply exampleuser backup
Start duply v1.5.7, time is 2012-07-03 21:02:33.
Using profile '/root/.duply/exampleuser'.
Using installed duplicity version 0.6.08b, python 2.6.6, gpg 1.4.10 (Home: ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan', bash '4.1.5(1)-release (x86_64-pc-linux-gnu)'.
Autoset found secret key of first GPG_KEY entry '7C6E958B' for signing.
Test - Encrypt to 7C6E958B & Sign with 7C6E958B (OK)
Test - Decrypt (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.4161.1341342154_*'(OK)

--- Start running command PRE at 21:02:34.459 ---
Running '/root/.duply/exampleuser/pre' - OK
--- Finished state OK at 21:02:34.535 - Runtime 00:00:00.075 ---

--- Start running command BKP at 21:02:34.599 ---
NcFTP version is 3.2.4
Reading globbing filelist /root/.duply/exampleuser/exclude
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Jul 3 20:47:50 2012
--------------[ Backup Statistics ]--------------
StartTime 1341342156.07 (Tue Jul 3 21:02:36 2012)
EndTime 1341342156.13 (Tue Jul 3 21:02:36 2012)
ElapsedTime 0.06 (0.06 seconds)
SourceFiles 50
SourceFileSize 13490043 (12.9 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 716 (716 bytes)
Errors 0
-------------------------------------------------

--- Finished state OK at 21:02:37.483 - Runtime 00:00:02.884 ---

--- Start running command POST at 21:02:37.556 ---
Skipping n/a script '/root/.duply/exampleuser/post'.
--- Finished state OK at 21:02:37.636 - Runtime 00:00:00.080 ---
root@server1:~#

To automate the backups, we can create a cron job (I create two cron jobs, one that runs daily and creates incremental backups, and one that runs once a month, creates a full backup and deletes old files):

crontab -e

# run the (incremental) backup each night at 03:23h
23 3 * * * /usr/local/bin/duply exampleuser backup

# do a full backup once per month & delete old backups
47 4 1 * * /usr/local/bin/duply exampleuser full && /usr/local/bin/duply exampleuser purge --force

 

6 Restoring A Backup With duply

Of course, you can use duply also to restore a backup (again, you won't be asked for a password). The syntax is very easy:

duply exampleuser restore /home/exampleuser

You can take a look at the duply help to learn what other options you have:

duply -h

root@server1:~# duply -h
VERSION:
  duply version 1.5.7
  (http://duply.net)

DESCRIPTION:
  Duply deals as a wrapper for the mighty duplicity magic.
  It simplifies running duplicity with cron or on command line by:

    - keeping recurring settings in profiles per backup job
    - enabling batch operations eg. backup_verify_purge
    - executing pre/post scripts for every command
    - precondition checking for flawless duplicity operation

  For each backup job one configuration profile must be created.
  The profile folder will be stored under '~/.duply/<profile>'
  (where ~ is the current users home directory).
  Hint:
   If the folder '/etc/duply' exists, the profiles for the super
   user root will be searched & created there.

USAGE:
  first time usage (profile creation):
    duply <profile> create

  general usage in single or batch mode (see EXAMPLES):
    duply <profile> <command>[_<command>_...] [<options> ...]

  Non duply options are passed on to duplicity (see OPTIONS).
  All conf parameters can also be defined in the environment instead.

PROFILE:
  Indicated by a path or a profile name (<profile>), which is resolved
  to '~/.duply/<profile>' (~ expands to environment variable $HOME).

  Superuser root can place profiles under '/etc/duply'. Simply create
  the folder manually before running duply as superuser.
  Note:
    Already existing profiles in root's profile folder will cease to work
    unless there are moved to the new location manually.

  example 1:   duply humbug backup

  Alternatively a _path_ might be used e.g. useful for quick testing,
  restoring or exotic locations. Shell expansion should work as usual.
  Hint:
    The path must contain at least one path separator '/',
    e.g. './test' instead of only 'test'.

  example 2:   duply ~/.duply/humbug backup

COMMANDS:
  usage      get usage help text

  create     creates a configuration profile
  backup     backup with pre/post script execution (batch: pre_bkp_post),
              full (if full_if_older matches or no earlier backup is found)
              incremental (in all other cases)
  pre/post   execute '<profile>/pre', '<profile>/post' scripts
  bkp        as above but without executing pre/post scripts
  full       force full backup
  incr       force incremental backup
  list [<age>]
             list all files in backup (as it was at <age>, default: now)
  status     prints backup sets and chains currently in repository
  verify     list files changed since latest backup
  restore <target_path> [<age>]
             restore the complete backup to <target_path> [as it was at <age>]
  fetch <src_path> <target_path> [<age>]
             fetch single file/folder from backup [as it was at <age>]
  purge [<max_age>] [--force]
             list outdated backup files (older than $MAX_AGE)
              [use --force to actually delete these files]
  purge-full [<max_full_backups>] [--force]
             list outdated backup files ($MAX_FULL_BACKUPS being the number of
             full backups and associated incrementals to keep, counting in
             reverse chronological order)
              [use --force to actually delete these files]
  cleanup [--force]
             list broken backup chain files archives (e.g. after unfinished run)
              [use --force to actually delete these files]

  changelog  print changelog / todo list
  txt2man    feature for package maintainers - create a manpage based on the
             usage output. download txt2man from http://mvertes.free.fr/, put
             it in the PATH and run 'duply txt2man' to create a man page.

OPTIONS:
  --force    passed to duplicity (see commands: purge, purge-full, cleanup)
  --preview  do nothing but print out generated duplicity command lines
  --disable-encryption
             disable encryption, overrides profile settings

PRE/POST SCRIPTS:
  All internal duply variables will be readable in the scripts.
  Some of interest might be

    CONFDIR, SOURCE, TARGET_URL_<PROT|HOSTPATH|USER|PASS>,
    GPG_<KEYS_ENC|KEY_SIGN|PW>, CMD_<PREV|NEXT>

  The CMD_* variables were introduced to allow different actions according to
  the command the scripts were attached to e.g. 'pre_bkp_post_pre_verify_post'
  will call the pre script two times, with CMD_NEXT variable set to 'bkp'
  on the first and to 'verify' on the second run.

EXAMPLES:
  create profile 'humbug':
    duply humbug create (now edit the resulting conf file)
  backup 'humbug' now:
    duply humbug backup
  list available backup sets of profile 'humbug':
    duply humbug status
  list and delete obsolete backup archives of 'humbug':
    duply humbug purge --force
  restore latest backup of 'humbug' to /mnt/restore:
    duply humbug restore /mnt/restore
  restore /etc/passwd of 'humbug' from 4 days ago to /root/pw:
    duply humbug fetch etc/passwd /root/pw 4D
    (see "duplicity manpage", section TIME FORMATS)
  a one line batch job on 'humbug' for cron execution:
    duply humbug backup_verify_purge --force

FILES:
  in profile folder '~/.duply/<profile>' or '/etc/duply'
  conf             profile configuration file
  pre,post         pre/post scripts (see above for details)
  gpgkey.*.asc     exported GPG key files
  exclude          a globbing list of included or excluded files/folders
                   (see "duplicity manpage", section FILE SELECTION)

IMPORTANT:
  Copy the _whole_ profile folder after the first backup to a safe place.
  It contains everything needed to restore your backups. You will need
  it if you have to restore the backup from another system (e.g. after a
  system crash). Keep access to these files restricted as they contain
  _all_ informations (gpg data, ftp data) to access and modify your backups.

  Repeat this step after _all_ configuration changes. Some configuration
  options are crucial for restoration.

SEE ALSO:
  duplicity man page:
    duplicity(1) or http://duplicity.nongnu.org/duplicity.1.html
root@server1:~#

 

7 Links


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Bill (not registered) on Fri, 2014-09-19 21:58.

pre
 
#!/bin/sh
DATE=`date +%d%b%Y`
DATABASE=ads
ARCHIVE_DIR=/home/pg-bkp/archives
SQL_BACKUP_FILE="$ARCHIVE_DIR/$DATABASE-db-$DATE.gz"
/usr/bin/pg_dump $DATABASE | gzip -c > $SQL_BACKUP_FILE

 post

 #!/bin/sh
DATE=`date +%d%b%Y`
DATABASE=ads
ARCHIVE_DIR=/home/pg-bkp/archives
SQL_BACKUP_FILE="$ARCHIVE_DIR/$DATABASE-mysqlbackup-$DATE.gz"
/bin/rm $SQL_BACKUP_FILE