ftplicity problems

[Ovidiu]

I started a backup again, with verbosity=9 and here is the last valid output:

Selecting /var/www/web2/user/web2_info/.quota.rc
Comparing ('var', 'www', 'web2', 'user', 'web2_info', '.quota.rc') and ('var', 'www', 'web2', 'user', 'web2_info', '.quota.rc')
Getting delta of (('var', 'www', 'web2', 'user', 'web2_info', '.quota.rc') /var/www/web2/user/web2_info/.quota.rc reg) and (('var', 'www', 'web2', 'user', 'web2_info', '.quota.rc') reg)
Generating delta - changed file: var/www/web2/user/web2_info/.quota.rc
Selecting /var/www/web2/user/web2_info/.spamassassin
Comparing ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin') and ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin')
Getting delta of (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin') /var/www/web2/user/web2_info/.spamassassin dir) and (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin') dir)
Generating delta - new file: var/www/web2/user/web2_info/.spamassassin
Selecting /var/www/web2/user/web2_info/.spamassassin/auto-whitelist
Comparing ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'auto-whitelist') and ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'auto-whitelist')
Getting delta of (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'auto-whitelist') /var/www/web2/user/web2_info/.spamassassin/auto-whitelist reg) and (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'auto-whitelist') reg)
Generating delta - changed file: var/www/web2/user/web2_info/.spamassassin/auto-whitelist
Selecting /var/www/web2/user/web2_info/.spamassassin/bayes_seen
Comparing ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_seen') and ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_seen')
Getting delta of (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_seen') /var/www/web2/user/web2_info/.spamassassin/bayes_seen reg) and (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_seen') reg)
Generating delta - changed file: var/www/web2/user/web2_info/.spamassassin/bayes_seen
Selecting /var/www/web2/user/web2_info/.spamassassin/bayes_toks
Comparing ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_toks') and ('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_toks')
Getting delta of (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_toks') /var/www/web2/user/web2_info/.spamassassin/bayes_toks reg) and (('var', 'www', 'web2', 'user', 'web2_info', '.spamassassin', 'bayes_toks') reg)
Generating delta - changed file: var/www/web2/user/web2_info/.spamassassin/bayes_toks
Saving duplicity-inc.2007-01-25T04:00:35+02:00.to.2007-02-24T11:40:57+02:00.vol42.difftar.gpg on FTP server
Traceback (most recent call last):
File "/usr/bin/duplicity", line 373, in ?
if __name__ == "__main__": main()
File "/usr/bin/duplicity", line 367, in main
else: incremental_backup(sig_chain)
File "/usr/bin/duplicity", line 170, in incremental_backup
bytes_written = write_multivol("inc", tarblock_iter, globals.backend)
File "/usr/bin/duplicity", line 89, in write_multivol
backend.put(tdp, dest_filename)
File "/usr/lib/python2.3/site-packages/duplicity/backends.py", line 349, in put
self.error_wrap('storbinary', "STOR "+remote_filename, source_file)
File "/usr/lib/python2.3/site-packages/duplicity/backends.py", line 335, in error_wrap
except ftplib.all_errors, e: raise BackendException(e)
duplicity.backends.BackendException
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe

thats how it ends - any ideas or suggestions?

[Ovidiu]

ok guys,

any hints for me? where to look for help. I would pay to solve this as I need my backup system running again.

Is it a ftplicity problem? Or a duplicity or gnupg or ncftp problem? Or is the problem with the ftp server? Or is a certain file to blame?

I have no clue, I jsut know the backup process stops like yo ucan see in the above logfile....

[falko]

I have no idea (otherwise I'd have replied...).

[Ovidiu]

THX.

This will mark this as a dead end :-(

[Ovidiu]

another hint from the duplicity page:

Currently duplicity supports deleted files, full unix permissions, directories, symbolic links, fifos, etc., but not hard links.

could this be a problem? I have not done anything with hardlinks so far, I am going to read on the net about them... find out how I can spot them and see if there are any inside the dirs I backup.

[martinfst]

Hard links are identified by having the same inode number.

ls -i

will show files with their i-node number. I don't recall a single command to find hardlinked files. I would write a some script to traverse the directory tree, sort it on i node and see if there are matches. Hardlinks only exist per filesystem, so that limits where to search. HTH.

[Ovidiu]

the hardlinks don't seem to be the problem, thx for the suggestion anyway.. :-(

[Ovidiu]

it seems there are news on this topic:

I found another of my threads here: http://www.howtoforge.com/forums/sho...ight=ftplicity

also a nice thread with a patch: http://serversupportforum.de/forum/p...tml#post104368
the last poster is also called falko, might be this falko?

also some other modified duplicities and ftplicity scripts:
http://www.weareroot.de/2007/08/17/f...icher-ablegen/
http://www.weareroot.de/2007/12/02/d...-046-released/


I am still testing all this new input, just posted here for other users.

[falko]

I've also written a tutorial about ftplicity: http://www.howtoforge.com/ftp-backup...ty-debian-etch

[Ovidiu]

yes I have found your tutorial, still, my problems changed from the "broken Pipe problem" which was solved by a patch in one of the threads I linked to here, to this one:

/usr/local/bin# ftplicity full
NcFTP version is 3.2.0
Reading globbing filelist /etc/ftplicity/exclude.conf
Last full backup date: Thu Jan 1 01:00:00 1970
Traceback (most recent call last):
File "/usr/bin/duplicity", line 421, in ?
if __name__ == "__main__": main()
File "/usr/bin/duplicity", line 410, in main
if action == "full": full_backup(col_stats)
File "/usr/bin/duplicity", line 150, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/bin/duplicity", line 83, in write_multivol
globals.gpg_profile,globals.volsize)
File "/usr/lib/python2.4/site-packages/duplicity/gpg.py", line 198, in GPGWriteFile
try: data = block_iter.next(bytes_to_go).data
File "/usr/lib/python2.4/site-packages/duplicity/diffdir.py", line 407, in next
result = self.process(self.input_iter.next(), size)
File "/usr/lib/python2.4/site-packages/duplicity/diffdir.py", line 487, in process
data, last_block = self.get_data_block(fp, size - 512)
File "/usr/lib/python2.4/site-packages/duplicity/diffdir.py", line 508, in get_data_block
buf = fp.read(read_size)
File "/usr/lib/python2.4/site-packages/duplicity/diffdir.py", line 338, in read
buf = self.infile.read(length)
IOError: [Errno 22] Invalid argument

FYI I have manually compiled duplicity version 0.4.6 as the current version in debian stable seems to be 0.4.2 if I am not mistaken. The links I listed showed problems solved in versions > 0.4.4 and recommend using those.

Do you have any clue about the current errors I posted in this postß