Looking at auth.log on my server revealed the problem: sshd was
refusing to use my public key because my home directory was
group-writable. In order to do pubkey auth, both the home directory
and the .ssh directory must writable only by the owner.
still, my backup restore fails with a Unable to read 4 bytes error