Rollback To A Working State With btrfs + apt-btrfs-snapshot On Ubuntu 12.10

This tutorial explains how you can revert failed apt operations (like apt-get upgrade) and roll back to the previous system state with apt-btrfs-snapshot on an Ubuntu 12.10 system that uses the btrfs file system. apt-btrfs-snapshot creates a snapshot of the system before the apt operation. Being able to easily restore the previous system state after a failed apt operation takes away much of the pain system administrators have to deal with normally and is one of the greatest features of the btrfs file system.

I do not issue any guarantee that this will work for you!

1 Preliminary Note

In this tutorial I have installed the whole system on a btrfs file system, i.e., there's no separate /boot partition on an ext file system. If you use a separate /boot partition and apt installs anything in that partition (like a new kernel), you cannot undo changes to the /boot partition with apt-btrfs-snapshot- only changes on the btrfs partition can be reverted.

My hard drive is named /dev/sda in this tutorial, my system partition is /dev/sda1.

A note for Ubuntu users:

Because we must run all the steps from this tutorial with root privileges, we can either prepend all commands in this tutorial with the string sudo, or we become root right now by typing

sudo su

2 Install apt-btrfs-snapshot

apt-btrfs-snapshot can be installed as follows:

apt-get install apt-btrfs-snapshot

To check if apt-btrfs-snapshot is able to create snapshots on apt operations, run

apt-btrfs-snapshot supported

It should display:

[email protected]:~# apt-btrfs-snapshot supported
Supported
[email protected]:~#

If it doesn't, your btrfs subvolume layout probably differs from Ubuntu's default layout which is as follows:

@ subvolume: mounted to /.
@home subvolume: mounted to /home.

This is the default Ubuntu subvolume layout:

btrfs subvolume list /

[email protected]:~# btrfs subvolume list /
ID 256 top level 5 path @
ID 258 top level 5 path @home
[email protected]:~#

If apt-btrfs-snapshot supports your system, you can proceed to chapter 3.

3 Do An apt Operation

Now let's do some apt operation like apt-get upgrade to test if we can rollback to the previous state.

Update your package database...

apt-get update

... and upgrade your system:

apt-get upgrade

[email protected]:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
  linux-headers-generic linux-image-generic
The following packages will be upgraded:
  apport base-files isc-dhcp-client isc-dhcp-common libwhoopsie0 linux-generic lsb-base lsb-release python3-apport python3-distupgrade python3-problem-report python3.2 python3.2-minimal
  ubuntu-release-upgrader-core vim vim-common vim-runtime vim-tiny whoopsie
19 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 14.4 MB of archives.
After this operation, 3,072 B of additional disk space will be used.
Do you want to continue [Y/n]? <-- ENTER
Get:1 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main base-files amd64 6.5ubuntu12 [69.6 kB]
Get:2 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main whoopsie amd64 0.2.7 [25.1 kB]
Get:3 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main libwhoopsie0 amd64 0.2.7 [7,054 B]
Get:4 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main lsb-base all 4.0-0ubuntu26.1 [10.3 kB]
Get:5 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main isc-dhcp-client amd64 4.2.4-1ubuntu10.1 [775 kB]
Get:6 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main isc-dhcp-common amd64 4.2.4-1ubuntu10.1 [836 kB]
Get:7 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main lsb-release all 4.0-0ubuntu26.1 [10.7 kB]
Get:8 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main python3.2 amd64 3.2.3-6ubuntu3.1 [2,585 kB]
Get:9 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main python3.2-minimal amd64 3.2.3-6ubuntu3.1 [1,798 kB]
Get:10 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main vim amd64 2:7.3.547-4ubuntu1.1 [1,051 kB]
Get:11 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main vim-tiny amd64 2:7.3.547-4ubuntu1.1 [413 kB]
Get:12 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main vim-runtime all 2:7.3.547-4ubuntu1.1 [6,317 kB]
Get:13 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main vim-common amd64 2:7.3.547-4ubuntu1.1 [85.7 kB]
Get:14 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main ubuntu-release-upgrader-core all 1:0.190.4 [27.7 kB]
Get:15 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main python3-distupgrade all 1:0.190.4 [141 kB]
Get:16 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main python3-problem-report all 2.6.1-0ubuntu6 [9,578 B]
Get:17 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main python3-apport all 2.6.1-0ubuntu6 [85.7 kB]
Get:18 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main apport all 2.6.1-0ubuntu6 [164 kB]
Get:19 http://de.archive.ubuntu.com/ubuntu/ quantal-updates/main linux-generic amd64 3.5.0.18.21 [1,714 B]
Fetched 14.4 MB in 2s (5,465 kB/s)
Supported
Create a snapshot of '/tmp/apt-btrfs-snapshot-mp-jnW7I_/@' in '/tmp/apt-btrfs-snapshot-mp-jnW7I_/@apt-snapshot-2012-11-22_11:50:38'
(Reading database ... 52666 files and directories currently installed.)
Preparing to replace base-files 6.5ubuntu11 (using .../base-files_6.5ubuntu12_amd64.deb) ...
Unpacking replacement base-files ...
Processing triggers for man-db ...
Processing triggers for install-info ...
Processing triggers for plymouth-theme-ubuntu-text ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.5.0-17-generic
Setting up base-files (6.5ubuntu12) ...
(Reading database ... 52666 files and directories currently installed.)
Preparing to replace whoopsie 0.2.5 (using .../whoopsie_0.2.7_amd64.deb) ...
whoopsie stop/waiting
Unpacking replacement whoopsie ...
Preparing to replace libwhoopsie0 0.2.5 (using .../libwhoopsie0_0.2.7_amd64.deb) ...
Unpacking replacement libwhoopsie0 ...
Preparing to replace lsb-base 4.0-0ubuntu26 (using .../lsb-base_4.0-0ubuntu26.1_all.deb) ...
Unpacking replacement lsb-base ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Setting up lsb-base (4.0-0ubuntu26.1) ...
(Reading database ... 52666 files and directories currently installed.)
Preparing to replace isc-dhcp-client 4.2.4-1ubuntu10 (using .../isc-dhcp-client_4.2.4-1ubuntu10.1_amd64.deb) ...
Unpacking replacement isc-dhcp-client ...
Preparing to replace isc-dhcp-common 4.2.4-1ubuntu10 (using .../isc-dhcp-common_4.2.4-1ubuntu10.1_amd64.deb) ...
Unpacking replacement isc-dhcp-common ...
Preparing to replace lsb-release 4.0-0ubuntu26 (using .../lsb-release_4.0-0ubuntu26.1_all.deb) ...
Unpacking replacement lsb-release ...
Preparing to replace python3.2 3.2.3-6ubuntu3 (using .../python3.2_3.2.3-6ubuntu3.1_amd64.deb) ...
Unpacking replacement python3.2 ...
Preparing to replace python3.2-minimal 3.2.3-6ubuntu3 (using .../python3.2-minimal_3.2.3-6ubuntu3.1_amd64.deb) ...
Unpacking replacement python3.2-minimal ...
Preparing to replace vim 2:7.3.547-4ubuntu1 (using .../vim_2%3a7.3.547-4ubuntu1.1_amd64.deb) ...
Unpacking replacement vim ...
Preparing to replace vim-tiny 2:7.3.547-4ubuntu1 (using .../vim-tiny_2%3a7.3.547-4ubuntu1.1_amd64.deb) ...
Unpacking replacement vim-tiny ...
Preparing to replace vim-runtime 2:7.3.547-4ubuntu1 (using .../vim-runtime_2%3a7.3.547-4ubuntu1.1_all.deb) ...
Unpacking replacement vim-runtime ...
Preparing to replace vim-common 2:7.3.547-4ubuntu1 (using .../vim-common_2%3a7.3.547-4ubuntu1.1_amd64.deb) ...
Unpacking replacement vim-common ...
Preparing to replace ubuntu-release-upgrader-core 1:0.190.1 (using .../ubuntu-release-upgrader-core_1%3a0.190.4_all.deb) ...
Unpacking replacement ubuntu-release-upgrader-core ...
Preparing to replace python3-distupgrade 1:0.190.1 (using .../python3-distupgrade_1%3a0.190.4_all.deb) ...
Unpacking replacement python3-distupgrade ...
Preparing to replace python3-problem-report 2.6.1-0ubuntu3 (using .../python3-problem-report_2.6.1-0ubuntu6_all.deb) ...
Unpacking replacement python3-problem-report ...
Preparing to replace python3-apport 2.6.1-0ubuntu3 (using .../python3-apport_2.6.1-0ubuntu6_all.deb) ...
Unpacking replacement python3-apport ...
Preparing to replace apport 2.6.1-0ubuntu3 (using .../apport_2.6.1-0ubuntu6_all.deb) ...
apport stop/waiting
Unpacking replacement apport ...
Preparing to replace linux-generic 3.5.0.17.19 (using .../linux-generic_3.5.0.18.21_amd64.deb) ...
Unpacking replacement linux-generic ...
Processing triggers for man-db ...
Processing triggers for mime-support ...
Processing triggers for ureadahead ...
Setting up libwhoopsie0 (0.2.7) ...
Setting up whoopsie (0.2.7) ...
whoopsie start/running, process 7859
Setting up isc-dhcp-common (4.2.4-1ubuntu10.1) ...
Setting up isc-dhcp-client (4.2.4-1ubuntu10.1) ...
Setting up lsb-release (4.0-0ubuntu26.1) ...
Setting up python3.2-minimal (3.2.3-6ubuntu3.1) ...
Setting up python3.2 (3.2.3-6ubuntu3.1) ...
Setting up vim-common (2:7.3.547-4ubuntu1.1) ...
Setting up vim-runtime (2:7.3.547-4ubuntu1.1) ...
Processing /usr/share/vim/addons/doc
Setting up vim (2:7.3.547-4ubuntu1.1) ...
Setting up vim-tiny (2:7.3.547-4ubuntu1.1) ...
Setting up python3-distupgrade (1:0.190.4) ...
Setting up ubuntu-release-upgrader-core (1:0.190.4) ...
Setting up python3-problem-report (2.6.1-0ubuntu6) ...
Setting up python3-apport (2.6.1-0ubuntu6) ...
Setting up apport (2.6.1-0ubuntu6) ...
apport start/running
Setting up linux-generic (3.5.0.18.21) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
[email protected]:~#

As you see, apt-btrfs-snapshot has automatically created as snapshot of our system (called @apt-snapshot-2012-11-22_11:50:38 in this example) before the upgrade. You can check that with...

btrfs subvolume list /

[email protected]:~# btrfs subvolume list /
ID 256 top level 5 path @
ID 258 top level 5 path @home
ID 260 top level 5 path @apt-snapshot-2012-11-22_11:50:38
[email protected]:~#

... and:

apt-btrfs-snapshot list

[email protected]:~# apt-btrfs-snapshot list
Available snapshots:
@apt-snapshot-2012-11-22_11:50:38
[email protected]:~#

4 Rollback

Now let's assume the last apt operation turned our working system into one that isn't working as expected anymore. That's why we want to restore the previous system state, i.e., we want to do a rollback.

Therefore we mount the btrfs filesystem to a separate location, e.g. /mnt:

mount /dev/sda1 /mnt

We can now see our subvolumes in the output of:

ls -l /mnt/

[email protected]:~# ls -l /mnt/
total 0
drwxr-xr-x 1 root root 230 Nov 22 10:46 @
drwxr-xr-x 1 root root 230 Nov 22 10:46 @apt-snapshot-2012-11-22_11:50:38
drwxr-xr-x 1 root root  26 Nov 22 10:57 @home
[email protected]:~#

@apt-snapshot-2012-11-22_11:50:38 is a snapshot of our working root filesystem (@) before the apt operation. In order to make the system boot from that working snapshot instead of from the current subvolume, we rename @ to something else and then @apt-snapshot-2012-11-22_11:50:38 to @:

mv /mnt/@ /mnt/@_badroot
mv /mnt/@apt-snapshot-2012-11-22_11:50:38 /mnt/@

Now reboot:

reboot

5 Check If The Rollback Was Successful

After the reboot we should check if the rollback was successful. To do this, we repeat the apt operation which made our system unusable, e.g.:

apt-get update

apt-get upgrade

If the rollback was successful, apt-get upgrade should show the same packages available for update as before (as this is just a check if the rollback was successful, don't install the updates again):

6 Delete The @ Subvolume (Optional)

If you are sure the rollback was successful and you don't need the old @ subvolume (now named @_badroot) anymore, you can delete it to free up some space.

mount /dev/sda1 /mnt

ls -l /mnt/

[email protected]:~# ls -l /mnt/
total 0
drwxr-xr-x 1 root root 230 Nov 22 10:46 @
drwxr-xr-x 1 root root 230 Nov 22 10:46 @_badroot
drwxr-xr-x 1 root root  26 Nov 22 10:57 @home
[email protected]:~#

btrfs subvolume delete /mnt/@_badroot

umount /mnt

7 Links

btrfs: https://btrfs.wiki.kernel.org/index.php/Main_Page
Ubuntu: http://www.ubuntu.com/

view as pdf | print

Share this page:

6 Comment(s)

Add comment

Comments

By: jpeg729

Seems to work fine in Ubuntu 12.04 as well.

Installation and use is exactly the same. I haven't tested very much but the commands are there and the snapshots do get created as they should.

By: Seanny123

You mention that this reverts the root file system. Does that mean it will revert my documents too? If so, would you mind including that in your introduction?

By: Anonymous

No, i won't rollback your documents in this setup. Your documents (should) live under /home, which will have a seperate subvolume in the default ubuntu btrfs setup.

Anything you'll change in /local or /usr or whereever might be rolled back.

By: mchid

On newer versions of ubuntu, you nolonger need to mount the filesystem to roll back to an apt-btrfs-snapshot. Just run:

~$ sudo apt-btrfs-snapshot list

to list the available snapshots and then run the following to revert to an older snapshot:

~$ sudo apt-btrfs-snapshot set-default @<name-of-snapshot>

Reboot for the changes to take effect.

Also, deleting snapshots is now much more reliable and simple. Again, to list snapshots, run:

~$ sudo apt-btrfs-snapshot list

Then, to delete a snapshot, run:

~$ sudo apt-btrfs-snapshot delete @<name-of-snapshot>

To list by date older than two days, for example, run:

~$ sudo apt-btrfs-snapshot list-older-than 2d

To delete all snapshots older than two days, for example, run:

~$ sudo apt-btrfs-snapshot delete-older-than 2d

The cool thing is that your old default will not be deleted so you can still roll back to that one if you have one set.

Deleting snapshots should take effect immediately.

Finally, if you ever run into the "no space left on device" problem when you boot up, insert a blank pin drive or external hard drive and mount the blank drive at /tmp using the command line. After doing so, you should be able to proceed with deleting older snapshots.

By: cheng

This documentation is useful for system admin who want to rollback root directory if apt-get broken system.

Thanks.

By: ste7233

Now it works.

mchid is wrong. I am using Kubuntu 16.04 LTS and when I trie what he said I get "Sorry, your system lacks support for the snapshot feature" plus I had to sudo apt-install apt-btrfs-snapshot, to install the package.

Log in or Sign up