btrfs, lxc and ISPConfig 3

[CSsab]

I have begun experimenting with ispconfig3 lxc and the new btrfs filesystem with considerable success and implications for making life much easier for us in the future. In my setup, the ISPConfig 3 master is installed on the physical host on the booting ext4 partition of my 1TB hard drive. Recently it has become possible to format your entire hard drive using the btrfs file system when installing ubuntu server (to boot from a btrfs partition) however the btrfs fsck facility is still under heavy development and I much prefer the stability of working from an ext4 environment for now. For the moment, this is the best of both worlds.

During server install I formatted my hard drive as follows:

<- / (ext4 file system) -><- swap -><- /btone (btrfs file system) -><- /bttwo (btrfs file system) ->

This is my fstab:

# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=2bc66489-e9a0-424a-8753-92ec87a9f3f5 / ext4 errors=remount-ro 0 0
# swap was on /dev/sda2 during installation
UUID=a93f94b1-6d78-4183-97e3-4839b77991b0 none swap sw 0 0
# /btone was on /dev/sda3 during installation
UUID=ec65eb1a-bede-4338-aa16-352a6783d27a /btone btrfs defaults 0 0
# /bttwo was on /dev/sda4 during installation
UUID=54cc801f-3e49-4071-83fd-f1164a1ed344 /bttwo btrfs defaults 0 0


It does not appear to be possible yet to mount a btrfs partition with anything other than the "defaults" (I tried errors=remount-ro and a few other things and the partition simply refuses to mount).

At time of writing, I have set up 5 lxc containers (servers in a multiserver setup) called ns1, ns2, web, mail and db on the partition /btone in 5 seperate btrfs subvolumes
I have then taken btrfs snapshots of the running servers and moved the servers to my other btrfs partition (/bttwo).

This is what I did (ns1 node only).

On the host:

configure the network bridge:
aptitude install bridge-utils
vi /etc/network/interfaces

Here is mine:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
# auto eth0
# iface eth0 inet dhcp

# Bridge Setup
auto br0
iface br0 inet static
address 192.168.1.XXX
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
bridge_ports eth0
bridge_fd 0
bridge_maxwait 0
bridge_stp off post-up
/usr/sbin/brctl setfd br0 0

Change the red to suit your setup.

aptitude install lxc debootstrap libcap-dev debian-archive-keyring libcap2 libcap2-bin
(this installed cgroup-lite for me which did not work as well as I had hoped so I removed cgroup-lite and rebooted the server before mounting cgroup the old way)
apt-get remove --purge cgroup-lite*

reboot

note that I mount cgroup on the same partition as the lxc-containers I am running (it may not matter which partition cgroup is mounted on)
mkdir /btone/cgroup
echo "none /btone/cgroup cgroup defaults 0 0" >> /etc/fstab
mount -a

cp /usr/lib/lxc/templates/lxc-debian /usr/lib/lxc/templates/lxc-debian.ORIG

vi /usr/lib/lxc/templates/lxc-debian

Edit the lxc-debian template to your liking - I usually change the locale settings from:

chroot $rootfs locale-gen en_US.UTF-8 UTF-8
chroot $rootfs update-locale LANG=en_US.UTF-8

to:

chroot $rootfs locale-gen en_AU.UTF-8 UTF-8
chroot $rootfs update-locale LANG=en_AU.UTF-8


and change the package list from:

packages=\
ifupdown,\
locales,\
libui-dialog-perl,\
dialog,\
dhcp3-client,\
netbase,\
net-tools,\
iproute,\
openssh-server


to:

packages=\
ifupdown,\
locales,\
libui-dialog-perl,\
dialog,\
dhcp3-client,\
netbase,\
net-tools,\
iputils-ping,\
ssh,\
cron,\
sudo,\
vim-nox,\
rsyslog,\
iproute,\
apt-utils,\
debconf,\
openssh-server

Create a subvolume on your btrfs partition

btrfs subvolume create /btone/ns1

Download a squeeze server into your new subvolume (this will be slowish the first time downloading a new distro but is very fast in subsequent same system containers since lxc builds an archived cache of the system and simply copies it accross in new servers)

/usr/lib/lxc/templates/lxc-debian -p /btone/ns1

Edit the configuration file of your new container and add the following lines:

# networking
lxc.utsname = ns1
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.ipv4 = 192.168.1.XXX/24 192.168.1.255 (change red to suit your setup)

My configuration files are slightly more complex and look something like this:

# networking
lxc.utsname = ns1
lxc.network.type = veth
lxc.network.veth.pair = vethns1 (this can be whatever you want to call the host side of the veth pair)
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.ipv4 = 192.168.1.XXX/24 192.168.1.255
lxc.network.ipv6 = XXXX::XXXX:XXXX:XXXX:XXXX/64
lxc.network.hwaddr = XX:XX:XX:XX:XX:XX (I prefer to let lxc decide what the mac address is and also let it work out my ipv6 address)

mkdir /var/lib/lxc/ns1
cp /btone/ns1/config /var/lib/lxc/ns1/config

Copy the configuration file accross to the host partition where lxc expects to find the container config file:

Start the container in daemon mode

lxc-start -n ns1 -d

Log in to your new ns1 node using putty or similar and configure server as you normally would.

Now take a snapshot of the server - note that there is no difference between a subvolume and a snapshot as far as btrfs is concerned.

mkdir /btone/snapshots
btrfs subvolume snapshot /btone/ns1 /btone/snapshots/ns1

lxc-info -n ns1
shows that ns1 is still running so lets just stop it while we move it over to a different partition.

lxc-stop -n ns1

Create a new subvoloume on the /bttwo partition

btrfs subvolume create /bttwo/ns1

Move the rootfs and config file across to the new subvolume

mv /btone/snapshots/ns1/rootfs /bttwo/ns1/rootfs
mv /btone/snapshots/ns1/config /bttwo/ns1/config

delete the empty snapshot

btrfs subvolume delete /btone/snapshots/ns1

Now it is very important to reflect the location changes in ns1 lxc container config file before re starting the container as follows:

rm /var/lib/lxc/ns1/config

vi /bttwo/ns1/config
and change the following lines from:


lxc.rootfs = /btone/ns1/rootfs
lxc.mount.entry=proc /btone/ns1/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=sysfs /btone/ns1/rootfs/sys sysfs defaults 0 0

to:

lxc.rootfs = /bttwo/ns1/rootfs
lxc.mount.entry=proc /bttwo/ns1/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=sysfs /bttwo/ns1/rootfs/sys sysfs defaults 0 0

cp /bttwo/ns1/config /var/lib/lxc/ns1/config

Now you can simply start the ns1 server like this:

lxc-start -n ns1 -d

or you can move cgroup to the new partition (like I do) and set up the containers to autostart before rebooting the host:


Edit /ect/default/lxc to look like this:


# Comment out to run the lxc init script
RUN=yes

# Directory containing the container configurations
CONF_DIR=/etc/lxc

#MIRROR="http://archive.ubuntu.com/ubuntu"

Symlink the configuration file for the container into the autostart directory like this:

ln -s /var/lib/lxc/ns1/config /etc/lxc/ns1.conf

Better to symlink rather than copy the configuration file at this point since fewer changes will be needed if you want to switch containers at a later date.

Now when you reboot the host, your container will automatically start - further we have left a complete backup of ns1 on a seperate partition and we have the potential to take snapshots of either subvolume using a script.

Again I stress that this is a highly experimental setup and documentation is thin on the ground at this stage. Any advice with regard to a useful snapshot/backup script for use with ispconfig 3 is appreciated. I have been looking at this:
https://github.com/mmehnert/btrfs-snapshot-rotation

Discussion very welcome.

Regards to all.

[till]

I use openVZ on my servers for a virtual setup in production for quite some time now. It is very stable and makes the creation of snapshots and backups very esay with vzdump. I had choosen OpenVZ over lxc as it was more mature and lxc had problems with quota support inside containers. Are the quota problems fixed in the meantime?

[CSsab]

I had choosen OpenVZ over lxc as it was more mature and lxc had problems with quota support inside containers. Are the quota problems fixed in the meantime?

The short answer is no ... I could not find any decent or well documented support for quota inside the containers.

The long answer is that I only use quota on the host where the master ISPConfig 3 web interface lives. In the container nodes I place a file called "repquota" in the /usr/sbin directory that does nothing when ISPConfig 3 looks to report quota from one of the container nodes like this:

IN THE CONTAINER (DON'T DO THIS ON THE HOST)

vi /usr/sbin/repquota

#!/bin/bash
exit 0

save and close ...

then:

chmod +x /usr/sbin repquota

Thanks again to falko for tip.

Okay so this works well in stifling errors.

I don't need quota on a nameserver but maybe there is an argument that I do need hard disk quota on a web server node. This is all managed from the master (where quota is installed and reported in the control panel) so far as I can tell.

Dynamic volume resizing is also possible with btrfs thereby implementing hard disk quota in other ways. I haven't experimented with resizing subvolumes/filesystems yet but from what I understand, when quota is reached, writes are no longer possible to the btrfs subvolume that has been sized.

For now here are a few more notes for anyone interested.

btrfs filesystem show
failed to read /dev/sr0 (we can ignore this - btrfs-tools is looking for a cd rom ? and this functionality has apparently not yet been implented ?)
Label: none uuid: ec65eb1a-bede-4338-aa16-352a6783d27a
Total devices 1 FS bytes used 3.09GB
devid 1 size 316.65GB used 12.04GB path /dev/sda3

Label: none uuid: 54cc801f-3e49-4071-83fd-f1164a1ed344
Total devices 1 FS bytes used 3.10GB
devid 1 size 319.25GB used 6.54GB path /dev/sda4

Btrfs Btrfs v0.19

btrfsck /dev/sda3

found 3321049088 bytes used err is 0
total csum bytes: 2976476
total tree bytes: 273137664
total fs tree bytes: 259792896
btree space waste bytes: 79237747
file data blocks allocated: 3074125824
referenced 3047546880
Btrfs Btrfs v0.19

I am still at learning stage with working out size of individual file systems and familiarizing myself with btrfs-tools:

Usage:
btrfs subvolume snapshot <source> [<dest>/]<name>
Create a writable snapshot of the subvolume <source> with
the name <name> in the <dest> directory.
btrfs subvolume delete <subvolume>
Delete the subvolume <subvolume>.
btrfs subvolume create [<dest>/]<name>
Create a subvolume in <dest> (or the current directory if
not passed).
btrfs subvolume list <path>
List the snapshot/subvolume of a filesystem.
btrfs subvolume find-new <path> <last_gen>
List the recently modified files in a filesystem.
btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t size] <file>|<dir> [<file>|<dir>...]
Defragment a file or a directory.
btrfs subvolume set-default <id> <path>
Set the subvolume of the filesystem <path> which will be mounted
as default.
btrfs filesystem sync <path>
Force a sync on the filesystem <path>.
btrfs filesystem resize [+/-]<newsize>[gkm]|max <filesystem>
Resize the file system. If 'max' is passed, the filesystem
will occupe all available space on the device.
btrfs filesystem show [<uuid>|<label>]
Show the info of a btrfs filesystem. If no <uuid> or <label>
is passed, info of all the btrfs filesystem are shown.
btrfs filesystem df <path>
Show space usage information for a mount point
.
btrfs filesystem balance <path>
Balance the chunks across the device.
btrfs device scan [<device> [<device>..]
Scan all device for or the passed device for a btrfs
filesystem.
btrfs device add <dev> [<dev>..] <path>
Add a device to a filesystem.
btrfs device delete <dev> [<dev>..] <path>
Remove a device from a filesystem.

btrfs help|--help|-h
Show the help.

Btrfs Btrfs v0.19

[CSsab]

I don't know why I didn't try this before (a bit slow sometimes!).

The base vm (Debian container) spits out an unconfigured fstab.

So .. to enable quota.

apt-get install quota quotatool

vi /etc/fstab


rootfs / rootfs rw,usrjquota=aquota.user,grpjquota=aquota.group,jq fmt=vfsv0 0 0


mount -a
touch /aquota.user /aquota.group
chmod 600 /aquota.*
quotaon -a


Heres the memory usage from the monitor module:


MemTotal: 8125480960
MemFree: 3394265088
Buffers: 125530112
Cached: 3934347264
SwapCached: 0
Active: 1000730624
Inactive: 3223650304
Active(anon): 179769344
Inactive(anon): 5820416
Active(file): 820961280
Inactive(file): 3217829888
Unevictable: 0
Mlocked: 0
SwapTotal: 17408454656
SwapFree: 17408454656
Dirty: 61440
Writeback: 0
AnonPages: 164446208
Mapped: 63606784
Shmem: 21082112
Slab: 356868096
SReclaimable: 328880128
SUnreclaim: 27987968
KernelStack: 2629632
PageTables: 17186816
NFS_Unstable: 0
Bounce: 0
WritebackTmp: 0
CommitLimit: 21471195136
Committed_AS: 1094057984
VmallocTotal: 35184372087808
VmallocUsed: 308191232
VmallocChunk: 35184060657664
HardwareCorrupted: 0
AnonHugePages: 0
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2097152
DirectMap4k: 56033280
DirectMap2M: 1822425088
DirectMap1G: 6442450944


and the Disk Usage:

Filesystem Type Size Used Available Use% Mounted on
tmpfs tmpfs 5.0M 4.0K 5.0M 1% /lib/init/rw
tmpfs tmpfs 1.6G 16M 1.5G 1% /tmp
tmpfs tmpfs 1.6G 0 1.6G 0% /run/shm
rootfs rootfs 249G 1.1G 245G 1% /
tmpfs tmpfs 775M 1020K 774M 1% /run
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock


The machine is not even running from any of the partitions listed above ...