btrfs, lxc and ISPConfig 3

Discussion in 'Tips/Tricks/Mods' started by CSsab, Oct 18, 2011.

  1. CSsab

    CSsab New Member

    I have begun experimenting with ispconfig3 lxc and the new btrfs filesystem with considerable success and implications for making life much easier for us in the future. In my setup, the ISPConfig 3 master is installed on the physical host on the booting ext4 partition of my 1TB hard drive. Recently it has become possible to format your entire hard drive using the btrfs file system when installing ubuntu server (to boot from a btrfs partition) however the btrfs fsck facility is still under heavy development and I much prefer the stability of working from an ext4 environment for now. For the moment, this is the best of both worlds.

    During server install I formatted my hard drive as follows:

    <- / (ext4 file system) -><- swap -><- /btone (btrfs file system) -><- /bttwo (btrfs file system) ->

    This is my fstab:

    # <file system> <mount point> <type> <options> <dump> <pass>
    proc /proc proc nodev,noexec,nosuid 0 0
    # / was on /dev/sda1 during installation
    UUID=2bc66489-e9a0-424a-8753-92ec87a9f3f5 / ext4 errors=remount-ro 0 0
    # swap was on /dev/sda2 during installation
    UUID=a93f94b1-6d78-4183-97e3-4839b77991b0 none swap sw 0 0
    # /btone was on /dev/sda3 during installation
    UUID=ec65eb1a-bede-4338-aa16-352a6783d27a /btone btrfs defaults 0 0
    # /bttwo was on /dev/sda4 during installation
    UUID=54cc801f-3e49-4071-83fd-f1164a1ed344 /bttwo btrfs defaults 0 0

    It does not appear to be possible yet to mount a btrfs partition with anything other than the "defaults" (I tried errors=remount-ro and a few other things and the partition simply refuses to mount).

    At time of writing, I have set up 5 lxc containers (servers in a multiserver setup) called ns1, ns2, web, mail and db on the partition /btone in 5 seperate btrfs subvolumes
    I have then taken btrfs snapshots of the running servers and moved the servers to my other btrfs partition (/bttwo).

    This is what I did (ns1 node only).

    On the host:

    configure the network bridge:
    aptitude install bridge-utils
    vi /etc/network/interfaces

    Here is mine:
    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).

    # The loopback network interface
    auto lo
    iface lo inet loopback

    # The primary network interface
    # auto eth0
    # iface eth0 inet dhcp

    # Bridge Setup
    auto br0
    iface br0 inet static
    address 192.168.1.XXX
    bridge_ports eth0
    bridge_fd 0
    bridge_maxwait 0
    bridge_stp off post-up
    /usr/sbin/brctl setfd br0 0

    Change the red to suit your setup.

    aptitude install lxc debootstrap libcap-dev debian-archive-keyring libcap2 libcap2-bin
    (this installed cgroup-lite for me which did not work as well as I had hoped so I removed cgroup-lite and rebooted the server before mounting cgroup the old way)
    apt-get remove --purge cgroup-lite*


    note that I mount cgroup on the same partition as the lxc-containers I am running (it may not matter which partition cgroup is mounted on)
    mkdir /btone/cgroup
    echo "none /btone/cgroup cgroup defaults 0 0" >> /etc/fstab
    mount -a

    cp /usr/lib/lxc/templates/lxc-debian /usr/lib/lxc/templates/lxc-debian.ORIG

    vi /usr/lib/lxc/templates/lxc-debian

    Edit the lxc-debian template to your liking - I usually change the locale settings from:

    chroot $rootfs locale-gen en_US.UTF-8 UTF-8
    chroot $rootfs update-locale LANG=en_US.UTF-8


    chroot $rootfs locale-gen en_AU.UTF-8 UTF-8
    chroot $rootfs update-locale LANG=en_AU.UTF-8

    and change the package list from:




    Create a subvolume on your btrfs partition

    btrfs subvolume create /btone/ns1

    Download a squeeze server into your new subvolume (this will be slowish the first time downloading a new distro but is very fast in subsequent same system containers since lxc builds an archived cache of the system and simply copies it accross in new servers)

    /usr/lib/lxc/templates/lxc-debian -p /btone/ns1

    Edit the configuration file of your new container and add the following lines:

    # networking
    lxc.utsname = ns1 = veth = up = br0 = eth0 = 192.168.1.XXX/24
    (change red to suit your setup)

    My configuration files are slightly more complex and look something like this:

    # networking
    lxc.utsname = ns1 = veth = vethns1 (this can be whatever you want to call the host side of the veth pair) = up = br0 = eth0 = 192.168.1.XXX/24 =
    XXXX::XXXX:XXXX:XXXX:XXXX/64 = XX:XX:XX:XX:XX:XX (I prefer to let lxc decide what the mac address is and also let it work out my ipv6 address)

    mkdir /var/lib/lxc/ns1
    cp /btone/ns1/config /var/lib/lxc/ns1/config

    Copy the configuration file accross to the host partition where lxc expects to find the container config file:

    Start the container in daemon mode

    lxc-start -n ns1 -d

    Log in to your new ns1 node using putty or similar and configure server as you normally would.

    Now take a snapshot of the server - note that there is no difference between a subvolume and a snapshot as far as btrfs is concerned.

    mkdir /btone/snapshots
    btrfs subvolume snapshot /btone/ns1 /btone/snapshots/ns1

    lxc-info -n ns1
    shows that ns1 is still running so lets just stop it while we move it over to a different partition.

    lxc-stop -n ns1

    Create a new subvoloume on the /bttwo partition

    btrfs subvolume create /bttwo/ns1

    Move the rootfs and config file across to the new subvolume

    mv /btone/snapshots/ns1/rootfs /bttwo/ns1/rootfs
    mv /btone/snapshots/ns1/config /bttwo/ns1/config

    delete the empty snapshot

    btrfs subvolume delete /btone/snapshots/ns1

    Now it is very important to reflect the location changes in ns1 lxc container config file before re starting the container as follows:

    rm /var/lib/lxc/ns1/config

    vi /bttwo/ns1/config
    and change the following lines from:

    lxc.rootfs = /btone/ns1/rootfs
    lxc.mount.entry=proc /btone/ns1/rootfs/proc proc nodev,noexec,nosuid 0 0
    lxc.mount.entry=sysfs /btone/ns1/rootfs/sys sysfs defaults 0 0


    lxc.rootfs = /bttwo/ns1/rootfs
    lxc.mount.entry=proc /bttwo/ns1/rootfs/proc proc nodev,noexec,nosuid 0 0
    lxc.mount.entry=sysfs /bttwo/ns1/rootfs/sys sysfs defaults 0 0

    cp /bttwo/ns1/config /var/lib/lxc/ns1/config

    Now you can simply start the ns1 server like this:

    lxc-start -n ns1 -d

    or you can move cgroup to the new partition (like I do) and set up the containers to autostart before rebooting the host:

    Edit /ect/default/lxc to look like this:

    # Comment out to run the lxc init script

    # Directory containing the container configurations


    Symlink the configuration file for the container into the autostart directory like this:

    ln -s /var/lib/lxc/ns1/config /etc/lxc/ns1.conf

    Better to symlink rather than copy the configuration file at this point since fewer changes will be needed if you want to switch containers at a later date.

    Now when you reboot the host, your container will automatically start - further we have left a complete backup of ns1 on a seperate partition and we have the potential to take snapshots of either subvolume using a script.

    Again I stress that this is a highly experimental setup and documentation is thin on the ground at this stage. Any advice with regard to a useful snapshot/backup script for use with ispconfig 3 is appreciated. I have been looking at this:

    Discussion very welcome.

    Regards to all.
    Last edited: Oct 18, 2011
  2. till

    till Super Moderator

    I use openVZ on my servers for a virtual setup in production for quite some time now. It is very stable and makes the creation of snapshots and backups very esay with vzdump. I had choosen OpenVZ over lxc as it was more mature and lxc had problems with quota support inside containers. Are the quota problems fixed in the meantime?
  3. CSsab

    CSsab New Member

    The short answer is no ... I could not find any decent or well documented support for quota inside the containers.

    The long answer is that I only use quota on the host where the master ISPConfig 3 web interface lives. In the container nodes I place a file called "repquota" in the /usr/sbin directory that does nothing when ISPConfig 3 looks to report quota from one of the container nodes like this:


    vi /usr/sbin/repquota

    exit 0

    save and close ...


    chmod +x /usr/sbin repquota

    Thanks again to falko for tip.

    Okay so this works well in stifling errors.

    I don't need quota on a nameserver but maybe there is an argument that I do need hard disk quota on a web server node. This is all managed from the master (where quota is installed and reported in the control panel) so far as I can tell.

    Dynamic volume resizing is also possible with btrfs thereby implementing hard disk quota in other ways. I haven't experimented with resizing subvolumes/filesystems yet but from what I understand, when quota is reached, writes are no longer possible to the btrfs subvolume that has been sized.

    For now here are a few more notes for anyone interested.

    btrfs filesystem show
    failed to read /dev/sr0 (we can ignore this - btrfs-tools is looking for a cd rom ? and this functionality has apparently not yet been implented ?)
    Label: none uuid: ec65eb1a-bede-4338-aa16-352a6783d27a
    Total devices 1 FS bytes used 3.09GB
    devid 1 size 316.65GB used 12.04GB path /dev/sda3

    Label: none uuid: 54cc801f-3e49-4071-83fd-f1164a1ed344
    Total devices 1 FS bytes used 3.10GB
    devid 1 size 319.25GB used 6.54GB path /dev/sda4

    Btrfs Btrfs v0.19

    btrfsck /dev/sda3

    found 3321049088 bytes used err is 0
    total csum bytes: 2976476
    total tree bytes: 273137664
    total fs tree bytes: 259792896
    btree space waste bytes: 79237747
    file data blocks allocated: 3074125824
    referenced 3047546880
    Btrfs Btrfs v0.19

    I am still at learning stage with working out size of individual file systems and familiarizing myself with btrfs-tools:

    btrfs subvolume snapshot <source> [<dest>/]<name>
    Create a writable snapshot of the subvolume <source> with
    the name <name> in the <dest> directory.
    btrfs subvolume delete <subvolume>
    Delete the subvolume <subvolume>.
    btrfs subvolume create [<dest>/]<name>
    Create a subvolume in <dest> (or the current directory if
    not passed).
    btrfs subvolume list <path>
    List the snapshot/subvolume of a filesystem.
    btrfs subvolume find-new <path> <last_gen>
    List the recently modified files in a filesystem.
    btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t size] <file>|<dir> [<file>|<dir>...]
    Defragment a file or a directory.
    btrfs subvolume set-default <id> <path>
    Set the subvolume of the filesystem <path> which will be mounted
    as default.
    btrfs filesystem sync <path>
    Force a sync on the filesystem <path>.
    btrfs filesystem resize [+/-]<newsize>[gkm]|max <filesystem>
    Resize the file system. If 'max' is passed, the filesystem
    will occupe all available space on the device.
    btrfs filesystem show [<uuid>|<label>]
    Show the info of a btrfs filesystem. If no <uuid> or <label>
    is passed, info of all the btrfs filesystem are shown.
    btrfs filesystem df <path>
    Show space usage information for a mount point
    btrfs filesystem balance <path>
    Balance the chunks across the device.
    btrfs device scan [<device> [<device>..]
    Scan all device for or the passed device for a btrfs
    btrfs device add <dev> [<dev>..] <path>
    Add a device to a filesystem.
    btrfs device delete <dev> [<dev>..] <path>
    Remove a device from a filesystem.

    btrfs help|--help|-h
    Show the help.

    Btrfs Btrfs v0.19
  4. CSsab

    CSsab New Member

    Update to "quota in lxc"

    I don't know why I didn't try this before (a bit slow sometimes!).

    The base vm (Debian container) spits out an unconfigured fstab.

    So .. to enable quota.

    apt-get install quota quotatool

    vi /etc/fstab

    rootfs  /       rootfs  rw,usrjquota=aquota.user,,jqfmt=vfsv0     0       0
    mount -a
    touch /aquota.user /
    chmod 600 /aquota.*
    quotaon -a
    Heres the memory usage from the monitor module:

    MemTotal:	8125480960
    MemFree:	3394265088
    Buffers:	125530112
    Cached:	3934347264
    SwapCached:	0
    Active:	1000730624
    Inactive:	3223650304
    Active(anon):	179769344
    Inactive(anon):	5820416
    Active(file):	820961280
    Inactive(file):	3217829888
    Unevictable:	0
    Mlocked:	0
    SwapTotal:	17408454656
    SwapFree:	17408454656
    Dirty:	61440
    Writeback:	0
    AnonPages:	164446208
    Mapped:	63606784
    Shmem:	21082112
    Slab:	356868096
    SReclaimable:	328880128
    SUnreclaim:	27987968
    KernelStack:	2629632
    PageTables:	17186816
    NFS_Unstable:	0
    Bounce:	0
    WritebackTmp:	0
    CommitLimit:	21471195136
    Committed_AS:	1094057984
    VmallocTotal:	35184372087808
    VmallocUsed:	308191232
    VmallocChunk:	35184060657664
    HardwareCorrupted:	0
    AnonHugePages:	0
    HugePages_Total:	0
    HugePages_Free:	0
    HugePages_Rsvd:	0
    HugePages_Surp:	0
    Hugepagesize:	2097152
    DirectMap4k:	56033280
    DirectMap2M:	1822425088
    DirectMap1G:	6442450944
    and the Disk Usage:
    Filesystem	Type	Size	Used	Available	Use%	Mounted on
    tmpfs	tmpfs	5.0M	4.0K	5.0M	1%	/lib/init/rw
    tmpfs	tmpfs	1.6G	16M	1.5G	1%	/tmp
    tmpfs	tmpfs	1.6G	0	1.6G	0%	/run/shm
    rootfs	rootfs	249G	1.1G	245G	1%	/
    tmpfs	tmpfs	775M	1020K	774M	1%	/run
    tmpfs	tmpfs	5.0M	0	5.0M	0%	/run/lock
    The machine is not even running from any of the partitions listed above ...

Share This Page