How to setup virtual containers with LXC and quota support on Debian 8

Quota support is an often requested feature in lxc. Linux filesystem quota is required when you want to give multiple users access to a container and want to control that one user is not using all the disk space. Quota is also required for web hosting servers, e.g. with ISPConfig 3, for the same reason: one website shall not be able to fill up the whole disk. This howto shows you, how you can use lxc with hard disk quota using qemu nbd with a qcow image file on Debian 8.

Prerequisites

To use lxc, you need the qemu utilities and the lxc package itself. Install them by calling:

apt-get install lxc qemu-utils

The installer will ask you to choose the directory where the lxc virtual machine images get installed later. This directory should be on a partition with a lot of free space. If you have enough space in /var, then accept the default /var/lib/lxc, otherwise, choose a free directory on your largest partition. When you use a non-default path, then ensure to change the path in all commands and config files below.

Preparing

Check if the kernel loop module is loaded with:

lsmod | grep '^loop'

If you get not result, you can enable the module by running:

modprobe loop

Create the virtual machine

Now we can start creating the VM. In this tutorial, I will use Debian Jessie in both the host and the container, but you can use other lxc templates of course, e. g. Debian wheezy or ubuntu.

lxc-create -B loop -t debian -n mydebianvm --fssize=20G -- -r jessie

The -t argument selects the main template, -r decides which release to use. To set the hard disk size for the virtual machine, you can alter the --fssize argument. Let's say you want to create a disk with 50 gigabytes, you'd change the argument to --fssize=50G.
The argument -n sets the name of the vm. I used mydebianvm in this tutorial. Please change the name in all following commands according to what you chose.

As we don't want to use a raw image file, we need to convert the disk image to the qemu qcow2 format. This is done by the following command

qemu-img convert -O qcow2 /var/lib/lxc/mydebianvm/rootdev /var/lib/lxc/mydebianvm/rootdev.qcow2

To make backup handling easier, later on, we create an image file set, i. e. a second file that records all changes to the device.

qemu-img create -f qcow2 -b /var/lib/lxc/mydebianvm/rootdev.qcow2 /var/lib/lxc/mydebianvm/rootdev-live.qcow2

You can now delete the original raw image file with:

rm /var/lib/lxc/mydebianvm/rootdev

Configure the Network Bridge

Install the bridge-utils:

apt-get install bridge-utils

Open the Debian Network configuration file /etc/network/interfaces in an editor

vim /etc/network/interfaces

and add the following lines:

auto br0 
iface br0 inet static 
      address 192.168.1.254 
      netmask 255.255.255.0 
      bridge_ports eth0 
      bridge_stp off 
      bridge_fd 2 
      bridge_maxwait 20

Replace the address and netmask with values for your local network.

Then activate the network bridge with the command:

ifup br0

Configure the VM

Backup the old config file:

mv /var/lib/lxc/mydebianvm/config /var/lib/lxc/mydebianvm/config_bak

And create the config for the container:

vim /var/lib/lxc/mydebianvm/config

And add the following content in the file:

lxc.rootfs = /var/lib/lxc/mydebianvm/rootfs
lxc.rootfs.options = usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0

lxc.hook.pre-start = /var/lib/lxc/prestart-nbd.sh
lxc.hook.post-stop = /var/lib/lxc/poststop-nbd.sh
 
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
 
# only if bridge is set up (or use other method)
lxc.network.type = veth 
lxc.network.name = veth0 
lxc.network.flags = up 
lxc.network.link = br0 
lxc.network.ipv4 = 192.168.1.101/24 
lxc.network.ipv4.gateway = 0.0.0.0
 
# Container specific configuration
lxc.mount = /var/lib/lxc/mydebianvm/fstab
lxc.utsname = debian8
lxc.arch = amd64
lxc.autodev = 1
lxc.kmsg = 0

Replace the IP-Address 192.168.1.101 with a free IP from your network.

Add the prestart script /var/lib/lxc/prestart-nbd.sh

vim /var/lib/lxc/prestart-nbd.sh

with the following content:

#!/bin/bash
CHK=$(lsmod | grep '^nbd');
if [[ "$CHK" == "" ]] ; then
 modprobe nbd nbds_max=64 max_part=8
fi
DEV=""

for D in /dev/nbd* ; do
F=$(basename $D)
if [[ $(lsblk | grep "^${F} ") == "" ]] ; then
DEV="$D"
break;
fi
done

echo "Next free NBD is $DEV";

CHK=$(lsof /var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2 | grep 'qemu-nbd' | awk '{ print $2 }');
if [[ "$CHK" == "" ]] ; then
if [[ "$DEV" == "" ]] ; then
print "No free nbd device found";
exit 1;
fi
echo "Connecting $DEV to /var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2"
qemu-nbd -c ${DEV} -n --aio=native /var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2
else
NBD=$(lsof -p ${CHK} | grep '/dev/nbd' | awk '{ print $9 }');
if [[ "$NBD" != "" ]] ; then
echo "/var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2 is already connected to $NBD"
DEV="$NBD";
else
echo "/var/lib/lxc/${LXC_NAME}/rootdev-live.qcow2 is used by suspicious PID";
exit 1;
fi
fi

CHK=$(mount | grep " /var/lib/lxc/${LXC_NAME}/rootfs ")
if [[ "$CHK" == "" ]] ; then
 echo "/var/lib/lxc/${LXC_NAME}/rootfs not mounted";
 echo "Mounting ${DEV} to /var/lib/lxc/${LXC_NAME}/rootfs"
 mount ${DEV} /var/lib/lxc/${LXC_NAME}/rootfs
fi
echo "${DEV} ${DEV:1} none bind,create=file,optional 0 0" > /var/lib/lxc/${LXC_NAME}/fstab

and make it executable:

chmod +x /var/lib/lxc/prestart-nbd.sh

Add the poststop script /var/lib/lxc/poststop-nbd.sh

vim /var/lib/lxc/poststop-nbd.sh

with the following content:

#!/bin/bash
CHK=$(mount | grep " /var/lib/lxc/${LXC_NAME}/rootfs " | awk '{ print $1 }')
if [[ "$CHK" != "" ]] ; then
 echo "Unmounting ${CHK} from /var/lib/lxc/${LXC_NAME}/rootfs"
 echo "Disconnecting ${CHK}"
 umount /var/lib/lxc/${LXC_NAME}/rootfs && qemu-nbd -d ${CHK}
fi

and make it executable:

chmod +x /var/lib/lxc/poststop-nbd.sh

Start the VM and set-up quota

We can now start the container in background mode by typing:

lxc-start -n mydebianvm -d

Install the neccessary packages for quota. We don't have to enter the container for this. Using lxc-attach we can run commands from outside the container.

lxc-attach -n mydebianvm -- apt-get -y update
lxc-attach -n mydebianvm -- apt-get -y install quota

It is not possible to activate quota through lxc-attach. So we create a bash script, that is executed on next boot of the container

vim /var/lib/lxc/mydebianvm/rootfs/opt/actquota.sh

with the following content:

#!/bin/bash
mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 /
touch /aquota.user /aquota.group
chmod 0600 /aquota.*
quotacheck -cmug /
quotaon -avug
echo '#!/bin/sh -e
exit 0' > /etc/rc.local
rm -f /opt/actquota.sh

The script will delete itself and empty the /etc/rc.local of the container afterwards.
Now make sure the bash script is executeable and called at start-up:

Make it executable:

chmod 700 /var/lib/lxc/mydebianvm/rootfs/opt/actquota.sh

Add the call to the vm's rc.local file:

echo '#!/bin/bash
if [[ -e "/opt/actquota.sh" ]] ; then
/opt/actquota.sh
fi' > /var/lib/lxc/mydebianvm/rootfs/etc/rc.local

With all pre-requisites set-up you can now restart the container. If you have followed the steps correctly, this will activate quota.

lxc-stop -r -n mydebianvm

Verify the results

You should now check if quota is working. Change to the container.

lxc-attach -n mydebianvm

Inside the container type:

repquota -avug

You should see used quota of users and groups now.

Destroying the virtual machine

It is very important to use the commands in the correct order. Before you can disconnect the nbd device you have to stop the container if it is running:

lxc-stop -n mydebianvm

Afterwards, you have to unmount the root fs.

umount /var/lib/lxc/mydebianvm/rootfs

The last step is to disconnect the nbd. Be sure to select the correct device number.
NEVER disconnect the nbd before you unmounted the rootfs. This will lead to a lot of problems and will require a full forced reboot of your host.

qemu-nbd -d /dev/nbd0

Backup the VM

Because we have created two files when creating the image file for the container, we can easily backup without stopping the vm. First we need to commit the changes that occured in the meantime to the base file.

qemu-img commit /var/lib/lxc/mydebianvm/rootdev-live.qcow2

The /var/lib/lxc/mydebianvm/rootdev.qcow2 now contains the current state of the vm's hard disk, so you can backup this file.