High Availability NFS With DRBD + Heartbeat
Ryan Babchishin - http://win2ix.ca
This document describes information collected during research and development of a clustered DRBD NFS solution. This project had two purposes:
- HA NFS solution for Media-X Inc.
- Develop a standard tool kit and documentation that Win2ix can use for future projects
Operating System
The standard operating for Win2ix is Ubuntu 12.04, therefore all testing was done with this as the preferred target.
Hardware
Because of the upcoming project with Media-X, computer hardware was chosen based on low cost and low power consumption.
A pair of identical systems for the cluster were used:
- SuperMicro SuperServer 5015A-EHF-D525 (default settings)
- Intel(R) Atom(TM) CPU D525 1.80GHz CPU (dual-core, hyper threaded)
- 4GB DDR3 RAM
- 2x750GB 2.5" Scorpio Black hard drives
- 3ware 9650 RAID-1 controller card with 128MB of RAM, without a battery backup
- 2x on-board Gigabit Ethernet
Partitioning and disk format
The disks were partitioned according to Win2ix standards.
- sda1 - / - ext4 - 20GB
- sda2 - /tmp - ext4 - 6GB
- sda3 - /var - ext4 - 6GB
- sdb5 - swap - swap - 2GB
- sda6 - drbd - drbd - 716GB
Networking
- eth0 was configured with a (unused) 192.168.0.[1|2] address for communication over a direct link between systems, with no switch
Bonding was tested, see below for more information.
- eth0 MTU was set to 9000
- eth1 was configured with a regular network IP address for SSH/NFS/etc... access on both systems
Bonding
Although not used due to lack of an extra PCI-E slot, Ethernet bonding was originally tested
Important notes when using bonding:
- To get true single connection load balancing in both directions, use bonding mode 0 (round robin) with NO SWITCH or find a switch that supports it. Direct connections between systems works well. Switches generally support trunking, LACP or some other Cisco variant. They will most likely only send traffic for different IP connections over separate links. This won't help with DRBD which uses a single connection
- Make sure you are seeing full throughput on your bonding device in both directions by testing with something like iperf
Sample working /etc/network/interfaces configuration segment:
iface bond0 inet static address 192.168.0.1 netmask 255.255.255.0 bond-mode 0 bond-miimon 100 bond-slaves eth0 eth1
Kernel Tuning
These sysctl changes seemed to make a small improvement, so I left them intact. This would need to be added to /etc/sysctl.conf.
# drbd tuning net.ipv4.tcp_no_metrics_save = 1 net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_wmem = 4096 87380 33554432 vm.dirty_ratio = 10 vm.dirty_background_ratio = 4
DRBD
DRBD is the tricky one. It doesn't always perform well despite what it's developers would like you to think.
DRBD Performance
- Our test RAID controller had no battery, but did have 128MB of ram
- DRBD flushes all writes to disk or uses barriers if you don't disable it. This is important for consistency but disks were not made to be used this way and it will not perform well in most circumstances.
- If you disable all flushing or barriers with no-md-flushes, no-disk-flushes, no-disk-barrier, DRBD performance will be nearly the native disk speed. However this means the DRBD device is more prone to corruption if it crashes, power goes out, etc...
- If you disable just meta-data flushing with no-md-flushes, performance is reasonable (about 80% native) and you still get some security. Meta-data flushing will make performance so bad that there is no point in using DRBD.
- If you have a battery backed RAID controller, disable all flushing and barriers. You will get near native performance and will not have to worry about data corruption
- Variable rate synchronization is excellent because it won't hurt DRBD performance and will use all bandwidth when it is free
- use-rle is a performance tweak that is enabled in the latest version be default, so I turned it on
- Basic tweaks like buffers, al-extents, etc... should all be enabled. They are well documented.
- Protocol C seems to perform almost the same as A and B (based on benchmarks). Protocol C provides the most protection.
- Our test systems had hard drives there were capable of write speeds very similar to that of 1Gb Ethernet transfer speeds. If faster disks/arrays were used, a faster link or bonding would be required for DRBD to keep up with writes.
Working Example
Working and well performing DRBD resource configuration
resource r0 {
net {
#on-congestion pull-ahead;
#congestion-fill 1G;
#congestion-extents 3000;
#sndbuf-size 1024k;
sndbuf-size 0;
max-buffers 8000;
max-epoch-size 8000;
}
disk {
#no-disk-barrier;
#no-disk-flushes;
no-md-flushes;
}
syncer {
c-plan-ahead 20;
c-fill-target 50k;
c-min-rate 10M;
al-extents 3833;
rate 35M;
use-rle;
}
startup { become-primary-on nfs01 ; }
protocol C;
device minor 1;
meta-disk internal;
on nfs01 {
address 192.168.0.1:7801;
disk /dev/sda6;
}
on nfs02 {
address 192.168.0.2:7801;
disk /dev/sda6;
}
}
Relevant section of /etc/fstab used with this configuration:
# DRBD, mounted by heartbeat /dev/drbd1 /mnt ext4 noatime,noauto,nobarrier 0 0
- 'nobarrier' makes a big different in performance (on my test systems) and still maintains filesystem integrity
- 'noatime' makes a small performance difference by disabling access time updates on every file read
- 'noauto' stops the init scripts (mount -a) from mounting it - heartbeat will manage this
Benchmarking
Before bothering with NFS or anything else, it is a good idea to make sure DRBD is performing well.
Benchmark tools
- atop - watches CPU load, IO load, IO throughput, network throughput, etc... for the whole system in one screen, run on both systems during your benchmarking to see what's going on
- iptraf - detailed network information, throughput, etc... if you need to dig further
- bonnie++ - performs many IO benchmarks, gives a good idea of actual disk performance. Must use a data set at least 2x larger than physical ram to be accurate
- postmark - great for over loading a system, performing many small write/reads/appends, directory creation, etc... good for testing NFS performance when everything is up. Gives you ops/sec performance results per test type
- dd - basic, initial benchmarking - see dd section
DD
There are some simple tests you can do to test performance of a storage device using DD. However, other tools should be used later for more accurate results (real world). When I'm benchmarking or trying to identify bottlenecks, I run atop on the same system in a separate terminal while dd is transferring data.
- Use direct access to write a sequentially to the filesystem (doesn't work on all filesystems)
dd if=/dev/zero of=testfile bs=100M count=20 oflag=direct
- Use regular access to write sequentially to the filesystem, and flush before exit
dd if=/dev/zero of=testfile bs=100M count=20 conv=fsync
- Use direct access to read sequentially from the filesystemto read sequentially from the filesystem (doesn't work on all filesystems)
dd if=testfile of=/dev/null bs=100M iflag=direct
- Use regular access to read sequentially from the filesystem. Drop system cache before doing this or you might just read the file from cache
dd if=testfile of=/dev/null bs=100M
- Drop system cache
sync
echo 3 > /proc/sys/vm/drop_caches
- Write directly to block device, bypassing filesystem. This will destroy data on the block device
dd if=/dev/zero of=/dev/sdXX bs=100M count=20 oflag=direct
- Read directly from block device, bypassing filesystem
dd if=/dev/sdXX of=/dev/null bs=100M count=20 oflag=direct
NFS Server
The only configuration was to '/etc/exports':
/mnt 192.168.3.0/24(rw,async,no_subtree_check,fsid=0)
- 'async' I found this particular setup performed much better (50%) with async rather than sync
- 'fsid=0' is a good thing to use in HA solutions. If all nodes use the same ID# (which is trivial) for the same mount, stale handles will be avoided after a fail over.
'/etc/idmapd.conf' may need to be adjusted to match your domain when using NFSv4 (on client and/or server)
NFS Client
In testing, I chose to use this command to mount NFS:
mount nfs:/mnt /testnfs -o rsize=32768,wsize=32768,hard,timeo=50,bg,actimeo=3,noatime,nodiratime
Explanation:
- rsize/wsize - set the read and write maximum block size to 32k, appropriate for the average file size of the customers data
- hard - cause processes accessing the NFS share to block for ever when it becomes unavailable unless killed with SIGKILL
- noatime - do not update file access times every time a file is read
- nodiratime - like noatime but for directories
- bg - continue NFS mount in the background, rather than blocking - mostly to prevent boot problems
- tcp - not specified because it's a default - if udp is used, data could be lost during a fail over, tcp will keep trying
- timeo - retry NFS requests after 5 seconds (specified in 10ths of a second)
Clustering
Heartbeat without Pacemaker was chosen. Pacemaker seemed too complex and difficult to manage for what was needed.
Heartbeat
It this test setup, heartbeat has 2 Ethernet connections to communicate between nodes. The first is the network/subnet/lan connection and the other is the DRBD direct crossover link. Having multiple connection paths is important so that one heartbeat node doesn't lose contact with the other. Once that happens, neither one knows which is master and the cluster becomes 'split-brained'.
STONITH
S.T.O.N.I.T.H = Shoot The Other Node In The Head
STONITH is the facility that Heartbeat uses to reboot a cluster node that is not responding. This is very important because heartbeat needs to know that the other node is not using DRBD (or other corruptible resources). If a node is really not responding at all, the other node will reboot it using STONITH, which uses IPMI (in the examples below) and then take over the resources.
When two nodes believe they are master (own the resources) it is called split-brain. This can lead to problems and sometimes data corruption. STONITH w/IPMI can protect against this.
Working Example
- Create below configuration files
- Install heartbeat (from repositories if possible)
- Start logd
- Start heartbeat
The ha.cf file defines the cluster and how its nodes interact.
/etc/ha.d/ha.cf:
# Give cluster 30 seconds to start initdead 30 # Keep alive packets every 1 second keepalive 1 # Misc settings traditional_compression off deadtime 10 deadping 10 warntime 5 # Nodes in cluster node nfs01 nfs02 # Use ipmi to check power status and reboot nodes stonith_host nfs01 external/ipmi nfs02 192.168.3.33 ADMIN somepwd lan stonith_host nfs02 external/ipmi nfs01 192.168.3.34 ADMIN somepwd lan # Use logd, configure /etc/logd.cf use_logd on # Don't move service back to preferred host when it comes up auto_failback off # If all systems are down, it's failure ping_group lan_ping 192.168.3.1 192.168.3.13 # Takover if pings (above) fail respawn hacluster /usr/lib/heartbeat/ipfail ##### Use unicast instead of default multicast so firewall rules are easier # nfs01 ucast eth0 192.168.3.32 ucast eth1 192.168.0.1 # nfs02 ucast eth0 192.168.3.31 ucast eth1 192.168.0.2
The haresources file describes resources provided by the cluster. It's format is: [Preferred node] [1st Service] [2nd Service]... services are started in the order they are listed and stopped in the reverse order. They will start on the preferred node when possible.
/etc/ha.d/haresources:
nfs01 drbddisk::r0 Filesystem::/dev/drbd1::/mnt::ext4 IPaddr2::192.168.3.30/24/eth0 nfs-kernel-server
The logd.conf file defines logging for heartbeat.
/etc/logd.conf:
debugfile /var/log/ha-debug logfile /var/log/ha-log syslogprefix linux-ha
Testing Fail-over
There are numerous tests you can perform. Try pinging the floating IP address while pulling cables, initiating heartbeat takover, killing heartbeat with SIGKILL, etc... But my favourite test is of the NFS service, the part that matters the most. /var/log/ha-debug will have lots of details about what heartbeat is doing during your tests.
Testing NFS Fail-over
- From another system, mount the NFS share from the cluster
- Use rsync --progress -av to start copying a large file (1-2 GB) to the share
- When the progress is 20%-30%, pull the network cable from the active node
- Rsync will lock up (as intended) due to NFS blocking
- After 5-10 seconds, the file should continue transferring until finished with no errors
- Do an md5 checksum comparison of the original file and the file on the NFS share
- Both files should be identical, if not, there was corruption of some kind
- Try the test again by reading from NFS, rather than writing to it