Setup ATA over Ethernet (AoE) on Debian 8 (Initiator and Target)

This tutorial shows you how to setup an AoE client (initiator) and server (target) on Debian 8 /Jessie). The term AoE stands for "ATA over Ethernet" which is a storage area network (SAN) protocol that allows AoE clients to use storage devices on the (remote) AoE server over a normal ethernet network. "Remote" in this case means "inside the same LAN" because AoE is not routable outside a LAN (this is a major difference compared to iSCSI). To the AoE client (initiator), the remote storage looks like a normal, locally-attached hard drive.

 

1 Preliminary Note

I'm using two Debian 8 servers here:

  • server1.example.com (Initiator): IP address 192.168.1.100
  • server2.example.com (Target): IP address 192.168.1.101

 

2 Load the AoE Kernel Module on both systems

server1/server2:

As a first step, we have to ensure that the Kernel on our servers supports ATA Over Ethernet. Run the following command as root user.

grep ATA_OVER /boot/config-`uname -r`

This should display something like this:

[email protected]:/tmp# grep ATA_OVER /boot/config-`uname -r`
CONFIG_ATA_OVER_ETH=m
[email protected]:/tmp#

This means that AoE was built in as a kernel module. Now we will check if the module is loaded:

lsmod | grep aoe

If you get nothing back, this means it's not loaded. In this case, we can load it as follows:

modprobe aoe

Let's check again if the module is loaded:

lsmod | grep aoe

[email protected]:/tmp# lsmod | grep aoe
aoe 51917 0
[email protected]:/tmp#

To have the module loaded automatically when the system boots, we add the aoe module to /etc/modules:

nano /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

aoe

 

3 Setting up the Target (server2)

server2:

First, we set up the AoE target (server2):

apt-get install vblade

We can use unused logical volumes, image files, hard drives (e.g. /dev/sdb), hard drive partitions (e.g. /dev/sdb1) or RAID devices (e.g. /dev/md0) for the storage. In this example, I will use an image file of 20GB that is stored in the folder /storage.

mkdir /storage
dd if=/dev/zero of=/storage/storage1.img bs=1024k count=20000

This creates the image file /storage/storage1.img with a size of 20GB.

If you want to use logical volume instead, then you can create one with a size of 20GB named storage1 in the volume group vg0 like this:

lvcreate -L20G -n storage1 vg0

Now we export our storage device as follows:

vbladed 0 1 eth0 /storage/storage1.img

The first number (0) is the shelf number (major), the second (1) the slot number (minor), change these numbers to your liking. Each AoE device is identified by a couple major/minor which must be unique (if you are exporting multiple devices), with major between 0-65535 and minor between 0-255. The eth0 part tells vbladed which ethernet device to use (if you ethernet device is eth1, then use eth1 - you can find out about your ethernet devices by running

ifconfig

).

To start the export automatically whenever you boot the target, open /etc/rc.local...

nano /etc/rc.local

... and add the following line to it (before the exit 0 line):

[...]
/usr/sbin/vbladed 0 1 eth0 /storage/storage1.img
[...]

4 Setting up the AoE Client / Initiator (server1)

server1:

On server1, we install the initiator:

apt-get install aoetools

Now we check what AoE storage devices are available:

aoe-discover

Don't worry, the command will not show any output. The command:

aoe-stat

should now show the storage devices:

[email protected]:/tmp# aoe-stat
e0.1 20.971GB eth0 1024 up
[email protected]:/tmp#

At this point we have a new block device available on the client box named /dev/etherd/e0.1. If we have a look at the /dev tree a new node appears:

ls -la /dev/etherd/

[email protected]:/tmp# ls -la /dev/etherd/
total 0
drwxr-xr-x 2 root root 160 Mar 22 08:46 .
drwxr-xr-x 19 root root 3160 Mar 22 08:34 ..
c-w--w---- 1 root disk 152, 3 Mar 22 08:34 discover
brw-rw---- 1 root disk 152, 0 Mar 22 08:46 e0.1
cr--r----- 1 root disk 152, 2 Mar 22 08:34 err
c-w--w---- 1 root disk 152, 6 Mar 22 08:34 flush
c-w--w---- 1 root disk 152, 4 Mar 22 08:34 interfaces
c-w--w---- 1 root disk 152, 5 Mar 22 08:34 revalidate
[email protected]:/tmp#

To use that /dev/etherd/e0.1 device, we must format it:

fdisk /dev/etherd/e0.1

[email protected]:/tmp# fdisk /dev/etherd/e0.1

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x2922f0be.

Command (m for help): <-- n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): <-- p
Partition number (1-4, default 1): <-- 1
First sector (2048-40959999, default 2048): <-- ENTER
Last sector, +sectors or +size{K,M,G,T,P} (2048-40959999, default 40959999): <-- ENTER

Created a new partition 1 of type 'Linux' and of size 19.5 GiB.

Command (m for help): <-- w

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Now we create a filesystem on /dev/etherd/e0.1p1...

mkfs.ext4 /dev/etherd/e0.1p1

[email protected]:/tmp# mkfs.ext4 /dev/etherd/e0.1p1
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 5119744 4k blocks and 1281120 inodes
Filesystem UUID: 2342cd83-bd45-4975-96c0-b0f366b73778
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

... and mount it for test purposes:

mount /dev/etherd/e0.1p1 /mnt

You should now see the new device in the outputs of...

mount

[email protected]:/tmp# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=125556,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=204220k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/etherd/e0.1p1 on /mnt type ext4 (rw,relatime,data=ordered)
[email protected]:/tmp#

... and

df -h

[email protected]:/tmp# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 57G 1.1G 54G 2% /
udev 10M 0 10M 0% /dev
tmpfs 200M 4.6M 195M 3% /run
tmpfs 499M 0 499M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 499M 0 499M 0% /sys/fs/cgroup
/dev/etherd/e0.1p1 20G 44M 19G 1% /mnt
[email protected]:/tmp#

You can unmount it like this:

umount /mnt

To have the device mounted automatically at boot time, e.g. in the directory /storage, we create that directory...

mkdir /storage

... and add the following line to /etc/fstab:

nano /etc/fstab

[...]
/dev/etherd/e0.1p1       /storage        ext4    defaults,auto,_netdev 0 0

This alone isn't enough to have the device mounted at boot time because the AoE stuff gets loaded after /etc/fstab is read. Therefore we open /etc/rc.local...

nano /etc/rc.local

... and add the following lines to it (before the exit 0 line):

[...]
aoe-discover
sleep 5
mount -a
[...]

For test purposes, you can now reboot the system:

reboot

After the reboot, the device should be mounted:

mount

[email protected]:/tmp# mount
[...snip...]
/dev/etherd/e0.1p1 on /storage type ext4 (rw,relatime,data=ordered)
[...snip...]

df -h

[email protected]:/tmp# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 57G 1.1G 54G 2% /
udev 10M 0 10M 0% /dev
tmpfs 200M 4.6M 195M 3% /run
tmpfs 499M 0 499M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 499M 0 499M 0% /sys/fs/cgroup
/dev/etherd/e0.1p1 20G 44M 19G 1% /storage

Now we can do a test write on the mounted partition:

touch /storage/test.txt

The check with ls command if the file has been written:

ls -la /storage

[email protected]:/tmp# ls -la /storage
total 24
drwxr-xr-x 3 root root 4096 Mar 22 09:06 .
drwxr-xr-x 23 root root 4096 Mar 22 09:05 ..
drwx------ 2 root root 16384 Mar 22 09:00 lost+found
-rw-r--r-- 1 root root 0 Mar 22 09:06 test.txt
[email protected]:/tmp#

The test.txt file has been written sucessfully on the volume that we mounted from server2.

Till Brehm

About Till Brehm

Over 20 years experience as Software Developer and Linux System Administrator. Till Brehm is the founder and lead developer of the ISPConfig Hosting Control Panel software (since 2000) and he founded HowtoForge in 2005 as a place to share Linux knowledge with other Linux enthusiasts.

Share this page:

Suggested articles

1 Comment(s)

Add comment

Comments

By: carlo

Thanks, very interesting. I did not even know about ata_over_eth !