Using ATA Over Ethernet (AoE) on CentOS 7 (Initiator and Target)
This guide explains how you can set up an AoE target and an AoE initiator (client), both running CentOS 7. AoE stands for "ATA over Ethernet" and is a storage area network (SAN) protocol which allows AoE initiators to use storage devices on the (remote) AoE target using normal ethernet cabling. "Remote" in this case means "inside the same LAN" because AoE is not routable outside a LAN (this is a major difference compared to iSCSI). To the AoE initiator, the remote storage looks like a normal, locally-attached hard drive.
1 Preliminary Note
I'm using two CentOS 7 servers here:
- server1.example.com (Initiator): IP address 192.168.1.100
- server2.example.com (Target): IP address 192.168.1.101
2 Enable Additional Repositories
server1/server2:
First, we import the GPG keys for software packages:
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY*
Then we enable the EPEL repository on our CentOS system as lots of the packages that we are going to install in the course of this tutorial are not available in the official CentOS 7 repository:
yum -y install epel-release
yum -y install yum-priorities
Edit /etc/yum.repos.d/epel.repo...
nano /etc/yum.repos.d/epel.repo
... and add the line priority=10 to the [epel] section:
[epel] name=Extra Packages for Enterprise Linux 7 - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch failovermethod=priority enabled=1 priority=10 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 [...]
Then we update our existing packages on the system:
yum -y update
3 Loading the AoE Kernel Module on both Systems
server1/server2:
Before we start, we must make sure that the kernel supports AoE:
grep ATA_OVER /boot/config-`uname -r`
This should display something like this:
[root@server1 ~]# grep ATA_OVER /boot/config-`uname -r`
CONFIG_ATA_OVER_ETH=m
[root@server1 ~]#
This means that AoE was built as a kernel module. Let's check if the module is already loaded:
lsmod | grep aoe
If you get nothing back, this means it's not loaded. In this case we can load it as follows:
modprobe aoe
Let's check again if the module is loaded:
lsmod | grep aoe
[root@server1 ~]# lsmod | grep aoe
aoe 54175 0
[root@server1 ~]#
To have the module loaded automatically when the system boots, we add the following line to /etc/rc.local:
nano /etc/rc.local
[...] modprobe aoe [...]
4 Setting up the Server (Target) on Server 2
server2:
First, we set up the target (server2):
rpm -i http://download.opensuse.org/repositories/home:/cyberorg:/ltsp/CentOS_7/x86_64/vblade-22-3.1.x86_64.rpm
We can use unused logical volumes, image files, hard drives (e.g. /dev/sdb), hard drive partitions (e.g. /dev/sdb1) or RAID devices (e.g. /dev/md0) for the storage.
In this example, I will use an image file of 10GB that is located in the storage folder /storage.
mkdir /storage
dd if=/dev/zero of=/storage/storage1.img bs=1024k count=10000
This will create an image file /storage/storage1.img with a size of 10GB.
If you want to use logical volume instead, then you can create one with a size of 10GB named storage1 in the volume group vg0 like this:
lvcreate -L10G -n storage1 vg0
Now we export our storage device as follows:
vbladed 0 1 ens33 /storage/storage1.img
The first number (0) is the shelf number (major), the second (1) the slot number (minor), change these numbers to your liking. Each AoE device is identified by a couple major/minor which must be unique (if you are exporting multiple devices), with major between 0-65535 and minor between 0-255. The ens33 part tells vbladed which ethernet device to use, you can find out the details about your ethernet devices by running
ifconfig
).
To start the export automatically whenever you boot the target, open /etc/rc.local...
nano /etc/rc.local
... and add the following line to it (after the modprobe aoe line!):
[...]
vbladed 0 1 ens33 /storage/storage1.img
[...]
5 Setting Up the Client (Initiator) on Server 1
server1:
On server1, we install the initiator. There is no package for CentOS 7 available, but the one from centOS 6 works fine, so we will use that here.
rpm -i http://download.opensuse.org/repositories/system:/aoetools/CentOS_CentOS-6/x86_64/aoetools-36-1.2.x86_64.rpm
Now we check what AoE storage devices are available:
aoe-discover
The command
aoe-stat
should now show the storage devices:
[root@server1 ~]# aoe-stat
e0.1 10.485GB ens33 1024 up
[root@server1 ~]#
At this point we have a new block device available on the client box named /dev/etherd/e0.1. If we have a look at the /dev tree a new node appears:
ls -la /dev/etherd/
[root@server1 ~]# ls -la /dev/etherd/
total 0
drwxr-xr-x. 2 root root 160 Jun 30 14:49 .
drwxr-xr-x. 21 root root 3240 Jun 30 14:32 ..
c-w--w----. 1 root disk 152, 3 Jun 30 14:32 discover
brw-rw----. 1 root disk 152, 0 Jun 30 14:49 e0.1
cr--r-----. 1 root disk 152, 2 Jun 30 14:32 err
c-w--w----. 1 root disk 152, 6 Jun 30 14:32 flush
c-w--w----. 1 root disk 152, 4 Jun 30 14:32 interfaces
c-w--w----. 1 root disk 152, 5 Jun 30 14:32 revalidate
[root@server1 ~]#
To use that device, we must format it:
fdisk /dev/etherd/e0.1
[root@server1 ~]# fdisk /dev/etherd/e0.1
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xed572fd4.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): <-- n
Command action
e extended
p primary partition (1-4)
<-- p
Partition number (1-4): <-- 1
First sector (2048-20479999, default 2048): <-- ENTER
Using default value 1
Last sector, +sectors or +size{K,M,G} (2048-20479999, default 20479999): <-- ENTER
Using default value 20479999
Partition 1 of type Linux and of size 9.8 GiB is set
Command (m for help): <-- t
Selected partition 1
Hex code (type L to list codes): <-- L
0 Empty 1e Hidden W95 FAT1 80 Old Minix bf Solaris
1 FAT12 24 NEC DOS 81 Minix / old Lin c1 DRDOS/sec (FAT-
2 XENIX root 39 Plan 9 82 Linux swap / So c4 DRDOS/sec (FAT-
3 XENIX usr 3c PartitionMagic 83 Linux c6 DRDOS/sec (FAT-
4 FAT16 <32M 40 Venix 80286 84 OS/2 hidden C: c7 Syrinx
5 Extended 41 PPC PReP Boot 85 Linux extended da Non-FS data
6 FAT16 42 SFS 86 NTFS volume set db CP/M / CTOS / .
7 HPFS/NTFS 4d QNX4.x 87 NTFS volume set de Dell Utility
8 AIX 4e QNX4.x 2nd part 88 Linux plaintext df BootIt
9 AIX bootable 4f QNX4.x 3rd part 8e Linux LVM e1 DOS access
a OS/2 Boot Manag 50 OnTrack DM 93 Amoeba e3 DOS R/O
b W95 FAT32 51 OnTrack DM6 Aux 94 Amoeba BBT e4 SpeedStor
c W95 FAT32 (LBA) 52 CP/M 9f BSD/OS eb BeOS fs
e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi ee GPT
f W95 Ext'd (LBA) 54 OnTrackDM6 a5 FreeBSD ef EFI (FAT-12/16/
10 OPUS 55 EZ-Drive a6 OpenBSD f0 Linux/PA-RISC b
11 Hidden FAT12 56 Golden Bow a7 NeXTSTEP f1 SpeedStor
12 Compaq diagnost 5c Priam Edisk a8 Darwin UFS f4 SpeedStor
14 Hidden FAT16 <3 61 SpeedStor a9 NetBSD f2 DOS secondary
16 Hidden FAT16 63 GNU HURD or Sys ab Darwin boot fb VMware VMFS
17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fc VMware VMKCORE
18 AST SmartSleep 65 Novell Netware b8 BSDI swap fd Linux raid auto
1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fe LANstep
1c Hidden W95 FAT3 75 PC/IX be Solaris boot ff BBT
Hex code (type L to list codes): <-- 83
Command (m for help): <-- w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@server1 ~]#
Afterwards, there's a new device /dev/etherd/e0.1p1 which you can see in the output of
ls -l /dev/etherd/
[root@server1 ~]# ls -l /dev/etherd/
total 0
c-w--w----. 1 root disk 152, 3 Jun 30 14:32 discover
brw-rw----. 1 root disk 152, 0 Jun 30 14:54 e0.1
brw-rw----. 1 root disk 152, 1 Jun 30 14:54 e0.1p1
cr--r-----. 1 root disk 152, 2 Jun 30 14:32 err
c-w--w----. 1 root disk 152, 6 Jun 30 14:32 flush
c-w--w----. 1 root disk 152, 4 Jun 30 14:32 interfaces
c-w--w----. 1 root disk 152, 5 Jun 30 14:32 revalidate
[root@server1 ~]#
Now we create a filesystem on /dev/etherd/e0.1p1...
mkfs.ext4 /dev/etherd/e0.1p1
... and mount it for test purposes:
mount /dev/etherd/e0.1p1 /mnt
You should now see the new device in the outputs of...
mount
[root@server1 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=930336k,nr_inodes=232584,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188060k,mode=700)
/dev/etherd/e0.1p1 on /mnt type ext4 (rw,relatime,seclabel,data=ordered)
[root@server1 ~]#
... and
df -h
[root@server1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 28G 1.3G 27G 5% /
devtmpfs 909M 0 909M 0% /dev
tmpfs 919M 0 919M 0% /dev/shm
tmpfs 919M 8.6M 910M 1% /run
tmpfs 919M 0 919M 0% /sys/fs/cgroup
/dev/sda1 497M 192M 306M 39% /boot
tmpfs 184M 0 184M 0% /run/user/0
/dev/etherd/e0.1p1 9.5G 37M 9.0G 1% /mnt
[root@server1 ~]#
You can unmount it like this:
umount /mnt
To have the device mounted automatically at boot time, e.g. in the directory /storage, we create that directory...
mkdir /storage
... and add the following line to /etc/fstab:
nano /etc/fstab
[...] /dev/etherd/e0.1p1 /storage ext4 defaults,auto,_netdev 0 0
This alone isn't enough to have the device mounted at boot time because the AoE stuff gets loaded after /etc/fstab is read. Therefore we open /etc/rc.local...
nano /etc/rc.local
... and add the following lines to it (after the modprobe aoe line!):
[...] aoe-discover sleep 5 mount -a [...]
For test purposes, you can now reboot the system:
reboot
After the reboot, the device should be mounted:
mount
[root@server1 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=930336k,nr_inodes=232584,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188060k,mode=700)
/dev/etherd/e0.1p1 on /mnt type ext4 (rw,relatime,seclabel,data=ordered)
[root@server1 ~]#
df -h
[root@server1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 28G 1.3G 27G 5% /
devtmpfs 909M 0 909M 0% /dev
tmpfs 919M 0 919M 0% /dev/shm
tmpfs 919M 8.6M 910M 1% /run
tmpfs 919M 0 919M 0% /sys/fs/cgroup
/dev/sda1 497M 192M 306M 39% /boot
tmpfs 184M 0 184M 0% /run/user/0
/dev/etherd/e0.1p1 9.5G 37M 9.0G 1% /mnt
[root@server1 ~]#
6 Links
- AoE Protocol Definition: http://www.coraid.com/RESOURCES/AoE-Protocol-Definition
- CentOS: http://www.centos.org/