There is a new version of this tutorial available for CentOS 7.2.

Using ATA Over Ethernet (AoE) On CentOS 6.3 (Initiator And Target)

This guide explains how you can set up an AoE target and an AoE initiator (client), both running CentOS 6.3. AoE stands for "ATA over Ethernet" and is a storage area network (SAN) protocol which allows AoE initiators to use storage devices on the (remote) AoE target using normal ethernet cabling. "Remote" in this case means "inside the same LAN" because AoE is not routable outside a LAN (this is a major difference compared to iSCSI). To the AoE initiator, the remote storage looks like a normal, locally-attached hard drive.

 

1 Preliminary Note

I'm using two CentOS 6.3 servers here:

  • server1.example.com (Initiator): IP address 192.168.0.100
  • server2.example.com (Target): IP address 192.168.0.101

 

2 Enable Additional Repositories

server1/server2:

First we import the GPG keys for software packages:

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY*

Then we enable the EPEL6 repository on our two CentOS systems:

rpm --import https://fedoraproject.org/static/0608B895.txt
cd /tmp
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
rpm -ivh epel-release-6-7.noarch.rpm
yum install yum-priorities

Edit /etc/yum.repos.d/epel.repo...

vi /etc/yum.repos.d/epel.repo

... and add the line priority=10 to the [epel] section:

[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
priority=10
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
[...]

The vblade package is available only from the EPEL5 repository, therefore we must add the following section to /etc/yum.repos.d/epel.repo, but only on server2:

server2:

vi /etc/yum.repos.d/epel.repo

Make sure to use priority=100 and to disable GPG checks with gpgcheck=0:

[...]
[epel5]
name=Extra Packages for Enterprise Linux 5 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/5/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-5&arch=$basearch
failovermethod=priority
enabled=1
priority=100
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

 

3 Loading The aoe Kernel Module On Both Systems

server1/server2:

Before we start, we must make sure that the the kernel supports AoE:

grep ATA_OVER /boot/config-`uname -r`

This should display something like this:

[root@server2 ~]# grep ATA_OVER /boot/config-`uname -r`
CONFIG_ATA_OVER_ETH=m
[root@server2 ~]#

This means that AoE was built as a kernel module. Let's check if the module is already loaded:

lsmod | grep aoe

If you get nothing back, this means it's not loaded. In this case we can load it as follows:

modprobe aoe

Let's check again if the module is loaded:

lsmod | grep aoe
[root@server2 ~]# lsmod | grep aoe
aoe                    26466  0
[root@server2 ~]#

To have the module loaded automatically when the system boots, we add the following line to /etc/rc.local:

vi /etc/rc.local
[...]
modprobe aoe
[...]

 

4 Setting Up The Target (server2)

server2:

First we set up the target (server2):

yum install vblade

We can use unused logical volumes, image files, hard drives (e.g. /dev/sdb), hard drive partitions (e.g. /dev/sdb1) or RAID devices (e.g. /dev/md0) for the storage. In this example I will create a logical volume of 20GB named storage1 in the volume group vg_server2:

lvcreate -L20G -n storage1 vg_server2

(If you want to use an image file, you can create it as follows:

mkdir /storage
dd if=/dev/zero of=/storage/storage1.img bs=1024k count=20000

This creates the image file /storage/storage1.img with a size of 20GB.

)

Now we export our storage device as follows:

vbladed 0 1 eth0 /dev/vg_server2/storage1

The first number (0) is the shelf number (major), the second (1) the slot number (minor), change these numbers to your liking. Each AoE device is identified by a couple major/minor which must be unique (if you are exporting multiple devices), with major between 0-65535 and minor between 0-255. The eth0 part tells vbladed which ethernet device to use (if you ethernet device is eth1, then use eth1 - you can find out about your ethernet devices by running

ifconfig

).

To start the export automatically whenever you boot the target, open /etc/rc.local...

vi /etc/rc.local

... and add the following line to it (after the modprobe aoe line!):

[...]
vbladed 0 1 eth0 /dev/vg_server2/storage1
[...]

 

5 Setting Up The Initiator (server1)

server1:

On server1, we install the initiator:

yum install aoetools

Now we check what AoE storage devices are available:

aoe-discover

The command

aoe-stat

should now show the storage devices:

[root@server1 ~]# aoe-stat
      e0.1        21.474GB   eth0 up
[root@server1 ~]#

At this point we have a new block device available on the client box named /dev/etherd/e0.1. If we have a look at the /dev tree a new node appears:

ls -la /dev/etherd/
[root@server1 ~]# ls -la /dev/etherd/
total 0
drwxr-xr-x.  2 root root     160 Dec 11 16:24 .
drwxr-xr-x. 20 root root    3620 Dec 11 16:00 ..
c-w--w----.  1 root disk 152,  3 Dec 11 16:00 discover
brw-rw----.  1 root disk 152, 16 Dec 11 16:24 e0.1
cr--r-----.  1 root disk 152,  2 Dec 11 16:00 err
c-w--w----.  1 root disk 152,  6 Dec 11 16:00 flush
c-w--w----.  1 root disk 152,  4 Dec 11 16:00 interfaces
c-w--w----.  1 root disk 152,  5 Dec 11 16:00 revalidate
[root@server1 ~]#

To use that device, we must format it:

fdisk /dev/etherd/e0.1

[root@server1 ~]# fdisk /dev/etherd/e0.1
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xed572fd4.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.


The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): <-- n
Command action
   e   extended
   p   primary partition (1-4)
<-- p
Partition number (1-4): <-- 1
First cylinder (1-2610, default 1): <-- ENTER
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): <-- ENTER
Using default value 2610

Command (m for help): <-- t
Selected partition 1
Hex code (type L to list codes): <-- L

 0  Empty           1e  Hidden W95 FAT1 80  Old Minix       bf  Solaris
 1  FAT12           24  NEC DOS         81  Minix / old Lin c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          82  Linux swap / So c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  83  Linux           c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c7  Syrinx
 5  Extended        41  PPC PReP Boot   85  Linux extended  da  Non-FS data
 6  FAT16           42  SFS             86  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi ee  GPT
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a5  FreeBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a6  OpenBSD         f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       a9  NetBSD          f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
Hex code (type L to list codes): <-- 83

Command (m for help): <-- w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@server1 ~]#

Afterwards, there's a new device /dev/etherd/e0.1p1 which you can see in the output of

ls -l /dev/etherd/
[root@server1 ~]# ls -l /dev/etherd/
total 0
c-w--w----. 1 root disk 152,  3 Dec 11 16:00 discover
brw-rw----. 1 root disk 152, 16 Dec 11 16:27 e0.1
brw-rw----. 1 root disk 152, 17 Dec 11 16:27 e0.1p1
cr--r-----. 1 root disk 152,  2 Dec 11 16:00 err
c-w--w----. 1 root disk 152,  6 Dec 11 16:00 flush
c-w--w----. 1 root disk 152,  4 Dec 11 16:00 interfaces
c-w--w----. 1 root disk 152,  5 Dec 11 16:00 revalidate
[root@server1 ~]#

Now we create a filesystem on /dev/etherd/e0.1p1...

mkfs.ext4 /dev/etherd/e0.1p1

... and mount it for test purposes:

mount /dev/etherd/e0.1p1 /mnt

You should now see the new device in the outputs of...

mount
[root@server1 ~]# mount
/dev/mapper/vg_server1-LogVol00 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/etherd/e0.1p1 on /mnt type ext4 (rw)
[root@server1 ~]#

... and

df -h
[root@server1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_server1-LogVol00
                      9.7G  1.7G  7.5G  18% /
tmpfs                 499M     0  499M   0% /dev/shm
/dev/sda1             504M   39M  440M   9% /boot
/dev/etherd/e0.1p1     20G  151M   19G   1% /mnt
[root@server1 ~]#

You can unmount it like this:

umount /mnt

To have the device mounted automatically at boot time, e.g. in the directory /storage, we create that directory...

mkdir /storage

... and add the following line to /etc/fstab:

vi /etc/fstab
[...]
/dev/etherd/e0.1p1       /storage        ext4    defaults,auto,_netdev 0 0

This alone isn't enough to have the device mounted at boot time because the AoE stuff gets loaded after /etc/fstab is read. Therefore we open /etc/rc.local...

vi /etc/rc.local

... and add the following lines to it (after the modprobe aoe line!):

[...]
aoe-discover
sleep 5
mount -a
[...]

For test purposes, you can now reboot the system:

reboot

After the reboot, the device should be mounted:

mount
[root@server1 ~]# mount
/dev/mapper/vg_server1-LogVol00 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/etherd/e0.1p1 on /storage type ext4 (rw,_netdev)
[root@server1 ~]#
df -h
[root@server1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_server1-LogVol00
                      9.7G  1.7G  7.5G  18% /
tmpfs                 499M     0  499M   0% /dev/shm
/dev/sda1             504M   39M  440M   9% /boot
/dev/etherd/e0.1p1     20G  151M   19G   1% /storage
[root@server1 ~]#

 

6 Links

Share this page:

0 Comment(s)