Using ATA Over Ethernet On Debian Etch

Imagine you have a machine with all of its disk full and another with unused gigabytes, and you don't want to move the data from one to the other. Why not using the second's disk on the first, you can do it with iSCSI but you can do it with ATA over Ethernet (AoE) too. It's the second method I'll explain in this article. All of this was made with two computers running Debian Etch.

Prepare The Kernel

First check if your running kernel has AoE, the config option name is CONFIG_ATA_OVER_ETH, have a look at fig1, my kernel has AoE as a module:

host:/# grep ATA_OVER /boot/config-`uname -r`

CONFIG_ATA_OVER_ETH=m
host:/#

If not, configure your kernel and activate AoE in core or in module like you prefer:

Device Drivers -->
  |- Block Devices --->
      |-  <m> ATA over Ethernet support

Ok now you have a kernel with AoE, just load the aoe module:

host:/# modprobe aoe

You can check your syslog to be sure AoE is available:

host:/#tail /var/log/syslog

Oct 10 11:54:07 host kernel: aoe: aoe_init: AoE v22 initialised.
host:/#

Now we'll call the client 'client' and the server 'server', funny isn't it?
In SAN vocabulary we call the client 'initiator' and the server 'target', I prefer to continue using simplest terms.

The Server Side (Target)

First we need to install the vblade package:

server:/# apt-get install vblade

Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  vblade
[...]
Unpacking vblade (from .../archives/vblade_11-1_i386.deb) ...
Setting up vblade (11-1) ...
server:/#

On our server we'll export the /dev/sdd5 partition which has a size of 5GB, exporting a block device is easy to do:

server:/# vbladed 0 1 eth0 /dev/sdd5

Some explanation about this command, each AoE device is identified by a couple Major/Minor, with major between 0-65535 and minor between 0-255. AoE is based just over Ethernet on the OSI models so we need to indicate which ethernet card we'll use.

In this example we export /dev/sdd5 with a major value of 0 and minor if 1 on the eth0 interface.

We are ready to use our partition on the network!

Client Side (Initiator)

The client needs the aoe kernel module too, so prepare your kernel as we saw.

The userland tools are present in the package aoetools:

client:/# apt-get install aoetools

Now discover what we can use over our network:

client:/# aoe-discover

client:/# aoe-stat

      e0.1         5.000GB   eth0 up
client:/#

At this point we have a new block device available on the client box named /dev/etherd/e0.1. If we have a look at the /dev tree a new node appears:

client:/# ls -al /dev/etherd/

total 4
drwxr-xr-x  2 root root     140 2007-10-10 13:30 .
drwxr-xr-x 16 root root   14660 2007-10-10 13:30 ..
c-w--w----  1 root disk 152,  3 2007-10-10 13:30 discover
brw-rw----  1 root disk 152, 16 2007-10-10 13:30 e0.1
cr--r-----  1 root disk 152,  2 2007-10-10 13:30 err
c-w--w----  1 root disk 152,  4 2007-10-10 13:30 interfaces
-rw-r--r--  1 root root       5 2007-10-10 13:00 revalidate

How To Use It

Simply make a filesystem on your block device like:

client:/# mkfs.ext3 /dev/etherd/e0.1

and use it like you do with your /dev/hd* or /dev/sd*. The only difference is that the block device is over the network!