Low Cost SAN - Page 2

5  Low level Building Block of SAN

This section provides brief description of low level building block of SAN. The very first thing comes in our mind is what hard disk we are using and what are the available protocol for proper communication (in terms of read and write operation over network) exists. Of course the corresponding support in OS is also required at protocol level. Choosing of a reliable protocol is necessary according to the corresponding hard disk.

Fiber Channel is much expensive and it has much extra overhead in terms of cost and resources. So, we have three choices to export our block devices over network: AoE, NBD (ENBD/GNBD), and iSCSI. The brief descriptions of these three are as follows:

•  AoE: ATA over Ethernet ( AoE ) is a network protocol developed by the Brantley Coile Company, designed for simple, high-performance access of SATA storage devices over Ethernet networks. It gives the possibility to build SANs with low-cost, standard technologies. AoE does not rely on network layers above Ethernet, such as IP and TCP. In this regard it is more comparable to Fiber Channel over Ethernet than iSCSI. While the non-routability means AoE cannot be accessed over the Internet or other IP networks, the feature makes AoE more lightweight (with less load on the host), easier to implement, provides a layer of inherent security, and offers higher performance. Its support is available on Linux, Windows, Mac OS X, Free BSD and for plan 9 from Bell Labs.

•  NBD: The Linux Network Block Device (NBD) is a device driver extension to the Linux kernel. With the NBD device driver you can create a TCP/IP network connection between your local Linux system and a server program on a remote (not necessarily Linux) computer. But NBD has some limitations in terms of read/write operation and using NBD as a root file system.

•  iSCSI: iSCSI is Internet SCSI (Small Computer System Interface), an Internet Protocol (IP)-based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. The protocol allows clients (called initiators ) to send SCSI commands (CDBs) to SCSI storage devices ( targets ) on remote servers. ISCSI requires TCP offload Engine and Host Bus Adapter. It has a large number of OS support available like Linux, Windows, HP-UX, AIX, NetWare etc-etc.

If we are thinking in terms of Low cost San having less overhead, then AoE is a good choice for us. The advantage of AoE is that you don't have the overhead of translating ATA to SCSI and then back to ATA (if you are using ATA drives). So there is a performance pickup. Server processing load for iSCSI is much higher than AoE for equivalent throughput. AoE can spare processing cycles. iSCSI requires TCP/IP and its requisite complexity.


6  Low level Components (AoE)

This section provides brief description of the low level components which are available on FOSS and in market. Since we are focusing on Low Cost SAN and we have to face all the storage networking challenges of clustering, so when it comes to choose an OS, we can go for Centos. Centos 5.2 is almost equivalent to RHEL5 having inherent RHCS (redhat clustering suite) and virtualization facility. We have to attach more and more number of hard disk having much capacity, and we have PCI card/ raid controller card available having 24 ports and 2 TB size of each disk. Therefore we have following table for our low level components:




SATA disks of >= 2TB capacity

PCI Card/ Raid Controller

24 port SATA controller PCI express Card, SATA || raid controller card

Nic Card / high quality Switch

Gigabyte multiport networking switch & gigabyte multiport nic card ( having jumbo frame support )


Linux/Windows (Preferably Centos 5.2)




7  High Level Building Block of SAN

This section provides brief description about high level building blocks of SAN. When we think to make our SAN more roust and to perform against all the available challenges of storage networking in terms of software then three things comes in our mind which are as follows :

•  Protocol: A reliable protocol is necessary. We can go for AoE. iSCSI and HyperSCSI can also be a choice.

•  Drivers (Targets & Initiators): Client side and server side drivers are necessary to export the block devices over network and to access them on client side. Generally on client side, there is a kernel module available and on server side it can be a kernel module as well as a user space application. These drivers are known as targets (on server side) and initiators (on client side).

•  HA Softwares: Redhat cluster suite can be used in many configurations in order to provide high availability, scalability, load balancing, file sharing, and high performance in SAN.


8  High level Components (AoE)

This section provides brief description of available high level AoE components like targets, initiators and HA softwares. Following table gives us the guideline to make a reliable high level building block of SAN.






Vblade, Ggaoed, Qaoed ( on GPL )


AoE driver, WinAoE driver ( on GPL )

HA Softwares

RHCS suite, DRBD, Heartbeat ( on FOSS )


8.1  Targets (AoE)

This section gives brief description about available AoE targets. The highly reliable and highly configurable AoE targets are Vblade, Ggaoed and Qaoed. Ggaoed and Qaoed are more configurable than Vblade, while Vblade is quite simple and can be easily ported on any platform. These three targets are user space targets. Following diagram shows the combined view of these targets:

Apart from above mentioned targets, some other targets are also available for AoE and they all are on GPL. Kvblade, Vblade-kernel and Aoeserver are kernel modules while rest others (vblade, Ggaoed, Qaoed and Sqaoed) are user space targets. Following table defines these targets.




Vblade is a software-based AoE target, a virtual EtherDrive Blade. It exports local block storage to hosts on an ethernet local area network. Hosts with an ATA over Ethernet (AoE) initiator, like the aoe driver for Linux, can then access the storage over the ethernet. It is available for Linux, FreeBSD and for Plan 9 of bell labs.


Ggaoed is an AoE ( ATA over Ethernet ) target implementation for Linux. It utilizes Linux kernel AIO, memory mapped sockets and other Linux features to provide the best performance. It requires Linux kernel 2.6.22 or greater. It's currently available for Linux only.


Qaoed is a multithreaded ATA over Ethernet storage target that is easy to use and highly configurable. It's available for Linux.


Aoeserver is an in-kernel Ata Over Ethernet Storage target driver used to emulate a Coraid EtherDriver Blade. It is partly based on vblade and the aoe-client from the Linux 2.6-kernel. It uses procfs

to control and command this target.



Kvblade is a kernel module implementing the target side of the AoE protocol. Users can command the module through sysfs to export block devices on specified network interfaces. The loopback device should be used as an intermediary for exporting regular files with kvblade.


Vblade-kernel is an AoE target emulator implemented as a kernel module for Linux 2.6.* kernels.


Qaoed is now ported on Solaris 10. Fubra people have done it and they call Qaoed as Sqaoed. But, the new Sqaoed has currently not coming with its configuration file. It is operated via command line.

Share this page:

0 Comment(s)

Add comment


From: krishna kumar at: Nov 20, 2009