How To Build A Low Cost SAN
Krishna Kumar
April 9, 2009
1 OBJECTIVE
In today's world there is a obvious need of information sharing in every department and network storage can help us to achieve this most growing challenge. Here in this article we are focusing our concentration to make a San which has following features:
- Low cost and easily affordable
- Ensured Scalability
- High Reliability
- Easily Manageable
- High Performance
- Ensured Security
- High availability
2 Available Options For SAN
There are some options available to make a reliable San which is quite complex and expensive. These are iSCSI, NBD, ENBD and FIBER CHANNEL. iSCSI, NBD, ENBD works on TCP/IP layer which has much overhead. Luckily we have a protocol which can easily serve our purpose in a affordable cost and less overhead. All we typically need is some dual-port Gig-E cards, a GiG-Ethernet switch and some disks. This is a very simple and lightweight protocol and it is known as ATA OVER ETHERNET. AoE comes with linux kernel as a kernel module. AoE does not rely on network layers above Ethernet, such as the IP and TCP that iSCSI requires. While this makes AoE potentially faster than iSCSI, with less load on the host to handle the additional protocols, it also means that AoE is not routable outside a LAN. AoE is intended for SANs only. In this regard it is more comparable to Fiber Channel over Ethernet than iSCSI. It export block devices (SATA HARD DISKS) over the network with a very high throughput when coupled with a quality Ethernet switch. A qaulity Ethernet switch can maximize throughput and minimize collisions through integrity checking and packet ordering. While using AoE in a large scalable enterprise environment, we can take the help of RED HAT cluster aware tools like CLVM, GFS, DLM, DRBD, HEARTBEAT etc-etc.
3 Cost Comparison among AoE, FC & iSCSI
Cost Comparison
| Technology | Speed | Server Interface | Switch | Cabling | Storage/TB |
| AoE | 2Gb | $99 | $15-$30 | $25-$35 | $400-$500 |
| iSCSI | 1Gb | $500-$1000 | $400-$600 | $25-$35 | $1000-$5000 |
| Fiber Channel | 4Gb | $1200-$2000 | $800-$3600 | $175-$225 | $4000-$10000 |
4 Comparison of AoE vs iSCSI
4.1 These are the following advantages of AoE over iSCSI
- AoE is cheap and simpler software stack. The advantage of AoE is that you don't have the overhead of translating ATA to SCSI then back to ATA if you are using ATA drives. So there is a performance pickup.
- Server processing load for iSCSI is much higher than AoE for equivalent throughput. AoE can spare processing cycles. iSCSI requires TCP/IP and its requisite complexity.
- AoE is not a routable protocol. Therefore it provides you inherent security.
- AoE Ethernet frames are passed by standard switches.
- AoE and iSCSI both have initiator support for Windows and Linux.
4.2 Disadvantages of AoE over iSCSI
- If you need features such as encryption, routability and user-based access in the storage protocol, iSCSI is a better choice.
- AoE is not much suitable for critical enterprise applications. AoE is not as scalable as iSCSI or Fiber Channel when you consider location i.e., with Fiber Channel and iSCSI, you can scale your storage throughout. This is primarily due to the inability of AoE to route AoE traffic.
- ATA disks are not as reliable as their SCSI counterparts.
5 Available AoE Targets
There are following AoE Targets (server) available on GPL:
- Kvblade
- Aoeserver
- Vblade-Kernel
- Vblade
- Ggaoed
- Qaoed
6 Feature Comparison of available AoE Targets
You can export your block-devices over the network by any of the available tar- gets. But the thing is how we can export our block devices in a much configured and manageable way so that it can help us to achieve our targets. These are the following features available by which you can configure your block devices, either by command-line or by configuration file. These are the acronym followed in this table for AoE targets:
- KV - Kvblade
- AOES - Aoeserver
- VB-KER - Vblade-kernel
- VB - Vblade
- GGOLD - Ggaoed base version
- GGNW - Ggaoed updated version
- QD - Qaoed base version
- SD - Sqaoed (ported version of qaoed on Solaris 10)
Small description of all the features are given in terms and terminology section.
Features
| FEATURE | KV | AOES | VB-KER | VB | GGOLD | GGNW | QD | SD |
| Shelf | Y | Y | Y | Y | Y | Y | Y | Y |
| Slot | Y | Y | Y | Y | Y | Y | Y | Y |
| Interface | Y | Y | Y | Y | Y | Y | Y | Y |
| Device-path | Y | Y | Y | Y | Y | Y | Y | Y |
| Conf-file | N | N | N | N | Y | Y | Y | N |
| MTU | N | N | N | N | Y | Y | Y | N |
| Mac-filtering | N | Y | N | Y | Y | Y | Y | N |
| ACL-listing | N | N | N | N | Y | Y | Y | N |
| Buffer Count | N | N | Y | Y | Y | Y | N | N |
| Sectors | N | N | Y | N | N | N | N | N |
| Queing | N | N | N | N | Y | Y | N | N |
| Logging-info | N | N | N | N | Y | Y | Y | N |
| Direct-Mode | N | N | N | Y | Y | Y | Y | N |
| Sync-Mode | N | N | N | Y | N | N | N | N |
| Read-only Mode | N | N | N | Y | Y | Y | Y | N |
| UUID | N | N | N | N | Y | Y | Y | N |
| Write Cache | N | N | N | N | N | N | Y | N |
| Policy | N | N | N | N | Y | Y | Y | N |
| Trace-i/o | N | N | N | N | Y | Y | N | N |
| Jumbo-Frames | Y | Y | Y | Y | Y | Y | Y | Y |
| On GPL | Y | Y | Y | Y | Y | Y | Y | Y |
| Reliability | Low | Med | Low | High | High | Med | High | Med |
| Usability | Low | Med | Low | High | High | Med | High | Med |