How To Build A Low Cost SAN - Page 3

8 AoE Initiator Setup

On client side you must have aoe driver and aoe-tools installed on your system (refer to section AoE Tools and Commands). You will find the corresponding exported block devices in your /dev/etherd directory. You just have to run the following command:

[[email protected] trunk]# modprobe aoe
[[email protected] trunk]# lsmod j grep aoe
[[email protected] trunk]# aoe-stat
[[email protected] trunk]# ls -l /dev/etherd/
[[email protected] trunk]# mkfs.ext3 /dev/etherd/e0.0
[[email protected] trunk]# mount /dev/etherd/e0.0 /mnt

So, finally you have access on your exported block device. Now you are free to do any type of read, write operation on this.

9 AoE Target Performance Graph

9.1 Hardware Used for Performance Measurement

In this experiment I have used following hardware to measure the disk i/o:

• 3 com super stack 24 port gigabyte switch
• gigabyte five port nic card
160 GB Sata hard disk ? Intel(R) Xeon(R) CPU E5450 @ 3.00 GHZ
• 24 GB RAM

My network supports following packet size:

[[email protected] ]# ifconfig eth0 mtu 9000

I have used FC7 and FC10 fedora OS because its' not possible to compile all the targets on same kernel.

9.2 Techniques to measure the disk i/o

After doing successful setup of all the available AoE targets, now its time to measure the disk i/o of exported block device. The available options to measure the i/o of block devices are hdparam, iostat, dd and some i/o tools like fio, bonnie, iozone, iometer etc.

9.3 Performance Graph with Fio

Here we have used fio (the i/o measurement tool) and plotted the performance graph. I have kept following configuration in surface-scan file:

[global]
ethread=1
bs=128k
direct=1
ioengine=sync
verify=meta
verify pattern=0xaa555aa5
verify interval=512
[write-phase]
filename=/dev/etherd/e0.1 ; or use a full disk, for example /dev/sda
rw=write
fill device=1
do verify=0
[verify-phase]
stonewall
create serialize=0
filename=/dev/etherd/e0.1
rw=read
do verify=1
bs=128k

After setting the appropriate parameter in this file, just run the following com- mand against all the available aoe targets to collect the required data for fol- lowing graph:

[[email protected] ]# fio surface-scan

The performance graph of AoE targets with fio in case of write operation is as follows:

Here X-axis denotes block-size in kilo-bytes and Y-axis denotes throughput in KB/sec (kilo-byte per second).

The performance graph of AoE targets with fio in case of read operation is as follows:

9.4 Performance Graph with Bonnie-64

We can setup bonnie-64 by following commands:

[[email protected] bonnie-64-read-only]# mount /dev/etherd/e0.0 /mnt
[[email protected] bonnie-64-read-only]#./Bonnie -d /mnt/ -s 128

The performance graph of AoE targets with bonnie in case of write operation is as follows:

Here X-axis denotes file-size in mega-bytes and Y-axis denotes throughput in M/sec.

10 AoE Tools & commands

There are some tools and commands available to analyze aoe-traffic on the net- work.Run the following commands to download and install these tools:

[[email protected] kk]# wget http://support.coraid.com/support/linux/aoe6-70.tar.gz
[[email protected] kk]# tar -xzvf aoe/root/Documents/Documents/diagram2.eps6- 70.tar.gz
[[email protected] kk]# cd aoe6-70
[[email protected] aoe6-70]# make
[[email protected] aoe6-70]# make install

Now you have installed the necessary aoe-tools and you are able to use the following commands:

• aoecfg - manipulate AoE configuration strings
• aoe-discover - tell aoe driver to discover AoE devices
• aoe-flush- flush the down devices out of the aoe driver
• aoe-interfaces - restrict aoe driver to specified network interfaces
• aoe-interfaces - restrict aoe driver to specified network interfaces
• aoe-mkdevs - create special device files for aoe driver
• aoe-mkshelf - create special device files for one shelf address
• aoeping - simple communication with AoE device
• aoe-revalidate - revalidate the disk size of an aoe device
• aoe-stat - print aoe device status report
• aoe-version - print AoE-related software version information

11 Making of a Small SAN

Till now, we have seen the available targets, corresponding features and their performances. Now its time to make a SAN based on the available disks and Gigabyte ethereal switch. So, following are the steps to make a SAN.Well, while writing this article, I don't have the extra hard disks. So, I have performed my experiment on 200 MB loop devices. In actual setup these loop devices will be replaced by actual 200 GB hard disks. The basic diagram of our SAN is as follows:

11.1 Server Setup For SAN

Export two devices from server0

[[email protected]]# dd if=/dev/zero of=file1.img bs=1M count=200
[[email protected]]# dd if=/dev/zero of=file2.img bs=1M count=200
[[email protected]]# losetup /dev/loop0 file1.img
[[email protected]]# losetup /dev/loop1 file2.img
[[email protected] vblade-19]# losetup -a
[[email protected] vblade-19]# ./vbladed 0 0 eth0 /dev/loop0
[[email protected] vblade-19]# ./vbladed 1 0 eth0 /dev/loop1

Export two devices from server1

[[email protected]]# dd if=/dev/zero of=file1.img bs=1M count=200
[[email protected]]# dd if=/dev/zero of=file2.img bs=1M count=200
[[email protected]]# losetup /dev/loop0 file1.img
[[email protected]]# losetup /dev/loop1 file2.img
[[email protected] vblade-19]# losetup -a
[[email protected] vblade-19]# ./vbladed 0 1 eth0 /dev/loop0
[[email protected] vblade-19]# ./vbladed 1 1 eth0 /dev/loop1

11.2 Client Setup For SAN

Exported block-devices at Client:

Make sure that at client side you have latest AoE driver and corresponding tools installed. You can check it out by following command:

[[email protected] krishna]# lsmod j grep aoe

If you don't have aoe driver at your box, you can download it from following mirror: http://support.coraid.com/support/linux/

[[email protected] krishna]# aoe-version

aoetools: 29
installed aoe driver: 70
running aoe driver: 70

Now run following commands to access the exported block device at client:

[[email protected] krishna]# modprobe aoe
[[email protected] krishna]# aoe-stat e0.0 0.209GB eth0 1024 up

e0.1 0.209GB eth0 1024 up
e1.0 0.209GB eth0 1024 up
e1.1 0.209GB eth0 1024 up

So, you can see the exported block devices on your box.

Creation of Raid Array:

Mirroring from e0.0 and e0.1:

[[email protected] krishna]# mdadm -C /dev/md0 -l 1 -n 2 /dev/etherd/e0.0 /dev/etherd/e0.1

Mirroring from e1.0 and e1.1:

[[email protected] krishna]# mdadm -C /dev/md1 -l 1 -n 2 /dev/etherd/e1.0 /dev/etherd/e1.1

Create the stripe over the mirrors:

[[email protected] krishna]# mdadm -C /dev/md2 -l 0 -n 2 /dev/md0 /dev/md1

So, now we have following configuration of Raid Array:

[[email protected] krishna]# cat /proc/mdstat

Personalities : [raid1] [raid0]
md2 : active raid0 md1[1] md0[0]
409344 blocks 64k chunks
md1 : active raid1 etherd/e1.1[1] etherd/e1.0[0]
204736 blocks [2/2] [UU]
md0 : active raid1 etherd/e0.1[1] etherd/e0.0[0]
204736 blocks [2/2] [UU]
unused devices: