How To Create A Cluster Testbed Using CentOS 5 Virtualization And iSCSI
A. Overview
This guide attempts to provide a Xen based test environment where you can practice setting up a two node cluster (cluster setup itself is not discussed here - I'm merely giving you what you need to set it up).
XEN can host two type of guest systems para-virtualized and fully-virtualized:
- for para-virtualized guests you require the Red Hat Enterprise Linux 5 installation tree available over NFS, FTP or HTTP.
- for fully-virtualized guest installations you will require DVD or CD-ROM distribution media or a bootable .iso file and a network accessible installation tree
For details, please refer to the RHEL5 Virtualization Manual.
I'll be using para-virtualized guests here in my setup. There will be three systems involved here:
- node00 - physical system
virtual IPs: 192.168.222.1 (public1 vlan)
192.168.100.1 (private1 vlan) - node01 - para-virtualized guest 1
virtual IPs: 192.168.222.10 (public1 vlan)
192.168.100.10 (private1 vlan) - node02 - para-virtualized guest 1
virtual IPs: 192.168.222.20 (public1 vlan)
192.168.100.20 (private1 vlan)
B. What I used
- an HP Blade bl25p machine with 4G of RAM (this is actually an AMD64 blade machine). A machine with decent amount of RAM and processing speed should do.
- Centos i386 5 update 1 www.centos.org DVD ISO downloaded HTTP, NFS and FTP installation sources were created from this iso. Also, the yum repository that can be used by host and guest systems will be generated from the centos iso image.
- logical volumes hosting the guests and the "virtual luns" via iscsi (you can also use disk partitions - please refer to the virtualization guide for details).
1. My LVM setup
The following is my LVM configuration. The lvLUN0* entries are the ones I used for iSCSI setup and will be shared by the two virtual guest systems.
lvs
LV VG Attr LSize Origin Snap% Move Log Copy% lvLUN01 Virtual00VG -wi-ao 50.00G lvLUN02 Virtual00VG -wi-ao 50.00G lvNODE01 Virtual00VG -wi-ao 30.00G lvNODE02 Virtual00VG -wi-ao 30.00G lvNODE03 Virtual00VG -wi-ao 15.00G lvsys00 vg00 -wi-ao 512.00M lvsys01 vg00 -wi-ao 8.00G lvsys02 vg00 -wi-ao 8.00G lvsys03 vg00 -wi-ao 512.00M lvsys04 vg00 -wi-ao 128.00M lvsys05 vg00 -wi-ao 1.00G lvsys06 vg00 -wi-ao 256.00M
C. Host Preparation
I'm assuming that you know how to install CentOS or other RHEL based distributions and that you are familiar with rpm installation. Since I do a lot of setup for test/dev environments at work, I already have an installation server making it easy to do a network based install via PXE. The kickstart file for node00 is provided below. You can do a local media install (you have the ISO so you can burn it to a DVD) and just refer to the kickstart file for some of the configuration. The list of packages I used is in the %packages section of node00's kickstart file. You can install them manually using yum, like:
# will list centos installation groups
yum grouplist
# will install Virtualization group
yum groupinstall Virtualization
1. ks file and installation
1.a kickstart file I use for the host (node00)
You'll have to modify the following to suit your setup.
## START node00_ks.cfg #modify for your own settings install nfs --server=remote_server --dir=/path/to/CENTOS5U1/i386 lang en_US.UTF-8 keyboard us skipx reboot network --device eth2 --bootproto static --ip a.b.c.1 --netmask 255.255.255.0 --gateway a.b.c.2 --nameserver x.y.z.n --hostname node00.example.com # grub and root password is a1s2d3f4g5 rootpw --iscrypted $1$3CXK2$CG9WlX2PuPpp7nxYMQGwP0 firewall --disabled authconfig --enableshadow selinux --disabled timezone Asia/Singapore bootloader --location=mbr --driveorder=cciss/c0d0 --append="rhgb quiet" --md5pass=$1$3CXK2$CG9WlX2PuPpp7nxYMQGwP0 clearpart --all --initlabel --drives=cciss/c0d0 part /boot --fstype ext3 --size=100 --ondisk=cciss/c0d0 part pv.100000 --size=100 --grow --ondisk=cciss/c0d0 --asprimary volgroup vg00 --pesize=32768 pv.100000 logvol /tmp --fstype ext3 --name=lvsys05 --vgname=vg00 --size=1024 logvol /opt --fstype ext3 --name=lvsys04 --vgname=vg00 --size=128 logvol /var --fstype ext3 --name=lvsys03 --vgname=vg00 --size=512 logvol /usr --fstype ext3 --name=lvsys02 --vgname=vg00 --size=8192 logvol swap --fstype swap --name=lvsys01 --vgname=vg00 --size=8192 logvol /home --fstype ext3 --name=lvsys06 --vgname=vg00 --size=256 logvol / --fstype ext3 --name=lvsys00 --vgname=vg00 --size=512 %packages @development-libs @editors @system-tools @text-internet @x-software-development @virtualization @dns-server @core @base @ftp-server @network-server @legacy-software-development @base-x @web-server @printing @server-cfg @sql-server @admin-tools @development-tools lsscsi createrepo audit net-snmp-utils iptraf tftp lynx mesa-libGLU-devel kexec-tools bridge-utils device-mapper-multipath vnc-server xorg-x11-server-Xnest xorg-x11-server-Xvfb imake openmotif -vim-enhanced -zisofs-tools -zsh -bluez-hcidump -sysreport ## END of node00_ks.cfg
2. Host configuration
Setting up an HTTP, NFS and FTP installation server:
2.a web server
#/etc/httpd/conf.d/centos5u1.conf Alias /centos5u1 /var/ftp/pub/centos5u1 <Location /centos5u1> Options Indexes FollowSymLinks MultiViews IndexOptions FancyIndexing Order deny,allow Deny from all Allow from 127.0.0.1 ::1 all <Location>
Then start the httpd service and make sure it does during startup:
service httpd start
chkconfig httpd on
2.b NFS server
Edit /etc/exports and put the following into it:
# /etc/exports /var/ftp/pub/centos5u1 192.168*(ro)
service nfs start
chkconfig nfs on
2.c FTP server
Since we already have the source in /var/ftp/pub/centos5u1, all that is needed is to start vsftpd:
service vsftpd start
chkconfig vsftpd on
2.d YUM repository
For this setup, I only use a local yum repository from the Centos DVD ISO I downloaded. First, I loopback mount it in /var/ftp/pub/centos5u1/i386/:
cd /var/ftp/pub/centos5u1/
mkdir temp
mount -o loop CentOS-5.1-i386-bin-DVD.iso temp
cp -pr temp i386
umount temp
createrepo -g i386
(i386/repodata/ will then be updated.)
For RHEL5, it's different:
createrepo -g repodata/comps-rhel5-server-core.xml Server
You need to do this inside the i386 directory (after loopback mounting and copying the whole directory structure).
2.d.1 the yum repo configuration:
I renamed the default repo files in /etc/yum.repos.d/ to *-repo (instead of *.repo) to disable them. I then created this file:
#/etc/yum.repos.d/CentOS5.repo [centos5-Server] name=CentOS5 Server baseurl=http://node00/centos5u1/i386 enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
node00 needs to be defined in /etc/hosts for the above file to work. Or just replace node00 with its IP address.
2.e VNC server
You won't be needing a vnc connection if you have local console access to the physical machine. All you need to do is switch into gui mode:
telinit 5
and open a gui terminal (like gnome-terminal or kde konsole). But since I do everything remotely, I use vncserver and vncviewer to do gui based stuff.
2.e.1 run vncserver:
This will bringvup a vncserver in node00 that is accessible via "vncviewer" at node00:1 (assuming node00 is resolveable from your vncviewer host).
vncserver
You will require a password to access your desktops. Password: Verify: xauth: creating new authority file /root/.Xauthority New 'node00.example.com:1 (root)' desktop is node00.example.com:1 Creating default startup script /root/.vnc/xstartup Starting applications specified in /root/.vnc/xstartup Log file is /root/.vnc/node00.example.com:1.log