Installing Xen 3.3 With Kernel 2.6.27 On Ubuntu 8.10 (x86_64) - Page 3

4 Creating LVM-Based Virtual Machines

This chapter explains how you can set up LVM-based virtual machines instead of virtual machines that use disk images. Virtual machines that use disk images are very slow and heavy on disk IO.

In this example I'm using an Ubuntu 8.10 host with the LVM volume group /dev/vg0 that has about 465GB of space. /dev/vg0 contains two logical volumes, /dev/vg0/root and /dev/vg0/swap_1 that consume about 48GB of space - the rest is not allocated and can be used to create logical volumes for our virtual machines:

vgdisplay

root@server1:~# vgdisplay
  --- Volume group ---
  VG Name               vg0
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  11
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               465.26 GB
  PE Size               4.00 MB
  Total PE              119106
  Alloc PE / Size       12396 / 48.42 GB
  Free  PE / Size       106710 / 416.84 GB
  VG UUID               8t178I-M2Fa-T9U6-8b76-Svjy-E7NW-fYIK1D

root@server1:~#

lvdisplay

root@server1:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg0/root
  VG Name                vg0
  LV UUID                CtAiSe-jr1F-Ibvj-80zR-DRnK-dgPz-OPoxnv
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                46.56 GB
  Current LE             11920
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

  --- Logical volume ---
  LV Name                /dev/vg0/swap_1
  VG Name                vg0
  LV UUID                n1DY62-rNBY-LLA9-jYZm-LypC-4LYW-Q4aiQB
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.86 GB
  Current LE             476
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

root@server1:~#

We will use xen-tools to create virtual machines. xen-tools make it very easy to create virtual machines - please read this tutorial to learn more: http://www.howtoforge.com/xen_tools_xen_shell_argo. The xen-tools package got installed as a dependency when we installed the ubuntu-xen-server package, so we don't need to install it anymore.

Now we edit /etc/xen-tools/xen-tools.conf. This file contains the default values that are used by the xen-create-image script unless you specify other values on the command line. I changed the following values and left the rest untouched:

vi /etc/xen-tools/xen-tools.conf

[...]
lvm = vg0
[...]
memory = 256Mb
[...]
dist   = lenny     # Default distribution to install.
[...]
gateway   = 192.168.0.1
netmask   = 255.255.255.0
broadcast = 192.168.0.255
[...]
passwd = 1
[...]
kernel      = /boot/vmlinuz-2.6.27-11-server
initrd      = /boot/initrd.img-2.6.27-11-server
[...]
mirror = http://ftp.de.debian.org/debian/
[...]
serial_device = hvc0
[...]
disk_device = xvda #default
[...]

In the kernel and initrd lines, we don't use our dom0 kernel that we built in the last chapter (2.6.27.5) because this kernel does not work in a domU. Please use one of the default Ubuntu kernels (take a look at the output of

ls -l /boot

), e.g. 2.6.27-11-server! The default Ubuntu kernels are built to work in Xen guests, so there's no need to build a special domU kernel.

Make sure that you uncomment the lvm line and fill in the name of your volume group (vg0 in my case). At the same time make sure that the dir line is commented out!

dist specifies the distribution to be installed in the virtual machines (Debian Lenny) (there's a comment in the file that explains what distributions are currently supported).

The passwd = 1 line makes that you can specify a root password when you create a new guest domain.

In the mirror line specify a Debian mirror close to you.

Make sure you specify a gateway, netmask, and broadcast address. If you don't, and you don't specify a gateway and netmask on the command line when using xen-create-image, your guest domains won't have networking even if you specified an IP address!

It is very important that you add the line serial_device = hvc0 (not xvc0 or tty1!) because otherwise the console of your virtual machines might not work properly!

Now let's create our first guest domain, xen1.example.com, with the IP address 192.168.0.101:

xen-create-image --hostname=xen1.example.com --size=4Gb --swap=256Mb --ip=192.168.0.101 --arch=amd64 --role=udev

Options that you specify on the command line override the settings in /etc/xen-tools/xen-tools.conf. Options that are not specified on the command line are taken from /etc/xen-tools/xen-tools.conf. Please make sure that you add --role=udev, or your virtual machine might not boot properly!

(To learn more about the available options, take a look at the xen-create-image man page:

man xen-create-image

)

The xen-create-image command will now create the xen1.example.com virtual machine for us. This can take a few minutes. The output should be similar to this one:

root@server1:~# xen-create-image --hostname=xen1.example.com --size=4Gb --swap=256Mb --ip=192.168.0.101 --arch=amd64 --role=udev

General Information
--------------------
Hostname       :  xen1.example.com
Distribution   :  lenny
Partitions     :  swap            256Mb (swap)
                  /               4Gb   (ext3)
Image type     :  full
Memory size    :  256Mb
Kernel path    :  /boot/vmlinuz-2.6.27-11-server
Initrd path    :  /boot/initrd.img-2.6.27-11-server

Networking Information
----------------------
IP Address 1   : 192.168.0.101 [MAC: 00:16:3E:BF:02:B9]
Netmask        : 255.255.255.0
Broadcast      : 192.168.0.255
Gateway        : 192.168.0.1


Creating swap on /dev/vg0/xen1.example.com-swap
Done

Creating ext3 filesystem on /dev/vg0/xen1.example.com-disk
Done
Installation method: debootstrap
Done

Running hooks
Done

Role: udev
        File: /etc/xen-tools/role.d/udev
Role script completed.

Creating Xen configuration file
Done
Setting up root password
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
All done


Logfile produced at:
         /var/log/xen-tools/xen1.example.com.log
root@server1:~#

As you see from the output, xen-create-image has created a new logical volume for our VM in the vg0 volume group, /dev/vg0/xen1.example.com-disk, for the VM's root filesystem. Take a look at

lvdisplay

and you will see that it has also created a second logical volume, /dev/vg0/xen1.example.com-swap, for the VM's swap:

root@server1:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg0/root
  VG Name                vg0
  LV UUID                CtAiSe-jr1F-Ibvj-80zR-DRnK-dgPz-OPoxnv
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                46.56 GB
  Current LE             11920
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

  --- Logical volume ---
  LV Name                /dev/vg0/swap_1
  VG Name                vg0
  LV UUID                n1DY62-rNBY-LLA9-jYZm-LypC-4LYW-Q4aiQB
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.86 GB
  Current LE             476
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

  --- Logical volume ---
  LV Name                /dev/vg0/xen1.example.com-swap
  VG Name                vg0
  LV UUID                0sVqfr-TUoJ-FLeo-qgVn-HIiy-lIeF-dSi3gn
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                256.00 MB
  Current LE             64
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:2

  --- Logical volume ---
  LV Name                /dev/vg0/xen1.example.com-disk
  VG Name                vg0
  LV UUID                otgxt7-T5cW-zaxd-y7x3-qwnp-gbmo-oriRLt
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                4.00 GB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:3

root@server1:~#

There should now be a xen1.example.com configuration file - /etc/xen/xen1.example.com.cfg. Take a look at it to become familiar with virtual machine configuration files:

vi /etc/xen/xen1.example.com.cfg

#
# Configuration file for the Xen instance xen1.example.com, created
# by xen-tools 3.9 on Fri Feb 13 00:47:09 2009.
#

#
#  Kernel + memory size
#
kernel      = '/boot/vmlinuz-2.6.27-11-server'
ramdisk     = '/boot/initrd.img-2.6.27-11-server'
memory      = '256'

#
#  Disk device(s).
#
root        = '/dev/xvda2 ro'
disk        = [
                  'phy:/dev/vg0/xen1.example.com-swap,xvda1,w',
                  'phy:/dev/vg0/xen1.example.com-disk,xvda2,w',
              ]


#
#  Hostname
#
name        = 'xen1.example.com'

#
#  Networking
#
vif         = [ 'ip=192.168.0.101,mac=00:16:3E:BF:02:B9' ]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

(If we had used disk images instead of logical volumes, the disk line would look similar to this one:

disk = [ 'file:/path/to/xen1.example.com/disk.img,xvda1,w', 'file:/path/to/xen1.example.com/swap.img,xvda2,w' ]

)

(Please note: if you have a dual-core or quad-core CPU and want the virtual machine to use all CPU cores, please add the line vcpus = '2' or vcpus = '4' to the configuration file.)

Before we start the virtual machine for the first time, we must first mount its storage device and copy the domU kernel to it (the xen-create-image command copies the dom0 kernel 2.6.27.5 to the guest, but as I mentioned before, our domU kernel is different from the dom0 kernel (2.6.27-11-server in this example)):

mount /dev/vg0/xen1.example.com-disk /mnt
cd /mnt/lib/modules/
cp -pfr /lib/modules/2.6.27-11-server/ .
cd
umount /mnt

To start the virtual machine, run

xm create /etc/xen/xen1.example.com.cfg

Run

xm console xen1.example.com

to log in on that virtual machine (type CTRL+] if you are at the console, or CTRL+5 if you're using PuTTY to go back to dom0), or use an SSH client to connect to it (192.168.0.101).

To get a list of running virtual machines, type

xm list

The output should look like this:

root@server1:~# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  3499     2     r-----   2930.0
xen1.example.com                            23   256     1     -b----      5.8
root@server1:~#

To shut down xen1.example.com, do this:

xm shutdown xen1.example.com

If you want xen1.example.com to start automatically at the next boot of the system, then do this:

ln -s /etc/xen/xen1.example.com.cfg /etc/xen/auto

Here are the most important Xen commands:

xm create -c /path/to/config - Start a virtual machine.
xm shutdown <name> - Stop a virtual machine.
xm destroy <name> - Stop a virtual machine immediately without shutting it down. It's as if you switch off the power button.
xm list - List all running systems.
xm console <name> - Log in on a virtual machine.
xm help - List of all commands.

A list of all virtual machines that were created with the xen-create-image command is available under

xen-list-images

root@server1:~# xen-list-images
Name: xen1.example.com
Memory: 256
IP: 192.168.0.101
root@server1:~#

To learn more about what you can do with xen-tools, take a look at this tutorial: http://www.howtoforge.com/xen_tools_xen_shell_argo

 

5 Links

Share this page:

18 Comment(s)

Add comment

Comments

From: Boris at: 2009-02-24 18:36:57
From: Ranjan at: 2010-04-05 17:08:04

Hi, I am able to compile successfully but when I do an ls -l I can't see an entry like the one below which means that for some reason this has not been downloaded into /boot. As such I can't boot into any kernel since this has to be loaded first.

-rw-r--r-- 1 root root 470144 2008-10-06 20:15 xen-3.3.gz

 

Has anyone faced this problem ?

Thanks

Ranjan

From: Detlef at: 2009-03-19 17:33:01

I followed this very useful description and successfully made my new XEN kernel for amd_64. However after compilation the kernel didn't boot on my rented webserver. As I have no console access during the bootphase the cause was not directly visible.

When I compared the size of the /lib/modules directory and the original initrd files  I realized that it was probably due to missing drivers (raid support etc). So I modified the kernel generation by copying the original kernel config file 

cp /boot/config-2.6.27-11-server ~/build/linux-2.6.27-xen/.config

before running the menuconfig. So all the kernel setting were similar to the normal Ubuntu kernel and the similar modules were compiled and added. However with the default setting the created initrd was 50 Mbytes in size compared to the normal 9Mbytes.

This can be controlled by changing the /etc/initramfs-tools/initramfs.conf The change of

MODULES=most

to 

MODULES=dep

before creating the new initrd let the system only included the used modules in the initrd.

Here the size of my installation

ls -la /boot/*2.6.27.5* /boot/*2.6.27-11*

-rw-r--r-- 1 root root 1258976 Mar 19 14:47 /boot/System.map-2.6.27.5
-rw-r--r-- 1 root root   87245 Mar 19 14:47 /boot/config-2.6.27.5
-rw-r--r-- 1 root root 9208093 Mar 19 14:49 /boot/initrd.img-2.6.27.5
-rw-r--r-- 1 root root 2195195 Mar 19 14:47 /boot/vmlinuz-2.6.27.5

-rw-r--r-- 1 root root 1354638 Jan 29 21:23 /boot/System.map-2.6.27-11-server
-rw-r--r-- 1 root root  504280 Jan 29 21:23 /boot/abi-2.6.27-11-server
-rw-r--r-- 1 root root   85313 Jan 29 21:23 /boot/config-2.6.27-11-server
-rw-r--r-- 1 root root 8984287 Mar 19 07:45 /boot/initrd.img-2.6.27-11-server
-rw-r--r-- 1 root root    1130 Jan 29 21:27 /boot/vmcoreinfo-2.6.27-11-server
-rw-r--r-- 1 root root 2341536 Jan 29 21:23 /boot/vmlinuz-2.6.27-11-server

 

 

From: Anonymous at: 2009-03-09 13:08:14

I get an error when building the kernel :(

WARNING: modpost: Found 1 section mismatch(es).
To see full details build your kernel with:
'make CONFIG_DEBUG_SECTION_MISMATCH=y'
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
arch/x86/kernel/built-in.o: In function `timer_interrupt':
/usr/src/linux-2.6.27-xen.hg/arch/x86/kernel/time_32-xen.c:469:
undefined reference to `__udivdi3'
make: *** [.tmp_vmlinux1] Error 1

From: Anonymous at: 2009-03-11 12:41:30

you need to use older (e.g. gcc-4.2) 

if it is possible dont use Ubuntu 8.10 as system where you would be compile xen kernel 

From: amir at: 2009-11-18 09:53:53

root@ubunzzo:/usr/src/linux-2.6.29.2-xen# vi /usr/src/linux-2.6.29.2-xen/arch/x86/kernel/time_32-xen.c

My attempt after reading reference (2) was successful, I inserted the asm(“”:”+r”(delta)) to the timer_interrupt function at the while scope at line number around 500.

        /* System-wide jiffy work. */
        if (delta >= NS_PER_TICK) {
                do_div(delta, NS_PER_TICK);
                processed_system_time += delta * NS_PER_TICK;
                while (delta > HZ) {

                        asm("":"+r"(delta));

                        do_timer(HZ);
                        delta -= HZ;
                }
                do_timer(delta);
        }

* This works because this asm fools gcc compiler not to use optimization.

source : http://mulps.wordpress.com/2009/05/29/compiling-xen-kernel-2-6-29-2/

From: Tapas at: 2009-11-09 11:00:30

I also had the same issue here is the solution some one from community helped

From: Wodin at: 2009-03-02 19:20:25

No need for "sudo su" when "sudo -i" will do.

From: Taner at: 2009-07-25 07:42:29

Hi,

I used your instruction  to install Xen 3.3 on top of my Ubuntu 9.04 Desktop amd64 (ext4). Unfortunately my Xen Kernel didn't boot up. There are the following errors.

Begin: Running /scripts/local-premount ...
19+0 records in
19+0 records out

kinit: name_to_dev_t(/dev/disk/by-uuid/874b696f-fd43-4c6d-9493-a8a01941bbc0) = dev(8,4)
kinit: trying to resume from /dev/disk/by-uuid/874b696f-fd43-4c6d-9493-a8a01941bbc0
kinit: No resume image, doing normal boot...
Done.
mount: mounting /dev/sda1 on /root failed: No such file or directory
Done.
mount: mounting /dev on /root/dev failed: No such file or directory
Done.
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. Try passing init= bootarg.


From: Anonymous at: 2010-05-26 16:54:58

I am having problem in this step "hg clone http://xenbits.xensource.com/ext/linux-2.6.27-xen.hg". I am behind proxy. I am getting error "destination directory: linux-2.6.27-xen.hg
abort: error: Connection timed out". Should i set http_proxy or anything else.

 Thank you.

From: at: 2009-04-07 17:02:56

I would like to thank you for your posting    “Installing Xen 3.3 With Kernel 2.6.27 On Ubuntu 8.10 (x86_64)”

I followed the steps mentioned in this, and was able to create the Xen server successfully. Thanks a lot for the nice Installation guide.

I have few questions after I did create the VM or domU on Xen

1 Can we create Windows domU on Xen-ubuntu server ?  How do we create it? If not is there any work around for this?

2 How do we duplicate or clone and existing VM( domU ) with and without it running?

It would be really great if you could answer this question or if you can direct to any place Where I can find information on this.

From: at: 2009-04-07 16:19:16

Have you Compiled your kernel with Xen sources and installed all the modules?

I got this error as I didnot do install of the new xen modules. Please check that

Thanks

Murali

From: at: 2009-04-07 17:27:18

Thanks for the Wonderful effort in bring the Xen installation guide to the whole world making it simple and easy. Thanks Falko Timme for this good effort.

From: chintalvaady at: 2009-03-31 09:23:11

Hi friends,

i have used this HOW TO and it has been very useful .however..i get the following error when i type "xm create /etc/xen/xen1.example.com.cfg"..

 

Can anyone here try to solve it

root@mamce-desktop:~# xm create /etc/xen/xen1.example.com.cfg
ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)
Traceback (most recent call last):
  File "/usr/sbin/xm", line 8, in <module>
    from xen.xm import main
  File "/usr/lib/python2.5/site-packages/xen/xm/main.py", line 61, in <module>
    xc = xen.lowlevel.xc.xc()
xen.lowlevel.xc.Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (2 = No such file or directory)')
root@mamce-desktop:~#

 

From: Anonymous at: 2010-12-15 17:31:17

i've the same problems. it seems, that you didn't boot the xen kernel...

From: at: 2009-02-27 10:42:27

Hello and thanks for the detailed instructions.

I am not sure why from the time I run update-grub nothing gets updates (mainly menu.lst) and I reboot, I have the same kernel as before!?

The files are there under /boot/

But again when I type any xm command or xm by itself I get:

>xm
ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)
Traceback (most recent call last):
  File "/usr/sbin/xm", line 8, in <module>
    from xen.xm import main
  File "/usr/lib/python2.5/site-packages/xen/xm/main.py", line 61, in <module>
    xc = xen.lowlevel.xc.xc()
xen.lowlevel.xc.Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (2 = No such file or directory)')


When I edit the menu.lst manually and insert any xen kernels available I either get a reboot/kernel panic, or freezes or once I was able to boot up in xen but No NIC (except for lo) so I had no network at all?!

For some reason eth0/1 and 2 all disappeared?!

Can you post a sample of a working menu.lst please?!

my server only loads  vmlinuz-2.6.27-7-server

 

Thank You!

 

 

From: Mick T. at: 2009-03-04 02:22:32

Good article, I've been something like this to help me get Xen running on Intrepid. If you do a revision of this article you might want to consider building images as DEB packages.

Thanks

Mick

From: Anonymous at: 2009-03-13 17:40:53

Dear author:

Thank you very much for your help.

Its a good tutorial!.

Best regards and best wishes for you

-W.F