What shoud be the correct settings for Xen Dom0 based on my resources ?

Discussion in 'Installation/Configuration' started by IanH, Jul 29, 2013.

  1. IanH

    IanH New Member

    Hi All,
    I'd like some advice with selecting the best settings for Xen.

    Currently I have 2 servers, each with 8 CPU Xeon cores, 12Gb memory and 1TB of Raid 5 storage. I had installed CentOS6 on both of these servers, and they were only used for a very trivial task. Having closely monitored the CPUs, I find that CPU0 is used on average around 40~60% and CPUs1~7 are barely used at all at less than 5%.
    I decided its time to make better use of my resourses, and I installed Xen.

    I followed the instructions from http://www.howtoforge.com/virtualiz...aravirtualization-and-hardware-virtualization
    And it seems that I have a successful installation.
    My boot/grub parameters are set up with
    kernel /xen.gz dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 dom0_vcpus_pin
    I installed the first PV DomU (snl3) it seemed fine. I gave it 3 CPUs and 4Gb of memory, and CentOS6.4
    I installed the 2nd PVDomU (snl4) it also seemed fine, I gave that 2CPUs and 3Gb of memory, CentOS6.4
    In both cases I created and started these DomUs with xl create (not using xm or virsh)
    I then installed the applications on each DomU, and this is where the problems began.

    Originally I ran a script on the dedicated server, it took just over 4 seconds. On the DomU snl3, its now taking over a minute to run, which is not good, because the script is executed by cron every minute.
    The script that I ran on DomU snl4, well, it just didn't finish, where as previously on dedicated server it took just over 2 mins.
    Upon checking xl vcpu-list, I see that Dom0 is using VCPU0=CPU0, and DomU(snl3) is using VCPU0=CPU0 , VCPU1=CPU2, VCPU2=6 and DomU(snl4) VCPU0=0 and VCPU1=4, there appeared to be no consistency, IE not choosing the next available unused CPU, however, refreshing xl cpu-list then showed it all changed again.
    I checked the CPU utilization of Dom0 and the DomUs, and in all cases, CPU0=100% utilized. (I was also very surprised that Dom0 had only 1 CPU, as I expected to see 8 CPUS from the SNMP stats, but I think I misunderstand how the hardware interacts with CentOS and Xen and the Dom0)

    I decided to make a couple of new CPU pools, "Pool-snl3", and "Pool-snl4". I removed CPUS 3~7 from Pool-0 and added 3,4,5 to Pool-snl3 and 6,7 to Pool-snl4.
    I assigned DomU (snl3) to pool-snl3 and then pinned the CPUs to 3,4,5
    I assigned DomU (snl4) to pool-snl4 and then pinned the CPUs to 6,7
    I ran the script on snl3 again, and this time it completed in under a minute, but only just, but still there were many times when it took longer than a minute.
    I checked the CPU utilization of the Dom0 and DomUs again and still they were at 100%.
    I tried to use taskset to get the scripts to run on a specific VCPU, but the results I got led me to believe that taskset was trying to run the task on the actual CPU specified rather than the VCPU.

    At this point, I decided to abandon what I had done and restart from new. Reading further made me believe that Dom0 controls how the DomUs are given access to the hardware CPU and memory resources, etc.
    I re-installed CentOS6.4 on my server, and re-installed Xen again.

    I changed the boot/grub
    kernel /xen.gz dom0_mem=12G cpufreq=xen dom0_max_vcpus=8 dom0_vcpus_pin
    It failed to start
    I changed it to
    kernel /xen.gz dom0_mem=11G cpufreq=xen dom0_max_vcpus=8 dom0_vcpus_pin
    it started, but then I could not create and DomUs
    I changed it to
    kernel /xen.gz dom0_mem=8G cpufreq=xen dom0_max_vcpus=6 dom0_vcpus_pin

    I have now created DomU(snl3), installed CentOS6.4, with 2 VCPUs (from Pool-0) and 2G memory, and the script how takes 10 seconds to execute. But, SNMP shows that VCPU0 is averaging at 97% while VCPU1 is at 9%.
    I've also checked Dom0 with SNMP and all CPU0 is under 4%, while CPUs1~5 are all under 1%
    xl vcpu-list shows that Dom0 VCPU0=0, V1=1, V2=3, V3=3, V4=4, V5=5, and DomU(snl3) is VCPU0=7, V1=5 (but again, refreshing xl vcpu-list shows that this changes, except for Dom0 that doesnt change)

    So now my head is quite messed up. I cant see the relationship between how the DomU is allocated a VCPU, sometimes it comes from a free CPU not allocated to the Dom0, and sometimes its allocated a CPU that is pinned to the Dom0.

    The bottom line is this, I have 8 CPU Cores, and 12G memory.
    Should I be allocating all my CPUs and memory to Dom0, thus to allow Dom0 to allocate its known resources to my DomUs ? Or should I stick with letting the Dom0 have 1G memory and 1 CPU, and allocate resources from what is left over to each of my DomUs ?
    Also, If I wish to pin a CPU to a DomU and give it exclusive, unscheduled and unshared access, how can I do this ?

    Please can you advise me, as I'm starting to lose sleep over this now.


Share This Page