I'd like some advice with selecting the best settings for Xen.
Currently I have 2 servers, each with 8 CPU Xeon cores, 12Gb memory and 1TB of Raid 5 storage. I had installed CentOS6 on both of these servers, and they were only used for a very trivial task. Having closely monitored the CPUs, I find that CPU0 is used on average around 40~60% and CPUs1~7 are barely used at all at less than 5%.
I decided its time to make better use of my resourses, and I installed Xen.
I followed the instructions from http://www.howtoforge.com/virtualiza...virtualization
And it seems that I have a successful installation.
My boot/grub parameters are set up with
kernel /xen.gz dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 dom0_vcpus_pin
I installed the first PV DomU (snl3) it seemed fine. I gave it 3 CPUs and 4Gb of memory, and CentOS6.4
I installed the 2nd PVDomU (snl4) it also seemed fine, I gave that 2CPUs and 3Gb of memory, CentOS6.4
In both cases I created and started these DomUs with xl create (not using xm or virsh)
I then installed the applications on each DomU, and this is where the problems began.
Originally I ran a script on the dedicated server, it took just over 4 seconds. On the DomU snl3, its now taking over a minute to run, which is not good, because the script is executed by cron every minute.
The script that I ran on DomU snl4, well, it just didn't finish, where as previously on dedicated server it took just over 2 mins.
Upon checking xl vcpu-list, I see that Dom0 is using VCPU0=CPU0, and DomU(snl3) is using VCPU0=CPU0 , VCPU1=CPU2, VCPU2=6 and DomU(snl4) VCPU0=0 and VCPU1=4, there appeared to be no consistency, IE not choosing the next available unused CPU, however, refreshing xl cpu-list then showed it all changed again.
I checked the CPU utilization of Dom0 and the DomUs, and in all cases, CPU0=100% utilized. (I was also very surprised that Dom0 had only 1 CPU, as I expected to see 8 CPUS from the SNMP stats, but I think I misunderstand how the hardware interacts with CentOS and Xen and the Dom0)
I decided to make a couple of new CPU pools, "Pool-snl3", and "Pool-snl4". I removed CPUS 3~7 from Pool-0 and added 3,4,5 to Pool-snl3 and 6,7 to Pool-snl4.
I assigned DomU (snl3) to pool-snl3 and then pinned the CPUs to 3,4,5
I assigned DomU (snl4) to pool-snl4 and then pinned the CPUs to 6,7
I ran the script on snl3 again, and this time it completed in under a minute, but only just, but still there were many times when it took longer than a minute.
I checked the CPU utilization of the Dom0 and DomUs again and still they were at 100%.
I tried to use taskset to get the scripts to run on a specific VCPU, but the results I got led me to believe that taskset was trying to run the task on the actual CPU specified rather than the VCPU.
At this point, I decided to abandon what I had done and restart from new. Reading further made me believe that Dom0 controls how the DomUs are given access to the hardware CPU and memory resources, etc.
I re-installed CentOS6.4 on my server, and re-installed Xen again.
I changed the boot/grub
kernel /xen.gz dom0_mem=12G cpufreq=xen dom0_max_vcpus=8 dom0_vcpus_pin
It failed to start
I changed it to
kernel /xen.gz dom0_mem=11G cpufreq=xen dom0_max_vcpus=8 dom0_vcpus_pin
it started, but then I could not create and DomUs
I changed it to
kernel /xen.gz dom0_mem=8G cpufreq=xen dom0_max_vcpus=6 dom0_vcpus_pin
I have now created DomU(snl3), installed CentOS6.4, with 2 VCPUs (from Pool-0) and 2G memory, and the script how takes 10 seconds to execute. But, SNMP shows that VCPU0 is averaging at 97% while VCPU1 is at 9%.
I've also checked Dom0 with SNMP and all CPU0 is under 4%, while CPUs1~5 are all under 1%
xl vcpu-list shows that Dom0 VCPU0=0, V1=1, V2=3, V3=3, V4=4, V5=5, and DomU(snl3) is VCPU0=7, V1=5 (but again, refreshing xl vcpu-list shows that this changes, except for Dom0 that doesnt change)
So now my head is quite messed up. I cant see the relationship between how the DomU is allocated a VCPU, sometimes it comes from a free CPU not allocated to the Dom0, and sometimes its allocated a CPU that is pinned to the Dom0.
The bottom line is this, I have 8 CPU Cores, and 12G memory.
Should I be allocating all my CPUs and memory to Dom0, thus to allow Dom0 to allocate its known resources to my DomUs ? Or should I stick with letting the Dom0 have 1G memory and 1 CPU, and allocate resources from what is left over to each of my DomUs ?
Also, If I wish to pin a CPU to a DomU and give it exclusive, unscheduled and unshared access, how can I do this ?
Please can you advise me, as I'm starting to lose sleep over this now.