Setting Up Xen pvops Dom0 on Fedora : Xen 3.4.2 + Kernel 2.6.31 with paravirt_ops in Fedora 12
Please refer to
for the latest stable Xen Dom0 solution.
How to set up Xen Dom0 with paravirt_ops technology in Fedora 12 will be introduced in this post. The Fedora 12 release has xen 3.4.2 which support paravirt_ops Dom0 and Domu but lack of Dom0 kernel. We use xen 3.4.2 in Fedora 12′s repository and build a paravirt_opsenabled Dom0 kernel by ourseves.
Hardware:
Dom0 hardware platform:
Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache
Linux system:
Fedora 12 x86_64
SELinux is disabled. Please refer here for detail: disabled SELinux on Fedora.
ext3 is recommended for the file system of disk partition for /boot.
Update system:
# yum update
Install xen
Install xen-3.4.2 from Fedora 12′s repository:
# yum install xen*
This is the packages that we have installed:
[root@localhost xen]# rpm -qa | grep xen xen-hypervisor-3.4.2-1.fc12.x86_64 xen-3.4.2-1.fc12.x86_64 xenner-0.47-3.fc12.x86_64 xen-libs-3.4.2-1.fc12.x86_64 xenwatch-0.5.4-3.fc12.x86_64 xen-doc-3.4.2-1.fc12.x86_64 xen-runtime-3.4.2-1.fc12.x86_64 xen-devel-3.4.2-1.fc12.x86_64
Install paravirt_ops technology enabled kernel
Download kernel 2.6.31.13
The kernel I used can be downloaded from here:
This kernel is downloaded from Jeremy’s git repository on Apr. 30, 2010:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git
Configuraton
The configuration file that I used can be downloaded from here:
Download config-kernel-xen-2.6.31.13
Just download this file, put it into the kernel source code file and rename it to .config .
You can also configure it by yourself using “make menuconfig” if you don’t want to use my configuration file or you need more components to be enabled.
Make sure you build the kernel with these components enabled:
Processor type and features --->
--- Paravirtualized guest support
[*] Xen guest support
(32) Maximum allowed size of a domain in gigabytes
[*] Enable Xen debug and tuning parameters in debugfs
[*] Enable Xen privileged domain support
[ ] Enable support for Xen PCI passthrough devices
[*] KVM paravirtualized clock
[*] KVM Guest support
-*- Enable paravirtualization code
[ ] Paravirtualization layer for spinlocks
Device Drivers --->
[*] Xen memory balloon driver
[*] Scrub pages before returning them to system
<*> Xen /dev/xen/evtchn device
[*] Backend driver support
<*> Block-device backend driver
<*> Block-device tap backend driver
<*> Xen backend network device
<*> PCI-device backend driver
PCI Backend Mode (Virtual PCI) --->
[ ] PCI Backend Debugging
<*> Xen filesystem
[*] Create compatibility mount point /proc/xen
[*] Create xen entries under /sys/hypervisor
Build kernel
# make -j16
Install modules and kernel
# make modules_install install
Configure grub
Add one entry in /boot/grub/grub.conf:
title Xen 3.4.2 - Fedora (2.6.31.13) root (hd0,0) kernel /xen-3.4.2.gz console=vga vga=ask noreboot module /vmlinuz-2.6.31.13 ro root=/dev/mapper/VolGroup-LogVol_root max_loop=32 noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us module /initramfs-2.6.31.13.img
The root and other parameters may be different depending on the configuration.
Configure xend
# cd /etc/xen # cp xend-config.sxp xend-config.sxp.bak # vim xend-config.sxp
Find:
#(network-script network-bridge) (network-script /bin/true)
Change it to:
(network-script network-bridge) #(network-script /bin/true)
After changing the configure file, reboot the system and choose xen + fedora entry which I have added in grub.
BIOS configuration
If xen stops at:
“I/O virtualization disabled.”
Enable VT and I/O virtualization in BIOS.
You may enable these in BIOS:
Intel (R) Virtualization Technology Intel (R) VT for Directed I/O Interrupt Remapping Coherency Support ATS Support
Enjoy it!
After booting the system, you can try to use xm to check xen info
[root@localhost xen]# xm info host : localhost.localdomain release : 2.6.31.13 version : #1 SMP Tue May 4 15:04:27 HKT 2010 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2266 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32705 free_memory : 3158 node_to_cpu : node0:0-15 node_to_memory : node0:3158 xen_major : 3 xen_minor : 4 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.4.2 20091027 (Red Hat 4.4.2-7) (GCC) cc_compile_by : mockbuild cc_compile_domain : (none) cc_compile_date : Fri Dec 11 05:58:50 EST 2009 xend_config_format : 4
Then xm command can be used to set up Domus.
This is the configuration file for one DomU that I use. It works well:
name="vm-10.0.1.201" memory=16384 disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ] vif = ['bridge=eth0'] bootloader = "/usr/bin/pygrub" vcpus=2 on_reboot = 'restart' on_crash = 'restart'
Here the loopback backed vbd device is used.
Making the performance more stable
I have test both Dom0 and DomU. The performance is stable but not as stable as the native Linux kernel. I saying not quite stable means that if we run a piece of code and we may find the time for it is a bit different from time to time. The difference may be up to 10%.
If you need a stable, you can configure xend to make Dom0 use one CPU. The Dom0 will change to UP mode. There are less virtual CPUs that physical CPUs. I don’t know the exact reason by now. But this method will make the performance stable.
We have also tested this solution. Both Dom0 and DomU (SMP mode of course) have as stable performance as native Linux.
The configuration:
# vim /etc/xen/xend-config.sxp
Find
(dom0-cpus 0)
Then change it to
(dom0-cpus 1)
Dom0 memory management
Initial memory size of Dom0
We can set the initial memory size of Dom0 by adding option dom0_mem (in KB) to xen in grub.conf.
If we set the initial memory size of Dom0 to 1GB, just change the line in grub entry to:
kernel /xen-3.4.2.gz console=vga vga=ask noreboot dom0_mem=1048576
Lowest permissible memory for dom0
The value dom0-min-mem (in MB) is the lowest permissible memory level for Dom0. The default value is 256. If we limit the memory size of Dom0 to larger than 1G, just set:
(dom0-min-mem 1024)
Problems
Limited number of loop devices
The default number of loop device in this kernel is 8. When we need to have more than 8 virtual machines, we need to add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.
initramfs related problem
We have found the initramfs image under /root generated by dracut doesn’t work on one of our servers (Use Xeon 5700 CPU). If you have this problem, you can try to use image generated by mkinitrd:
1) Generate initrd-2.6.31.13.img using mkinitrd
mkinitrd /boot/initrd-2.6.31.13.img 2.6.31.13
2) Edit entry in /boot/grub/grub.conf
Change
module /initramfs-2.6.31.13.img
to
module /initrd-2.6.31.13.img
drm related problem
On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.
The kernel command line in /boot/grub.conf will change to:
module /vmlinuz-2.6.31.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset max_loop=32 noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
Build kernel on 32bit platform
You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.
You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”
I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)
Error message about ksm
We can disable service ksm and ksmtuned to eliminate these error messages:
# chkconfig ksm off # chkconfig ksmtuned off
Update history:
Updated on Jun 18. 2010. Correct the error in hardware configuration. The memory is 24GB.
Updated on Jun 12. 2010. Add drm related problem.
Updated on May 5. 2010. Add download link of kernel 2.6.31.13.
Updated on May 5. 2010. Add info that SELinux is disabled.
Updated on May 6. 2010. Add info that the /boot partition should in ext3 format rather than ext4.
Updated on May 6. 2010. Add info about dracut generated initramfs’s problem on one of our platform.
Updated on May 6. 2010. Add method to add more loop devices for setting up more than 8 virtual machines.
Updated on May 6. 2010. Add info that PAE should be enabled first before building 32bit Dom0 kernel.
Updated on May 10, 2010. Add the Dom0 memory management part.
Updated on May 16, 2010. Add ksm related problem.
Tags: dom0, Fedora, Server config, Tutorial, xen