Setting Up Stable Xen Dom0 with Fedora: Xen 4.0.0 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

Please refer to
Xen solution
for the more latest stable Xen Dom0 solution.

---

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 4.0.0 from xen.org and Xenified Linux kernel 2.6.32.13. This solution have been tested quite stable in our cluster.

 

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 4.0.0

$ wget http://bits.xensource.com/oss-xen/release/4.0.0/xen-4.0.0.tar.gz
$ tar xf xen-4.0.0.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries" -y;
yum groupinstall "Development Tools" -y;
yum install transfig texi2html
libaio-devel dev86 glibc-devel
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial -y

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

The structure of these directories are as follows:

|
|-linux-2.6.32.13
|    |
|    `- patch.sh
` xen-patches-2.6.32-2
      |
      `- 6*.patch1

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 4.0.0 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-4.0.0.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

We may need to enable VT and I/O virtualization in BIOS.

These options can be enabled in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=16384
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

Here is a list of problem that may occur during the configuration.

Limited number of loop devices

The default number of loop device in this kernel is 8. When we are using blkback backed VBDs and we need to have more than 8 virtual machines, we should add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

The initramfs image under /root generated by dracut doesn’t work on some servers. If you have the similiar problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.32.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.32.13.img 2.6.32.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.32.13.img

to

module /initrd-2.6.32.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.

You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

The clocksource=jiffies kernel parameter may be needed. Please refer to Larry Matter’s reply below. He installed Xen with 32bit kernel successfully.

Error message about ksm

If you get lots error messages generated by ksm, you can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

More problems and solutions can be found here: Problems during Installing Xen Dom0 in Fedora

Update history
Aug. 10, 2010. Add clocksource kernel parameter for 32bit; Add dependency list;
May. 6, 2011. Add directory structure when compiling Xen and kernel.