Linux and Virtualization » dom0 http://fclose.com/b/linux Clear solutions, tutorials and tips on Linux and virtualization from the author's experience with clusters. Sat, 04 Feb 2012 04:40:21 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Simple Introduction to paravirt_ops for Xen http://fclose.com/b/linux/2683/simple-introduction-to-paravirt_ops-for-xen/ http://fclose.com/b/linux/2683/simple-introduction-to-paravirt_ops-for-xen/#comments Wed, 20 Oct 2010 04:50:50 +0000 Zhiqiang Ma http://pkill.info/b/?p=2683 The is a simple introduction to paravirt_ops in Linux kernel for Xen, VMware, etc. We make this introduction from the view of code. We use the function raw_local_irq_disable() and raw_local_irq_enable() functions in Linux kernel
to introduce paravirt_ops for Xen and Xenified kernel.

Please download the introduction to paravirt_ops pdf file:

introduction-to-pv-ops-v3.pdf

]]> http://fclose.com/b/linux/2683/simple-introduction-to-paravirt_ops-for-xen/feed/ 0 Xen DomU’s I/O Performance of LVM and loopback Backed VBDs http://fclose.com/b/linux/2260/xen-domus-io-performance-of-lvm-and-loopback-backed-vbds/ http://fclose.com/b/linux/2260/xen-domus-io-performance-of-lvm-and-loopback-backed-vbds/#comments Sun, 17 Oct 2010 16:04:15 +0000 Zhiqiang Ma http://pkill.info/b/?p=2260 This posts list benchmark (using bonnie++) result of I/O performance of Xen LVM and loopback backed VBDs.

The configuration of machines

Dom0

VCPU: 2 (Intel(R) Xeon(R) CPU E5520  @ 2.27GHz)
Memory: 2GB
Xen and Linux kernel: Xen 3.4.3 with Xenified 2.6.32.13 kernel

DomU

VCPU: 2
Memory: 2GB
Linux kernel: Fedora (2.6.32.19-163.fc12.x86_64)

DomU’s profile:

name="10.0.1.200"
vcps=2
memory=2048
disk = ['phy:vg_xen/vm-10.0.1.150/vmdisk0,xvda,w']
#disk = ['tap:aio:/lhome/xen/vm0-f12/vmdisk0,xvda,w']
#disk = ['file:/lhome/xen/vm0-f12/vmdisk0,xvda,w']

vif=['bridge=eth0']
bootloader="/usr/bin/pygrub"
#extra="single"
on_reboot='restart'
on_crash='restart'

The “disk” lines is changed depending on the driver used.

Benchmark method

We use Bonnie++ to test the performance of I/O:

# bonnie++ -u root

We run bonnie++ on one single VM. We also test the performance change after making a snapshot of LVM for a new VM. For the file backed VMs, we also run two VMs together on the same hard disk and run bonnie++ on them.

Bonnie++’s result is in this format:

Version 1.03e       ------Sequential Output------ --Sequential Input- --Random-
 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.locald 4G 76999  98 107423  21 47522  13 73347  91 159847  16 266.0   0
 ------Sequential Create------ --------Random Create--------
 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
localhost.localdomain,4G,76999,98,107423,21,47522,13,73347,91,159847,16,266.0,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

We only list the last line which contains all the result in the result section in this post.

Benchmark result

LVM backed VBD

localhost.localdomain,4G,76999,98,107423,21,47522,13,73347,91,159847,16,266.0,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,79588,98,120078,22,46140,13,75343,94,150167,15,248.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,81942,98,113617,22,47736,13,75947,94,152110,15,262.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

New LVM logical volume made by snapshot

localhost.localdomain,4G,11846,15,12044,2,27133,7,71510,92,141408,14,262.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,12200,15,18147,3,33086,9,66687,89,146550,14,251.9,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,58521,73,58482,10,33880,9,69399,90,144237,14,267.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,62553,78,57576,11,32755,9,70037,89,143462,14,259.9,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,66031,84,65640,12,34357,9,66036,86,152171,15,266.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,58666,75,60092,11,34826,9,72821,91,141328,14,259.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

File backed VBD

vm112,4G,20865,27,23559,4,32913,9,63006,81,128395,13,217.9,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,23022,30,18611,3,30086,8,63784,82,125736,13,197.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,21485,27,20366,3,29587,8,72130,92,140957,14,239.6,0,16,+++++,+++,+++++,+++,30751,52,+++++,+++,+++++,+++,+++++,+++
vm112,4G,22375,32,21716,3,30300,8,65488,87,128625,13,221.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,21968,28,19298,3,29007,8,68469,88,122111,12,222.5,0,16,26967,94,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,21477,28,20463,3,38395,10,49312,63,154206,15,241.0,0,16,+++++,+++,+++++,+++,32699,56,+++++,+++,+++++,+++,+++++,+++

Two VMs on the same disk running together

A:

vm112,4G,10645,13,9498,1,9606,2,30866,41,86911,8,100.1,0,16,9181,22,20583,6,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,10623,13,10143,1,10485,2,26013,35,77362,7,116.3,0,16,25701,66,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,10824,14,9558,1,12028,3,27503,36,57196,5,92.1,0,16,9679,28,+++++,+++,9294,15,+++++,+++,+++++,+++,+++++,+++
vm112,4G,15098,19,10485,1,12536,3,22771,30,64679,6,142.2,0,16,32006,82,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,11315,14,9674,1,12052,3,26453,35,68206,7,121.1,0,16,23789,62,32446,13,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,11865,15,11564,2,11508,3,24945,34,61946,6,102.6,0,16,13297,34,21805,7,+++++,+++,+++++,+++,+++++,+++,+++++,+++

B:

vm119,4G,8963,11,9255,1,10909,3,36446,48,70485,7,125.1,0,16,19701,52,23649,8,9574,16,+++++,+++,+++++,+++,+++++,+++
vm119,4G,9074,12,8410,1,12898,3,35266,47,68469,7,107.4,0,16,6585,17,3206,1,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm119,4G,9151,13,8664,1,10285,2,20120,28,58011,5,90.9,0,16,22894,59,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm119,4G,9053,11,10406,1,12852,3,27618,37,55405,5,108.8,0,16,10144,24,22599,7,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm119,4G,8987,11,11464,1,12123,3,19278,26,59274,6,104.8,0,16,5357,13,23010,7,+++++,+++,7922,18,+++++,+++,+++++,+++
vm119,4G,9593,12,11450,1,30598,8,57078,73,119884,12,222.2,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Conclusion

From the evaluation result, we can see that LVM backed VBD of Xen DomU has a much better performance that file backed. From out experiment in our cluster, LVM backed VBD is also quite stable.

]]> http://fclose.com/b/linux/2260/xen-domus-io-performance-of-lvm-and-loopback-backed-vbds/feed/ 0 An Introduction to Xen Source Code Structure and Disk in Xen http://fclose.com/b/linux/2600/a-introduction-to-xen-source-code-structure-and-disk-in-xen/ http://fclose.com/b/linux/2600/a-introduction-to-xen-source-code-structure-and-disk-in-xen/#comments Wed, 22 Sep 2010 05:16:39 +0000 Zhiqiang Ma http://pkill.info/b/?p=2600 I have created slides from introducing Xen’s source code structure and backend and frontend drivers in Xen. Please find the PDF version here:

xen-code-disk-v2.pdf

Animation is enabled which may help to understand it.

Update history
Dec. 10, 2010. Update pdf to version 2. Animation is added.

]]> http://fclose.com/b/linux/2600/a-introduction-to-xen-source-code-structure-and-disk-in-xen/feed/ 2 Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://fclose.com/b/linux/2405/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://fclose.com/b/linux/2405/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Fri, 27 Aug 2010 00:55:23 +0000 Zhiqiang Ma http://pkill.info/b/?p=2405 This is the latest stable and recommended stable Xen Dom0 solution on Fedora 12. No serious bug found till now and we will fix the bugs by ourselves if some appears. It also works on Fedora 14 as well. It should not be hard to use this solution on other versions of Fedora or other Linux distribution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.32.13. This is a very stable Dom0 solution for Fedora 12. Lot’s DomUs have been tested on this platform.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum install make gcc -y;
yum groupinstall "Development Libraries" -y;
yum groupinstall "Development Tools" -y;
yum install transfig texi2html
dev86 glibc-devel glibc-devel.i686
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial ed -y

The we can make Xen and Xen tools:

$ make xen
$ make tools

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number.

You may need to install patch if it isn’t installed:

# yum install patch

This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

The structure of these directories are as follows:

|
|-linux-2.6.32.13
|    |
|    `- patch.sh
` xen-patches-2.6.32-2
      |
      `- 6*.patch1

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 < > Block-device tap backend driver 2 (NEW)
 <*> Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 < > PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code) --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -jX

X can be two times of the number of the processor. We use this to let make invoke compilation work in X-way parallel.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

Please refer to
Xen solution
for the DomU solution and more Xen Dom0 solution.

Update history
Sep. 3, 2010. blktap driver is used instead of blktap2; PCI backend driver is disabled.
Sep. 8, 2010. Add install patch; highlight PCI part in make menuconfig.
Sep. 11, 2010. Delete DomU configure file which is not used.
Oct. 19, 2010. Host config in this site.
May. 1, 2011. Fix corrupted URL.
May. 6, 2011. Add directory structure of Xen patch and Kernel src.
Aug. 11, 2011. Add glibc-devel.i686 for depended packages.

]]> http://fclose.com/b/linux/2405/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 7 Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://fclose.com/b/linux/2412/setting-up-stable-xen-dom0-with-fedora-xen-4-0-1-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://fclose.com/b/linux/2412/setting-up-stable-xen-dom0-with-fedora-xen-4-0-1-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Thu, 26 Aug 2010 18:51:15 +0000 Zhiqiang Ma http://pkill.info/b/?p=2412 Please refer to
Xen solution
for the more latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 4.0.1 from xen.org and Xenified Linux kernel 2.6.32.13. This solution have been tested quite stable in our cluster.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 4.0.1

$ wget http://bits.xensource.com/oss-xen/release/4.0.1/xen-4.0.1.tar.gz
$ tar xf xen-4.0.1.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries" -y;
yum groupinstall "Development Tools" -y;
yum install transfig texi2html
libaio-devel dev86 glibc-devel
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial -y

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 4.0.1 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-4.0.1.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=16384
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

Problems and solutions can be found here: Problems during Installing Xen Dom0 in Fedora

]]> http://fclose.com/b/linux/2412/setting-up-stable-xen-dom0-with-fedora-xen-4-0-1-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 2 Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://fclose.com/b/linux/2388/setting-up-stable-xen-dom0-with-fedora-xen-3-4-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://fclose.com/b/linux/2388/setting-up-stable-xen-dom0-with-fedora-xen-3-4-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Wed, 25 Aug 2010 05:21:24 +0000 Zhiqiang Ma http://pkill.info/b/?p=2388 Please refer to

Xen solution

for the more latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.2 from xen.org and Xenified Linux kernel 2.6.32.13. This is a very stable Dom0 solution for Fedora 12. Lot’s DomUs have been tested on this platform.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.2

$ wget http://bits.xensource.com/oss-xen/release/3.4.2/xen-3.4.2.tar.gz
$ tar xf xen-3.4.2.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries";
yum groupinstall "Development Tools";
yum install transfig texi2html
libaio-devel dev86 glibc-devel
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server)) --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*> Block-device tap backend driver 2 (NEW)
 <*> Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code) --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.2 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-3.4.2.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

]]> http://fclose.com/b/linux/2388/setting-up-stable-xen-dom0-with-fedora-xen-3-4-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 0 Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12 http://fclose.com/b/linux/2356/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-31-12-in-fedora-12/ http://fclose.com/b/linux/2356/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-31-12-in-fedora-12/#comments Sun, 15 Aug 2010 16:05:54 +0000 Zhiqiang Ma http://pkill.info/b/?p=2356 Please refer to

Xen solution

for the latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.31.12.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries";
yum groupinstall "Development Tools";
yum install transfig texi2html
libaio-devel dev86 glibc-devel
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.31.12

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.12.tar.bz2
$ tar xf linux-2.6.31.12.tar.bz2

Download 2.6.31 Xen patches v14

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-14.tar.bz2
$ mkdir xen-patches-2.6.31-14
$ tar xf xen-patches-2.6.31-14.tar.bz2 -C xen-patches-2.6.31-14

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.31-14/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-2.6.31.12-xenified

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 
Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 <*>     Network-device loopback driver (NEW)
 < >   PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 < >   SCSI backend driver (NEW)
 < >   USB backend driver (NEW)
 < > Block-device frontend driver
 < > Network-device frontend driver
 < > SCSI frontend driver (NEW)
 < > USB frontend driver (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (3.0.2 and later)  --->

Build kernel

$ make -j16

-jN: N may be 16 or other numbers depending on the number of processors in the system.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.31.12
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.31.12 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.31.12.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

]]> http://fclose.com/b/linux/2356/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-31-12-in-fedora-12/feed/ 0 Problems during Installing Xen Dom0 in Fedora http://fclose.com/b/linux/2354/problems-during-installing-xen-dom0-in-fedora/ http://fclose.com/b/linux/2354/problems-during-installing-xen-dom0-in-fedora/#comments Sun, 15 Aug 2010 11:58:36 +0000 Zhiqiang Ma http://pkill.info/b/?p=2354 Here is a list of problem that may occur during installing and configuring Xen Dom0 in Fedora. It is found originally in Fedora systems, but the tips in this post should also be helpful for installing Xen Dom0 on other platforms.

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

We may need to enable VT and I/O virtualization in BIOS.

These options can be enabled in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Limited number of loop devices

The default number of loop device in this kernel is 8. When we are using blkback backed VBDs and we need to have more than 8 virtual machines, we should add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

The initramfs image under /root generated by dracut doesn’t work on some servers. If you have the similiar problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.32.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.32.13.img 2.6.32.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.32.13.img

to

module /initrd-2.6.32.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

If Linux kernel can’t boot even nomodeset has been set which may happen when use Intel i915 series hardware, a tricky solution is setting Xen’s vga option to let Xen use gfx console. This method can bypass the drm problem. Just replace “vga=ask” in Xen’s options with:

vga=gfx-1280x1024x32

to use 1280 x 1024 resolution with 32 bit color.

Xen’s graphical video mode codes are different from Linux Kernel’s. This is a list of Xen’s vga option from [1]:

 'vga=<mode-specifier>[,keep]' where <mode-specifier> is one of:

   'vga=ask':
      display a vga menu of available modes

   'vga=text-80x<rows>':
      text mode, where <rows> is one of {25,28,30,34,43,50,60}

   'vga=gfx-<width>x<height>x<depth>':
      graphics mode, e.g., vga=gfx-1024x768x16

   'vga=mode-<mode>:
      specifies a mode as specified in 'vga=ask' menu
      (NB. menu modes are displayed in hex, so mode numbers here must
           be prefixed with '0x' (e.g., 'vga=mode-0x0318'))

 The option 'keep' causes Xen to continue to print to the VGA console even
 after domain 0 starts to boot. The default behaviour is to relinquish
 control of the console to domain 0.

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.

You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

The clocksource=jiffies kernel parameter may be needed.

Error message related to ksm

If you get lots error messages generated by ksm, you can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

Xend conflicts with netplugd

Xend will report error when netplugd is started. Xend will print out a error message after about one minutes:

/etc/xen/scripts/network-bridge: line 240: "sigerr" command not found.

And the network interface of Linux doesn’t change. The new bridge can’t be started and the physical interface will be change name to pethx.

After disable netplugd, xend can start successfully:

# chkconfig netplugd off

References

[1] http://lists.xensource.com/archives/html/xen-changelog/2007-06/msg00206.html

–
Update history
Aug. 22, 2010. Add drm tricky and xen video mode; add netplugd caused problem.

]]> http://fclose.com/b/linux/2354/problems-during-installing-xen-dom0-in-fedora/feed/ 17 Setting Up Stable Xen Dom0 with Fedora: Xen 4.0.0 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://fclose.com/b/linux/2252/setting-up-stable-xen-dom0-with-fedora-xen-4-0-0-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://fclose.com/b/linux/2252/setting-up-stable-xen-dom0-with-fedora-xen-4-0-0-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Mon, 26 Jul 2010 16:39:39 +0000 Zhiqiang Ma http://pkill.info/b/?p=2252 Please refer to
Xen solution
for the more latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 4.0.0 from xen.org and Xenified Linux kernel 2.6.32.13. This solution have been tested quite stable in our cluster.

 

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 4.0.0

$ wget http://bits.xensource.com/oss-xen/release/4.0.0/xen-4.0.0.tar.gz
$ tar xf xen-4.0.0.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries" -y;
yum groupinstall "Development Tools" -y;
yum install transfig texi2html
libaio-devel dev86 glibc-devel
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial -y

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

The structure of these directories are as follows:

|
|-linux-2.6.32.13
|    |
|    `- patch.sh
` xen-patches-2.6.32-2
      |
      `- 6*.patch1

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 4.0.0 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-4.0.0.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

We may need to enable VT and I/O virtualization in BIOS.

These options can be enabled in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=16384
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

Here is a list of problem that may occur during the configuration.

Limited number of loop devices

The default number of loop device in this kernel is 8. When we are using blkback backed VBDs and we need to have more than 8 virtual machines, we should add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

The initramfs image under /root generated by dracut doesn’t work on some servers. If you have the similiar problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.32.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.32.13.img 2.6.32.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.32.13.img

to

module /initrd-2.6.32.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.

You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

The clocksource=jiffies kernel parameter may be needed. Please refer to Larry Matter’s reply below. He installed Xen with 32bit kernel successfully.

Error message about ksm

If you get lots error messages generated by ksm, you can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

More problems and solutions can be found here: Problems during Installing Xen Dom0 in Fedora

Update history
Aug. 10, 2010. Add clocksource kernel parameter for 32bit; Add dependency list;
May. 6, 2011. Add directory structure when compiling Xen and kernel.

]]> http://fclose.com/b/linux/2252/setting-up-stable-xen-dom0-with-fedora-xen-4-0-0-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 2 Managing Xen Dom0′s CPU and Memory http://fclose.com/b/linux/2258/managing-xen-dom0s-cpu-and-memory/ http://fclose.com/b/linux/2258/managing-xen-dom0s-cpu-and-memory/#comments Mon, 26 Jul 2010 15:12:54 +0000 Zhiqiang Ma http://pkill.info/b/?p=2258 The performance of Xen’s Dom0 is important for the overall system. The disk and network drivers are running on Dom0. I/O intensive guests’ workloads may consume lots Dom0′s CPU cycles. The Linux kernel calculates various network related parameters based on the amount of memory at boot time. The kernel also allocate memory for storing memory metadata (per page info structures) is also based on the boot time amount of memory. After ballooning down Dom0′s memory, the network related parameters will not be correct. Ballooning down busy Dom0′s memory sometimes cause SSH to die from our observation, which is a nightmare for the administrator since SSH is usually the only way for remote control of the server. Another bed effect is that it’s a waste of memory with a large memory metadata for a smaller memory amount.

Please refer to Xen solution for the more latest stable Xen Dom0 solution.

Now let’s look at how to menage Xen Dom0′s CPU and memory in a better way.

Dedicate a CPU core for Dom0

Dom0 will have free CPU time to process the I/O requests from the DomUs if it has dedicated CPU core(s). Better performance may be achieved by this since there are less CPU context switches to do in Dom0.

We can dedicate CPU core for Dom0 by passing “dom0_max_vcpus=X dom0_vcpus_pin” options to Xen hypervisor (xen.gz) in /boot/grub/grub.conf. X is the number of vcpus dedicated to Dom0.

As hyperthreading technology is enabled in most modern CPUs, we need to specify two processors to dedicate one CPU core. So the “X” above should usually be 2 for one CPU core.

kernel /xen.gz console=vga vga=ask noreboot dom0_max_vcpus=2 dom0_vcpus_pin

After booting the system, the VCPU list can be got on Dom0 by this command:

# xm vcpu-list

Even after booting the system, the VCPU number can be configured by xm command. We can set Domain-0 have two VCPUs and processor 0 and 1 to be dedicated to Dom0 by these commands:

# xm vcpu-set Domin-0 2
# xm vcpu-pin Domain-0 0
# xm vcpu-pin Domain-0 1

Dedicate memory for Dom0

We should always dedicate fixed amount of memory for Xen Dom0.

We can set the initial memory size of Dom0 by passing “dom0_mem=xxx” (in KB) option to Xen hypervisor (gen.gz) in /boot/grub/grub.conf. “xxx” is the amount of memory for Dom0 in KB.

If we set the initial memory size of Dom0 to 2GB, just change the entry in grub.conf to:

kernel /xen.gz console=vga vga=ask noreboot dom0_max_vcpus=2 dom0_vcpus_pin dom0_mem=2097152

Set lowest permissible memory for Dom0

The option dom0-min-mem in Xend configuration file /etc/xen/xend-config.sxp is used to specify the lowest permissible memory for Dom0.

The value of dom0-min-mem (in MB) is the lowest permissible memory level for Dom0. The default value is 256. If we limit the memory size of Dom0 to 2G, just set:

(dom0-min-mem 2048)

Preventing dom0 memory ballooning

The “enable-dom0-ballooning” option in Xend configuration file is used to specify whether Dom0′s memory can be ballooned out. Setting “enable-dom0-ballooning” to “no” will make sure Xend never takes any memory away from Dom0:

(enable-dom0-ballooning no)
]]> http://fclose.com/b/linux/2258/managing-xen-dom0s-cpu-and-memory/feed/ 0