The configuration of machines

Dom0

VCPU: 2 (Intel(R) Xeon(R) CPU E5520  @ 2.27GHz)
Memory: 2GB
Xen and Linux kernel: Xen 3.4.3 with Xenified 2.6.32.13 kernel

DomU

VCPU: 2
Memory: 2GB
Linux kernel: Fedora (2.6.32.19-163.fc12.x86_64)

DomU’s profile:

name="10.0.1.200"
vcps=2
memory=2048
disk = ['phy:vg_xen/vm-10.0.1.150/vmdisk0,xvda,w']
#disk = ['tap:aio:/lhome/xen/vm0-f12/vmdisk0,xvda,w']
#disk = ['file:/lhome/xen/vm0-f12/vmdisk0,xvda,w']

vif=['bridge=eth0']
bootloader="/usr/bin/pygrub"
#extra="single"
on_reboot='restart'
on_crash='restart'

The “disk” lines is changed depending on the driver used.

Benchmark method

We use Bonnie++ to test the performance of I/O:

# bonnie++ -u root

We run bonnie++ on one single VM. We also test the performance change after making a snapshot of LVM for a new VM. For the file backed VMs, we also run two VMs together on the same hard disk and run bonnie++ on them.

Bonnie++’s result is in this format:

Version 1.03e       ------Sequential Output------ --Sequential Input- --Random-
 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.locald 4G 76999  98 107423  21 47522  13 73347  91 159847  16 266.0   0
 ------Sequential Create------ --------Random Create--------
 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
localhost.localdomain,4G,76999,98,107423,21,47522,13,73347,91,159847,16,266.0,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

We only list the last line which contains all the result in the result section in this post.

Benchmark result

LVM backed VBD

localhost.localdomain,4G,76999,98,107423,21,47522,13,73347,91,159847,16,266.0,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,79588,98,120078,22,46140,13,75343,94,150167,15,248.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,81942,98,113617,22,47736,13,75947,94,152110,15,262.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

New LVM logical volume made by snapshot

localhost.localdomain,4G,11846,15,12044,2,27133,7,71510,92,141408,14,262.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,12200,15,18147,3,33086,9,66687,89,146550,14,251.9,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,58521,73,58482,10,33880,9,69399,90,144237,14,267.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,62553,78,57576,11,32755,9,70037,89,143462,14,259.9,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,66031,84,65640,12,34357,9,66036,86,152171,15,266.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
localhost.localdomain,4G,58666,75,60092,11,34826,9,72821,91,141328,14,259.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

File backed VBD

vm112,4G,20865,27,23559,4,32913,9,63006,81,128395,13,217.9,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,23022,30,18611,3,30086,8,63784,82,125736,13,197.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,21485,27,20366,3,29587,8,72130,92,140957,14,239.6,0,16,+++++,+++,+++++,+++,30751,52,+++++,+++,+++++,+++,+++++,+++
vm112,4G,22375,32,21716,3,30300,8,65488,87,128625,13,221.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,21968,28,19298,3,29007,8,68469,88,122111,12,222.5,0,16,26967,94,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,21477,28,20463,3,38395,10,49312,63,154206,15,241.0,0,16,+++++,+++,+++++,+++,32699,56,+++++,+++,+++++,+++,+++++,+++

Two VMs on the same disk running together

A:

vm112,4G,10645,13,9498,1,9606,2,30866,41,86911,8,100.1,0,16,9181,22,20583,6,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,10623,13,10143,1,10485,2,26013,35,77362,7,116.3,0,16,25701,66,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,10824,14,9558,1,12028,3,27503,36,57196,5,92.1,0,16,9679,28,+++++,+++,9294,15,+++++,+++,+++++,+++,+++++,+++
vm112,4G,15098,19,10485,1,12536,3,22771,30,64679,6,142.2,0,16,32006,82,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,11315,14,9674,1,12052,3,26453,35,68206,7,121.1,0,16,23789,62,32446,13,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm112,4G,11865,15,11564,2,11508,3,24945,34,61946,6,102.6,0,16,13297,34,21805,7,+++++,+++,+++++,+++,+++++,+++,+++++,+++

B:

vm119,4G,8963,11,9255,1,10909,3,36446,48,70485,7,125.1,0,16,19701,52,23649,8,9574,16,+++++,+++,+++++,+++,+++++,+++
vm119,4G,9074,12,8410,1,12898,3,35266,47,68469,7,107.4,0,16,6585,17,3206,1,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm119,4G,9151,13,8664,1,10285,2,20120,28,58011,5,90.9,0,16,22894,59,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm119,4G,9053,11,10406,1,12852,3,27618,37,55405,5,108.8,0,16,10144,24,22599,7,+++++,+++,+++++,+++,+++++,+++,+++++,+++
vm119,4G,8987,11,11464,1,12123,3,19278,26,59274,6,104.8,0,16,5357,13,23010,7,+++++,+++,7922,18,+++++,+++,+++++,+++
vm119,4G,9593,12,11450,1,30598,8,57078,73,119884,12,222.2,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Conclusion

From the evaluation result, we can see that LVM backed VBD of Xen DomU has a much better performance that file backed. From out experiment in our cluster, LVM backed VBD is also quite stable.

LVM volumes as backing for DomU’s file system is an appealing solution to Xen VBD. LVM volumes can dynamically grow/shrink and snapshot. These features make it simple and fast to duplicate DomU and adding storage to DomU. LVM backed DomU is recommended.

Dom0 installation and configuration

Stable Dom0 solution:

Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

DomU installation and configuration

LVM backed Xen DomU is quite stable with high performance that uses unmodified Fedora pv_ops kernel:

Setting Up LVM Backed Xen DomU

Setting up Stable Xen DomU with Fedora: Unmodified Fedora 12 on top of Xenified Fedora 12 Dom0 with Xen

Setting Up Ubuntu DomU on Xen: Use Ubuntu 10.10 on Fedora Xen Dom0

Dom0 management

Dom0′s CPU and memory configuration advises:

Managing Xen Dom0′s CPU and Memory

DomU management

Duplicating LVM Backed Xen DomU

Duplicating and Backing Up LVM Backed Xen DomU from a Remote Server

Create and manage virtual machines on Xen

How to Duplicate Xen DomU Virtual Machines

Automatically backing up Xen File-backed DomU

Unified Xen DomU configuration file

LVM with Xen

A summary of tutorials related to Xen and LVM:

Xen with LVM

Problems

Some problems that may happen and their solution:

Problems during Installing Xen Dom0 in Fedora

Xen solution

for the latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.31.12.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries";
yum groupinstall "Development Tools";
yum install transfig texi2html
libaio-devel dev86 glibc-devel
e2fsprogs-devel gitk mkinitrd
iasl xz-devel bzip2-devel
pciutils-libs pciutils-devel
SDL-devel libX11-devel gtk2-devel
bridge-utils PyXML qemu-common
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.31.12

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.12.tar.bz2
$ tar xf linux-2.6.31.12.tar.bz2

Download 2.6.31 Xen patches v14

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-14.tar.bz2
$ mkdir xen-patches-2.6.31-14
$ tar xf xen-patches-2.6.31-14.tar.bz2 -C xen-patches-2.6.31-14

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.31-14/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-2.6.31.12-xenified

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 
Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 <*>     Network-device loopback driver (NEW)
 < >   PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 < >   SCSI backend driver (NEW)
 < >   USB backend driver (NEW)
 < > Block-device frontend driver
 < > Network-device frontend driver
 < > SCSI frontend driver (NEW)
 < > USB frontend driver (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (3.0.2 and later)  --->

Build kernel

$ make -j16

-jN: N may be 16 or other numbers depending on the number of processors in the system.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.31.12
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.31.12 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.31.12.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

The hardware platform:

DomU:

CPU: 2 x Intel(R) Xeon(R) CPU E5520 @ 2.27GHz

Memory: 1G

HD:

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
              ext4     16G  2.0G   13G  14% /
/dev/xvda1    ext3    194M   23M  162M  13% /boot
tmpfs        tmpfs    517M     0  517M   0% /dev/shm

Dom0:

The raw image file is stored on a ext4 partition.

More details can be found here.

Test method

Bonnie++ 1.03c

Using default parameter.

Result

Loopback driver backed:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm101         2064M 25511  35 18075   3 199488  47 71094  98 937880  86 +++++ +++
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
vm101,2064M,25511,35,18075,3,199488,47,71094,98,937880,86,+++++,+++,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

blktap driver backed:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm101         2064M 69438  96 93549  20 38118  10 54955  76 131645   8 249.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 29488  79 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
vm101,2064M,69438,96,93549,20,38118,10,54955,76,131645,8,249.1,0,16,29488,79,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

From the result we can see that the loopback backed VBD has better read performance with high CPU usage while it has worse write performance. The blktap backed VBD has a more balanced performance. It has a much better write speed than the loopback backed one. With a bit worse read performance, we can get a much better over all performance. So from the view of performance, blktap driver is better than the loopback driver for Xen DomU’s VBD usage.

There are some other benefits we can get by using blktap driver. The loopback file-backed VBDs may not be appropriate for backing I/O-intensive domains because this approach is known to experience substantial slowdowns under heavy I/O workloads, due to the I/O handling by the loopback block device used to support file-backed VBDs in dom0 [1]. Another reason is the blktap can provides better scalability than loopback backed driver. Linux only support at most eight loopback file-backed VBDs across all domains by default. If we want to have more than eight loopback devices, the max_loop=n boot option should be passed to the kernel or module depends on whether CONFIG_BLK_DEV_LOOP is conpiled as a module of the Dom0 kernel. The method can be found here. And some other advantages such as easily support for metadata disk formats such as Copy-on-Write, encrypted disks, sparse formats and other compression features, avoid the flushing dirty pages problem which are present in the Linux loopback driver, and some more [2].

Referrences

[1] http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/
[2] http://wiki.xensource.com/xenwiki/blktap

This is the method:

Add these lines to the ~/.mplayer/config file:

For setting the startup default volume:

volume=28

For setting the startup position to (0,0) (top left):

geometry=0:0

I have done some simple performance test on DomU and Dom0 and compare with the performance on physical machines. These test are simple, but it can provides some performance factor of xen. I test the CPU bound, memory write and memory read performance of Dom0, DomU and compare them with the result of the test on physical machine. Each test are done for 50 times and the the Average (E), Standard deviation (SD) and SD/E are calculated.

Test method:

I only past the C code here because I think it is clear enough.

1) CPU bound:
The code:

  const int test_limit = 100000;
  cout << "-----------------------------" << endl;
  cout << "cpu bound test begins." << endl;
  clock_t begin_time = clock();
  register int a = 0;
  for (register int i = 0; i < test_limit; i++) {
    for (register int j = 0; j < test_limit; j++) {
      a += i + j;
    }
  }
  clock_t end_time = clock();
  do_nothing(a);
  int time = (end_time - begin_time) / (CLOCKS_PER_SEC / 1000);

2) Memory read:

The code:

  const int test_times = 1000000;
  const int data_size = 1000000000;
  const int data_interval = 1000000;
  int* array_read = new int[data_size];
  cout << "----------------------" << endl
       << "Memory read bound test begins." << endl;

  register int read_value = 0;
  long read_begin =0;
  long read_end = 0;
  int data_range = data_size - test_times;

  read_begin = clock();
  for (register int t = 0; t < test_times; t++) {
    for (register int i = 0; i < data_range; i += data_interval) {
      read_value = array_read[i + t];
    }
  }

  read_end = clock();
  do_nothing(read_value);
  int time = (read_end - read_begin) / (CLOCKS_PER_SEC / 1000);

3) Memory write:

The code:

  const int test_times = 1000000;
  const int data_size = 1000000000;
  const int data_interval = 1000000;

  int* array_write = new int[data_size];

  cout << "----------------------" << endl
       << "Memory write bound test begins." << endl;

  long write_begin = 0;
  long write_end = 0;
  int data_range = data_size - test_times;
  write_begin = clock();

  for (register int t = 0; t < test_times; t++) {
    for (register int i = 0; i < data_range; i += data_interval) {
      array_write[i + t] = 528283;
    }
  }

  write_end = clock();
  int time = (write_end - write_begin) / (CLOCKS_PER_SEC / 1000);

The result:

Performance result on the physical machine:

Performance result on Dom0:

Performance result on DomU:

From the test, we can see that the CPU performance is nearly 100%. But the memory performance is not so good especially for the DomU. This maybe because of the memory validation of Xen. But the CPU and memory performance of Xen VM is pretty good as a virtual machine.