Adjusting Dom0 and Xenheap Settings in XenServer

Summary

This article describes the correct method for configuring Dom0 memory and Xenheap settings in XenServer 5.5 as described in KB article CTX124086 – XenServer Single Server Scalability with XenDesktop.

Procedure

This procedure only applies to the XenServer 5.5 and updates 1 and 2. In version 5.5, which uses Xen 3.3, the Xenheap is statically allocated, defaulting to 16 MiB.

Note also that a host reboot is necessary for this change to take effect.

1. Using the example in CTX124086, modify the Xen command-line found in /boot/extlinux.conf to contain “xenheap_megabytes=24” and “dom0_mem=2940M”.

2. Log on to dom0 as root (either through the console in XenCenter or by SSH access).

3. Open the file /boot/extlinux.conf in an editor. Notice that this file contains a header followed by several sections, each introduced by a label. In each section, there is a line prefixed “append”. This line specifies the arguments that are passed to Xen. The one that is usually used is the one with label “xe”. To be sure, make the change in each section.

4. Insert “xenheap_megabytes=24” into the “append” line in each section. Put it before the first “—”. Also change the existing “dom0_mem” parameter to read “dom0_mem=2940M”.

For example, change the following:

label xe # XenServer kernel mboot.c32 append /boot/xen.gz dom0_mem=752M lowmem_emergency_pool=16M crashkernel=64M@32M console=/dev/null vga=mode-0x0311 — /boot/vmlinuz-2.6-xen root=LABEL=root-kzmhdirt ro quiet vga=785 splash — /boot/initrd-2.6-xen.img

to:

label xe # XenServer kernel mboot.c32 append /boot/xen.gz dom0_mem=2940M lowmem_emergency_pool=16M xenheap_megabytes=24 crashkernel=64M@32M console=/dev/null vga=mode-0x0311 — /boot/vmlinuz-2.6-xen root=LABEL=root-kzmhdirt ro quiet vga=785 splash — /boot/initrd-2.6-xen.img

Adjusting xenheap Settings:

• The “xenheap_megabytes=24” change is a workaround to a known issue with the version 5.5 release and its updates that would otherwise cause an artificial ceiling to be reached even though the host should be capable of starting more VMs. You would see error messages such as the following appearing in /var/log/xensource.log when trying to start a VM:
Xc.Error(“creating domain failed: hypercall 36 fail: 12: Cannot allocate memory (ret -1)”) even though the host has enough memory to start the VM.

• The default xenheap size is 16 megabytes. Changing it to 24 megabytes was done in this particular environment to start that number of VMs. In general, setting xenheap_megabytes to (12 + max-vms/10) is probably a good rule to use.

Adjusting Dom0 Memory Settings

• The “dom0_mem=2940M” change gives more memory to dom0 which means that it can better handle large numbers of VMs. After changing this setting and rebooting, you then need to instruct dom0 to consume all of the memory allocated to it.

• On XenServer 5.5, including updates 1 and 2, the following sequence of commands can be used to achieve this:
. /etc/xensource-inventory
staticmax=`xe vm-param-get uuid=$CONTROL_DOMAIN_UUID param-name=memory-static-max`
echo staticmax=$staticmax
xe vm-param-set uuid=$CONTROL_DOMAIN_UUID memory-dynamic-max=$staticmax
xe vm-memory-target-set uuid=$CONTROL_DOMAIN_UUID target=$staticmax

You can confirm it worked by examining the contents of /proc/xen/balloon. The sum of ‘Current allocation’, ‘Low-mem balloon’ and ‘High-mem balloon’ should be equal to (or be pretty close to) the value you specified in /boot/extlinux.conf. (Usually the latter two of these parameters are zero.)
You only need to execute those commands once. The values you set for the dom0 memory target are saved in the xapi database which persists across reboots.

• There are some (rare) circumstances in which XenServer adjusts the memory target automatically (adds more memory to the host), but these should not affect you because you have set the target as high as static-max.
This means that if you want to go back to the old behavior, with a standard-sized dom0, you must reverse your changes – rebooting does not revert your configuration. If you undo the edits to /boot/extlinux.conf and reboot, executing the same commands should have the effect of causing dom0 to revert to 752 MiB.

Caution! The default value for the “dom0_mem” parameter is 752 MB. There is a historic reason for this value. It may be that on some machines going above this value causes poor performance. Modifying the dom0_mem parameter is therefore only supported in XenDesktop 4 deployment scenarios as described in CTX124086 – XenServer Single Server Scalability with XenDesktop.