• Sales: 1-800-961-2888
  • Support: 1-800-961-4454

Using a Custom Kernel with pv-grub


Contents

Obligatory Disclaimers

  • pv-grub is an advanced feature. It should only be used by experienced Linux administrators with a demonstrated need to run their own kernels.
  • Not all kernels are supported. Your kernel must have pv-ops support for running in a Xen domU. Also, kernels compressed with bzip or LZMA will not work.
  • Follow the instructions as listed, in the order listed. Do not blindly copy/paste these commands, as some of them will require editing.
  • Do not attempt these instructions on a production system. If they are applied incorrectly, your Cloud Server may become unbootable.

 

Grub configuration

Every instance booted with pv-grub requires a working grub configuration file in /boot/grub/menu.lst:

default=0
timeout=5
 
title=DISTRO-NAME "KERNEL VERSION STRING"
    root (hd0)
    kernel /boot/KERNEL-VMLINUZ-FILENAME ro console=hvc0 root=/dev/xvda1
    initrd /boot/KERNEL-INITRD-FILENAME

 

Distribution-specific adjustments

Arch

pacman -Sy pacman
pacman -Sy base-devel
pacman -Syu
sed -i 's/sda/xvda/' /etc/fstab
pacman -Sy kernel26
mkdir -p /boot/grub
cp /usr/src/KERNEL-VERSION/vmlinux /boot/vmlinux26

Note: The stock arch kernel is bzipped and won't work with pv-grub. Use the uncompressed vmlinux instead.

Now add (or uncomment) the following line in /etc/inittab for console:

h0:2345:respawn:/sbin/agetty -8 38400 hvc0 linux

CentOS 5.x

yum -y upgrade
mkdir -p /boot/grub
echo "alias eth0 xennet" > /etc/modprobe.conf
echo "alias eth1 xennet" >> /etc/modprobe.conf
echo "alias scsi_hostadapter xenblk" >> /etc/modprobe.conf
echo "xvc0" >> /etc/securetty
sed -i 's/sda/xvda/' /etc/fstab
yum -y install kernel-xen

Now add (or uncomment) the following line in /etc/inittab for console:

8:2345:respawn:/sbin/mingetty xvc0

Debian 5.x

apt-get update; apt-get -y upgrade
mkdir /boot/grub
sed -i 's/sda/xvda/' /etc/fstab
apt-get -y install linux-image-xen-amd64

Now add (or uncomment) the following line in /etc/inittab for console:

8:2345:respawn:/sbin/getty 38400 hvc0

Fedora 13

mkdir -p /boot/grub
echo "alias eth0 xennet" > /etc/modules
echo "alias eth1 xennet" >> /etc/modules
echo "alias scsi_hostadapter xenblk" >> /etc/modules
sed -i 's/sda/xvda/' /etc/fstab
echo "hvc0" >> /etc/securetty
yum -y install kernel

Gentoo

emerge --sync
emerge --update portage
emerge gentoo-sources genkernel #Update /usr/src/linux symlink if necessary
sudo sed -i 's/sda/xvda/' /etc/fstab
genkernel --menuconfig all

Enable the following in the kernel configuration:

Processor Type and Features -> Paravirtualized guest support -> "Y"
Hit enter, go to "Xen guest support" -> "Y"

Then exit the configuration, save, and build.

Now add (or uncomment) the following line in /etc/inittab for console:

h0:12345:respawn:/sbin/agetty 38400 hvc0 linux

RHEL 5.x

yum -y upgrade
mkdir -p /boot/grub
echo "alias eth0 xennet" > /etc/modprobe.conf
echo "alias eth1 xennet" >> /etc/modprobe.conf
echo "alias scsi_hostadapter xenblk" >> /etc/modprobe.conf
echo "xvc0" >> /etc/securetty
sed -i 's/sda/xvda/' /etc/fstab
yum -y install kernel-xen

Now add (or uncomment) the following line in /etc/inittab for console:

8:2345:respawn:/sbin/mingetty xvc0

Ubuntu 9.10/10.04

mkdir /boot/grub
cp /etc/init/tty1.conf /etc/init/hvc0.conf
sed -i 's/tty1/hvc0/' /etc/init/hvc0.conf
sed -i 's/sda/xvda/' /etc/fstab
apt-get -y install linux-virtual

Ubuntu 10.10

mkdir /boot/grub
cp /etc/init/tty1.conf /etc/init/hvc0.conf
sed -i 's/tty1/hvc0/' /etc/init/hvc0.conf
apt-get -y install linux-virtual

Kernel upgrade

Contact Rackspace Cloud support and request the pv-grub feature, informing them that you have completed these steps and would like to use your own kernel. Your Cloud Server will be rebooted, and -- if all goes well -- will come back up with your new kernel. You may verify this with 'uname -a'.



© 2011-2013 Rackspace US, Inc.

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License


See license specifics and DISCLAIMER

13 Comments

Do you mean pygrub here? pv-grub is totally different than pygrub.

Nope, the article does mean pv-grub. We've started using pv-grub as the bootloader for newer instances (Fedora 15 and I think any we release after that). You can also ask support to switch an older instance to pv-grub - that situation is mainly what this article covers, talking about changes that should be made to the system when switching from using a kernel external to the instance to using one inside the instance using pv-grub.

In Archlinux, it's important to add "xen-blkfront xen-fbfront xenfs xen-netfront xen-kbdfront" to the /etc/mkinitcpio.conf MODULES section (or some combination therein, I just added every xen module), and then regenerate the initrd, or else it won't boot correctly :-)

Thanks Bryce. I haven't tried compiling my own kernel on Arch, I admit, but if you use the stock kernel package you can get away with skipping the mkinit stuff. You just have to make sure you use the uncompressed kernel instead of the bzipped one they install by default, since (last time I tried it) our Xen appears to have trouble with anything other than uncompressed or gzipped.

For latest stock kernel in Arch, I had to set barrier=0 in /etc/fstab e.g.:

/dev/xvda1 / ext3 defaults,barrier=0 0 1
/dev/xvda2 swap swap defaults,barrier=0 0 0

Otherwise, root fs will get mounted read-only very often with error messages like:

[ 8.389389] end_request: I/O error, dev xvda1, sector 1053144
[ 8.389406] end_request: I/O error, dev xvda1, sector 1053144
[ 8.389415] Buffer I/O error on device xvda1, logical block 131643
[ 8.389444] Aborting journal on device xvda1.
[ 8.390160] EXT3-fs (xvda1): error: ext3_journal_start_sb: Detected aborted journal
[ 8.390178] EXT3-fs (xvda1): error: remounting filesystem read-only

Curious. Thanks, Santo. We'll look into that and get the article updated appropriately.

This is also needed in gentoo 3.1+ kernels.

Hi Jered,

I second Santo's comment. I just got linux 3.1 to install on my node today, and only by setting barrier=0 was I able to make any writes to disk. There may be a better way to make the disk writable, however I am unaware of it.

Also, a note: The uncompressed image does _not_ work with linux 3.1, while the vmlinuz-linux bzImage (installed by the Arch package "linux") _does_ work (my node is currently running on vmlinuz-linux; however, it kernel panicked with the uncompressed image).

Thanks!

Matt

Additional note: I found the Arch wiki's <a href="https://wiki.archlinux.org/index.php/Xen">Xen page</a> to be extremely helpful, in addition to this page.

I would like to see a complete step-by-step for Ubuntu, with an explanation of each step. The procedure offered in the article is very abbreviated and contrast rather sharply with the one at http://failverse.com/setting-up-pv-grub-cloud-servers/

Why not just have pv-Grub enabled for instances by default? Some type of failover is needed to boot the original virtual image if the customer/new kernel does not boot.

Our newer images actually do use pv-grub by default now, so as new versions are released for various distributions you should see them all with pv-grub support right off the bat.

We have to port this article over to Rackspace Cloud still, but you can follow the instructions in this Slicehost article since the procedure is the same:

http://articles.slicehost.com/2010/10/27/using-pv-grub-to-run-custom-kernels-overview

Access to the command console and rescue mode would be done through the RS control panel, of course, but otherwise the instructions should still apply.

Here's what I did for moving to pv-grub on Ubuntu 12.04, 3.2 series
kernel on a first generation server:

Setup /boot/grub/menu.lst as above, but add rootflags=nobarrier to the
kernel argument line:

kernel /boot/vmlinuz-3.2.0-35-virtual ro console=hvc0 root=/dev/xvda1 rootflags=nobarrier

Replace sda with xvda in /etc/fstab, and add a nobarrier mount option
on the root filesystem:

/dev/xvda1 / ext3 defaults,errors=remount-ro,noatime,relatime,nobarrier 0 1

In /etc/init, copy tty1.conf to hvc0.conf and replace tty1 with hvc0
in the new file.

On 3.2 series kernels, if you leave out or typo the nobarrier option
your root filesystem will error on boot and get mounted read-only.

My swap device turned out to be /dev/xvda2, confirmed with blkid before running swapon or modifying my fstab.

In case people are are curios about what kernel options are needed, you can enable all the things (for xen that is) and generally be good. (for 3.7)

CONFIG_XEN=y
# CONFIG_XEN_PRIVILEGED_GUEST is not set
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PRIVCMD=y
CONFIG_GRKERNSEC_CONFIG_VIRT_XEN=y

Add new comment