The kernel

Deep at the core of any Linux installation is the kernel, the bundle of instructions that ties the whole system together. Without the kernel Linux isn't Linux - indeed, technically Linux is just the kernel. Everything else comes from the distribution.

Rackspace made a change to how the kernel is bundled with our newer distribution images. The kernel used to be kept outside the instance; now it will be kept in the file system on instances built from the new images.

The change is as simple as that. The kernel has always been around, so changing where it's kept won't change how servers work. No extra hassle is introduced with the kernel relocation.

Most users will never want to mess with their kernel wherever it's kept. However, some users do want to change their kernel, and this change benefits them. For those people, and for others who are simply curious about what's changed, let's look at what's what.

Where is it?

To find your current kernel, or to see if you've installed an instance with an internal kernel, check the "/boot" directory:

ls /boot

If the directory is empty or doesn't exist then your kernel is outside the instance. It's possible to change the location of the kernel for an existing instance, but it takes a bit of work (and some help from support staff). We'll post detailed instructions for getting that set up soon.

On the other hand, if you got a listing back you'll see names in there like "grub", "vmlinuz", and "System.map".

The kernel itself is the "vmlinuz" file (or, sometimes, "vmlinux"). There might be more than one kernel file.

Most of the rest are supporting files, but the "grub" directory is worthy of special attention.

grub

Grub isn't the kernel, but it is the "boot loader" - it tells the system what kernel to use.

Grub displays a menu at boot time if you have more than one kernel choice available. You'll only see it if you're in the web console when the server starts up (described a bit more in the "Troubleshooting" section further on).

menu.lst

Grub gets the kernel and menu information from a file in the grub directory, "menu.lst".

The menu.lst file is the place to look to find out which kernel file the system wants to boot from. Let's look at a simple menu.lst file and interpret its contents:

default=0
timeout=5
 
title=Kernel 2.6.32-25
     root (hd0)
     kernel /boot/vmlinuz-2.6.32-25-server ro console=hvc0 root=/dev/xvda1
     initrd /boot/initrd.img-2.6.32-25-server

The "default" entry tells grub which kernel choice to boot with by default - "0" is the first listed kernel (in this case, the only one).

On a related note, the "timeout" line tells grub how long to wait for someone to use the menu via the web console before booting the default kernel. Five seconds should give you enough time to use the menu without delaying a regular boot by too much.

The rest of our sample file is a kernel description. The "title" line is exactly that. When it's shown in a menu the kernel is identified by that title, so make it distinctive.

The "root" line tells grub what disk partition holds the kernel. That should be "hd0".

The "kernel" line is, well, the kernel. The first bit is where the kernel file is located. The rest are kernel arguments you won't generally want to change (the default terminal console and the partition holding the Linux root file system, respectively).

The "initrd" line points to an initial RAM disk to help the system boot - usually a file with "initrd" or "initramfs" in the name that gets installed with the kernel.

Changing menu.lst

If you install another kernel you don't need to get rid of the one you have - just add a new entry in "/boot/grub/menu.lst" for the newcomer. Copy the existing kernel entry (from the "title" line on, in the sample) and change the details to match the new kernel.

For example, let's add a new kernel entry to our sample file and make it the new default:

default=1
timeout=5
 
title=Kernel 2.6.32-25
     root (hd0)
     kernel /boot/vmlinuz-2.6.32-25-server ro console=hvc0 root=/dev/xvda1
     initrd /boot/initrd.img-2.6.32-25-server
 
title=Custom kernel 2.6.39.3
     root (hd0)
     kernel /boot/vmlinuz-2.6.39.3 ro console=hvc0 root=/dev/xvda1
     initrd /boot/initrd.img-2.6.39.3

By keeping the old kernel in the menu you have a fallback you know will work in case the new kernel doesn't.

You can get an in-depth look at menu.lst and how it controls grub from the official grub documentation.

Kernel packages

Now that you know where the kernel is let's look at how your system maintains it.

By default your instance will use a kernel drawn from its distribution's package repository. For most distributions that means your kernel might be updated when you run a general system update.

It can be a trick working out which kernel package your system is using, however. Different distributions use different package names. Sometimes those names change between distribution versions.

Here are some search commands that should return lists of kernel packages for different distributions. Check the search results to see which package is listed as "installed".

Fedora

yum search kernel

CentOS

yum search kernel

Ubuntu

aptitude search linux-

Most Ubuntu kernels start with "linux-image", but not all of them - "linux-virtual" being one example at the time of this writing.

Debian

aptitude search linux-image

Red Hat Enterprise Linux

yum search kernel

Arch

pacman -Ss kernel

Gentoo

emerge --search genkernel

Gentoo is more involved than that, actually - genkernel is just a tool to help compile a kernel yourself. You can use other approaches too, but their guide for genkernel is a decent start.

Update menu.lst

Some distributions will update /boot/grub/menu.lst after installing a newer kernel, but always make sure before rebooting and putting it to the test. Verify that the "default" value and the corresponding kernel entry match your expectations. If not you may need to make the addition manually.

If you compile your own kernel you'll almost certainly need to add a menu entry for it yourself.

Speaking of...

Compiling your own kernel

Compiling a kernel for yourself can be complicated and fraught with peril. We won't talk about it in detail here, but we do have a couple recommendations for the adventurous.

First, run "make localmodconfig" on your kernel source to grab the config for the kernel you're already running. That can be helpful as a base.

Enable Xen options in the config. Xen guest support is particularly important.

Use gzip compression for the kernel. At the time of this writing bzip compression can keep the system from reading the kernel properly. If a gzipped kernel isn't available use an uncompressed kernel instead (created as "vmlinux" in the root of the source directory after compiling).

Troubleshooting

If your server isn't coming up after installing a new kernel, the first thing to do is to hop into the web console. You can get to it through the Cloud Control Panel by selecting the overview for the server you're working on and clicking the "Console" button.

The web console acts like an ssh session, just a bit less responsive and it stays up through a restart. If you reboot the server with the web console open (use another browser window to trigger the reboot) you can watch for kernel-related error messages when the system tries to come up.

Even better, in the web console you'll get to see the grub menu. You can use that to boot from your old kernel - the one you know will work.

Rescue Mode

If you didn't keep a working kernel option in the grub menu, or if you altered other parts of the system that would keep the old one from working (like device names or fstab entries), you'll need to make some changes to the instance before you can boot normally. That's where rescue mode comes in.

This article describes rescue mode in-depth, but to summarize the important parts: Go into the server overview in the control panel, click "Rescue Mode", wait for the server to become available, then ssh in with the root password provided.

Once you're in you can mount your file system.  The device holding your root file system may vary depending on the image it was built from, and that's detailed a bit more in the article linked above.  In simplest terms, run:

fdisk -l

Then look for a device with the same size as your instance's disk.  It should be either /dev/sda1, /dev/sdb1, or /dev/xvdb1.

To mount your device, replace "/dev/sda1" with the proper device name in the following command:

mount /dev/sda1 /mnt

...And then make the changes you need. Just remember to put "/mnt" in front of the usual file paths since you're mounting your file system under the rescue mode's host environment.

If all else fails

As a last-ditch effort, if you need a new working kernel you can try spinning up a new server using the same base image as the troubled instance. Bring the misbehaving server up in rescue mode, copy the kernel and associated files over from the new instance, update menu.lst, then restart.

Don't forget to delete the temporary instance you created when you're done.

Summary

It bears repeating: If you don't want to mess with your kernel you don't have to. Your instance will find its kernel without you needing to do anything, wherever it's being kept.

But if you do want to change kernels - for a particular feature or module, or to use a newer version than comes with your distribution - you've got that freedom. Just remember to keep a known kernel handy and update your menu.lst.