This site is now 100% read-only, and retired.

Booting Xen 3.0 guests using NFS

Posted by Steve on Thu 8 Mar 2007 at 16:18

Tags: , ,

One of the nice things about using Xen is that it doesn't require much setup to create new guests - just a loopback file or two, or an LVM partition. If you use NFS to store your remote systems you don't even need that. Here we'll give a quick example of booting Xen guests which will mount their root file-systems via NFS.

To boot Xen domains over the network you'll need several things working:

  1. A Xen kernel with NFS Root support.
  2. A remote NFS server
  3. A Xen configuration file setup to use NFS

Xen domU kernel with NFS ROOT support

Unfortunately the Debian Xen packages do not contain kernels which have support for NFS mounting of their root filesystem, so if you want to use NFS-mounted domU upon Debian you must compile your own kernel.

Building your own kernel doesn't take too long, and you're still able to use the other Debian-supplied Xen packages (such as the hypervisor, and the tools).

Since I wanted NFS root support, and a recent kernel, I chose to use the xen-unstable repository, which is the development release of Xen.

To pull this down you'll need the mercurial revision control package:

apt-get install mercurial

Once you have it you can fetch the most current version of the source by running:

hg clone

Change into the directory and now you're ready to build the kernel. Since we have the Debian packages we don't want to build anything else, not the hypervisor, nor the Python tools.

First of all configure the source, then build and install it:

# make linux-2.6-xen-config CONFIGMODE=menuconfig
# make linux-2.6-xen-build
# make linux-2.6-xen-install

When configuring the source you want to select the following options :

 Network Filesystems
  [*] NFS file system support
  [*] Root file system on NFS

This should ensure that CONFIG_ROOT_NFS is set to "y" in your configuration file.

Any other choices which suit your environment are fine, but these are the reason for rebuilding in the first place so don't forget them!

All being well you'll find a new kernel and modules installed upon your dom0 system, and we can proceed to setup the NFS server with our new system.

With the rebuild, providing you compile NFS support into the kernel you'll not need an initrd image.

The NFS Server

I'm going to setup a machine called "" ( to serve the root filesystem for my Xen guest - since I have only one guest I will use a simple scheme which doesn't involve significant customisation of the exported filesystem.

If you were using this kind of system in production you would probably want to have either one NFS share per-client, or do some other fancy organisation.

To start with we'll need an NFS server installed, if we don't have one already:

apt-get install nfs-user-server portmap

Now we'll create a directory to export:

mkdir -p /mnt/etch

Add this to the /etc/exports file, so that it can be served via NFS:

/mnt/etch (rw,sync,no_root_squash)

Note: here we've exported this filesystem such that any host within the range has full read & write access to it. This is overly permissive.

Now we want to populate the filesystem with something the client can boot. We've previously shown how to use debootstrap to create new Debian installations, so we'll gloss over the details:

debootstrap etch /mnt/etch/

This gives us a basic system, but it won't be very usable since it is missing some files. We'll simply copy some files from the host to make it more complete:

cp /etc/resolv.conf /mnt/etch/etc
cp /etc/hosts /mnt/etch/etc
echo "" >> /mnt/etch/etc/hostname

Now we'll add an SSH server package, and a couple more:

chroot /mnt/etch/ /usr/bin/apt-get install openssh-server nfs-common portmap

Finally we need to make sure that the /etc/fstab file upon the guest is correct - so we'll add the following to it: /    nfs rw    0 0
proc          /proc proc     rw,nodev,nosuid,noexec              0     0
sys          /sys   sysfs     rw,nodev,nosuid,noexec              0     0

The last step is to copy the built kernel modules from /lib/modules/2.6.18-xen on the Xen host to the NFS-exported filesystem, so that module loading works correctly when our client boots.

Note: is the IP address of the NFS server.

Now we should be almost complete - we have:

  • The NFS server which is exporting a root filesystem.
  • The Xen domU kernel which should allow us to mount that remotely.

Whats missing is the Xen configuration file.

NFS Xen configuration file

Each xen domain (domU) typically has a configuration file beneath the directory /etc/xen, so we'll create a new one for this NFS-booting client, /etc/xen/

# Common things.
kernel  = '/boot/vmlinuz-2.6.18-xen'
memory  = '64'

#  Name
name     = ''
hostname = ''

# Networking basics
vif     = [ 'ip=' ]
netmask   = ''
gateway   = ''
ip        = ''
broadcast = ''

# NFS option
nfs_server = ''
nfs_root   = '/mnt/etch'
root       = '/dev/nfs'

Here the kernel is the kernel we've previously built and I've setup the client to have a static address. If you wanted you could use DHCP too - I just like static IPs for Xen guests.

Now that we have everything we can try starting this Xen guest domain:

# xm create -c

All being well you should see your machine boot up with the static address we've allocated in the configuration file, mount its root filesystem via NFS, and boot as normal.



Re: Booting Xen 3.0 guests using NFS
Posted by dom (217.147.xx.xx) on Thu 8 Mar 2007 at 17:19
It might be worth explicitly mentioning that running NFS servers is something that should only ever be done on trusted networks (unless with fancy v4 authentication features).


[ Parent ]

Re: Booting Xen 3.0 guests using NFS
Posted by Steve (80.68.xx.xx) on Thu 8 Mar 2007 at 17:23
[ View Weblogs ]

Good catch. I should probably have been more explicit when I warned that opening it up to was overly permissive / a security hole waiting to happen.

Still per-host should be reasonably safe.


[ Parent ]

Re: Booting Xen 3.0 guests using NFS
Posted by Anonymous (81.192.xx.xx) on Thu 12 Apr 2007 at 16:34
Actually, even the per host restriction fails to protect from arp spoofing, which is a realistic attack from someone w/ a laptop per se.
damn that nfsv4 gotta be easier to deploy!

[ Parent ]

Re: Booting Xen 3.0 guests using NFS
Posted by Anonymous (220.214.xx.xx) on Wed 20 Feb 2008 at 14:07
Hi Steve,

Thanks for this step-by-step instruction. It cannot be easier and more clear than this. But with this I got the NFS version 2 working. I need to work on version 3. I even included nfsvers=3 in the /etc/fstab file. It looks like this / nfs rw,nfsvers=3,hard,intr,async 0 0

But still, its going in version 2. I just did apt-get install nfs-common portmap while installing nfs. Is it that i need to install further packages to support nfs version 3??

Thanks in Advance for your help.


[ Parent ]

Re: Booting Xen 3.0 guests using NFS
Posted by Anonymous (131.113.xx.xx) on Mon 25 Feb 2008 at 08:00

AS I had suspected this was just a small piece of missing configuration in config file. All we need to do was to mention the version and transport protocol in the config file as follows:

nfs_root = 'xxx/xx/xx ,v3,tcp'


[ Parent ]

Re: Booting Xen 3.0 guests using NFS
Posted by Anonymous (125.160.xx.xx) on Wed 4 Jun 2008 at 07:19

how to limited disk space when we using /dev/nfs ?
when i check on ubuntu domU using /dev/nfs i couldn`t found information about diskpace.

and how to we create swap ?

root@10:/# df -h
Filesystem Size Used Avail Use% Mounted on
varrun 129M 28K 129M 1% /var/run
varlock 129M 0 129M 0% /var/lock
udev 129M 8.0K 129M 1% /dev
devshm 129M 0 129M 0% /dev/shm

please advice

[ Parent ]

Re: Booting Xen 3.0 guests using NFS
Posted by Anonymous (194.145.xx.xx) on Sat 26 Jul 2008 at 21:45
I guess you must first mount it as a mounted FS. So add to your /etc/fstab something like:
NFSERVER:/path/to/root / nfs defaults 0 0

[ Parent ]