This site is now 100% read-only, and retired.

An introduction to custom Xen networking

Posted by Steve on Mon 27 Feb 2006 at 05:16

I've been running Xen for a few weeks now and until now I've been happy with the default networking setup installed. Only when I decided to install Xen upon the server which is hosting this website did I need to explore the way Xen sets up networking.

Xen is pretty good at giving a working network setup for most common cases. By default it sets up virtual instances so they communicate with the network via the host's eth0 device, using NAT.

There are several other supported configurations for the times when this setup isn't appropriate these are:

  • bridged networking.
  • routed networking.

Each of the available network setups are implemented via a pair of scripts inside the directory /etc/xen/scripts. For example the bridge setup is implemented in the two files:

  • /etc/xen/scripts/network-bridge
  • /etc/xen/scripts/vif-bridge

In order to change the network setup you simply tell the xend daemon which script(s) it should be using via the configuration file /etc/xen/scripts/xend-config.sxp.

To switch to the bridge setup, for example, you'd make sure the following settings were present:

(network-script network-bridge)
(vif-script vif-bridge)

When I needed to host some Xen instances upon this machine I needed to use the routed setup, which could be done by setting:

(network-script network-route)
(vif-script vif-route)
  • My server has the public IP address 80.68.80.176.
  • My hosting provider gave me an additional range 80.68.86.192/29 for use by Xen.

The new IP addresses 80.68.86.192/29 were each routed via my existing public address - and in this scenario I couldn't use NAT, since these are public addresses.

Instead I needed to use a bridge. The range I've been given /29 means that I have 6 public IP addresses. One of these must be allocated to the bridge - leaving me with 5 addresses for use by Xen machines.

The following diagram shows what I needed to do:

This image was created using Dia; source file available

The way that bridging works is that the host gains an IP address in the new range, and the virtual instances use this IP address as their default gateway, or route, to the internet.

To setup the bridge required two steps:

  • Setup the xend-config.sxp file, as previously discussed.
  • Make sure the host has an IP address allocated on the bridge.

To give the host an IP address on the bridge we can't use virtual addresses so we manage it by using the dummy driver:

skx2:~# modprobe dummy
skx2:~# ifconfig dummy0 80.68.86.193 up

To make sure that these settings persist we add the name 'dummy' to the file /etc/modules - and add the dummy network address to the file /etc/network/interfaces as follows:

auto dummy0
iface dummy0 inet static
        address 80.68.86.193
        broadcast 80.68.86.199
        netmask 255.255.255.248

Once this is done we've got the host setup correctly. Restarting xend will allow it to create the bridge:

root@skx2:~# /etc/init.d/xend restart

Now we just need to create the virtual instances giving them IP addresses from the range 80.68.86.192/29 ensuring that they use the IP address 80.68.86.193 as their gateway.

One minor complication was that initially my Xen instances were unable to see the network. It turned out that adding a new line to the configuration file(s) in /etc/xen/ for each instances was required:

kernel = "/boot/xen-linux-2.6.12.6-xen"
memory = 128
name   = "vm1.steve.org.uk"
root   = "/dev/sda1 ro"
vif    = [ 'ip=80.68.86.194' ]
disk   = ....

Adding the vif line allowed things to work correctly.

 

 


Re: An introduction to custom Xen networking
Posted by Anonymous (213.33.xx.xx) on Mon 27 Feb 2006 at 11:06
Is it possible with such configuration to monitor traffic on every domUs ip?

How to prevent stealing others IP from inside domU?

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (84.133.xx.xx) on Mon 27 Feb 2006 at 15:51
"""How to prevent stealing others IP from inside domU?"""

ebtables

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (84.186.xx.xx) on Fri 10 Mar 2006 at 17:18
I'm somewhat puzzled by the term "bridge" here because it is commonly used to describe a technique to bond multiple physical interfaces on layer2 (one broadcast domain) whereas this setup looks like a normal "routed" setup. That is, the router behind dom0 (next hop) needs to know about the /29 network behind the public IP of dom0 right?

cheers
Paul

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (80.176.xx.xx) on Tue 21 Mar 2006 at 20:02
Does the dummy0 (80.68.86.193) require its own ip? i mean this is wasted since its not actually being used by anything.

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Steve (82.41.xx.xx) on Tue 21 Mar 2006 at 20:04
[ View Weblogs ]

Yes. For routing, that IP is the bridge out to the internet - since the other machines have IPs on ranges where they couldn't reach the gateway..

Steve

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (80.176.xx.xx) on Tue 21 Mar 2006 at 20:16
Can you not use 80.68.80.176?

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Steve (82.41.xx.xx) on Tue 21 Mar 2006 at 20:23
[ View Weblogs ]

No:

vm1:/home/steve# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
80.68.86.192    *               255.255.255.248 U     0      0        0 eth0
default         80.68.86.193    0.0.0.0         UG    0      0        0 eth0

vm1:/home/steve# route del default gw 80.68.86.193

vm1:/home/steve# route add default gw 80.68.80.176 
SIOCADDRT: Network is unreachable

Because it is on a different network to the IP of the virtual host it can't be reached with the given netmask/broadcast address.

Steve

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (80.176.xx.xx) on Tue 21 Mar 2006 at 20:43
That clears up a lot of confusion!! thanks :)

This is a bit off topic.
But is there a way to give each vm's their own public IP's which aren't in a nice subnet?

Becuase i've bought couple of IP's from my isp and all of them are jsut random ip's, in no particular order. Also they are all set to 255.255.255.255 netmask.

How would i go about doing this?

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Steve (212.20.xx.xx) on Wed 22 Mar 2006 at 09:40
[ View Weblogs ]

Assuming that you have something upstream of your IPs doing the routing for you (so that packets will reach your virtual machines) you might not need to do anything special at all.

Setup the machine(s) with the IP addresses you've been given, set the default route to whichever gateway you're supposed to use and hope for the best!

(Really it is hard to say without a lot more details and knowlege of your setup. But if your provider has sold you IPs I'd expect them to be routable and you to have all the details you need.)

Steve

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (203.126.xx.xx) on Fri 4 Aug 2006 at 09:14
I've read this page several times over the past few days while banging my head with Xen networking (my ignorance more than anything else). Then finally read down through all the comments and realised that your suggestion above is probably what I'm looking for ... I didn't pick up the significance (if I'm not wrong), until now, of you saying that your additional IPs were routed via your existing one. Whereas I'm assuming that if I'm given public IPs and a gateway I can just configure via vif in the config file and then setup my networking in the DomU as I normally would i.e. IP, gateway and DNS?

James

[ Parent ]

Not really true
Posted by Anonymous (219.94.xx.xx) on Wed 9 May 2007 at 17:22
You just have to tell it how to find the router.

route add 80.68.80.176 dev eth0
route add default gw 80.68.80.176

[ Parent ]

Re: Not really true
Posted by Anonymous (62.143.xx.xx) on Tue 10 Jul 2007 at 23:17
My host is in the same subnet like the rest of my ips. What is the configuration for a bridge?

Do i also have to
skx2:~# modprobe dummy
skx2:~# ifconfig dummy0 ip.of.my.hostgateway up

I set
(network-script network-bridge)
(vif-script vif-bridge)
in /etc/xen/scripts/xend-config.sxp.

My domains .cfg looks like this
vif = ['']
and the vm networkconfiguration is unconfigured.

Starting vm is ok, but after configuring and starting vms network i loose the connection and have to reboot the host.

Can somebody help?

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (203.81.xx.xx) on Wed 24 May 2006 at 18:18
How is it possible to route multiple IPs in the domU? I've been trying for hours, but it doesn't work to simply list the IPs using space, while the vif-route script seems to accept this parameter... Strange!

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (82.135.xx.xx) on Sat 3 Jun 2006 at 18:22
Maybe anyone here can give me a hint - since my intended Setup is quite similar.
I have one NIC in the Server and two IPs. I want one for dom0, the other for domU.
The Problem is that my hoster is dropping my Switchport as soon as there arrive Packets with MACs different from my MAC in the physical NIC.
I had a look at ebtables but am not quite shure how exactly to use them in this special case.
tia

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (193.180.xx.xx) on Wed 2 Aug 2006 at 13:35
Great Article!

Does Xen also support bridging on several physical interfaces? This, I guess, would allow virtual hosts on the same CPU belonging to different subnets.

Thanks
Mikael

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (65.89.xx.xx) on Tue 9 Jan 2007 at 23:07
Is this outdated or am I doing something wrong? I set the dummy device and Xen never does anything with it, the bridge doesn't get set with an ip. Then when I load the Xen domain it can see anything. Though if I set an IP on eth1 I can see it from the domain, but it can't access eth0 on the Dom0.

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (70.165.xx.xx) on Sun 22 Apr 2007 at 03:17
A super tutorial!!! Really great stuff, judging by the comments it leaves people wanting more! Did you load test your DomU? I tried reloading the page a few times and it seemed snappy enough.

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (71.48.xx.xx) on Wed 5 Sep 2007 at 20:45
This was a great help when I set up initially on Suse, but six months later after a reboot, things stopped working.

For me it was necessary to do:

echo "1" > /proc/sys/net/ipv4/ip_forward

To make sure your box is set to forward IP traffic from the bridge to the DomU

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (213.136.xx.xx) on Sat 22 Mar 2008 at 12:14
Hi all,

I recently started with Xen (3.1.0) on my new server running Ubuntu Gutsy (7.10) and I'm experiencing very poor network performance on my domU's.

Since my colo provider offers me a /28 of routed public IPs next to one interconnection IP to their routers, I built a Xen configuration as described in this article (thanks for writing it, it was very useful!). This works as expected: all domU's are accessible from the internet.

The network performance of the Dom0 is fine: I can saturate my 100Mb/sec uplink both by uploading and downloading. Performance towards the domU is fine as well. Performance from the DomU's to the internet however is dramatic: speeds start at 2-3MB/sec but quickly drop to just a few KB/sec (!). Transfer of large files never end at all.

The server I'm running this on is a Supermicro H8SSL-i2 with and AMD Opteron 1218 Dual Core proc, 4GB RAM, the NIC is a Broadcom 1G interface connected to a 100Mb/sec switchport at my ISP.

I've tried searching these archives (and the internet) for any clue what might cause this poor upload performance, but failed to find anything really useful.

If anyone has hints and tips on how to make my DomU's uplink speed normal, I'd really appreciate it. I really have no idea where to look now :(

TIA,

Teun

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (213.136.xx.xx) on Sat 22 Mar 2008 at 12:41
And somewhere else I got the answer:
ethtool -K eth0 tx off
on the domU solves this.

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (213.165.xx.xx) on Tue 17 Feb 2009 at 11:58
I know it's almost been a year but I just found this page and wanted to thank you for posting the answer.
I am wrestling with a xen problem right now·
Regards;

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (90.155.xx.xx) on Fri 4 Jun 2010 at 14:55
Hi There,
Just a couple of questions:
Why does the bridge need an IP address? Can't you just remove this? Also, how do you prevent the DomUs from accessing sensitive services on the Dom0?

Thanks

[ Parent ]

Re: An introduction to custom Xen networking
Posted by Anonymous (24.37.xx.xx) on Fri 15 Oct 2010 at 19:50
Steve, using your guide I managed to have things working the exact same way you did, I also have a hosting provider which game me 6 additionnal IPs.
My question is: until I rebooted my Dom0 for the first time in months two days ago, Dom0 firewall didn't have any influence on the traffic going to a VM IP, but now I got to open ports on the Dom0 for the traffic to reach the VM. Any clue ?

[ Parent ]