This site is now 100% read-only, and retired.

Working with network block devices

Posted by Steve on Thu 2 Aug 2007 at 09:43

There are times when you have a machine, or two, which is short of disk space and yet you have spare capacity elsewhere upon your LAN. For these times using a Network Block Device could come in handy. This allows you literally export files as block devices remotely.

As another useful tip because the disk activity is going over the network you can filter or rate-limit IO in a simple fashion.

For example imagine you were running a lot of Xen guests upon a single host and you didn't want a greedy client performing a lot of I/O operations to adversely affect the other users of the machine. This is hard to manage by traditional methods, but using iptables you could easily manage it.

There are drawbacks to using network block devices, and I'm sure they are obvious:

  • You're limited by the speed of your network - which is probably lower than the speed of your disks.
  • You must care about security a lot.

In the system we're going to describe we'll have two machines and The first will export a single file, which will be mounted upon the second. We'll use port 1234 for this, and we must firewall this port such that only the client system which is supposed to connect to it can.

Server Setup

Setting up the server is pretty simple. We'll need to install the software, prepare a device, and then export it.

To install the server run:

root@mine:/# apt-get install nbd-server

Now we'll create a file to export:

root@mine:/# modprobe nbd
root@mine:/# mkdir -p /home/exported
root@mine:/# dd if=/dev/zero of=/home/exported/trial.img count=256 bs=1024k
root@mine:/# mkfs.ext3 /home/exported/trial.img

Starting the server should be simple, but I found that it immediately segfaulted on my machine - so we'll demonstrate two commands for this:

root@mine:/# nbd-server 1234 /home/exported/trial.img

If this errors like mine did run this instead:

root@mine:/# touch /root/empty
root@mine:/# nbd-server 1234 /home/exported/trial.img -C /root/empty

(There is a global configuration which can be used to list exports /etc/nbd-server/config - however everytime I tried to use this file I received a segfault from the server process so I can't tell you anything useful about it.)

Setup The Client

Setting up the client is very similar to setting up the server, we need to install the relevant software then mount the remote image.

root@yours:/# apt-get install nbd-client

To mount the system we'll run:

root@yours:~# nbd-client 1234 /dev/nbd0
Negotiation: ..size = 262144KB
bs=1024, sz=262144

root@yours:~# mkdir /mnt/remote
root@yours:~#  mount /dev/nbd0 /mnt/remote

Now we can play:

for i in $(seq 1 100) ; do echo $i > /mnt/remote/$i; done

Unmount the volume:

root@yours:/# umount /mnt/remote 

Now try mounting it back upon the server, to make sure that those files have persisted and been created as we expect:

root@vain:~#  nbd-client 1234 /dev/nbd0
root@vain:~# mkdir /tmp/foo
root@vain:~#  mount /dev/nbd0 /tmp/foo
root@vain:~#  ls /tmp/foo/
1    14  2   25  30  36  41  47  52  58  63  69  74  8   85  90  96
10   15  20  26  31  37  42  48  53  59  64  7   75  80  86  91  97
100  16  21  27  32  38  43  49  54  6   65  70  76  81  87  92  98
11   17  22  28  33  39  44  5   55  60  66  71  77  82  88  93  99
12   18  23  29  34  4   45  50  56  61  67  72  78  83  89  94  
13   19  24  3   35  40  46  51  57  62  68  73  79  84  9   95  lost+found

Fun, huh?

Working With Xen

I spent a while trying to get this working with Xen, but only found success when using the /dev/nbdN devices.

For this to work I had to use the following Xen configuration, which is less than ideal:

disk        = [ 'phy:vain-vol/,sda1,w', 
                'phy:/dev/nbd0,sda3,w' ]

I had hoped to be able to specify the host + port inside the disk stanza, so that I could be sure of device ordering - but I couldn't make that work.

Any suggestions welcome..



Re: Working with network block devices
Posted by wuzzeb (64.5.xx.xx) on Thu 2 Aug 2007 at 10:23
for i in $(seq 1 100) ; do echo $i > /mnt/nbd/$i; done

Should probably be echo $i > /mnt/remote/$i

Secondly, (I haven't tried nbd yet), but for the xen configuration it seems like you pass the device file name to use to nbd-client... why not just put the server and port into the device filename... something like

# ndb-client server 1234 /dev/ndb.server.1234

And then in the xen configuration use /dev/ndb.server.1234,sda3,w

[ Parent ]

Re: Working with network block devices
Posted by Steve (80.68.xx.xx) on Thu 2 Aug 2007 at 10:27
[ View Weblogs ]

Thanks - I've fixed that error now. (That'll teach me to switch from the real paths I was using to 'friendlier' ones!)

I'll try your Xen suggestion this evening and see how it works out. It's not quite as nice as having:

nbd:/ 1234,sda3,w

But preferable to what I ended up with.


[ Parent ]

Re: Working with network block devices
Posted by Anonymous (217.7.xx.xx) on Wed 8 Aug 2007 at 13:50
I'm wondering what are advantages of this method over NFS. I understand that this is more efficient, but it's complicated and not easy to maintain. Moreover, basic security is already provided by NFS.
Anyway, I like this thing that demonstrate the power Unix/Linux world, where all things are files :)

[ Parent ]

Re: Working with network block devices
Posted by Steve (80.68.xx.xx) on Wed 8 Aug 2007 at 16:12
[ View Weblogs ]

The simple answer is that both could be used - I like this solution because it is explicit about one IP per device, which would allow the firewalling to be simpler.


[ Parent ]