This site is now 100% read-only, and retired.

XML logo

Shared NFS root filesystems & debian packages
Posted by endecotp on Tue 9 Oct 2007 at 19:29
Tags: none.
For a while I have been running a few Debian boxes from NFS root filesystems - mainly NSLU2s and similar little ARM things. But what I have never successfully done is to run them simultaneously from a single shared NFS root filesystem. I'm now trying to do this.

The idea is that they share a single root but have their own /tmp and /var. /home is also an NFS mount.

But there are files in /var/lib/dpkg, and elsewhere in /var, that presumably need to be kept in-sync with the contents of /bin, /usr and so on. So some parts of /var need to be shared too.

So, what I need is a list of the subdirectories of /var that should be shared (like /var/lib/dpkg) and the subdirectories that should be per-machine (like /var/tmp and /var/run). Is it safer to default to "shared" or "per-machine"?

I'm particularly worried about the dpkg and apt-related stuff, since its operation is "magic" to me and getting it wrong could leave everything horribly screwed up...

Here's the editted output of "tree -L 2" in /var :

|-- backups
| |-- dpkg.status.0
| |-- group.bak
| |-- gshadow.bak
| |-- infodir.bak
| |-- passwd.bak
| `-- shadow.bak
|-- cache
| |-- apt
| |-- debconf
| |-- locate
| `-- man
|-- lib
| |-- apt
| |-- aptitude
| |-- dhcp3
| |-- dpkg
| |-- initramfs-tools
| |-- initscripts
| |-- logrotate
| |-- misc
| |-- urandom
| |-- usbutils
| `-- vim
|-- local
|-- lock
|-- log
| [snip]
|-- mail
|-- opt
|-- run
| |-- crond.pid
| |-- crond.reboot
| |-- klogd.pid
| |-- motd
| |-- network
| |-- sshd
| |-- sshd.pid
| |-- syslogd.pid
| `-- utmp
|-- spool
| |-- cron
| `-- mail -> ../mail
`-- tmp
[snip]

 

Comments on this Entry

Re: Shared NFS root filesystems & debian packages
Posted by ajt (204.193.xx.xx) on Wed 10 Oct 2007 at 10:20
[ View Weblogs ]
The Filesystem Hierarchy Standard tells you where everything goes and which parts are shareable and which are not. It's a good place to start looking.

http://www.pathname.com/fhs/

--
"It's Not Magic, It's Work"
Adam

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by endecotp (86.6.xx.xx) on Wed 10 Oct 2007 at 11:43
[ View Weblogs ]
Thanks Adam, that's a great start.

Unfortunately the FHS says that /usr is shareable while /etc is not shareable. Hmm. So if I apt-get install something on one machine that puts files in /usr/bin and /etc, then it will be broken on the other machines - they will see the binaries but not the configuration files.

I presume that people doing virtual machines with shared filesystems encounter the same problems. Or maybe no-one actually tries to share anything...

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by Steve (80.68.xx.xx) on Wed 10 Oct 2007 at 13:19
[ View Weblogs ]

When I share root filesystems over NFS I give every machine a particular export ... Which means I duplicate space on the NFS server, but don't have to worry about other side-effects. (I'm sure with bind mounts, or union I could cut down on the disk space required; but I've never needed to do that yet.)

The only thing I share would be /home. That way I avoid these kind of issues.

Steve

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by endecotp (86.6.xx.xx) on Wed 10 Oct 2007 at 14:56
[ View Weblogs ]
Hi Steve,

Yes, I think that most people just replicate everything. Disk is cheap. Then they discover that they want to do the same install or upgrade on all their machines, and they look for tools like the multiple-ssh things described in a recent article here. It's a trade-off.

I am now running two machines sharing everything except /var/backups,cache,lock,log,run,tmp and /tmp, and I'm waiting to see how well it works.

One thing that I've noticed is that an NFS-mounted /var is treated specially by the init scripts - /etc/init.d/mountnfs will wait for NFS mounts of /usr, /usr/*, /var and /var/* to complete before continuing, since subsequent scripts may want to access /var/run, /var/log etc. But other NFS mounts continue in the background. This breaks if your per-client directories are done using symlinks pointing out of /var; my first attempt had a /per_client in the root directory. The solution is to have a structure like this on the server:

var
|---lib [shared]
|---opt [shared]
|---run ---> /var/per_client/run [symlink]
|---tmp ---> /var/per_client/tmp [symlink]
`---per_client [empty dir as mount point]

Then mount server:/path/to/`hostname`/var on /var/per_client. Because the mount point is in /var, mountnfs.sh will wait for it.

Phil.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by mcortese (213.70.xx.xx) on Wed 10 Oct 2007 at 17:11
[ View Weblogs ]

There's one thing that is not clear to me: do you want to do the maintenance work from the small systems (let's call them clients), or from the central machine that exports the shares (let's call it the server)?

In the first case, what is your expectation when you apt-get install something on one client? The same package (with all its config files) should be propagated to the other clients? This suggests that /etc should be shared, but what about mtab, hostname, etc.?

On the other hand, if all the administrative tasks like installing packages are done centrally on the server, why do you bother about clients not being able to see /var/lib/dpkg? You should not run apt-get on the clients in the first place!

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by endecotp (86.6.xx.xx) on Wed 10 Oct 2007 at 17:37
[ View Weblogs ]
> There's one thing that is not clear to me: do you want to do the
> maintenance work from the small systems (let's call them clients),
> or from the central machine that exports the shares (let's call it
> the server)?

Preferably from one of the clients, with the others shut down if necessary, but doing from the server (e.g. with a chroot) could be an option if it made something simpler.

> In the first case, what is your expectation when you apt-get install
> something on one client? The same package (with all its config files)
> should be propagated to the other clients?

Yes.

> This suggests that /etc should be shared, but what about mtab,
> hostname, etc.?

Right, well that's another list of things to worry about....

If you just delete /etc/hostname (or make it empty?) then the init scripts will trust that the kernel alrady knows its hostname; in my case, it got it already from DHCP.

mtab can be made a symlink to /proc/mounts for most purposes. (There is some obscure case where /proc/mounts has less info than a real mtab would provide, but I can't recall the details.)

I haven't found anything else to worry about, yet. Let me know if you think of any more examples.


Phil.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by Anonymous (37.4.xx.xx) on Wed 2 May 2012 at 11:54
Hi, could you please tell us what you found out, did it work?

From reading this, it seems only a slight adaption to the Filesystem Hierachy Standard would be needed to allow that / (root) can be on a network share or synced filesystem/harddisk and used by all nodes (or laptop and desktop).

It would only need to define some places to separate out the things that need to be kept individually per kernel, i.e.:
/etc/kernel-specific (/etc/fstab, mtab, hostname) and
/var/kernel-specific (/var/cache, lock, log, run, tmp)
/tmp (maybe with a /tmp/shared-root to allow continuing jobs on other nodes)

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by Anonymous (37.4.xx.xx) on Wed 2 May 2012 at 12:04
Maybe a better wording could be: per kernel-run/ or machine/ or node/ specific.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by endecotp (86.171.xx.xx) on Wed 2 May 2012 at 12:53
[ View Weblogs ]
> could you please tell us what you found out, did it work?

I gave up. If you deviate too far from what others consider "normal", something will bite you.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by Anonymous (37.4.xx.xx) on Wed 2 May 2012 at 13:26
> If you deviate too far from what others consider "normal", something will bite you.

Wow, thanks for answering that fast! Uh, these days I'd consider it a perfectly normal to wish to be able to work on and modify a system regardless of the machine you're sitting at. And further so to do it with the right tools at the fundamental level, rather the added complexity of high level "cloud" stacks.

Was there more to it than the FHS not achieving a proper separation of sharable local (lan) and host (node) specific files in /etc and /var/?

According to wikipedia (Single_system_image), shared root seems to be something all SSI tools for clusters seem to implement, but that seems overkill if you just need to sync up (nfs, csync, unison) a couple machines, and do not need to migrate processes or share IPs.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by endecotp (86.171.xx.xx) on Wed 2 May 2012 at 17:18
[ View Weblogs ]
> I'd consider it a perfectly normal to wish to be able
> to work on and modify a system regardless of the machine
> you're sitting at.

Sure, that's what ssh lets you do.

What's not "perfectly normal" is to want to share /usr/bin etc. Sharing /home is a bit more common, but even that fails surprisingly often because e.g. NFS doesn't support things like inotify and some kinds of locking, and other things like mmap()d files have poor performance.

> Was there more to it than the FHS not achieving a proper
> separation of sharable local (lan) and host (node) specific
> files in /etc and /var/?

I honestly don't recall. (This was 5 years ago.) I note that I mentioned some init script issues above, i.e. assumptions about mount order.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by Anonymous (37.4.xx.xx) on Wed 2 May 2012 at 18:39
I see. Thank you for sharing. At that time there was probably still much more machine specific configuration required. I notice it has become a no-brainer today to swap and boot a hard disk in different PCs. The OS boots flawlessly, thanks to the hotplug support.

It's time for new experiments on sharing the root filesystem (syncing laptops/desktops) and probably a refinement of the FHS.

With auto synced local filesystems (dropbox-style), fortunately, inotify and mmap() are no issues. And the awareness about the need to separate cache/temp/run etc. stuff from persistent user data in the home directories is also raising, due to auto syncing becoming more and more popular.

[ Parent ]

Re: Shared NFS root filesystems & debian packages
Posted by endecotp (86.171.xx.xx) on Wed 2 May 2012 at 19:02
[ View Weblogs ]
> swap and boot a hard disk in different PCs

That has never been a problem. The difficulty is with concurrent access.

[ Parent ]