This site will turn read-only at the end of September 2017.

Automatic host installation using Viper

Posted by docelic on Mon 10 Aug 2009 at 15:16

The purpose of this article is to give you a straight-forward, Debian-friendly way of installing and configuring Viper, a system for completely automated installation and configuration of Debian GNU based systems.

Introduction

Viper is a system for completely automated installation and configuration of Debian GNU based systems.

It is a newly developed system that had the first public release in August 2009. This article should serve as an early introduction, focusing on the results. For more on the underlying logic and in-depth treatment of various topics, refer to official Viper documentation.

Viper uses a modern approach (debian-installer, Puppet etc.), works with existing Debian packages, and does not require any custom patching of either install media, client or server.

The basis of the whole system is a LDAP server, containing all configuration data.
If you are not familiar with LDAP and related terminology, glance over my other article, OpenLDAP installation on Debian, found here or here.

LDAP has the potential to be enormously successful in configuration management, but its flat data structure, absence of any kind of dynamic behavior, and heaps of duplicated data make it a mediocre tool.
To build on LDAP's great strengths while solving its deficiencies, a custom OpenLDAP backend has been implemented.

The setup explained here will get you to a point that you can perform an automatic installation of a Debian host. The configuration part is done using Puppet, but is not covered here; expect it soon as another article on debian-administration.org and the official website.

Viper download and setup

The instructions shown should quickly set you up with a Viper install server, using fully-working example configuration shipped with Viper, that you can test with and without actual client hosts. Then, after that, you should be able to start with writing your own site and hosts configuration.

It would be best if you can install Viper on a clean system without important data. However, the install script will only modify OpenLDAP, DHCP and Puppet server config so you should be able to install it on any system that does not already use one of those.

That said, let's move onto the setup.

Download Viper:

apt-get install git-core
mkdir -p /etc/ldap
cd /etc/ldap
git clone git://github.com/docelic/Viper.git viper 

Now, to set things up, you should just have to run scripts/viper-setup.sh. It should do everything automatically.

However, for the first time, I suggest you read the script (it contains comments) and execute commands manually.

Before running the script, you should do just one thing related to the DHCP server -- you should create a network interface with IP address 10.0.1.1, to fit in the example subnet used by our default config. Then, you can run the install script with the name of the physical interface. Here's the session transcript:

ifconfig eth0:1 inet 10.0.1.1 netmask 255.255.255.0

sh scripts/viper-setup.sh eth0
To configure eth0:1 on every boot, add it to /etc/network/interfaces with a stanza like this:
allow-hotplug eth0:1
iface eth0:1 inet static
  address 10.0.1.1
  netmask 255.255.255.0

Net::LDAP::FilterMatch fix

Net::LDAP's FilterMatch module contains a bug that you have to patch manually, at least until it is fixed in the official distribution.

The patch file is included in downloaded files, so you just need to execute:

patch -p0 < support/FilterMatch.pm.patch

And then re-start slapd with invoke-rc.d slapd restart.

Testing the installation

The above simple procedure should have set you up with a working Viper-enabled LDAP server, DHCP server, and default configuration data you can query from the command line.

Tests you can run:

With ldapsearch

ldapsearch -x -b ou=dhcp
ldapsearch -x -b ou=defaults
ldapsearch -x -b ou=clients

ldapsearch -x -b cn=h2,ou=hosts,o=c1.com,ou=clients

ldapsearch -x -b cn=popularity-contest/participate,ou=hosts,ou=defaults
ldapsearch -x -b cn=debian-installer/locale,cn=h2,ou=hosts,o=c1.com,ou=clients
ldapsearch -x -b cn=ntp/servers,cn=h2,ou=hosts,o=c1.com,ou=clients

Test results

Ldapsearch query for cn=h2,ou=hosts,o=c1.com,ou=clients is a pretty good way of determining if everything is working alright. Here's how the output from the command should look like:
$ ldapsearch -x -b cn=h2,ou=hosts,o=c1.com,ou=clients

# extended LDIF
#
# LDAPv3
# base  with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# h2, hosts, c1.com, clients
dn: cn=h2,ou=hosts,o=c1.com,ou=clients
objectClass: top
objectClass: device
objectClass: dhcpHost
objectClass: ipHost
objectClass: ieee802Device
objectClass: puppetClient
cn: h2
ipHostNumber: 10.0.1.8
macAddress: 00:11:6b:34:ae:8d
puppetclass: test
puppetclass: ntp::server
dhcpHWAddress: ethernet 00:11:6b:34:ae:8d
dhcpOption: host-name "h2"
dhcpOption: routers 10.0.1.1
dhcpOption: domain-name-servers 192.168.1.254
dhcpOption: nis-domain "c1.com"
dhcpOption: domain-name "c1.com"
dhcpOption: subnet-mask 255.255.255.0
dhcpOption: broadcast-address 10.0.1.255
dhcpStatements: fixed-address 10.0.1.8
hostName: h2
ipNetmaskNumber: 255.255.255.0
clientName: c1.com
ipNetworkNumber: 10.0.1.0
ipBroadcastNumber: 10.0.1.255
domainName: c1.com

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

With scripts/node_data

perl scripts/node_data h2.c1.com

With scripts/preseed

QUERY_STRING=ip=10.0.1.8 perl scripts/preseed

Installing a client host

Adjust defaults and host's macAddress

After you've got the Viper server running and the tests are successful, you can attempt installing a host.

To do so, go to /etc/ldap/viper/ldifs/ and edit two LDIF files, 0-defaults.ldif and c1.com.ldif.

In the defaults file, you should go over all the values (search for occurences of "value:") and adjust them if any need change in your setup.

In the c1.com.ldif file, you should basically just modify the macAddress of host h1, and set it to the MAC address of the host you want to install.

Then, you should run 'make' in the current directory to load all .ldif files into LDAP.

Set up Apache to handle requests for preseed.cgi

Apache setup will already be done correctly if your cgi-bin directory is the standard /usr/lib/cgi-bin/ and accessible through default vhost at location /cgi-bin/. If not, adjust accordingly.

On the client host

To install the client, boot from official boot media, and simply select Advanced --> Automatic installation.

If your host has multiple network interfaces, you might also add interface=ethX on the boot line, or answer that question manually when asked by the installer. (The d-i installer should supposedly auto-detect the interface, but it seems this isn't working 100% well yet).

The results

The client host should install automatically and reboot into the newly installed system.

Comments, questions and requests for clarification welcome here as the article followup, or on the Viper mailing list.

For more about Viper and further documentation, refer to Viper resources:

Viper homepage
Viper mailing list
Viper @ GitHub



Davor Ocelic
http://www.spinlocksolutions.com/

Copyright (C) 2007,2008 Davor Ocelic, <docelic@spinlocksolutions.com>
Spinlock Solutions, http://www.spinlocksolutions.com/

 

 


Re: Automatic host installation using Viper
Posted by ptman (88.192.xx.xx) on Mon 10 Aug 2009 at 16:48
Can you please give an overview of Viper vs. FAI?

[ Parent | Reply to this comment ]

Re: Automatic host installation using Viper
Posted by docelic (78.134.xx.xx) on Mon 10 Aug 2009 at 18:03
I can't tell much about FAI, so I hope someone who skims over Viper docs can make a more detailed comment.

Viper tends to be a transparent central information and interoperability point, where other tools that make up the complete picture can search for information they need to do their job (DHCP for IPs, d-i for installation, Puppet for configuration etc.).

So it basically does the following things:

1) Provides central information store with support for dynamic values (unlike regular LDAP which only stores "flat" data)

2) Provides glue code or scripts used by other components to work with LDAP (such as preseed.cgi script for d-i's HTTP preseed method, or node_data script for Puppet's external_nodes)

3) Provides recipes on how to combine everything (Viper, dhcp, d-i, puppet) into a complete system

4) Provides a collection of Puppet modules to do typical configuration tasks (as of now, modules yet have to be written)

[ Parent | Reply to this comment ]

Scripted daemons strike again... Sigh.
Posted by Anonymous (86.214.xx.xx) on Mon 10 Aug 2009 at 21:57
Though I recognize LDAP integration is very neat, one thing will make me never ever use any scripted daemon on every and each of my machines. Never. Ever (again).

I tried Puppet and, yeah, it was infinitely nicer to handle than the venerable CfEngine (v2). But this is a ressources blackhole. As of today, all of my services are separated on different virtualized ("semi-virtualized", more particularly, thanks to the godly OpenVZ) instances. Hence, if I don't care that much for some services that eat a lot of memory, I will certainly not make them part of the skeleton of all my containers. There is not any use in having a scripted crap that ends up eating hundreds of MB on dozens and dozens of machines (which run on the same hardware). Such a thing is a big no-no: 24-32MB of RAM is more than enough for such a skeleton (the rest of what I allocate is for the daemon[s] the machine is for); no Perl+Ruby juggernauted daemons in each and every corner: what I earn for not having them allows me to have more memory for useful things, which allows me to segregate more efficiently. Less-is-more.

So I'll stay with FAI and CFengine v2. Definately. Well, up until CFEngine 3 comes to Debian, of course.

[ Parent | Reply to this comment ]

Re: Scripted daemons strike again... Sigh.
Posted by Anonymous (69.17.xx.xx) on Tue 11 Aug 2009 at 04:50
Yeah, it was kind of a downer to realize that puppetd sucked up 40megs of ram on every one of my VMs.

But then I realized that the dollar cost of the time saved by using puppet was worth the dollar cost of the RAM (even including the power to run it).

If you're trying to make do on a limited hardware budget, and you already have a CFengine setup, then ok, puppet may not be for you.

But if you are running 40 VMs on a 32gig box (which is a reasonable size, depending on your workloads you might be lower or higher in population), the
RAM spent on puppet is about 5% of your total. Or about $50.

RAM is cheap enough that most people don't benefit from optimizing for it.

[ Parent | Reply to this comment ]

Re: Scripted daemons strike again... Sigh.
Posted by Anonymous (86.214.xx.xx) on Tue 11 Aug 2009 at 06:39
Well, I tested it during Lenny stabilization, and I came across 200-300MB usage after a few days on some instances (and it was still getting bigger, if I let it ; I precize I speak about global memory usage, including buffers and such, about which I have to independently allocate memory for each VM anyway). I don't know where the leak did come from (and if it even was one), but still, with the loading of ruby and all, I learnt to consider 40MB as a very minimum. And once you plan for 200-300 additional MB on, say, 100-200 VMs, it is 20-30GB of memory eaten up: even on a 32GB node, it gets ridiculous, when good ol' CFEngine would eat 1-2GB at the very most in the same case, from the experience I have of it. Not to speak about ruby very strange and ununderstandable errors and crashes.

I know RAM is always getting cheaper, but still: when the skeletons of my VMs eat a lot more of memory than most of the daemons they run (I said it: I also got juggernauts, like Slapd and such, but they are a minority in my case; most of my daemons are as lightweight as possible), I'm convinced there is something really wrong.

Well, sure, as I said: I recognize CFEngine is very old, necessitates a lot of DIY, and Puppet syntax is very nice. But still: I hadn't tested scripted daemons for a long time, and Puppet reminded me this was a thing I did not want. I'll wait for CFEngine v3, which I've tested and been impressed with, to come on Debian. It'll probably still be to young when Squeeze comes out, but I am not so stressed.

[ Parent | Reply to this comment ]

Re: Scripted daemons strike again... Sigh.
Posted by oxtan (82.95.xx.xx) on Sat 26 Sep 2009 at 19:52
[ View Weblogs ]
amen to that. Cfengine + FAI is debian admin's nirvana.

I only disagree with you in that puppet is nicer to handle than cfengine 2. You may like it better (lots of people do) but lots of people do not want to touch pupet with a 3 meter long stick. Cfengine is a proven technology, it works really well and it will stay here for a very long time. I have yet to start with cfengine 3 because 2 works for me.

Besides that, I think that misleading customers in your corporate website about what puppet does that cfengine doesn't is not a nice way of running a busines ...

[ Parent | Reply to this comment ]

Re: Automatic host installation using Viper
Posted by Anonymous (134.95.xx.xx) on Wed 26 Aug 2009 at 16:32
Does Viper suppert network installations? I cannot find any references to PXE in your source.

> Viper uses a modern approach (debian-installer,...
Mmm, d-i is not a modern approach, it's more like a hack to fit into small RAM.
Using debconf preseeding for partitioning is totally ugly IMO.
IMO Vipers only seems to be a LDAP backend for d-i and puppet.

[ Parent | Reply to this comment ]

Re: Automatic host installation using Viper
Posted by Anonymous (78.134.xx.xx) on Wed 26 Aug 2009 at 17:07
You may wish to double-check your statement that d-i is not a modern installer.

Using debconf preseeding for partitioning is a legitimate feature of the d-i, carried out by partman. There are patches available for partman (should be on its way into the distribution) that add RAID support in combination with other methods (LVM, crypto), which was the only important missing piece.

Viper is not "only a LDAP backend for d-i and puppet". But I do consider it a plus that it serves a specific purpose and uses successful existing software where possible, instead of reinventing the wheel.

Regards,
-doc

[ Parent | Reply to this comment ]

Re: Automatic host installation using Viper
Posted by kodzoman (93.103.xx.xx) on Fri 25 Sep 2009 at 10:30
[ View Weblogs ]
Looks very interesting and promising! I really like the entire idea and will surely try it out. I think it's a move in the right direction. Thank you docelic for sharing this!

[ Parent | Reply to this comment ]

Re: Automatic host installation using Viper
Posted by Anonymous (134.95.xx.xx) on Fri 1 Oct 2010 at 12:15
The software development of viper seems to be stalled. No mail since oct 2009, no commit since dec 2009.

[ Parent | Reply to this comment ]