Do you use let's encrypt?





6369 votes ~ 21 comments

 

2/2: An introduction to using Puppet

Posted by Steve on Fri 25 May 2007 at 11:39

In our previous introduction to Puppet we covered installing the client and server, such that a small LAN could be configured centrally. Here we'll demonstrate some of the things that can be done with such a setup.

In our brief introduction to installing Puppet we demonstrated a very simple manifest file, called site.pp. This is the master configuration file which is pushed by puppet to each managed host, and will be used to specify what actions to carry out. (What is a manifest file? Well it is a list of instructions on what to carry out in a declarative fashion which is why it isn't called a script or a program file.)

There are many actions which may carried out by puppet:

  • Installing / Removing / Updating Debian packages.
  • Copying files from a central server.
  • Applying configuration changes.

Last time we demonstrated the following manifest file (Saved in /etc/puppet/manifests/site.pp):

# fixup permissions on sudo
class sudo {
    file { "/etc/sudoers":
        owner => root,
        group => root,
        mode  => 440,
    }
}

node default {
     include sudo
}

This example shows several things:

  1. Basic class definition with "class sudo ..".
  2. The use of the "file" object.
  3. The definition of actions to carry out on all hosts.

The definition of the class we'll gloss over, as it is treated pretty comprehensively in Puppet documentation, however the file section is worthy of note.

The file type is one of several types that Puppet understands - in much the same way that CFEngine understands the use of copy: sections. It allows you to do things such as:

  • Copy files from the central puppet server.
  • Fix permissions of existing files.
  • Fix ownership of existing files.

(The Puppet type reference shows more types with examples for several of them.)

Copying Files

Copying files from the master server can be conducted with the file type which we've just introduced. To perform a copy is very similar to fixing up permissions which we saw in the example above. All we need to do is add a source for the file, which will be somewhere upon our master server.

Previously we enabled the Puppet server, running upon the host vain.my.flat, to serve files to clients from the directory /etc/puppet/files so we'll continue working with that prefix.

Assuming we wanted each host upon our LAN to receive a copy of /etc/sudoers from our central server we'd write the following in /etc/puppet/manifests/site.pp:


#
#  Class to pull /etc/sudoers from our master server.
#
class sudo {
    file { "/etc/sudoers":
        owner  => root,
        group  => root,
        mode   => 440,
        source => "puppet://vain.my.flat/files/etc/sudoers"
    }
}

node default {
     include sudo
}

Now we just need to place the file in the correct location upon the master server:

root@vain:~# mkdir -p /etc/puppet/files/etc/
root@vain:~# cp /etc/sudoers  /etc/puppet/files/etc/

This will now ensure that the file sudoers is available beneath our file-sharing root, and can be installed upon all machines.

If you wanted to have an action carried out after copying a file, but only if it was changed, you could use something like this:

class aptsetup {

      file { "/etc/apt/sources.list",
             owner  => root, group => root, mode => 644,
             source => "puppet://puppet/files/etc/apt/sources.list.etch 
      }

      exec { subscribe-echo:
               command     => "/usr/bin/apt-get -q -q update",
               logoutput   => false,
               refreshonly => true,
               subscribe   => file["/etc/apt/sources.list"]
      }
}

Now you can just "include aptsetup" in the definition of your node(s) and they will receive a copy of the sources.list file from the remote server. If that is updated upon the server then they will receive a refreshed copy and they will automatically run "apt-get update".

Including Classes

So far we've seen a very simple layout with all the content located in the single site.pp file. That can quickly become unmanageable, so it is recommended that you break your configuration down into a set of self-contained classes.

A basic site.pp could look like this:

import "classes/*.pp"

node default {
     include sudoers
     include aptsetup
}

Now we can create the files "/etc/puppet/manifests/classes/sudo.pp" & "/etc/puppet/manifests/classes/aptsetup.pp" with the contents we've seen before and all will continue to work properly.

Defining Nodes

One thing that should be apparent in our example so far is that we've not really split our code up to handle any complex situations where managed nodes differ. Instead we've just demonstrated the use of "node default" which means that our actions have been carried out upon all machines using this server.

Here is an example of a more complex setup for two machines (etch-builder.my.flat and sid-builder.my.flat).

Each machine will get the appropriate sources.list file setup, in addition to some common activity applied to both machines, and any other machines which have been configured by not named separetely:

class sudo {
    file { "/etc/sudoers":
        owner  => root,
        group  => root,
        mode   => 440,
        source => "puppet://vain.my.flat/files/etc/sudoers"
    }
}

class common {
   # Remove upstream sources.list
   # We manage this in sources.list.d per distro
   tidy { "/etc/apt/sources.list":
          age => '0s',
        }
}

class etch {
   file{ "/etc/apt/sources.list.d/etch.list":
         mode   => 644,
         source => "puppet://vain.my.flat/files/etc/apt/sources.etch"
       }
}

class sid {
   file{ "/etc/apt/sources.list.d/sid.list":
         mode   => 644,
         source => "puppet://vain.my.flat/files/etc/apt/sources.sid"
       }
}

node default {
     # all host get these actions
     include sudo
}

node etch-builder inherits default {
     include common
     include etch
}
node sid-builder inherits default {
     include common
     include sid
}

Making Puppet more CFEngine-like

Several actions familiar to the users of CFengine are missing, such as DeleteLinesMatching and AppendIfNoSuchLine which allow simple text file manipulation.

Thanks to this Puppet wiki page we can find a simple solution which covers most cases.

Create the file /etc/puppet/manifests/classes/cfengine.pp with the following contents:


define append_if_no_such_line($file, $line, $refreshonly = 'false') {
   exec { "/bin/echo '$line' >> '$file'":
      unless      => "/bin/grep -Fxqe '$line' '$file'",
      path        => "/bin",
      refreshonly => $refreshonly,
   }
}

define delete_lines($file, $pattern) {
   exec { "sed -i -r -e '/$pattern/d' $file":
      path   => "/bin",
      onlyif => "/bin/grep -E '$pattern' '$file'",
   }
}

Now in your site.pp file you can have the following:

import "classes/*.pp"

node default {
   delete_lines{ removecfengine:
           file    => "/etc/motd",
           pattern => "^.*CFEngine.*$" }

   append_if_no_such_line{ motd:
           file => "/etc/motd",
           line => "Configured with Puppet!"}
}

I'll close the second part of this introduction here, even though I've barely scratched the surface of what Puppet can do for you. If you're interested in learning more I'd suggest you look at the documentation at the Puppet Wiki.

The IRC channel was friendly and helpful when I joined; allowing me to get my first change committed to the project in only minutes!

 

 


Re: 2/2: An introduction to using Puppet
Posted by Thorsten (84.58.xx.xx) on Fri 25 May 2007 at 13:43
Another really great and usefull article Steve!

But one little typo under headline "copying files": it should be "include sudo" instead of "include sudoers"

best wishes
/thorsten

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Steve (80.68.xx.xx) on Fri 25 May 2007 at 14:08
[ View Steve's Scratchpad | View Weblogs ]

Thanks! I received a large example from Jorge O. Castro prior to writing this, which is why the examples here are more complete than those I'd usually use.

(I fixed the sudoers error you spotted).

Steve

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by practicalclouds (195.234.xx.xx) on Tue 26 Feb 2013 at 12:29
I would recommend that you keep your configurations in your version control system, where they belong, and deploy them via puppet. Puppet has many modules available via it's puppet forge and vcsdeploy allows you to deploy directly from your vcs repositories, see more http://www.practicalclouds.com/content/guide/pclouds-vcsdeploy-de ploy-stuff

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by etptupaf (158.227.xx.xx) on Fri 25 May 2007 at 17:37
Steve,

This article was great and very timely for me, as I am thinking of deploying a tool to manage a small number (+-12) of computers. I had been thinking of CFengine all along. Puppet seems to be a worthy alternative.

Could you, or some of your readers, comment on the relative strengths of those two tools --and any other--- so I can make an informed choice?

Thank you again, ft.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Steve (62.30.xx.xx) on Fri 25 May 2007 at 22:49
[ View Steve's Scratchpad | View Weblogs ]

The puppet wiki does have a brief comparision, and I guess to make a good decision you should probably look at the documentation for both projects a fair bit yourself ..

From my point of view:

  • CFEngine is mature, stable, reasonably complete and very well understood.
  • Puppet is more capable (ie. native support for installing packages easily) actively developed, but there are areas where documentation is weak and finding good examples can be hard.

For a large installation I'd go with CFEngine, because it has been around longer. For a small installation I think I'd probably use Puppet now. 12 is kinda in-between so I have no real preference.

One deciding factor would be Sarge - if you're using that I'd suggest CFEngine.

(Potential bias: I use Puppet on five nodes at home; I use CFengine on 100+ nodes at work.)

Steve

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (69.17.xx.xx) on Fri 25 May 2007 at 23:44

Thanks for the article, Steve; I've blogged about it, and I'll put a link to it on the wiki, so new users can see multiple introductions.

I disagree that Puppet's only appropriate for small sites, though -- I've got lots of users managing hundreds of machines with it, and many of those users either replaced Cfengine, or chose Puppet in preference to it, largely because Puppet is more functional but also drastically easier to maintain (e.g., less code duplication).

I agree on the difficulty in finding good examples; we're working on it, of course, and the documentation has gotten dramatically better in the last six months. Users are now starting to publish their configurations, and there are quite a few recipes.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Steve (62.30.xx.xx) on Sat 26 May 2007 at 00:45
[ View Steve's Scratchpad | View Weblogs ]

It's not that I think it is unsuitable for large installations; more that I think it is relatively recent and untested.

(Whilst there may well be lots of users running upon large networks I'm not personally familiar with them and seeing the website talking about random Ruby seg-faults isn't really reassuring!)

The big plus for Puppet is the responsiveness of the development team, and of the project activity as a whole. By comparison CFEngine is glacial.

The more I use puppet the more I like it, but still I'd be reluctant to replace a large-scale installation of CFEngine just yet. In the next few months, or next year it would probably be likely, but not immediately.

Steve

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (69.17.xx.xx) on Sat 26 May 2007 at 21:21
Well, Puppet's been being used in production for more than a year, so I wouldn't consider it untested, nor would those who have been running it that way. It's definitely not "done" yet, but this is a huge problem space and I can't imagine what the solutions will look like in the end.

There have been problems with ruby, but that's always individual compiles of Ruby on a specific platform, not Puppet itself.

As to replacing a Cfengine installation, you can start slowly by using Puppet's cfengine module, which allows you to use Puppet for the things that Cfengine doesn't do well (packages, services, cron, users, groups, etc.) but still use Cfengine's classing and such.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by mwr (24.158.xx.xx) on Sat 26 May 2007 at 00:06
[ View Weblogs ]

From my point of view, the pro-Puppet stuff:

  • Puppet rules on dependencies. If you need to ensure that changes to rule A happen before changes to rule B, you can do it.
  • Package management.
  • Abstraction: there are native types for services, cron jobs, and other common administrative resources. I can define any of these programmatically and not have to sling around files everywhere. Example:
    class dual-boot-cluster-host inherits cluster-host {
        # A dual-boot cluster host needs:
        # - cron jobs to reboot it before classes or open lab hours start
        cron { sunday-reboot:
            command => "/sbin/shutdown -t10 -r now",
            ensure  => present,
            user    => root,
            minute  => 30,
            hour    => 13,
            weekday => 0,
        }
        cron { weekday-reboot:
            command => "/sbin/shutdown -t10 -r now",
            ensure  => present,
            user    => root,
            minute  => 30,
            hour    => 7,
            weekday => [ 1, 2, 3, 4, 5 ],
        }
    }
    
    In the above case, I don't have to drop a file into cron.d. And if I need the same jobs to run on Debian, Solaris, OS X, etc., Puppet's cron abstraction should handle the details of how to enter a cron job into any of those systems, whether or not they support cron.d directories. And if I need to override those jobs on hosts that inherit dual-boot-cluster-host, I should be able to make a new subclass that inherits dual-boot-cluster-host, including:
        Cron["sunday-reboot"] { ensure  => absent, }
    

I don't think the size of the deployment matters as much. By the time I get done, I'll probably have Puppet out on 100 or so nodes. One major difference is that I didn't have more than a few days of time invested in cfengine. If you're in a similar situation to mine and you're trying to create an managed infrastructure from scratch, it's worth looking at Puppet.

And yes, puppet goes best with Etch or later. I don't think I'd try it on Sarge. I'm running a backport of the version from unstable at the moment.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (81.224.xx.xx) on Mon 28 May 2007 at 03:23
I would consider FAI for handling a one computer and bigger installation. You can use any installation script with it. Like CFengine, Puppet or any script you have.
FAI is mainly for installation, but can also upgrade an installations nodes with softupdate instead of installation.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by oDn (83.197.xx.xx) on Fri 25 May 2007 at 17:52
Thanks Steve,

I've been currently using csync2 for a week now.
I guess it's just a simpler alternative to Puppet or Cfengine.

Have any of you used csync2? It really doesn't seem to be widespread and is only available on the unstable repos as far as I know ...

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Steve (62.30.xx.xx) on Fri 25 May 2007 at 18:41
[ View Steve's Scratchpad | View Weblogs ]

I've never heard of it! If you had the time to give a simple introduction then I'm sure our readers would appreciate it..

For me I really do need something which works well with Etch - if the backport is simple I'm happy enough to do it myself.

Steve

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by mwr (24.158.xx.xx) on Fri 25 May 2007 at 23:29
[ View Weblogs ]

Csync2 is in stable. It looks good enough for synchronizing config files among a group of systems, and running arbitrary commands after particular files get updated, but doesn't look to be much simpler than a puppet installation that does equivalent tasks. But if it works for you, stick with it until it doesn't.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by oDn (86.197.xx.xx) on Sat 26 May 2007 at 18:29
My mistake.
Indeed csync2 is in stable.
All our servers are running oldstable, that's why I got all confused ...

I haven't gone through all the possibilities that both csync2 and puppet offer but I get the feeling csync2 is really appropriate for synching machines in a cluster that only require one poxy little config file and a restart.

But as you say, what works for me might not work for everyone.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by MikeMc (67.190.xx.xx) on Sat 26 May 2007 at 06:40

If you are evaluating a system administration/configuration management tool such as this, I'd recommend taking a look at Bcfg2 in addition to Puppet. As my collection of servers and virtual servers grows (quickly, since I'm using both Xen and Linux VServer), I decided to implement some automation using Cfengine, since it is widely known and has been around for quite some time. However, upon doing some research on alternatives, I learned that Bcfg2 and Puppet seem to be where the leading-edge is right now, as they both seem to have learned lessons from Cfengine.

I ultimately went with Bcfg2 over Puppet and Cfengine after reading some of the following sites:

  • This blog post contains a detailed conversation between John Goerzen (a Debian Developer) and Narayan Desai (one of the lead authors of Bcfg2), with some small comments from our own Steve Kemp and Luke Kaines (author of Puppet).
  • The Configuration Management mailing list archives of LOPSA (League of Professional System Administrators). Both Kaines (Puppet) and Desai (Bcfg2) are prolific writers and frequent commenters on this list. Browse through a few months of the archives and you'll have a good sense about the tools and how they approach similar problems.

I'm happy with Bcfg2 because of its "Dry Run" and "Interactive" modes that let me carefully see what changes it wants to apply to a specific host given the configuration specified on the server. I like the fact that its lead authors seem to use Debian as one of their primary development/administration platforms, so your can build a Debian package right from the bleeding edge SVN repository. I found the academic articles published by the authors of Bcfg2 to be useful in understanding the problem space and how they address it. I'm thrilled with how quickly the two minor problems I reported via Bcfg2's Trac were resolved (one within hours, and on a weekend, the other by the next business day).

So while I picked Bcfg2, I believe both are good candidates. I'd say that if you manage three or more servers, it's more important that you start using one of them than which one you use. Both are under active development, and I suspect that both will converge on the same set of "good ideas" over time.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (69.17.xx.xx) on Sat 26 May 2007 at 21:12
Puppet also has a dry run mode -- we call it noop. I think all tools in this space have this mode; at least, I'd be shocked if they didn't.

I agree, though, that it's more important to choose any tool than which specific tool you choose, although I'd prefer people picked a tool that made it easy to share configurations, and I don't see any other tools that really think about this part of the problem.

And my last name is spelled "Kanies", btw. :)

--Luke

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (83.67.xx.xx) on Mon 8 Feb 2010 at 13:24

There's an interesting discussion about Puppet's 'dry-run' mode in my "Puppet vs Chef" article:

Puppet vs Chef

Specifically, how valuable is a dry-run or 'noop' mode when the system is unconfigured and we have to assume that all dependent actions were successful - as opposed to simply running the code against a VM or other disposable test machine?

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by eric (194.2.xx.xx) on Wed 6 Jun 2007 at 15:29
[ View Weblogs ]
How do Puppet know if a client is a sarge or an etch machine? Or how do you classify the clients between sarge and etch?

:eric:
http://blog.sietch-tabr.com

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Steve (80.68.xx.xx) on Wed 6 Jun 2007 at 15:35
[ View Steve's Scratchpad | View Weblogs ]

In my case it doesn't - I know I'm not running it upon Sarge machines!

Still using the "facts" system you can determine such things at run-time on a per-host basis. The documentation does have some good examples of that.

Steve

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by eric (194.2.xx.xx) on Thu 7 Jun 2007 at 16:40
[ View Weblogs ]
Okay, in fact I was talking about etch and sid, from your example. I didn't see the "facts" system but will look at it.

:eric:
http://blog.sietch-tabr.com

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (138.47.xx.xx) on Fri 22 Jun 2007 at 01:25
Install the lsb-release package on Debian (sarge/etch/*) and use Facter to list all the available facts for a particular machine.
> aptitude install lsb-release
> facter
[...]
lsbdistcodename => etch
lsbdistdescription => Debian GNU/Linux 4.0r0 (etch)
lsbdistid => Debian
lsbdistrelease => 4.0r0
lsbrelease => core-2.0-noarch:core-3.0-noarch:core-3.1-noarch:core-2.0-ia32:cor e-3.0-ia32:core-3.1-ia32
[...]
-- Michael

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by jparrella (65.254.xx.xx) on Wed 13 Jun 2007 at 20:44
[ View Weblogs ]
I wrote a little introduction to Puppet in spanish in my blog -- we're know managing 300+ hosts with Puppet, adding 10-15 a day until we reach 4.5K-5K machines by 2008, I guess. I didn't really bothered to try cfengine, and we didn't have time to try it, so we started with Puppet/Facter and we're very pleased now.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (89.16.xx.xx) on Tue 19 Jun 2007 at 18:35
That's a lot of hosts, and a /lot/ of additions per day ;) Must be fun!

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (66.9.xx.xx) on Fri 2 Nov 2007 at 21:45
Steve,

I've been finding it easier to get my head around Puppet by reading your articles than by using their wiki.. One question that I am having trouble finding the answer to is how to puppet is used to keep a machine up to date, like with security updates.

As a newbie to utilities like Puppet and CfEngine type utilities could you please expand your coverage of puppet to include this aspect of managing multiple machines?

My machines are using etch.

Thanks

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by henning (85.177.xx.xx) on Tue 28 Apr 2009 at 01:37
On Lenny, it seems to me that the explicit import is not necessary - even though I pulled out a class into the classes directory (which I needed to create beforehand), it's actions seem to be executed.

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (193.65.xx.xx) on Tue 7 Sep 2010 at 07:41
Great article on getting puppet up and running quickly - I had never used it before but had it up and running almost straight away. Better than the actual Puppet docs!

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Anonymous (62.16.xx.xx) on Thu 7 Apr 2011 at 13:56
Puppet is not a production material at all!!
- poor design and no scalability

just wasted my precious time!!

[ Parent | Reply to this comment ]

Re: 2/2: An introduction to using Puppet
Posted by Steve (82.41.xx.xx) on Thu 7 Apr 2011 at 13:58
[ View Steve's Scratchpad | View Weblogs ]

Many people are using it in production; if you believe it is not up to such use you'll need to explain why to have people take you seriously.

Steve

[ Parent | Reply to this comment ]