Client Application Management (Part 1, for .deb packages)

(Original infrastructures.org writeup here.)

Wow, this part has been a learning experience. The things I’ve picked up out of this stage:

• aptitude is not apt-get. Obvious, yes. But how different they are was not apparent until this weekend.
• pkgsync is great, and does exactly what it claims, but read its claims very carefully, since it can wreak legally-precise and well-defined havoc on your system.
• Xen instances beat actual hardware for testing puppet configurations. As long as the dom0 is accessible, I can pull up a console for a domU and investigate why my host fell off the face of the earth. Doing the same investigation on a real system requires a bit of a drive.

More after the jump.
I only run Debian GNU/Linux on our Linux systems, and I don’t think you can beat its package management policies and tools. Running primarily the stable release, and occasionally the testing release, I’ve never run into dependency hell. So I was pretty confident that anything to automate my usual route of apt-get install package1 package2 ... packageN would be a benefit.

When I was going through making cfengine work as my configuration management tool, I had started to use Steve’s autoapt scripts. I liked the way they worked: “for this hostname or system class, make sure the following packages are installed, and make sure the following are uninstalled.” No problems there, but if I was probably migrating away from cfengine to puppet, I’d take a look around and see what else was available.

While looking around, I found this recipe for pkgsync. Looked to be identical to what I had gotten used to with autoapt. And it almost was. Things I discovered, in relative order of occurrence:

1. Why isn’t this pkgsync recipe doing anything? I see it rebuilding the musthave list, but nothing’s getting installed. Oh, the recipe had a -s flag on pkgsync, causing it to just simulate what was going to be installed. Well, what’s the point of that? Delete that flag, and let’s go!
2. Why did my system drop off the face of the earth? It pings, but no ssh. Since it’s one of our dual-boot systems, maybe it got rebooted back into Windows via a power outage. No, nmap doesn’t show the usual Windows ports open on it. Maybe it’s back at a PXE debian-installer prompt, so drive in and see what’s up. Hmm, nope, it’s at regular login prompt. Log in. ps aux | grep ssh gives me ps: Command not found. Really? dpkg -l | grep -v 'ii' | less gives me less: Command not found. After reinstalling less, it’s obvious there’s a lot of stuff missing now. A little apt-get, dpkg, grep, and awk magic later, all my packages are back. So what happened? Oh, pkgsync has a -k flag, too. If you leave that off, it’ll remove lots of stuff. And it says so on the package page: “pkgsync will take care of meeting the demands put down in the lists, and then removing everything that is not in the ‘must’ or ‘may’ list and is not necessary for their operations (as determined by aptitude).” No -k flag, no guarantee that anything other than what is explicitly required by your musthave list will be installed after we’re done. aptitude is a harsh mistress, a mean mistreater in blues parlance. But on the positive side, I had a very lean system with xemacs21, build-essential, rsync, ssmtp, and stow installed.
3. Ok, so now all that’s cleared up. Go do other things, and pick back up on it in the morning. Hey, why’d the system fall off the face of the earth again? pkgsync has been fixed for hours, and I know nothing changed on the gold server while I was asleep. Oh, there’s a nightly cron job for pkgsync, and it doesn’t have a -k flag in it either. Let’s fix that, too. That would also explain why the original pkgsync recipe had a -s flag for pkgsync in it — probably not a mistake, but my assumption is that the author was happy with letting the cron job handle all the package updates.

So the final configurations:

/etc/puppet/manifests/classes/pkgsync.pp:

# $Id$
Exec { path => "/usr/bin:/bin:/usr/sbin:/sbin" }

class pkgsync_setup {
  package { pkgsync: ensure => installed }

  file { ["/etc/pkgsync/musthave", "/etc/pkgsync/mayhave", "/etc/pkgsync/maynothave"]:
    ensure => present
  }
  file { "/etc/cron.daily/nightly-pkgsync":
    owner => root, group => root, mode => 755,
    source => "puppet://gold.cae.tntech.edu/files/apps/pkgsync/nightly-pkgsync",
    require => Package[pkgsync]
  }
  file { [ "/var/lib/puppet/pkgsync", "/etc/pkgsync"]: ensure => directory }
  file { "/var/lib/puppet/pkgsync/musthave.d":
    ensure => directory,
    purge => true,
    backup => false,
    recurse => true,
    notify => Exec[pkgsync-musthave-rebuild]
  }
  file { "/var/lib/puppet/pkgsync/mayhave.d":
    ensure => directory,
    purge => true,
    backup => false,
    recurse => true,
    notify => Exec[pkgsync-mayhave-rebuild]
  }
  file { "/var/lib/puppet/pkgsync/maynothave.d":
    ensure => directory,
    purge => true,
    backup => false,
    recurse => true,
    notify => Exec[pkgsync-maynothave-rebuild]
  }
  exec { "pkgsync-musthave-rebuild":
    command => "find /var/lib/puppet/pkgsync/musthave.d/ -type f | xargs cat > /etc/pkgsync/musthave",
    refreshonly => true,
    notify => Exec[pkgsync]
  }
  exec { "pkgsync-mayhave-rebuild":
    command => "find /var/lib/puppet/pkgsync/mayhave.d/ -type f | xargs cat > /etc/pkgsync/mayhave",
    refreshonly => true,
  }
  exec { "pkgsync-maynothave-rebuild":
    command => "find /var/lib/puppet/pkgsync/maynothave.d/ -type f | xargs cat > /etc/pkgsync/maynothave",
    refreshonly => true,
  }
  exec { "pkgsync": refreshonly => true, command => "pkgsync -k" }
}

define pkgsync($ensure) {
  include pkgsync_setup
  file { "pkgsync_$ensure_$name":
    name => $ensure ? {
      present => "/var/lib/puppet/pkgsync/musthave.d/$name",
      absent => "/var/lib/puppet/pkgsync/maynothave.d/$name",
      optional => "/var/lib/puppet/pkgsync/mayhave.d/$name"
    },
    ensure => present,
    content => "$namen",
    notify => $ensure ? {
      present => Exec[pkgsync-musthave-rebuild],
      absent => Exec[pkgsync-maynothave-rebuild],
      optional => Exec[pkgsync-mayhave-rebuild]
    }
  }
}

Host definition from site.pp:

class cae-host {
  include timezone-central
  include ntp, ntpdate, sudo, openssh-server, syslog-ng
}

node ch208e {
  include cae-host
  include pkgsync

  # Use pkgsync for end-user apps with little to no configuration questions.
  # Things like syslog-ng, ntp, and others with persistent daemons,
  # non-default configuration files, etc. probably need to be broken out into
  # their own classes above.
  pkgsync {
    [ "xemacs21", "build-essential", "rsync", "stow", "ssmtp" ]: ensure => present
  }
}