Authentication Servers, the Next Generation

I’m mildly embarrassed by my previous setup authentication servers, but this one should be a vast improvement. A reminder of the existing constraints and conditions:

  • Lots of Linux systems and a few Solaris systems, some of which dual-boot and aren’t accessible as *nix systems during normal hours of the day
  • An Active Directory already in place
  • An overworked administration team that has lots of other stuff to do besides create user accounts

As I had indicated at the end of the previous authentication server post, my experience with winbind has gotten decidedly more positive lately. My previous experience with it in 2001-2002 was entirely negative, since either due to design limitations or my own incompetence, I was unable to get multiple systems running winbind to agree on what the users’ UIDs and GIDs were, which made NFS unworkable. As of 2007, things have improved vastly. Continue reading “Authentication Servers, the Next Generation”

The New File Server: Puppet and Modules

On to Puppet. I’ve not yet factored everything about the new server out into separate modules or classes; that’ll come later. But things that will either get reused on other systems (e.g., Active Directory ties) or things that need to be generated consistently and repeatedly (e.g., Amanda configurations) have been factored out. The new server’s manifest looks like: Continue reading “The New File Server: Puppet and Modules”

The New File Server: Preseeding and LVM

Remember that no one cares if you can back up — only if you can restore.

So we’ve got a new file server in the middle of initial installation and configuration. The file server is one of our most mission-critical systems — if mail goes down, a half-dozen people care. If the web server goes down, a few more would care, but it’s not a life-or-death issue. But the file server? That’s important. Students I’ve never met, students who wouldn’t know the difference between a shell prompt and a hole in the ground, students who couldn’t care less about parallel computing or anything else I put effort into on this area — they’ll notice the file server being down.

And all things considered, I like it that way. I know that a student or faculty member is statistically more likely to lose data on their local hard drive, their flash drive, or their removable media of choice than I am to lose it on a RAID-5, hot-spare-ready, redundant power supplies drive array connected to a RAID-1, redundant power supplies server. We’ve had one data loss experience since 2001 when I started doing this. And we only lost data because of

  • Human error in moving the external RAID in our server rack
  • Having Amanda holding disk space on the external RAID in addition to what was on the system’s internal drives

and I’m not too keen to repeat it. I didn’t get more than 2 hours of sleep at a time for most of a week while I was constantly having to load a different tape in the changer. Thankfully, I didn’t have to camp out in the server room the whole time, since I could manipulate the changer via ssh. But it was both embarrassing and a major drag away from anything I’d have rather been doing at the time.

But the new file server is physically ready, and >90% ready as far as configuration and software are concerned. More details on this after the jump. Continue reading “The New File Server: Preseeding and LVM”

I Love Spam Karma 2

Back many months before I ever set up a blog server, one of the ITS people asked me in passing, “guess what happens if you search for viagra?” I had no idea. The answer (at the time) was a ton of entries in a guestbook or similar for one of the faculty. The answer (now) is the guestbook for a student organization, plus some articles about a TTU alumnus who was involved in the development process for said drug.

When I did set up some starter blogs on another platform, I didn’t have anything to post there, and neither did anyone else I set one up for. So they languished, got crawled, and then got spammed. But nobody else noticed, so I deleted those blogs and watched the 404 errors pile up in the web server logs.

So let me know confess my undying love for the goodness that is Spam Karma. Lots of spams eaten, few to none passed onto the actual pages. Most recently caught:
Thank you, “free mp3 ringtones” and “downloadable ringtones”, for your kind words. I tried to make my site nice, and I’m happy that it waw’ed you. “omegax”, who a found me site in google, thanks for stopping by. I think I know “traspoolkae”, a classmate of the nearly-famous Mark V. Shaney.

But most of all, thank you Dave of Unknown Genius. This plugin reduces my daily list of things to worry about by one.

My Own Private Debian Repository

So now that I’ve got all these .deb files made from non-free commercial software packages, and some more packages from unstable (since ANSYS depends on libopenmotif, and I needed a more current version of puppet), and a deb package of Torque based off the work of the nice folks at SARA, I need a place to put them. I already had a local Debian mirror, since many of my systems don’t get off-campus Internet access, and it’s used by other Debian users on campus.

So I made a new Debian-style repository off to the side of my regular Debian mirror. Features include:

  • A pool directory where all the binaries and sources actually reside
  • A release directory for etch, including binaries for both i386 and amd64 architectures, and sources
  • Gpg-signed, so I can convince apt to download from it without warning me about unauthenticated sources
  • Protected from unauthorized access via Apache directive, proftpd directive, and keyed ssh access.

This covers some of the same territory as Roberto Sanchez’s 2005 howto, but goes a bit further in some areas. Details after the jump. Continue reading “My Own Private Debian Repository”

Making Debian Packages from Commercial Software

One of my main goals for a managed infrastructure is to make sure I have consistent versions of end-user applications installed everywhere. My users aren’t too picky about the version of xemacs installed, but they’ve got pretty stringent requirements on having a particular version of ANSYS, Abaqus, Fluent, Maple, Matlab, and other large non-free/no-source-available software packages. And they don’t all agree on which version should be loaded, so I keep several loaded at once.

Previously, I’d install the software on one candidate system, test it, rsync it to the other systems that needed it, and hopefully put it in the appropriate system image later. And the nightly rsyncs on my dual-boot labs were a pain (can’t rsync multiple repositories to one target, so let’s have 42 clients simultaneously rsync two NFS-mounted trees to their local disks). So all things considered, it was time to make Debian-style packages of the big, expensive third-party stuff.

Goals (using Matlab as an example):

  1. The ability to run ‘apt-get install matlab74’, and get the latest packaged version of Matlab 7.4.
  2. The ability to keep Matlab 7.4, Matlab 7.2, and other versions loaded simultaneously.
  3. The ability to run any version of Matlab currently loaded on-demand — if I type ‘matlab74’, I expect Matlab 7.4. If I type ‘matlab72’, I expect Matlab 7.2.
  4. For the less picky users, ‘matlab’ with no numbers attached should still run some version of Matlab.
  5. I need to use the same “source” tree for both i386 and amd64 packages. I don’t want to duplicate my packaging efforts.

More details on how I’m doing this after the jump. Continue reading “Making Debian Packages from Commercial Software”

Client OS Update

(Original post here.)

For the moment, I’m working on Debian GNU/Linux. Everything bought new (since sometime last fall) has the current stable release (4.0r0, or “etch”) installed, and everything older has the previous stable release (3.1r6, or “sarge”). Assuming that I keep apt sources for both the primary Debian archives and their security updates, the puppet recipe for managing apt upgrades should keep me running.

Things won’t get much more complicated when I bring some Ubuntu desktops into the infrastructure, since Ubuntu uses the same basic package management tools as Debian. I’ll have to read up some more on Solaris patch and upgrade management when I get ready to convert our few Solaris workstations.

Client Application Management (Part 2, for stow packages)

UPDATE: this page largely superceded by the stowedpackage puppet definition.

Back in part 1, I outlined how I’m getting a consistent package load on my various hosts with pkgsync and puppet. This works great for things that are already included in Debian. And I’ll make .deb packages of some of our third-party commercial applications, too (Matlab, Abaqus, Ansys, etc.), mostly for the ease of running ‘apt-get install matlab74’ or making one entry into a pkgsync definition. But some things are more of a pain to package up, and it’s easier to let them install into /usr/local. But since I don’t want /usr/local to become an unmanageable mess, I’ve been using GNU Stow to manage it. The basics of stow is that a package (for example, Torque) that wants to install into /usr/local instead gets installed into /usr/local/stow/torque-2.1.6, and stow symlinks from /usr/local/sbin/pbs_mom to /usr/local/stow/torque-2.1.6/sbin/pbs_mom, and for all other files and directories contained in all your stowed packages.

So I needed a way to ensure that a particular set of packages was deployed consistently to a set of machines, and minor modifications to autoapt seemed to be the easiest and most robust way to go at the time. Details after the jump. Continue reading “Client Application Management (Part 2, for stow packages)”

Directory Servers

(Original here.)

This is a relatively quick task on my end. The central ITS department handles DNS, I don’t use automount, and the vast majority of UID/GID mappings I already covered in the Authentication Servers post, though it may technically belong here. One other thing at the bottom of the post bears repeating, even if you don’t do the rest of the infrastructure business:

We tend to use hostname aliases in DNS, and in our scripts and configuration files, to denote which hosts currently offer which services. This way, we don’t have to edit scripts when a service moves from one host to another. For example, we might create CNAMEs of ‘sup’ for the SUP server, ‘gold’ for the gold server, and ‘cvs’ for the CVS repository server, even though these might all be the same machine.

This sidesteps the whole “what’s your naming scheme on servers” religious war, and makes things much more maintainable long-term. For the longest time, we had this name disconnect between systems that the outside world would see, and systems that were only used on-campus. As long as it was just a small number of people having to manage things, there wasn’t much of a problem. But when you start having students wanting to work on their own laptops on the campus wireless network, or when you’ve got ad hoc support people trying to fix a problem, the lines between manager, user, and outside client get blurred. It just makes good sense to have your DNS names and your documentation correspond to specific services rather than something seemingly random. Everybody expects to find the web server at www. Eventually, everybody came to expect the mail server at pop, mail, or smtp. I’m starting to push more on the front of “the file server is reachable at files, license servers are reachable at ls01, ls02, etc.” No unnecessary rewrites of documentation, you can migrate services from one physical or virtual server to another with minimal downtime and plenty of time for testing, and other advantages become apparent.

Ad Hoc Change Tools

Most of the cost of desktop ownership is labor [gartner], and using ad hoc change tools increases entropy in an infrastructure, requiring proportionally increased labor. If the increased labor is applied using ad hoc tools, this increases entropy further, and so on — it’s a positive-feedback cycle. Carry on like this for a short time and all of your machines will soon be unique even if they started out identical. This makes development, deployment, and maintenance of applications and administrative code extremely difficult (and expensive).

Ad hoc tools are easy enough to get started with, but they’ll chew you up over time. They work great for small one-off problems, but they’re entirely too tempting to use for everything, and by the time you notice, the frog’s already boiled. That having been said, it was time for me to update my old looped-ssh script with something more robust. So I’m installing dsh on my administrative server, and defining groups mostly corresponding to my Nagios hostgroups. Eventually, I may dig into some scripting that will generate Nagios configurations and dsh configurations from one common dataset.