Expect for DevOps
If you’re in DevOps, you need to know about Expect. If you’re in DevOps, accustomed to working on nitty-gritty back-end details of datacenter configuration, and you want a quick career boost, you truly need to learn a little Expect.
Expect is the real “Internet duct tape”. While that label has been applied to everything from a blog to the popular computing language Perl, it’s Expect that fits the metaphor best. Duct tape dominates transient hacks: nearly anything you do with duct tape can be done in a different way better and more permanently. The point is that duct tape fits for those situations that are important enough to get right, but urgent or frugal enough not to allow for a fully-engineered solution. Folklore claims that the largest single modern use of duct tape is not taping ducts, but set-up at theatrical performances, where execution has to be good enough to reward $100 ticket holders at the same time as they’re flexible enough to switch from tonight’s opera to tomorrow’s hockey game.
All of these aspects have parallels in use of the Expect computing technique. From its creation over twenty years ago by National Institute of Standards and Technology computer scientist Don Libes, Expect has been hard to explain because it’s most valuable for getting out of situations you don’t particularly want to be in. It’s not directly comparable to a language like Java or an application like Firefox, which concentrate on solving all the problems in their respective domains. Although Expect can be packaged as a general-purpose computing language, or a stand-alone application, or in a couple of other manifestations, its value is highest when it only fills a small gap left by other facilities.
Here’s an example: I recently needed to scan a few hundred Linux servers to ensure that particular applications were current, correctly installed, and had the right permissions. The installations couldn’t be identical, as it happened, but the variations were easily computed.
Long term, the solution is obvious–we need to use standard Debian-style packaging, along with a tool like Puppet (or cfengine or Maven or …) in a rationalized network environment controlled by exchange of cryptographic keys. That’s the right way to do things, and we’ll eventually have all those pieces in place. The organization and I would probably not make it to those long-term horizons, though, if we waited for the perfected solution. When the need arose, it was for a solution that day.
The next alternative was “manual” labor: having a skilled sysad or three check each server “by hand”. Any time a task’s tedium repels you, especially when it involves repeated log-ins, look for a way Expect can take over for you. The heart of Expect is automation of keystroke interactions that otherwise would be done by human fingers. While Expect is often identified with Libes’ original Tcl-based implementation, what’s important is the idea of keystroke automation. The technical label for this is
pty, for “pseudo-terminal“, an abstraction of keyboard entry. While Libes’ original implementation had certain technical advantages that haven’t yet been surpassed, nearly all common languages, including the sh-bash-ksh family, now support at least one workable Expect interface.
For my example case, scripting a comprehensive Expect solution took about forty minutes to write and another ten to run, and will be adequate until we roll out all the packaging and key management we plan. Fifty minutes was a big improvement on the weeks a polished solution will require, or even the five hours I estimate a “manual” approach would have cost (without accounting for the errors to which human operators are prone, or the times we’re likely to need to re-scan before any “final solution”). Those hours I saved were “pure profit”, as far as I’m concerned.
For more details about ways Expect helps in common networking and sysad situations, see other articles I’ve written on the topic.