That’s not the impression many executive-level decision-makers have, of course; they truly believe that the next dashboard or console will substitute for expertise, and allow them to staff datacenters with entry-level screen-watchers. From their perspective, “deskilling” is a reasonable expectation for digital products, and many don’t have enough background in software to understand why it doesn’t work that way.

Consider a model incident: because you have analyzed your needs well and prepared wisely, when a particular mission-critical application starts to slow down for several of your branch offices, you immediately see an alarm on your desktop that alerts you to the problem. The “disconnect” comes in interpretation of what happens next. Even if the monitor is advanced enough to diagnose a root cause, and the particular symptom at hand is sufficiently standard to permit such a diagnosis, it’s a mistake to believe that anyone can repair the damage and restore functionality throughout the organization. Suppose the monitoring product detects a Java resource leak, or a transient SQL slowdown, or a partial failure in one of your RAIDs. Even in this ideal circumstance, when the monitoring product is operating at its best, it still requires expertise, insight, and judgment to determine whether the RAID should be re-built with resources at hand, or it’s better to order replacement components and define a new storage array.

That’s typical of real-life situations. Monitoring products can help slash the time to detect and occasionally diagnose problems from hours and days down to seconds and minutes. Naive responses to the diagnosis can create bigger problems that demand additional days and weeks to repair, though. The big win comes when good automated monitoring is paired with seasoned experience.

Automated monitoring is good for you, too: more of your time goes to solving hard problems, and creation of thoughtful solutions, with less of it going to tedious, mundane detection tasks such as calling different sites to ask what they’ve seen, or distinguishing between network and storage slowdowns.

As “The Programmers Shall Always Be With You” explained earlier, this is not about whether executives or entry-level operators or anyone else has the capacity to read manuals, configure monitors properly, and decide on remedial action. That’s not what they’re best at, though; you are. Better monitoring tools only accentuate your advantage in core devops skills.

Expect is the real “Internet duct tape”. While that label has been applied to everything from a blog to the popular computing language Perl, it’s Expect that fits the metaphor best. Duct tape dominates transient hacks: nearly anything you do with duct tape can be done in a different way better and more permanently. The point is that duct tape fits for those situations that are important enough to get right, but urgent or frugal enough not to allow for a fully-engineered solution. Folklore claims that the largest single modern use of duct tape is not taping ducts, but set-up at theatrical performances, where execution has to be good enough to reward $100 ticket holders at the same time as they’re flexible enough to switch from tonight’s opera to tomorrow’s hockey game.

All of these aspects have parallels in use of the Expect computing technique. From its creation over twenty years ago by National Institute of Standards and Technology computer scientist Don Libes, Expect has been hard to explain because it’s most valuable for getting out of situations you don’t particularly want to be in. It’s not directly comparable to a language like Java or an application like Firefox, which concentrate on solving all the problems in their respective domains. Although Expect can be packaged as a general-purpose computing language, or a stand-alone application, or in a couple of other manifestations, its value is highest when it only fills a small gap left by other facilities.

Server-room example

Here’s an example: I recently needed to scan a few hundred Linux servers to ensure that particular applications were current, correctly installed, and had the right permissions. The installations couldn’t be identical, as it happened, but the variations were easily computed.

Long term, the solution is obvious–we need to use standard Debian-style packaging, along with a tool like Puppet (or cfengine or Maven or …) in a rationalized network environment controlled by exchange of cryptographic keys. That’s the right way to do things, and we’ll eventually have all those pieces in place. The organization and I would probably not make it to those long-term horizons, though, if we waited for the perfected solution. When the need arose, it was for a solution that day.

The next alternative was “manual” labor: having a skilled sysad or three check each server “by hand”. Any time a task’s tedium repels you, especially when it involves repeated log-ins, look for a way Expect can take over for you. The heart of Expect is automation of keystroke interactions that otherwise would be done by human fingers. While Expect is often identified with Libes’ original Tcl-based implementation, what’s important is the idea of keystroke automation. The technical label for this is pty, for “pseudo-terminal“, an abstraction of keyboard entry. While Libes’ original implementation had certain technical advantages that haven’t yet been surpassed, nearly all common languages, including the sh-bash-ksh family, now support at least one workable Expect interface.

For my example case, scripting a comprehensive Expect solution took about forty minutes to write and another ten to run, and will be adequate until we roll out all the packaging and key management we plan. Fifty minutes was a big improvement on the weeks a polished solution will require, or even the five hours I estimate a “manual” approach would have cost (without accounting for the errors to which human operators are prone, or the times we’re likely to need to re-scan before any “final solution”). Those hours I saved were “pure profit”, as far as I’m concerned.

For more details about ways Expect helps in common networking and sysad situations, see other articles I’ve written on the topic.

Recently an article was written on the impact “Application-Aware Network Performance Management”* has on business operations. Organizations that utilize important information technology (IT) resources such as mission critical applications and systems have long known that optimized IT resources mean optimal business productivity. By ensuring the fastest response time and availability of applications, systems and networks an organization will run efficiently from top to bottom. To read the entire article click here, below we’ve summarized three important aspects from the article:

1) Do the applications improve user experience?

If business critical applications are utilized by non-IT pros daily, their experience at the end level is crucial for success. The ability to ensure that their experience, and those behind the scenes, is running optimally means that business productivity should run smoothly. Proactive approaches to make sure no down time occurs is therefore, critical for success.

2) Does the business model align with the IT department’s core mission?

If an organization is trying to keep overhead costs low by utilizing the resources they already have, perhaps that new server and hardware upgrade isn’t the best option at the current point in time. Organizations need to make sure everyone is communicating, particularly if applications running an organization can be continually used and not upgraded or changed frequently.

3) Is the integration and performance management* of applications critical to the business?

Business are created to solve a solution or administer a service, on the back end of that they are also tasked with generating revenue for its employees and stakeholders. This means that if organizations can highlight issues quickly and easily inside an IT department, what it will do for productivity as a whole? Some organizations have the luxury of separating and delegating their applications in different departments to do different things, some need a unified window at which to utilize everything.

Monitoring applications inside the IT infrastructure is critical for business success. Whether or not increasing application usage, the spread of applications throughout the organization or even reducing the complexity of those applications is critical for organization efficiency needs to be decided and discussed. Want to lean more on how Heroix can deliver application performance information on an iPad, iPhone or Android? Visit us on www.heroix.com or send us an email at info@heroix.com  Problems are displayed in easy-to-understand, color-coded pie charts, with drill down access to the details you need to quickly and easily fix any issues in the IT infrastructure.

*APMDigest

Many of the potentials of a “DevOps” orientation are familiar by now among systems administrators. Not so well known are several of the time-tested techniques programmers have long used to make the most of their software-oriented development efforts, and that also apply to administrative work.

It’s not that devops haven’t heard of parametrization, logging, version control, and peer review, or that programmers are smarter for making more use of them; mostly it’s just that these techniques haven’t yet become part of the routine culture of system administration. It’s time not just to enjoy the advantages of software-defined devops, but also to manage the technical debt it too-often incurs.

DRY for better configuration

To configure a new virtual machine or deploy a new service across a server farm with just a few lines of VMX, Puppet, or so on is exciting, and even liberating, when compared to the tedious chore of fumbling with a physical machine’s limitations and point-and-click clumsiness. Even the most mission-critical applications are increasingly living in a virtualized, software-defined world. Is there “boilerplate” in your configuration definitions, though? Do you have to be careful to make sure that the ‘MY_TEST_SERVER_37‘ hostname in line 4 matches the IP address of line 17?

Automate your way out of such error-prone situations. Programmers often intone “DRY”: don’t repeat yourself. The principle applies equally to any other textual realm, including configuration definition. Many modern configuration languages include variables with which to parametrize definitions. Those that don’t can be generated with pre-processors or similar facilities. The particular technology you use–anything from XSLT in a browser to a Cobol program running on a legacy CPM/86 antique–is secondary to the idea of leveraging the computing power available to you to wring the repetitive work out of your tasks.

Logs of all shapes and sizes

Sysads are experts in logs. Many of us think of Apache or ssh logs as recreational reading. Too often, though, we concentrate on their consumption, and forget the opportunity we have to produce them.

When you’re working through a configuration puzzle, don’t dissipate your brainpower with assumptions about state and flow; instead, have your processes automatically record these variables in systematic logs, so you can examine them at your leisure. Also recognize your freedom to treat different values with different responses: I like to

• log everything,
• e-mail myself notices about events that might bear on current projects, and
• SMS (telephone-text) myself when certain alarms fire,

for instance. A good logging framework makes all this easy.

Version control is a simple necessity

All your configurations belong in version control (VC). Period. That’s non-negotiable, as far as I’m concerned–and what’s checked in better correspond to what is “live”! Read a good book like Eric Sink’s on what VC does for you, and how to make the most of it. Sink’s focus is on distributed programming teams who use a tool like Git, while I generally recommend SVN or another more “conservative” VC system for devops. Those details don’t matter nearly as much, though, as the decision to use some VC, and use it well.

More eyeballs

A final lesson devops can learn from programmers is to make good use of your teammates. Sysads’ pride in solving problems on their own is a good instinct; it’s easy to take it too far, though. Programmers have led the way in practices from “pair programming” to “code review“; sysads need to learn about these forms of co-operation, especially for assignments realized as delivery of textual artifacts such as configuration definitions.

It’s not just AA, of course. Technical choices and management that sound like matters purely for specialists end up affecting hundreds of millions of government documents, over a hundred thousand customers of the Royal Bank of Scotland, or millions of users of tech powerhouse Google.

AA’s episode on 16 April 2013 resulted in cancellation of what was variously reported as between 978 and 1950 flights, representing apparent receipts of at least $20 million, based on a daily run-rate at the operating company in excess of $50 million.

Where did all that loss–conceivably over $100 million total, once all impacts are summed–originate? We might never know. AA originally blamed the independent Sabre computerized reservation system. A few hours later, AA apologized for what it admitted was an error, and Sabre issued a press release that it was “operating as normal” on its side. Within the next days, AA Chief Executive Officer Thomas Horton identified the problem as a network outage that was understood internally and “… will not recur”.

It’s hard to confirm details from the outside. AA operations have been strained in recent years, from anecdotal drama which interferes with flight-crew co-operation, to more systemic difficulties in employee relations leading to poor on-time performance. Clint Boulton almost certainly has it right when he emphasizes that real-world software quality is not about one application showing the right colors for its display; the more important point is that enterprise-class programs inevitably co-ordinate and interact with many different systems. Mishandling a small error response in a Sabre request easily propagates to the extent that numerous physical components–airplanes, luggage, crews, and more–are all effectively lost or crippled.

This is why “Real User Monitoring” consistently heralds the importance of automated application performance management (APM) and allied technologies. Although an application might begin its life in a well-defined and well-understood role, if it’s successful and useful it inevitably grows beyond the point that any one expert or even team can keep up with all its connections and responsibilities. Only automated monitoring techniques can cope with and mitigate the cascades of errors that, in cases such as AA’s, disrupt thousands of lives and dissipate millions of dollars. Even at smaller companies than AA, unplanned outages cost thousands of dollars every minute. Too often, “disaster recovery” at best accounts for only “static” assets and doesn’t properly account for all the systemic relations on which mission-critical workflows rely. While APM itself isn’t a perfected service, it’s the best handle IT has on management of crucial operations.

The curse of software: it does what it’s told, not what was intended

An information technology (IT) department has successfully brought its server provisioning under control with Puppet, the popular declarative open-source configuration management tool. “Success” here means that the devops team has confidence its Puppet recipes maintain servers in desired states, with software at specified versions, the ability to roll changes back or forward readily and consistently, the capacity to simulate configuration changes without performing them, and so on. These are all positive qualities, and the team rightly feels pride in the security this infrastructure provides.

Yet technical debt lurks within. The Puppet recipes duplicate information, including SSH authorized keys, lists of related servers, and locations of sources for various resource. The inevitable result: occasionally, when a configuration is updated–for example, a new server is introduced in the pool of licensed file converters for a particular bottleneck–one of the lists is left un-updated. At that point, definitions that were supposed to be copies of each other become near-copies. The consequence: it’s only weeks or months later that someone notices the organization is paying for under-utilized (or even inactive!) resources.

No harsh criticism is intended. Programmers know that even the best practitioners are prone to such mishaps. As discussed before, technical debt is not all bad. What’s new now, though, is that a cohort of sysads and network managers have just begun with software-defined configurations, and they have little background in use of variables, conditionals, and other technical devices to bring technical debt under control.

That’s the message, then, for this wave of virtualizers, infrastructure-as-a-service consumers, and datacenter automators: recognize that professionalism requires not only careful configuration so that systems work correctly. Beyond that, the configurations also demand on-going maintenance, adjustment, and refinement. That unglamorous follow-up pays the technical debt unavoidable when systems are first designed and implemented.

Realization of a metaphor

That quote is from Holford’s recent “Can Technical Debt Be Quantified? The Limits And Promise Of The Metaphor“, which also has the virtue of mapping limits of the metaphor. For Holford, “[f]inancial debt has but one form: Money owed”, while technical debt has at least a dozen dimensions he usefully lists. You’ll want to read his thoughtful, nuanced description for yourself; make sure you also scan the Comments for the reference to Don O’Neill’s earlier managerial perspective on technical debt and its impact on planning.

For me, the financial metaphor has a few more insights to supplement Holford’s description. Alexandra Szynkarski, likes Holford, aims to quantify technical debt. In her “Prioritizing Your Technical Debt“, she observes that individual items of financial debt have not only a principal–Holford’s “money owed”–but also an interest. One doesn’t strictly determine the other. Szynkarski implicitly recommends a return-on-investment (ROI) model for decision-making: “If there is an item that has a very high interest with a relatively low principal, then that’s the one you want to focus on fixing first.”

Similar advice is common from financial advisors: households with multiple debts should pay off loans with high interest rates first, while of course staying current with other items. That’s what I always practiced and preached in my personal life. In the software lab, that means “paying the minimum” to keep applications afloat, while simultaneously focusing on a single “high-interest” area of technical debt where the development team focuses all its surplus attention.

Smallest debt first

Dave Ramsey has a variation on that counsel that programmers need to consider. Ramsey is a mass-market financial counselor who targets people controlled by, rather than in control of, their personal finances. One element of his elaborate “Total Money Makeover” particularly deserves our attention in thinking about technical debt: Ramsey’s followers pay off their smallest debts first, rather than those with highest interest rates.

That sounds like an arithmetic mistake. On paper, Ramsey’s approach incontestably costs more than a highest-interest-rate-first plan. Ramsey himself, though, writes, “The math seems to lean more toward paying the highest interest debts first, but what I have learned is that personal finance is 20% head knowledge and 80% behavior. You need some quick wins in order to stay pumped enough to get out of debt completely. When you start knocking off the easier debts, you will start to see results and you will start to win in debt reduction.”

There’s a lot of wisdom in Ramsey’s approach, enough to extend to its application in the technical debt metaphor. As important as measurement is in software development, and as clear as the benefits of ROI planning are, I often attack technical debt with a heuristic that looks for early and easy wins. When struggling households are getting their finances under control, complex strategies are no benefit; they need a simple way to create sustainable behavior, stay current with their creditors, and simplify their balance sheet. Similarly, what’s most important for development teams under stress is not a complicated dependency diagram of subtly-prioritized programming tasks. Instead, programmers most need a simple plan that minimizes surprises, adequately copes with the daily surprises of stakeholder issues, and frees up a little time to focus first on a few easy problems. “Take out the trash” is one of the images I have for this early stage in cleaning up technical debt: pay off the lowest debts first, and thus clear away their distraction so that it becomes possible to target bigger problems more effectively, and with more clarity.

In analytic terms, think of the combinatorics of debugging: it’s much harder to solve four interacting errors simultaneously, than one or two in sequence. That difficulty is not only cognitive–it’s hard keeping all the variables clearly in mind–but attitudinal, in that it’s a psychological strain to keep moving forward with no clear “wins”.

Win now

Quantify what you can of your technical debt, and plan with your decision-makers in terms of rational priorities. At the same time, identify small, achievable goals, and start with successes you can achieve quickly.

Technical debt is also a marker of a kind of vitality, as I’ll show below. One of the on-going consequences of virtualization and cloud migration is that whole new realms of computing are entering technical debt, and that is a good thing! More precisely, it has the potential to be a good thing. Here’s why:

Debt and its metaphors

Whether in the form of US student loans or South Asian farm peonage, debt is associated with misery and despair to the point of suicide. Whole religions abjure debt. Financial calamities–and banking crises specifically–inevitably result from “excess leverage“. The strongest high-tech (and other) companies are (financially) debt-free“.

Still, sophisticated economists compute that optimal debt is generally a positive value, not just zero. Debt invested has an entirely different character from debt incurred to fund current consumption. Financial lines-of-credit provide a kind of insurance to improve flexibility and protect against certain risks.

Programming incidents in the last week reminded me of these economic realities. In reviewing the source code behind applications in production, I found expressions that aren’t to my standards: programs filled with magic numbers and fragile calculations. On the other hand, these same applications meet my most important standards: they serve our end-users well, at least for now, and they were constructed in a way that made it easy for maintenance programmers to understand. Once we identified the problems, it was quick enough to replace inscrutable hard constants with human-meaningful symbolic values, and fortify the programs against future change.

If “technical debt” were an unalloyed evil, we should never have committed the original source code to our repository, and certainly not put the code into production. As Olve Maudel pointed out several years ago, however, “Technical Debt is Good“, at least to the extent it enables earlier delivery of value to customers. The metaphor itself is apt because financial and technical debt share so many characteristics: debt is oppressive when too large, but rigid adherence to freedom from debt makes work too clumsy and slow-moving to serve creative developers or the consumers of their programs well.

Software-defined agility

Whole new populations of DevOps will learn these lessons as industry moves to such new and more agile techniques as software-defined networking (SDN) and datacenter-as-a-service (DCaaS). Commenters who complain, for example, that SDN slows down packet delivery are right, but typically miss the point. Third-and fourth-generation programming languages are slower than assembler, in naive benchmarks, but we use higher-order programming languages because we’re able to achieve more with them. Similarly, SDN, DCaaS, and other virtualization highlights will help make us more nimble in rolling out effective solutions.

At the same time, debt needs to be kept in balance. In coming weeks, I’ll illustrate a few common instances of technical debt in SDN and other emerging technologies, and how good DevOps can keep technical debt from growing too large.

Every month or so, someone–Gartner last summer, Luke Stevens, Mike James, and others in the fall, up to the present–announces that the HTML5 emperor is more skimpily dressed than is seemly. “Hype or Hope?” is a headline that’s been re-used multiple times.

HTML5 is a clumsy subject, and certainly doesn’t fit comfortably in a single dimension of “hypeness”. It’s not a natural successor to HTML4, for instance; when a prospective developer tells me of her expertise with HTML5, I expect her to be conversant with JavaScript and CSS3 to a depth that HTML4 or XHTML never suggested.

HTML5 is ungainly enough to be almost useless as a project specification. HTML5 is “future” for some audiences; I work with some organizations that still require Internet Explorer (IE) 6 solutions. While, in a pinch, compatibility shims can bring HTML5 even to IE6, I generally don’t fight that battle. On the other hand, HTML5 is a “done deal” for other audiences, and as practical as can be: nearly all browsers for Web-ready telephones in the US give HTML5 capabilities, and I feel free to assume HTML5 for such mobile apps.

“HTML5″ is only the beginning of a technical description, though. Any real-world project must detail exactly what parts of HTML5 it requires–geolocation, video playback, and form fields are three much different parts of the sprawling HTML5 standard that tend to be used in different ways by applications. Effective creation and maintenance of an “HTML5-based” application inevitably requires more precise examination of what users do with the application, and how those users’ specific browsers support different parts of HTML5.

As much as deep Web history and themes of openness interest any of us strategically, there’s no need to take a position at that level to benefit tactically from HTML5 today. The current generation of Web browsers build in capabilities that would have seemed extraordinary just five years ago, and, in broad terms, HTML5 is the right starting point to unlock them.

A lot of buzz in the information technology (IT) world has been created by Gartner’s APM model, published back in August of 2012.  IT Centralization recently published an article composed by LarryDragich, which prioritized the APM framework and allowed those who are in the market for an APM (application performance management) solution the ability to see the dimensions, focus and potential benefits* associated with application performance management. Today we’d like to quickly discuss how one can monitoring VMware’s vMotion when performing a live migration in vShpere, an important aspect we found in an APM solution.

One of the most important aspects we found in this article relates around the end user experience* and coincidentally, live migration. Heroix couldn’t agree more, as end user experience was one of the primary priorities when selecting an APM solution.  In fact, the article state that 80% of the APM value comes from “application visibility for the business and helps lay the foundation for performance trending and predictive analysis.”* Having the ability to predict these trends is an important one, as it allows for (but is not limited to): capacity planning, trend analysis for migration and disaster recover (DR) systems as well as proactive resource allocation.

VMware’s vMotion: Having the ability to be proactive when addressing upcoming issues is massive, especially if addressing capabilities as vSphere’s  vMotion, where reporting on a resource move after it has occurred and then correlating the data with the virtual machine resources and application performance come into play.  From there, the proper APM can be administered.

A monitoring solution should have the ability to play out a scenario in advance, allowing the end user visibility into what could happen if resources were moved from one host to another. It should also include a capable reporting capability, including historical usage and context. This would allow the end user to see where and when resources were affected after a migration or DR failover.

Having the ability to take the metrics which are important for IT departments: CPU capacity, memory, disk, workloads, etc. and then quantifying them into value for an organization is imperative. This is what an APM solution does, and which is why it can ease many woes of the IT professional’s world.

Want to learn more on the proactive approach towards an APM (application performance monitoring) solution? Check out www.heroix.com or send us an email at info@heroix.com