Getting to the Data That Matters Most

By now, you’ve probably heard of the term “Big Data,” which describes the huge data sets that are available into today’s enterprise environment. On a micro level, you could argue that there’s too much monitoring data to process and it’s resulting in a forest/trees scenario where you’re not sure what you’ve got.

In most instances, when it comes to monitoring you probably want to get as much information as you can. If you’re doing a post-mortem to figure out what went wrong when your mission critical application crashed, you want to get down to the transaction level and see where it all went wrong, so you can fix it and try to find ways to prevent a similar break-down from happening again in the future.

That’s all to the good, but can that much information get in the way sometimes? It probably can and it requires a way to filter through the noise to get to the issues that matter most to you. In recent months HP bought enterprise search vendor Autonomy and just last week Oracle announced it had bought enterprise search vendor, Endeca.

In fact, I’ve argued that this wasn’t a coincidence. It’s because both companies recognize that with increasing amounts of data, companies need ways to get to the nuggets of information that matter most to them. Sophisticated enterpise search tools like those from Endeca and Autonomy give companies a fighting chance to do just that.

When it comes to monitoring, Jacob Ukelson believes we need tools to that do this on a performance monitoring level because as he sees it, we get too many false positives, so much so that we often ignore alerts because they are so often mundane that we miss it when a good one comes along.

He writes:

What we need are brains that distill the important essence from the sea of available performance data, and give it to IT in a way that provides complete, accurate, actionable alerts – along with domain specific root cause analytics.

And it’s possible that tools like the ones from Endeca and Autonomy could help do that. Endeca, for instance lets you build applications on top of the search platform to find specific types of data. You could create an application that pulls data from your application performance monitoring software, whatever that software is, and provide you with a dashboard of the data you consider most important.

Because it’s customized, you could refine it over time to accommodate issues as they come up inside your organization.

This is just one way to think about approaching this problem, but it’s clear that there can be such a thing as too much information and we need to find ways to get at the monitoring information that matters most to us and lets us take action when needed to prevent or solve problems.