Survey Finds Organizations Struggle to Harness Big Data

As companies gather increasing amounts of data from internal and external sources, they recognize the value of big data, but are struggling with how to harness it.

While organizations clearly see the value of big data, many are struggling to get a grip on  the tools that will help them get at the information they know is there — so says a new report from AIIM called Big Data – extracting value from your digital landfill.

The report found that Big Data as a trend is very real. In fact, when asked when they plan to make big data tool investments, more than half of respondents reported within three years with 10 percent reporting in the next 12 months. More than 5 percent have already started.

How they plan to go about it though turns out to be a mix of products for the most part. These could include open source,  so-called best of breed, custom development or using business intelligence or analytics product. Of these, the mix of products answer was by far the most popular with more than 40 percent. The next closest was best of breed with around 18 percent, followed by analytics or BI tools with around 15 percent.

As the report states, “Big data is very much in the early adopter stage. Many organizations are convinced of the benefits and are poised to jump in as soon as more packaged products are available with simpler user interfaces and greater applicability across different data repositories.”

Yet companies clearly understand that big data could add value if they can only harness it. In fact, half of respondents indicated they want to use big data for real-time prediction or incident prevention by analyzing data from a variety of sources including social media sentiment, web behavior and help desk calls.

But right now, when asked what’s holding companies back from taking advantage of big data, the most popular answers were cost and lack of in-house expertise at just over 35 percent for each one. The lack of expertise part is one area that IT pros can exploit to help their organizations

And this shows, big data is clearly something you as an IT pro need to be thinking about and acting as a thought leader at your company. In many ways, the role of the IT pro is changing as organizations allow more consumer-style choices, self-serve service portals and bring your own devices (BYOD), but IT can still lead when it comes to technology choices and Big Data should be a subject you are learning about today.

But getting the systems in place to analyze and understand the data is only the first step, then you need to figure out what data to analyze and what questions to ask. One of the biggest areas companies are curious about is something you are probably worried about too, and that’s finding ways to analyze your systems logs and failure messages. More than 80 percent of respondents said this is something they hope to be able to do with big data analytics tools.

And this confirms what I reported last week in the post, Too Much Information: Monitoring Becomes a Big Data Problem. You may recall that in that post, I reported on the EMC World keynote from VMware CEO Paul Maritz. He stated that systems were becoming so complex that it was impossible for humans to keep up with the reporting data and the data inside your logs was in fact becoming a big data issue.

The AIIM survey was conducted between March 30th and April 25, 2012. It involved 402 respondents from the AIIM community across a variety of geographic locations, industry sectors, job titles and company sizes. Check out Appendix 1 of the report for a complete breakdown of the respondent demographics.

When it comes to analyzing monitoring data, this is a case of big data really hitting home for IT Pros, but as this survey shows it’s certainly not just an IT problem. It’s a business problem and your job as an IT pro will be to guide your business units to the right solutions to find the answers they need inside that growing pool of data.

Leave a Reply

Your email address will not be published. Required fields are marked *