The promise of big data still looms, but execution lags

When something is hyped as much as the notion of big data, there’s bound to be disappointment when results don’t meet expectations right this second.

That realization — that implementation of big data analytics and related technologies hasn’t matched expectations — is a common thread across a recent spate of research reports. While corporate execs now “get” the possible impact of aggregating and analyzing all the data their companies generate, very few companies have realized that potential.

Most adoption remains at pilot or test stage

A new McKinsey Quarterly report acknowledged that earlier predictions that retailers would parlay big data analytics to boost operating margins by more than 60 percent, and that the healthcare sector could likewise use the technology to slice costs 8 percent, haven’t played out.

While massively scaled companies like [company]Amazon[/company] and [company]Google[/company] use data analytics to wring out significant costs, data analytics success at most legacy companies is limited to a few test projects or narrow pieces of the overall business. Very few of those accounts “have achieved what we would call ‘big impact through big data,’ or impact at scale,” according to McKinsey.

CEOs have an inflated view of their own big data projects

According to a just-released survey of 362 executives worldwide, there is a pronounced disconnect between what the CEO of a company perceives is happening with that company’s big data efforts and what the rest of the organization sees.

From the key findings of the work conducted The Economist Intelligence unit for [company]Teradata[/company],:

While 47 percent of CEOs believe that all employees have access to the data they need, only 27 percent of all respondents agree that they do. Similarly, 43 percent of CEOs think relevant data are captured and made available in real time, compared to 29 percent of all respondents. CEOs are also more likely to think that employees extract relevant insights from data – 38 percent of them hold this belief, as compared to 24 percent of all respondents and only 19 percent of senior vice presidents, vice presidents and directors.

So, while there is indeed a ton of available data, it’s still hard for most companies to parlay it into useful insights. More than half (57 percent) of the total respondents said their companies do a poor job in this respect, although almost everyone agreed that access to data and the ability to wring actionable insights out of it are critical.

Paranoia isn’t driving enough action

A whopping 60 percent of 226 execs surveyed by [company]Capgemini[/company] Consulting, in yet another report, said that big data will “disrupt” their industry in the next three years but only 13 percent of them have big data implementations in production. And, less than a third of those responding (27 percent) described their own big data initiatives as successful; 8 percent described them as very successful.

So while progress has been made — as Derrick Harris and Fast Forward Labs’ Founder Hilary Mason noted on this week’s Structure Show podcast — executives definitely know what Hadoop is now and have a grasp of what big data can do, much heavy lifting still needs to be done to realize the benefits of analyzing all that data.

For more on this topic of the latest and greatest technologies and how they are put to use — and the cognitive dissonance between big data hype and real-world adoption — will be discussed by top names in the field at Gigaom’s Structure Data event  in March.

Big data implementations status


To hear Fast Forward Labs’ founder Hilary Mason on how that company combs academia, open-source and “outsider art” sources for big data applications, check out the second half of the podcast below:

[soundcloud url=”” params=”color=ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false” width=”100%” height=”166″ iframe=”true” /]

AppDynamics launches new real-time analytics service

Application-performance monitoring startup AppDynamics now has a real-time analytics service called Application Analytics, it said Monday. The service lets organizations analyze all of their application and transactional data so that IT staff can spot errors and management can make better business decisions. In July, the startup landed $120 million in funding, highlighting the hot application-monitoring space market in which companies like New Relic and AppNeta compete. New Relic recently unveiled its own real-time analytics service called Insights.