Structure Data 2014: So we’ve got all this data. Now, what do we do with it?

As anyone who can fog a mirror knows by now, businesses are awash in data — from sales documents, support tickets, sensors and social networks. This is information about how their products and services are performing or how they are perceived to be performing in the field.

The more-evolved among these companies, many of whom spoke at or attended Structure Data 2014 this week in New York, are taking advantage of that flood to find the right data that will let them tighten the feedback loop between product delivery and product improvements. And tech vendors are banking — or hoping — that there are enough of those forward-thinking companies to support substantial businesses around storing, analyzing and manipulating that data.

Some of the most advanced of those customers have gotten to the point that they’re willing to toss out old IT infrastructure and fully embrace a new IT system. For them it may well be a matter of survival, depending on what their industry is and what their competitors are doing, as Pivotal CEO Paul Maritz said on Wednesday.

Are big customers ready to junk the old to bring in the new?

“Some players … have clear enough line of sight to competitive opportunities that they’re willing to build afresh, leave IT legacy behind and build a new platform, said Maritz “There’s nothing like the gallows to focus the mind.”

Pivotal is the company spun off from EMC(s emc) and VMware(s vmw) to build a next-generation big-data platform using technologies acquired by its two parent companies.

GigaOM Structure Data event at Pier 60, Chelsea Piers in New York, NY on Wednesday March 19, 2014. (© Photo by Jakub Mosur).

GigaOM Structure Data event at Pier 60, Chelsea Piers in New York, NY on Wednesday March 19, 2014. (© Photo by Jakub Mosur).

The promise is that companies glean incoming data of all types, analyze it, and then tweak their applications or user experiences according to what the data tells them; which ideally tightens up a virtuous cycle of fast-response development. So, if you’re at an ATM and it is out of money, the bank will know that and issue some sort of apology or make-good at your very next interaction. (Yeah, right, I don’t believe that either.)

Well, how about the next time your carrier drops a call, it applies an immediate credit to your bill? That could happen, at least around contract renewal time.

Of course, getting old-line mainstream companies to adopt technology that was cooked up at Yahoo or Google may not be the easiest sale for many of these customers that are used to buying technology from IBM or HP, but hey; IBM and HP are in on this act as well. And Hadoop trailblazers Hortonworks and Cloudera are doing their best to harden Hadoop and ancillary tools to make them truly enterprise-ready.

Cloudera, which just raked in $160 million in new investment, is buying into that rapid iteration philosophy. CEO Tom Reilly told Structure Data attendees that the company implemented its own Enterprise Data Hub last year, which enabled it to build a predictive support model. That means if one customer experiences a failure in a cluster, Cloudera can identify other customers with similar implementations and reach out to them to warn them of what may be coming down the pike and to fix their cluster before it goes bad. Stopping failures before they happen is one hard-to-measure but potentially invaluable plus of the rapid feedback loop big data applications can enable.

Tightening up the feedback loop to near-zero

Speaking of faster feedback loops, startups Expect Labs and Swiftkey don’t think anticipatory search — when Google(s goog) watches your keystrokes to suggest what you might be looking for — is enough. They’re working on anticipatory analysis of your searches. Swiftkey, for example, scrapes all phrases entered on the internet to build language models and then analyzes that phrasing for users of its predictive keyboard  for Android devices. Products like that and ExpectLabs Mindmeld, promise a world in which your device is at your command, even before you type or speak that command.

One word: Whoa.

Photo courtesy Jakub Mosur.

Structure Data Ticker