Seemingly overnight, big data became the behemoth to conquer. But the truth is, tried and true technologies have been tackling the problem for years. Versant’s Robert Greene gives respect to three unsung heroes of big data.
Hardware rarely comes up in discussions about big data, save for those centered on data warehouse appliances. But the omission hardly means hardware is irrelevant. In fact, big gear might become a big deal as companies look to bolster the performance of their big data systems.
One of the statements that struck me most from Structure: Big Data was CA CTO Donald Ferguson’s notion that big data represents a “very promising” opportunity for startups, particularly those targeting specific target use cases. I think he’s right, particularly with regard to the latter part.
I continue to watch the Data Marketplace trend with interest, and touched on some of the issues in two weekly updates here on GigaOM Pro last month. Two marketplaces with very different perspectives are those from Infochimps and Microsoft. I’ve spoken separately with Microsoft’s Moe Khosravy and Infochimps’ Flip Kromer, and even recorded one of my early conversations with Flip as a podcast. At last week’s Structure Big Data, Stacey Higginbotham was lucky enough to get them both together on stage for a chat. Clearly aiming high, Flip’s response to a question about the data marketplace opportunity was to suggest “We’re the next Google.” It’s easy to dismiss such a flippant remark as hubris, but it would also be unwise to underestimate the scale of this opportunity. Infochimps, Microsoft, and a host of others are racing to stake out their part of the emerging data web, and to make sure their share is as big as possible.
The GigaOM team spent the past week in New York City for a number of events, including our first (and definitely not last) East Coast conference, Structure Big Data. It’s all part of our plan to expand our footprint into new markets, including New York.
Mapr, a stealth-mode start-up with about 30 employees is developing a version of Hadoop and plans to compete with the likes of Cloudera. The company is likely to launch later this year and has been funded by Lightspeed Venture Partners and NEA.
Underlying all the useful applications, like Hadoop, that have emerged out of the big data ecosystem, there’s a fundamental assumption: The data that companies want will be able to be accessed when companies want and need it, explained Michelle Munson, CEO and co-founder of Aspera.
Estimates say that 90 percent of all data was created in the last two years alone. That staggering figure can lead to analysis paralysis for some organizations, but those that can sift through, analyze and take action on information faster than others will enjoy competitive advantages.
Pete Warden got famous for scraping 220 million Facebook profiles and then analyzing this data to unearth U.S.-wide user connection trends. Now he wants you to be able to do the same with a new web service released at GigaOM’s Structure Big Data conference today.
Google may have more distributed data than any other company but it still takes user input to create smarter machines. Google’s Voice Search speech recognition, for example, began to improve when the service started to train itself and improve accuracy through the use of end-user data
