Hadoop: “It’s damn hard to use”

When Continuuity CEO and founder Todd Papaioannou was setting up Yahoo’s(s yhoo) architecture it was like his home-brew computing club days all over again. As Chief Cloud Architect, Papaioannou and his 120-person team were tasked with setting up 45,000 Hadoop servers in Yahoo’s 400,000 node private cloud.
The idea was to create a stable platform over which 5,000 Yahoo developers could build their applications. What Papaioannou wound up with was “a bunch of redneck architecture.” Every team took bits of code from here libraries from there and cobbled together something a clever teenager might cobble together in his garage — except on a massive scale. Once the architecture was in place it took developers four or five months to launch applications and innovation stalled.
“In case you haven’t been paying attention, Yahoo may have had a few product issues from time to time,” Papaioannou told an audience Thursday at GigaOM’s Structure:Data conference in New York City.
What Papaioannou learned from the experience was one big lesson:
“Hadoop is hard – let’s make no bones about it,” Papaioannou said. “It’s damn hard to use. It’s low-level infrastructure software, and most people out there are not used to using low-level infrastructure software.”
That experience with Yahoo led him to found Continuuity, which creates an abstraction layer called AppFabric on top of Hadoop and HBase, which Papaioannou said would help companies and developers avoid the problems he encountered at Yahoo.
Check out the rest of our Structure:Data 2013 live coverage here, and a video embed of the session follows below:
[youtube http://www.youtube.com/watch?v=dWagSVBkaJk&w=560&h=315]
A transcription of the video follows on the next page