Joyent goes big on big-data computing and storage with Manta service

Joyent, the Infrastructure-as-a-Service (IaaS) provider, is coming out with an object-storage service that lets customers compute right where the data is being stored. The purpose of the project is to provide more storage where it’s needed, but it’s also an admission that the storage industry has failed to address the growing needs of huge data consumers such as scientists.
It’s a novel concept, designed for the types of big-data work that lots of companies are interested in doing, potentially in the cloud, and other IaaS vendors could feel compelled to replicate it.
Joyent CTO Jason Hoffman talked about the exponential growth of data and the need for compute to keep up at our Structure conference last week, in what now looks like a precursor to this announcement:

What we’re really going to start right now, as far as human-generated content, a data set from machines that is so much larger than what we’ve ever dealt with in our lives, we have to sit down and say, “Well, what happens when compute and data begins to converge?”

What happens is performance of big-data computation improves. Instead of having to move objects from storage nodes to compute nodes, everything happens while the data is sitting in the server.
But it’s not just a matter of faster computing, said Bryan Cantrill, Joyent’s senior vice president of engineering (pictured). “When you can operate on petabytes of data in situ, it’s more accurate to say that it made things possible that were economically impossible before,” Cantrill said.
Customers pay for computing work running on the new service, called Manta, by the second, Cantrill said — .004 cents per GB of DRAM per second, to be exact. Still, storage and data bandwidth for Manta do have tiered price lists.
Hardware-wise, the approach is helped along by storage-heavy commodity servers not unlike Sun’s old Thumper gear. Inside the servers, Manta can get primo compute performance from Joyent’s SmartOS operating system.
Manta costs customers about as much money to use as Amazon (s amzn) Web Services’ S3 object-storage service, Cantrill said. But rather than try to appeal to customers on price alone, Joyent is putting on its competitive face this time by focusing on performance and specialized use cases. Those use cases include transcoding images and finding patterns in log files — both dealing with large amounts of data.
Manta sounds like a cloud service optimized for rapidly processing large volumes of unstructured data — so much that relying on DRAM for it all is out of the question. Expect the list of possible applications to multiply as companies kick Manta’s tires and resolve their doubts about big-data possibilities on public infrastructure. Issues around speed and cost could fall by the wayside.