Pivotal open sources its Hadoop and Greenplum tech, and then some

Pivotal, the cloud computing and big data company that spun out from EMC and VMware in 2013, is open sourcing its entire portfolio of big data technologies and is teaming up with Hortonworks, IBM, GE, and several other companies on a Hadoop effort called the Open Data Platform.

Rumors about the fate of the company’s data business have been circulating since a round of layoffs began in November, but, according to Pivotal, the situation isn’t as dire as some initial reports suggested.

There is a lot of information coming out of the company about this, but here are the key parts:

  • Pivotal is still selling licenses and support for its Greenplum, HAWQ and GemFire database products, but it is also releasing the core code bases for those technologies as open source.
  • Pivotal is still offering its own Hadoop distribution, Pivotal HD, but has slowed development on core components of MapReduce, YARN, Ambari and the Hadoop Distributed File System. Those four pieces are the starting point for a new association called the Open Data Platform, which includes Pivotal, [company]GE[/company], [company]Hortonworks[/company], [company]IBM[/company], Infosys, Pivotal, SAS, Altiscale, [company]EMC[/company], [company]Verizon[/company] Enterprise Solutions, [company]VMware[/company], [company]Teradata[/company] and “a large international telecommunications firm,” and which promises to build its Hadoop technologies using a standard core of code.
  • Pivotal is working with Hortonworks to make Pivotal’s big data technologies run on the Hortonworks Data Platform, and eventually on the Open Data Platform core. Pivotal will continue offering enterprise support for Pivotal HD, although it will outsource to Hortonworks support requests involving the guts of Hadoop (e.g., MapReduce and HDFS).

Sunny Madra, vice president of the data and mobile product group at Pivotal, said the company has a relatively successful big data business already — $100 million overall, $40 million of which came from the Big Data Suite license bundle it announced last year — but suggested that it sees the writing on the wall. Open source software is a huge industry trend, and he thinks pushing against it is as fruitless as pushing against cloud computing several years ago.

“We’re starting to see open source pop up as an RFP within enterprises,” he said. “. . . If you’re picking software [today] . . . you’d look to open source.”


The Pivotal Big Data Suite.

Madra pointed to Pivotal’s revenue numbers as proof the company didn’t open source its software because no one wanted to pay for it. “We wouldn’t have a $100 million business . . . if we couldn’t sell this,” he said. Maybe, but maybe not: Hortonworks isn’t doing $100 million a year, but word was that Cloudera was doing it years ago (on Tuesday, Cloudera did claim more than $100 million in revenue in 2014). Depending how one defines “big data,” companies like Microsoft and Oracle are probably making much more money.

However, there were some layoffs late last year, which Madra attributed to consolidation of people, offices and efforts rather than a failing business. Pivotal wanted to close some global offices and bring the data team and Cloud Foundry teams under the same leadership, and to focus its development resources on its own intellectual property around Hadoop. “Do we really need a team going and testing our own distribution?” he asked, troubleshooting it, certifying it against technologies and all that goes along with that?

EMC first launched the Pivotal HD Hadoop distribution, as well as the HAWQ SQL-on-Hadoop engine, with much ado just over two years ago.

The deal with Hortonworks helps alleviate that engineering burden in the short term, and the Open Data Platform is supposed to help solve it over a longer period. Madra explained the goal of the organization as Linux-like, meaning that customers should be able to switch from one Hadoop distribution to the next and know the kernel will be the same, just like they do with the various flavors of the Linux operating system.

Mike Olson, Cloudera’s chief strategy officer and founding CEO, offered a harsh rebuttal to the Open Data Platform in a blog post on Tuesday, questioning the utility and politics of vendor-led consortia like this. He simultaneously praised Hortonworks for its commitment to open source Hadoop and bashed Pivotal on the same issue, but wrote, among other things, of the Open Data Platform: “The Pivotal and Hortonworks alliance, notwithstanding the marketing, is antithetical to the open source model and the Apache way.”

The Pivotal HD and Hawq architecture

Much of this has been open sourced or replaced.

As part of Pivotal’s Tuesday news, the company also announced additions to its Big Data Suite package, including the Redis key-value store, RabbitMQ messaging queue and Spring XD data pipeline framework, as well as the ability to run the various components on the company’s Cloud Foundry platform. Madra actually attributes a lot of Pivotal’s decision to open source its data technologies, as well as its execution, to the relative success the company has had with Cloud Foundry, which has always involved an open source foundation as well as a commercial offering.

“Had we not had the learnings that we had in Cloud Foundry, then I think it would have been a lot more challenging,” he said.

Whether or not one believes Pivotal’s spin on the situation, though, the company is right in realizing that it’s open source or bust in the big data space right now. They have different philosophies and strategies around it, but major Hadoop vendors Cloudera, Hortonworks and MapR are all largely focused on open-source technology. The most popular Hadoop-ecosystem technologies, including Spark, Storm and Kafka, are open source, as well. (CEOs, founders and creators from many of these companies and projects will be speaking at our Structure Data conference next month in New York.)

Pivotal might eventually sell billions of dollars worth of software licenses for its suite of big data products — there’s certainly a good story there if it can align the big data and Cloud Foundry businesses into a cohesive platform — but it probably has reached its plateau without having an open source story to tell.

Update: This post was updated at 12:22 p.m. PT to add information about Cloudera’s revenue.

Apprenda teams with Piston to gain OpenStack support

Apprenda, which started out as a .NET-and-Windows-focused Platform-as-a-Service but has since opened up to other languages and technologies, continues to broaden its horizon. A new partnership with Piston Cloud gives it an entry into the OpenStack camp.

Here’s the PR spin from the announcement:

Together, Apprenda and Piston will deliver a tightly integrated solution that enables agile software development teams to build Java and .NET cloud applications and microservices faster in a true hybrid cloud environment. With more enterprise developers turning to both PaaS and OpenStack solutions than ever before, it makes sense to deliver a powerful joint solution.

Piston co-founder Chris MacGown said the deal makes sense given that both Piston CloudOS and Piston OpenStack are meant to be vendor agnostic in terms of underlying hardware. “We believe that developers want to consume platform-as-a-service, and that there’s not yet a one-size fits all approach to meet that need. This means we need to provide, integrate, and partner with PaaS solutions tailored for specific use-cases,” he said via email.

Since Cloud Foundry never focused on the Windows arena, this partnership brings .NET-focused platform to Piston OpenStack, he added.

If you follow vendor shenanigans, this is an interesting turn because Piston and its other co-founder, OpenStack pioneer Joshua McKenty, have been fairly tightly aligned with Cloud Foundry, the open-source PaaS backed by Pivotal, [company]IBM[/company], [company]HP[/company] and others. In fact, McKenty recently left Piston for Pivotal — which offers a PivotalCF, a commercial version of Cloud Foundry.

Cloud Foundry claims to bring PaaS capabilities to your public cloud of choice, and Apprenda CEO Sinclair Schuller has been openly dismissive of public PaaS adoption in general. Apprenda paints itself as the enterprise-class private PaaS, a contention that the PivotalCF folks can’t stand.

Apprenda billboard on Pivotal's SF building

So it’s fair to say that Apprenda and Pivotal are not tight. Apprenda recently plastered a billboard on the side of Pivotal’s San Francisco headquarters building (pictured above).

Back to Piston and Apprenda: If this partnership delivers what it promises, customers can get — as Piston CEO Jim Morrisroe put it in a statement — “a scalable turnkey [OpenStack] IaaS and [Apprenda] PaaS out of the box.”

Disclosure: Piston is backed by True Ventures, a venture capital firm that is an investor in the parent company of Gigaom.

Note: This story was updated at 5:55 a.m. PST, December 24 with Chris MacGown’s comments.


Cloud Foundry Foundation brings in reinforcements

The Cloud Foundry Foundation, established in February,  will get a new marquee name Tuesday with [company]Intel[/company] coming aboard as a Platinum member.

Other companies, including [company]Akamai[/company], [company]Fujitsu[/company], [company]Hortonworks[/company], [company] Mendix,[/company] [company]SAS[/company] and others also joined the effort to push Cloud Foundry as an open framework for cross-cloud Platform as a Service (PaaS).

If it achieves its goals a business customer could “write an app once and run it in many places with lots of services,” said Leo Spiegel, SVP of strategy and corporate development at Pivotal, the company behind the commercial PivotalCF version of the PaaS. Those “many places” would include PivotalCF, or [company]IBM[/company] BlueMix or [company]HP[/company] Helion or whatever iteration of Cloud Foundry it deems most appropriate. That would alleviate corporate concerns of platform or vendor lock-in but is also a tall order.

Another (unspoken) part of the foundation’s aim is to show that Cloud Foundry is not under the control of one vendor — Pivotal or its parent company [company]VMware[/company] — so bringing in these big names and their code contributions is a plus. But then again, all those vendors want to wring competitive advantage out of Cloud Foundry, which is at odds with providing customers an easy offramp to someone else’s version of the PaaS.

To expedite logistics, the Cloud Foundry leadership signed the Linux Foundation to provide staffing, legal assistance and help with events.

“Why recreate things that exist?” Spiegel asked. “We raised plus or minus $6 million a year — we want to use that cost-effectively and not hire separate legal and event teams that are already in place.”

HP’s Helion OpenStack is here

If you want an HP-branded OpenStack or an HP-branded Cloud Foundry, you are in luck: Both are available today with support from friends Docker, New Relic, MongoDB.

Pivotal nabs OpenStack co-creator and Piston co-founder Joshua McKenty

Joshua McKenty, one of the early architects of OpenStack while at NASA, and a co-founder of OpenStack startup Piston, has joined Pivotal at field CTO for Cloud Foundry. He hopes to make Cloud Foundry, running on OpenStack, into what NASA envisioned several years ago.