“Backbreaking” OpenStack migrations hinder enterprise upgrades

Technology updates are never a day at the beach, but customers trying to upgrade to the latest release of OpenStack from its predecessor are often encountering bigger-than-expected problems. That is definitely not good news for OpenStack as it tries to gain traction among big,  reference-type accounts at a time that Amazon Web Services is ramping up its enterprise push and VMware(s vmw) and Microsoft(s msft) are tuning up their own enterprise-focused cloud efforts.

Even some in the OpenStack community admit that upgrades and migrations are more difficult than they should be, especially if the customer is not using a packaged version of OpenStack from one of the major vendors. Of course those vendors all talk up their own OpenStack implementation as the key to success. For this story I talked to several executives in the OpenStack ecosystem.

One major issue is that the Grizzly-to-Havana upgrade often requires that users shut down their applications and restart them when it’s over. One reason is that the major OpenStack modules — Nova compute, Cinder block storage, Quantum networking — have their own databases. “As you build new features, the database schema gets changed and that makes it extremely hard to upgrade in a rolling fashion. You have to wipe out your system and reinstall,”  said one executive with an OpenStack vendor. To be clear, the issue of rolling upgrades isn’t even solved in mainline versions of the Linux kernel.

scaffoldingStill, “wipe out and re-install?” That noise you heard is the scream from IT departments across the land.

One company that would not be named, also had to rejigger those applications that had run on Grizzly post-migrationto get them to run on Havana at all.

Easing migrations and management are major selling points behind  “enterprise grade” releases of OpenStack like Red Hat’s new Enterprise Linux OpenStack Platform 4.0. Tools like HP’s(s hpq) TripleO and Mirantis Fuel are also geared to ease migrations and deployments.

Upgrade problem is “unresolved”

But as things stand now, “the upgrade problem is real and unresolved,” said the OpenStack vendor  executive.

One issue is that OpenStack, at just over 3 years old, is still very young but is rushing to add features and functions to compete with AWS. “All of that upstream development to add features is part of a mentality that isn’t focused on basic things that enterprises need — like backwards compatibility,” he added.

Remember: Cloud is hard

Cloud computing is complicated and a technology built and deployed by multiple, often-competing vendors, gets more complicated. Things take longer to get set in stone and adopted. While the promise of open-source cloud infrastructure is that customers needn’t worry about vendor lock-in (to AWS, to VMware etc.) sometimes having one vendor set the requirements, makes deployment easier. That’s the whole promise behind Oracle’s “cloud in a box” strategy and one reason there have been calls among some cloud pundits for a “benevolent dictator” to take over OpenStack strategy.

To be fair, others acknowledge difficulties in Openstack deployments but that they can be mitigated by choosing deployment partners wisely. Jonathan Bryce, executive director of the OpenStack foundation said via email:

“OpenStack can be used and deployed in many different ways. Commercial and community support is available through a variety of distributions, products and services to meet different application and user requirements. With that choice and breadth of use cases, upgrade experiences can vary, but we have consistently heard that Havana has been the smoothest upgrade yet.”

Dreamhost did the Grizzly-to-Havana migration in a week, said Jonathan LaCour, VP of product and development for the Los Angeles-based web hosting company.

“We use continuous integration and deployment so it was a matter of rolling it out to smaller test and staging clusters first. We had some data migration issues that could be improved upon. If we’d used Ubuntu or Red Hat that would have been handled but we have a lot of specific stuff we need to handle so we went our own way,” Lacour said in an interview.

He recommends that less tech-savvy businesses go for a packaged release of OpenStack as opposed to rolling their own. “There is no shortage of fully supported OpenStack packages that make it easier on the enterprise,” LaCour said in an interview.

Randy Bias, co-founder and CTO of Cloudscaling, another OpenStack purveyer agreed. “Upgrades with OpenStack will always be hard. People have to use an actual product to get smooth upgrades — OpenStack is a big distributed software system with lots of interdependencies. Rolling your own OpenStack deployment using DIY is like making your own Linux distribution,” he said.

Probably so. But another software exec at another of the major OpenStack backers — an IT giant with tons of resources — admitted (privately for obvious reasons) that his own company was challenged by its Havana upgrade.

Too many cooks?

Why is this happening even to very tech-savvy companies? He blamed the proliferation of separate projects — like Ceilometer for billing systems, Marconi for queuing and notification, etc. — which he said are not aligned back to OpenStack core projects like Nova and Glance. Those multiplying projects means, in his opinion, not enough follow-through on promises to fix key issues of scale, manageability, performance and stability.

He also cited not-so-far-beheath the surface infighting between the various OpenStack groups and backers. “That looks horrible to the customers that are trying to run businesses on the platform that could care less about the soap opera drama,” he said via email.

Ouch. A cloud pundit who is watching developments carefully but also asked for anonymity because he advises a couple of the OpenStack players, said many of the current OpenStack cadre — HP, Cisco, etc.– have mixed agendas and aren’t all that hot on selling OpenStack just yet.

“Sure they talk up OpenStack but in reality they’re all trying to sell their own converged data center hardware where they can make money. In two or three years they’ll get their acts togehter and start selling OpenStakc and making it more attractive to run atop all those data center goodies they’ve sold,” he said.

For more on why OpenStack is a tough enterprise sale, check out Gartner analyst Alessandro Perilli’s take.

Meanwhile, there remain just a handful of really big OpenStack implemenations — CERN being the poster child. Meanwhile, rumors of big new customers — Salesforce.com(s crm), for example — surface not through the alleged customer itself but via OpenStack proponent — Rackspace(s rax). It smacks of desperation.

So we’re back to the worry that’s nagged OpenStack for more than a year. At the OpenStack Summit this fall, the foundation touted new users including Workday, Concur, Shutterstock and Fidelity Investments. But there is still concern about whether there are enough big companies deploying OpenStack at scale to push it into the mainstream. It sounds like that won’t happen until upgrades and migrations are proven to be much, much easier than they’ve been to date.

In other words, give it a few more years. God knows what Amazon Web Services can do in that time.

Feature photo courtesy of Flickr user idigit_teddy