Verizon explains its seamless cloud upgrade except not really

In a bid to be more transparent about last weekend’s 40-hour closure of the Verizon Cloud, the company just posted more about that massive upgrade. But, to be honest, it was more a history lesson about Verizon’s cloud ambitions than any new detail about what exactly it did to its cloud.

Mostly the post recapped the message from October, 2013, when Verizon first talked up its planned enterprise-class cloud, and explained how it was built from scratch to be completely virtualized and network addressable at all levels — all the endpoints, all the orchestration, firewalls, load balancers, identity management etc. Thursday’s post was written by Verizon CTO Kevin Clarke who, by the way, stepped in for his predecessor John Considine, who left Verizon last fall in a move that has not been reported but which the company confirmed on Thursday.

What all that virtualized goodness means, according to Clarke is:

Updates and maintenance, even unscheduled maintenance like those required with the Xen hypervisor security flaw last Fall, can be performed while client virtual machines continue to run, without requiring a two-zone setup or a machine reboot. It’s a little like performing brain surgery on yourself while you’re awake. Tricky? Yes, but necessary to keep our commitment to enterprise customers

Soooo, what exactly was done over the 40 hours last weekend to make sure this is the last massive maintenance closure? No clue. We’re just told that “Seamless Upgrade was a key addition” to achieve that aforementioned vision.

To be fair, the post was called: “Why seamless upgrades matter” but in reality everyone already knows that.

[company]Verizon[/company], [company]AT&T[/company], [company]CenturyLink[/company] and other telco-rooted companies moving into cloud have some heavy lifting to do to sell to many customers who think of cloud as [company]Amazon[/company] Web Services, full stop. There is a bias among many of these customers that telephone companies, despite their networking expertise — and, face it, networking is key — are not really in this game.

For that reason alone, Verizon had better hope that its seamless upgrade, whatever it actually was, works as promised.


40 hours later, Verizon says its cloud is back …

After taking a ton of heat for what ended up being a 40-hour maintenance shutdown over the past two days, Verizon said the work it did will prevent these sorts of stoppages in the future.

This maintenance work, added what Verizon called “seamless upgrade functionality” that will enable similar major upgrades to happen without service interruption, according to a press release posted Sunday afternoon.

Going forward, Verizon said, “virtually all maintenance and upgrades to Verizon Cloud will now happen in the background with no impact to customers.” There wasn’t a ton of information about how this will work, but there you have it.

Last weekend, when Verizon advised customers a week in advance of  what it said could be a 48-hour shutdown for planned maintenance, all sorts of things hit the fan. The prevailing opinion was that cloud computing vendors should be able to handle upgrades and maintenance with a lot less downtime than that.

Verizon is trying to make a name for itself in enterprise-class cloud infrastructure.  In that market it must contend not only with other telco-rooted companies– [company]CenturyLink[/company], [company]AT&T[/company] et al — which are trying to pitch the same customers but with public cloud giant [company]Amazon[/company] Web Services, which has proved serious about winning corporate workloads.

As if that’s not enough, legacy IT powers like [company]Microsoft[/company] and [company]IBM[/company], which are already in virtually every enterprise account are pitching their own respective clouds aggressively.

In a cloud melee like that, Verizon, which launched this new Verizon Cloud last fall, had better make good on no-more-upgrade-shutdowns because people will be watching.

Verizon Cloud shut down passes 30-hour mark

As of about 6 a.m. EST Sunday morning, Verizon’s Cloud was on hour 30 of what could be a 48-hour planned shutdown to rollout a major service upgrade. The provider had hoped — and its PR team had expressed — that most similar closures last 24 hours or less.

One customer Kenneth White, a security architect who designs systems for non-profit and federal clients, followed through on plans to live tweet the outage.

[company]Verizon[/company] has said that the major upgrade being implemented now will make future upgrades less painful to customers. Updates on the work should post on this Verizon customer forum.

While maintenance and service upgrades are to be expected, and users appreciate getting advanced warning on such events, the prevailing sentiment is that a full day — let alone two days — of no service is beyond the pale.

There are several ways cloud providers can offer high-availability service including hot patching and live migration. It was unclear if these technologies are part of Verizon’s plan going forward.

Data center construction fire

In other cloud news last week, there was a fire Friday morning at the construction site of a new [company]Amazon[/company] Web Services data center under construction in Reston, Virginia. According to CNNMoney and other reports, Loudoun County firefighters were sent in at 10:12 a.m. and had the fire, reportedly started by welding equipment on the roof, under control in about an hour.

Mirantis makes its OpenStack plug-in friendly

OpenStack upstart [company]Mirantis[/company] last week rolled out its OpenStack based on the new Juno OpenStack release. Mirantis has also worked with Tesoro to certify its Trove database-as-a-service in Mirantis OpenStack 6.0. ONe of the key goals is to make it easy for third parties to develop plug-ins that will integrate easily with this release.

In October, Mirantis scored $100 million in Series B funding from Insight Venture Partners with August Capital, [company]Intel[/company], WestSummit Capital [company]Ericsson[/company] and [company]SAP[/company], bringing total funding to about $120 million.

Structure Show!

Steve Herrod, who helped [company]VMware[/company] become, well, VMware, is now at General Catalyst Partners where he’s looking for startups in proactive security technology, and other areas that will be key in next-gen enterprise infrastructure.

Given the current availability of capable open source software and cheap (near-free) cloud infrastructure, there’s never been a better time to be a startup, Herrod notes. But listen to the whole show. He knows whereof he speaks.

[soundcloud url=”” params=”color=ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false” width=”100%” height=”166″ iframe=”true” /]


Hosts: Barb Darrow and Derrick Harris.

Download This Episode

Subscribe in iTunes

The Structure Show RSS Feed


Structure 2012: Steve Herrod - CTO and SVP of R&D, VMware

Structure 2012: Steve Herrod – CTO and SVP of R&D, VMware

Cloud upgrades no big deal if deployment is done right

When Verizon said it might shutter its new Verizon Cloud for 48 hours this weekend for a major upgrade, users were shocked. Some promised to live-tweet the event which starts Saturday at 1 p.m. EST. Verizon’s subsequent qualifications — that 48 hours is the worst-case scenario, that customers received ample notice, that this work would make future upgrades less traumatic — eased the situation a bit.

Cloud vendors take these occasions — whether the issue is a planned-in-advance upgrade or more hastily put together patch  — to educate users about best deployment practices. They tick off a range of tips — that workloads should be deployed across availability zones and regions, for example. And third-party tools and management vendors likewise parlay these events to promote the use of their own products.

To mitigate downtime, plan, plan, plan

Cedexis, which offers a cross-cloud load balancing service, sent out an email Thursday with its own list of best practices. From the email:

Whatever your opinion on Verizon Cloud and the way they are rolling this upgrade out, what is true is that there is no reason for this type of system maintenance to impact the correctly configured enterprise. It is time for architects and designers to realize that cloud outages are a fact of life — just like Data Center outages.

In cloud, as in past IT deployment models,  disaster recovery “relies on the use of geographically diverse deployment of applications. Why would anyone adopting Cloud think single-homing an application is a reasonable practice?,” according to Cedexis.

Deploy across zones, regions and vendors

Cedexis goes further than the multi-availability zones mantra to say enterprises should use multiple cloud vendors as well. That’s something you probably won’t hear from [company]Amazon[/company] or [company]Microsoft[/company].

Vendor specific outages — whether the vendor is Amazon Web Services, Microsoft, Rackspace, IBM or Verizon — are “more common than ‘acts of God,'” according to Cedexis, so selecting multiple vendors with (of course) global load balancing protects the user from these events. [company]RightScale[/company] is another vendor who provides these cross-cloud management and deployment capabilities.

So go easy on [company]Verizon[/company], said Carl Brooks, an analyst with 451 Research. In a comment on Gigaom’s earlier story, Brooks wrote that the hubbub around Verizon’s planned upgrade was way overblown.

[company]Verizon[/company] is “telling a small subset of its customers to get the heck  out while they do some planned maintenance, which is miles better than the usual practices from cloud providers,” Brooks wrote.

“I’d bet real money they could do most of this upgrade without kicking users off … [company]Microsoft[/company] has planned monthly outages to patch and tells you if you’re in the target zone ahead of time. This isn’t so different, just a bit more ham handed. You know what AWS says to its [most of its] users before it upgrades and reboots a bunch of its platform? Nothing.” Brooks later amended his statement via email to say that most customers never notice routine AWS maintenance.

Update: A Verizon customer pointed out that at least one Verizon Cloud (compute and storage) shut down in September did, in deed, last two days. It kicked off at 11 a.m. CT on September 11 and services were not back on till 11:38 p.m. two days later, according to this Verizon support post.

Note: This story was updated at 1:43 p.m. PST with information on Verizon’s September 11 shut down and againa t 7:19 p.m. PST January 9 with Carl Brooks’ amended statement.