Verizon gets 'black eye' in long cloud shutdown
For an industry that generally measures downtime in minutes or several hours, this was a long shutdown. But Verizon says the outage means it won't have to interrupt users' service again for future upgrades.
Even so, one user found to be too little too late.
"When a company claims enterprise-grade global services, they are asking customers to entrust their most important data & systems," said Kenneth White, principal security architect with CBX Group, a Washington, D.C.-based company that designs production cloud platforms for non-profit and federal clients. "A 40-plus hour world-wide outage of all services signals that a provider either simply doesn't understand the space, or worse, they are just trying to close sales, at the expense of others' businesses. Any CIO worth their salt would do well to look at actual performance, and not simply marketing claims of 'next time we'll do better.'"
White said he had been beta testing Verizon Cloud since early last year but after this past weekend's shut own, he has "ruled out" any production deployments.
Verizon, which said the downtime affected about 10% of users, shut its new enterprise cloud service down at 1 a.m. ET on Saturday, Jan. 10. (It had warned users to shut down their virtual machines an hour ahead of that time.)
Verizon reported in a blog post that it completed its maintenance at 5 p.m. Sunday.
The company had warned users ahead of time that the shutdown was coming and could last up to 48 hours.
"Any downtime on a cloud service isn't good, of course," said Dan Olds, an analyst with The Gabriel Consulting Group. "And an outage of more than 40 hours, even if it's announced ahead of time, is very bad.... The way Verizon has handled this has given them a huge black eye in the market."
Verizon, though, noted that this should be the last time it has to take the service down for maintenance.
"The seamless upgrade functionality allows Verizon to conduct major system upgrades without interrupting service or limiting infrastructure capacity," the company wrote. "Traditionally, updates have been made via rolling maintenance and other methods. Many cloud vendors require customers to set up virtual machines in multiple zones or upgrade domains, which can increase the cost and complexity. Additionally, those customers must reboot their virtual machines after maintenance has occurred.
"Verizon eliminates these requirements, since virtually all maintenance and upgrades to Verizon Cloud will now happen in the background with no impact to customers," the company said.
That, noted Olds, is important.
"What's getting lost in all of this is that the outage was used to give Verizon a chance to make a major infrastructure upgrade that is supposed to eliminate all maintenance downtime from now on," he said. "But Verizon could have done this in a way that would have sidestepped this whole [shutdown]."
Trying to do the upgrade without a lengthy shutdown might have taken Verizon weeks and it probably would have cost it a lot more money.
"...My view is that this has set Verizon back considerably in terms of entering a highly competitive market," said White. "So, yes, my testing is done for now. I would consider reassessing in perhaps 18 months, but they lost a lot of credibility the past few months."