Microsoft performance update caused Azure to stumble Tuesday

Microsoft detailed Wednesday just what exactly went on in its data centers that led to its Azure cloud experiencing a series of hiccups on Tuesday.

Microsoft attributed the problem to an Azure Storage performance update, which inadvertently led to to “an issue that resulted in storage blob front ends going into an infinite loop,” wrote Jason Zander, [company]Microsoft[/company] corporate vice president of Azure, in a blog post.

From Zander’s post:
[blockquote person=”Microsoft” attribution=”Microsoft”]Yesterday evening Pacific Standard Time, Azure storage services experienced a service interruption across the United States, Europe and parts of Asia, which impacted multiple cloud services in these regions. I want to first sincerely apologize for the disruption this has caused. We know our customers put their trust in us and we take that very seriously.[/blockquote]

This infinite loop problem meant that storage blobs — essentially, bundled up collections of binary data — were unable to handle additional traffic, so services built on them suffered.

Microsoft has since undone the update and restarted the storage front ends, although a few customers are still experiencing some problems.

Zander wrote that both engineering and support teams are helping those unlucky clients and Microsoft will issue a root cause analysis to explain in more detail what went wrong and how it plans to make improvements.