Microsoft Azure is experiencing an outage which is affecting our websites.

Incident Details:

Update 2:50 pm (ETD) – All services have been restored.  We are continuing to work through any delayed text messages and emails sent by our services while we were down.  We want to assure you that there was no data loss experienced during this outage.

We are working with Microsoft Support to further understand the issues from their side and have also gained valuable insights into how we can improve the time to bring our backup servers online during these kinds of outages and will be working over the next few days to implement that learning.

We really hate it when we let our customers down for any reason so we’re grateful for your patience while we worked through this event.

—-

Update 12:20 PM (ETD) – We are seeing signs of recovery and systems are slowly coming back online.  We are cheering on Microsoft for a fast recovery but are continuing our own methodical efforts to bring systems online from our backups in the event we get there first.

---

Update 11:24 AM (EDT) – We received the following update from Microsoft:
After further investigation, we have found some unhealthy compute nodes impacting our service on East US region as well. We continue directing all our efforts towards bringing the affected nodes back to a healthy state. The next update will be provided in 60 minutes, or as events warrant.

We are using our own engineers to begin bringing services online one at a time using backup facilities.  This is more time consuming than we would like and are taking extensive notes on how to speed this process in the future if required.

Update 10:17 AM (EDT)

Microsoft is actively working to apply a mitigation to this.   Our engineers are monitoring closely and also preparing now to bring select services online from our backup systems should this issue persist.

We understand how important our services are you to you, especially as many of you prepare for services this weekend and are actively working as fast as we can to restore services.  We really appreciate your understanding and patience.

—

Starting at 07:33 UTC on 16 Sep 2023, you have been identified as a customer using SQL Database in East US who may experience issues accessing services. New connections to databases in this region may result in an error or timeout. Existing connections remain available to accept new requests, however if those connections are terminated then re-established, they may fail.