Microsoft Azure Outage Knocks Ubuntu VMs Offline After Buggy Update

Microsoft Azure customers’ virtual machines (VMs) running Ubuntu 18.04 have been taken offline by an ongoing outage caused by a faulty systemd update.

The outage started nine hours earlier, around 06:00 UTC, after the affected customers upgraded to systemd version 237-3ubuntu10.54 and their VMs started experiencing DNS errors, with no DNS resolver addresses available on impacted systems.

Microsoft says in an incident report published on the Azure status page that these DNS issues only affect VMs running Ubuntu 18.04 (Bionic Beaver).

Also Read: How can businesses protect their enterprise from Business Email Compromise (BEC) attacks?

“Starting at approximately 06:00 UTC on 30 Aug 2022, a number of customers running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version,” the company revealed.

According to the Azure status page, services currently impacted worldwide by this outage include Azure Kubernetes Service (AKS), Azure Monitor, Azure Sentinel, Azure Container Apps, and more.

Microsoft has confirmed downstream impact to other Azure services, with Azure Kubernetes Service (AKS) being the most affected across multiple regions. 

A potential fix for this systemd bug requiring a switch to a fallback DNS server and a restart is available on Canonical’s Launchpad software collaboration platform.

Microsoft provides an additional workaround for affected Azure customers that involves rebooting the impacted Ubuntu virtual machines.

Also Read: 6 ways to recognize potential phishing scam and what to do if you receive one

“An additional potential workaround customers can consider is to reboot impacted VM instances so that they receive a fresh DHCP lease and new DNS resolver(s),” the company said.

Redmond also implies that those who haven’t yet been affected should consider disabling security updates until the bug has been mitigated.

“If you are not experiencing impact on your Ubuntu 18.04 images, but you have unattended security updates enabled, we recommend you review this setting until the Ubuntu issue is mitigated,” Microsoft added.

The company’s Azure customers were hit by another global scale outage (tracking ID 0NC_-L9G on the status page) that took down Windows-based Azure VMs for roughly eight hours on October 13, 2021, due to resource migration issues.