CrowdStrike, Microsoft continue to address global IT outage

The ripple effect of the global IT outage that hit Microsoft July 19 due to a software update from cybersecurity company CrowdStrike has spurred the company to release a new technique to accelerate impacted system remediation.

“We’re in the process of operationalizing an opt-in to this technique. Customers are encouraged to follow the tech alerts for latest updates as they happen and they will be notified when action is needed,” CrowdStrike said in a release. “We have published a video outlining the steps required to self-remediate impacted remote Windows laptops. We will continue to provide updates here as information becomes available and new fixes are deployed.”

Several airlines, ports, hospitals, banks and other companies using the Microsoft platform  were impacted Friday when a software update from CrowdStrike caused an IT outage, causing delays, cancellations and operational issues.

Not all airlines were directly affected by the outage. WestJet, for example, said in a statement that it was aware of the issue impacting Microsoft products and services and that its IT team was monitoring the situation closely.

“At this time, there has been no direct impact to WestJet’s IT systems or operations and any flight cancellations across our network are unrelated to the CrowdStrike outage,” WestJet said in a statement Friday. “We are expecting there may be some delays across our network as we operate to and from airports that have been affected. Additionally, guests travelling via WestJet’s network and connecting onwards with an alternative airline may be impacted.”

FedEx said it was working to lessen the impact of the Microsoft outage.

“The FedEx network is operating across the globe, and we are working diligently to lessen any potential service impacts resulting from a global IT outage experienced by a third-party software vendor,” the company said. “We appreciate our customers’ patience.”

CrowdStrike president and CEO George Kurtz said on social media that his company was working with partners to resolve the incident and had released a technical overview of the events.

“On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform,” the technical overview states. “This configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems. The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024 05:27 UTC. This issue is not the result of or related to a cyberattack.”

Systems running Linux or macOS do not use Channel File 291 and were not impacted by the outage.

“We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. This effort will be ongoing,” stated CrowdStrike. “We are committed to identifying any foundational or workflow improvements that we can make to strengthen our process. We will update our findings in the root cause analysis as the investigation progresses.”

David Weston, vice-president, enterprise and OS security for Microsoft said in a post that the company estimates about 8.5 million Windows devices – less than one per cent – were impacted by the outage.

Weston said several steps have been taken to fix the issue, including:

• Engaging with CrowdStrike to automate their work on developing a solution. CrowdStrike has recommended a workaround to address this issue and has also issued a public statement. Instructions to remedy the situation on Windows endpoints were posted on the Windows Message Center.
• Deploying hundreds of Microsoft engineers and experts to work directly with customers to restore services.
• Collaborating with other cloud providers and stakeholders, including Google Cloud Platform (GCP) and Amazon Web Services (AWS), to share awareness on the state of impact we are each seeing across the industry and inform ongoing conversations with CrowdStrike and customers.
• Quickly posting manual remediation documentation and scripts.
• Keeping customers informed of the latest status on the incident through the Azure Status Dashboard.