On July 19, 2024, a widespread Windows outage was triggered by a faulty update from CrowdStrike, which led to disruptions across various global IT systems. This report details the vulnerabilities involved, the impact, and the subsequent remediation efforts.
Incident Overview:
- Date of Incident: July 19, 2024
- Affected Systems: Windows systems running CrowdStrike Falcon sensor version 7.11 and above
- Cause: Faulty sensor configuration update from CrowdStrike
Vulnerabilities Involved:
Channel File 291:
- Description: This configuration file is part of CrowdStrike’s Falcon sensor, designed to manage named pipe executions on Windows systems. A logic error in the update led to system crashes and BSOD (Blue Screen of Death).
- Location:
C:\Windows\System32\drivers\CrowdStrike\
- Impact: Triggered operating system crashes on Windows devices.
Technical Details:
Configuration Update:
- Released on July 19, 2024, at 04:09 UTC.
- Intended to address newly observed malicious named pipes used in cyberattacks.
- The logic error in Channel File 291 resulted in a system crash upon execution.
- Remediation implemented on July 19, 2024, at 05:27 UTC.
Impact:
- Global Reach: Affected approximately 8.5 million Windows devices.
- Industries Impacted: Businesses, airlines, banks, and other critical infrastructure providers.
- Systems Affected: Systems that were online and downloaded the configuration update between 04:09 UTC and 05:27 UTC on July 19, 2024.
Remediation and Response:
1. CrowdStrike Actions:
- Issued a corrected configuration update.
- Released public statements and remediation instructions.
- Conducted a thorough root cause analysis to prevent future occurrences.
2. Microsoft Actions:
- Collaborated with CrowdStrike to automate and expedite remediation.
- Deployed engineers to assist customers directly.
- Provided technical guidance and support to bring disrupted systems back online.
- Maintained communication with cloud providers (AWS and GCP) to manage the broader impact.
- Published manual remediation documentation and scripts.