On July 19th, 2024, a CrowdStrike, a cybersecurity company, pushed a software update that caused major issues for Windows users around the world who use Crowdstrike cybersecurity products. The update triggered the “blue screen of death” on many computers, causing them to crash and enter a reboot loop.
*It is important to note that this is NOT a cyber threat or atttack, rather it is an update gone wrong.*
What Happened:
The issue was caused by a faulty “channel file” in CrowdStrike’s Falcon Sensor product, rather than a full software update.
The outage affected a wide range of organizations, including airports, businesses, and broadcasters like Sky News.
Microsoft acknowledged the problem and said it was taking “mitigation actions” to address the service issues.
How to Resolve It:
CrowdStrike provided a manual workaround for non-Bitlocker-encrypted devices:
Boot Windows into Safe Mode or WRE
Navigate to C:\Windows\System32\drivers\CrowdStrike
Locate and delete the file matching “C-00000291*.sys”
Boot the system normally
However, this manual fix is not scalable for large organizations with many affected systems. CrowdStrike is working to pull the faulty update and instruct older agents not to update until the issue is resolved.
For critical systems, the best option may be to restore from a backup or use the built-in Windows recovery features.
The CrowdStrike outage highlights the importance of comprehensive testing and quality assurance for software updates, especially for security products that are widely deployed. While CrowdStrike is working to resolve the issue, this incident serves as a reminder for organizations to have robust backup and recovery procedures in place to mitigate the impact of such incidents.
For IT and Security Teams, here are some things you can do:
Prevent automatic updates outside of critical patches/vulnerabilities
Test and validate encryption keys
Look into backup solutions for critical infrastructure
Conduct table top exercises with leadership with topics focused on bad updates
