CrowdStrike'd and BitLocker'd
With yesterday's CrowdStrike outage. I'm sure a number of you are probably wondering "what does BitLocker have to do with any of this?" Well, it all has to do with an added layer of security many organizations use to keep data from being stolen if a computers ends up in the hands of an "unauthorized user."
To start, let me briefly explain what the CrowdStrike driver issue did and what the fix for it is.
After the update was automatically installed to computers running the CrowdStrike Falcon Sensor, a faulty driver file caused the Windows kernel on those computers to crash and display a Blue Screen of Death. How Windows typically handles a crash like this is to create a crash log file, then perform a reboot. Since this driver would launch shortly after Windows finished booting, it would cause the operating system to crash and reboot again. When two crashes have occurred in sequence, Windows will automatically boot into Recovery Mode. Hence why we say several pictures of the Recovery Mode screen across social media yesterday.
Unfortunately, this update was automatically pushed out to around 8.5 million computers across several organizations, causing widespread chaos within the matter of a few hours. And the fix for this issue had to be performed by physically accessing each computer, which required those of us working in I.T. to have to run around several facilities, locate each affected computer, and apply the fix one by one.
The short and simple of the fix is either one of two options. You can either use the Recovery Mode that Windows was already booted into to navigate to Start-Up Setting, and launch Safe Mode. Once Windows boots into Safe Mode, the technician can navigate to C:\Windows\System32\Drivers\CrowdStrike, and delete the file C00000291-*.sys. After that the computer can be rebooted as normal, and the crashing will stop.
Or the technician can open System Restore from Recovery Mode. And, assuming there is a recovery point, restore the computer back to a good known working state.
Now, this may all seem simple enough. So why were so many organizations having trouble running this fix? That all has to do with BitLocker. As an added security measure, many organizations use BitLocker on their computers to perform a full hard disk encryption. This is done so that if a computer ends up being lost or stolen, and ends up in the hands of an "unauthorized user" they will not be able to access any of the data stored on the computer without a password to log into the computer, or the computer's BitLocker recovery key.
This presented a problem when trying to restore all of these affected computers, because when either trying to launch Safe Mode or System Restore, the user would be prompted for the BitLocker recovery key.
In my organization's case, we found we could save time by providing our end user the steps to perform one of the two fixes on their own to save time. But the problem we still continue to run into is the need for these BitLocker recovery keys. In my case, I've been fielding several calls where I've had end users walking from one computer to another while I provide them with the key as they walk through reverting back to a previous restore point.
Again, a pretty long story, but hopefully that can provide some context as to what's been happening over the past 48 hours.