Hacked - surprise on Windows servers
Finally, it has happen to me, I got hacked on two of my Windows servers. It escalated quickly... This morning, a customer called me, because he has problems while accessing one of my servers. Usually this happens, when the customer has no internet connection (a lot of G4 or G5 network users and connection problems are not rare...). But nonetheless, I want to make sure, there is no problem on my side, so I checked the server. Hm, it is not really responding. I connected via my hosters VNC console and ... the server was running, but why is there another user account than mine on the login sceen? The machine has only one account as far as I knew. Reboot... why the bluescreen? Booting with last known working configuration, it works. But the management console on Windows said after this, something with the IIS is wrong. I went to the webroot... and... "!!! YOUR FILES HAS BEEN ENCRYPTED !!.txt" Inside this file, the offer to get my server decrypted again with a bitcoin donation and the offer to prove the possibility by sending one file to test the decryption. Oh, f*ck. I'm hacked. But also: I don't negotiate with extortionists, I don't pay hackers, I'm not a victim. It was a question of time getting hacked On the one side, that I got hacked, was only a question of time, because the server has a big problem: The OS is Windows 2008 R2. Very old, not supported anyomore and of course no security updates. I had some security, but obviously not enough. Unfortunatelly, I could not migrate to a more advanced Windows version, because I got this server as an image file. There is very specific software on the machine running, build from someone I don't know. Ready installed software with no available installer or documentation, all is from a niche or self-made (not by me). So, even if I want, I cannot build a running service with these services on another installation. The only alternative would be to build new client AND new server software. I don't really want to do this. Maybe this incident is a warning to do so against my will. It was also strange from my point of view, that the 2008 R2 was a vhd drive (a virtual drive), which was bootable from the Windows boot manager. There is an underlying OS, Windows Server 2016, but it is not really running, when I boot from the vhd. As I booted the W2016.... this version was hacked/encrypted too! I couldn't imagine that someone could break out of a running vhd to the host system, which is not really active. Even if you boot, you must have access the virtual console (only accessable via VNC console) to boot into W2016. I'm cautious, so I have a second backup server with identical configuration in another network segment. Ok... encrypted too. Some kind of desperation arises in my chest. What to do? Important and not importent at the same time On the other side, this is not a server with living data, which is in a status of change. It was mostly a static server, so no backup was really necessary. Not very important data, no data, which could be lost. For my customers, the functionality was very important - if the server don't run, 5,000+ customers have useless software and hardware for about 1000 Euro each. So, I need a plan. Things to be done first The server is mostly unchangend since I got it. So, an image from 2018 will fit my first needs. I upladed it to my hosters server management tool (upload took 1 hr) and began to copy the image to the backup machine. That took some time (45 min) and the server booted fine. First of all, I killed all inbound roules in the Advanced Firewall. I let only port 80 and port 443 open, I disabled all other rules. That was too much, because the server don't get his IP addresses via DHCP, no internet connection at all. Some kind of trial and error to figure ot, which rules I have to enable again, finally it was working. I bought a failover IPv4 from my hoster several years ago, so I connected it to the domain for the service and added the IP address to the server. Because it was a DNS change, it took some time for worldwide annoncement of the new configuration. In the meantime, I made some configurations in the ISS, installed missing Windows update since the image from 2018. After the server was running correctly from the technical view, I tried to connect with the client software. Yeah, the client is running again. Next things to be done after been hacked From the final configuration, I saved a snapshot and exported it as image (1.5 hr). I download the image to another server for backup. Then I installed the image on the other machine (1 hr), little adjustment for the network interface and... running. So, I can use my failover IP only by pointing to the server I wamt to be active without another DNS change. Then, I avoided one bad failure I made before. Instead of hot standby by the backup system, I shut it down. No one can hack a server without switching it on. In case of hacked next on the running machine, I can take it down, starting the second machine, redirect the failover IP and I'm back in business in short term and can reinstall the first machine, configure and switch off. But here is the problem As I said, the server is mostly static data. With one exception: The clients have to authenticate itself while connecting to the server. The authentication is be done by a service on the Windows server and checked against a SQL CE database. So, If an user bought access, I have to put his credentials in this database. In this database, I don't have an actual backup, so this database has credentials until Dec 2018, but not newer. No mercy without backup... fortunately, the problem can be handled by manual work. A lot of work indeed. So I looked through the invoices from 2018 to 2023 and entered the missing credentials manually. I spend another 2.5 hrs on this task, but I want to reduce the amount of phone calls tomorrow significantly. Maybe I did not found all users, but if there are some left, it can be maximal 20 users and the users are not using on a daily basis, so the calls will be distributed well over time. After the database was updated, I made a backup of it (yeah! Finally) - at least, I have to copy the database to the cold standby system. Next step is a decision, getting hacked or not? Today I learned, that servers, which become insecure from passing time alone, are a well liked target for criminals - and get hacked. I cannot go away from the configuration I have with the software I have. My decision has to be: - act as nothing happend and repeat it with high possiblity in the next future - exchange all software, which means primarely a new client software, which can work with a new server OS and server software/services The decision is hard, because I want to give up this business, but don't want to hit the customers and keep the servers alive for some more years. But even if I can get some money for the new software (and fulfill some wishes from the customers), that is not enough money to pay the development. If I don't do this, I could lose the business early than expected... there were minutes today, in which I thought, I already lost it... but I made it again and I'm back in business again. Read the full article
