Network Issues: A Status Update
In the past few days, you may have noticed that a few of our sites have been going offline for short periods. There is good news and bad news around this. The good news is that it is nothing to do with us, or for the most part, our providers. The other good news is it is not down to a bug*, accidental breakage or otherwise. The bad news however is it is not something we can do anything about.
Put simply, the network which hosts the sites is experiencing higher than normal levels of traffic. This isn't just "We've been featured somewhere" traffic, this is "Um, can you not spam us?" traffic. Not at all the kind of traffic you want. Now, due to the volume of it, it is essentially blocking all of the genuine traffic, meaning you are not able to get through to the sites. Pretty crappy eh? Unfortunately, we are not able to do a thing about it. Our providers are however working to do what they can and are so far doing a great job at it.
In the meantime, have a look at this
Have you seen a girl with hair like this?**
Now, for the complicated version. Also frankly a little scary. I suggest grabbing a pillow or hiding behind a Dalek.
Information from our providers has shown the cause of the connectivity issues to be the result of a Denial Of Service (DoS) Attack. Specifically a form of NTP Amplification attack. The precise target of this is unknown, but what we can say is that it isn't one of our sites. Now, this is the part where we need to make some things clear. NO Data has been compromised, at all. Not a thing. This is just an attempt to stop people accessing sites on that network, nothing more.
Onto the gritty details. This form of attack relies on exploiting a vulnerability in some NTP (Network Time Protocol) Servers. This bug, or whatever you wish to call it, allows disreputable people to use a built in function to flood a target with requests. In normal operation this function is provided for administrators use when debugging or monitoring and simply provides a list of IPs of recent users. Nothing particularly sinister about that. However in the case of this attack, the identity of the user is being spoofed, meaning instead of the data being sent to the user who actually requested it, it is being sent to the target. That means for each call of the function the target is receiving a list of 600 IPs. Under normal circumstances that could be handled, however this is not normal circumstances. The function is instead being called hundreds, if not thousands of times, generating huge amounts of traffic. Try dealing with this data and you soon start struggling. It will either start clogging up your network, or keep your server too busy to deal with legitimate requests, or both. In the case of this attack, around 10 Gbps of traffic was being seen, which in simple terms is a hell of a lot. Not the smallest however, one of the larger attacks of this nature was recorded as involving 400Gbps. For more information, have a look at this from US-CERT and this, if you are so inclined. Beware of brain hurt.
Long story short, some crazy person/persons if causing these issues. Nothing we can do, but our providers are working on it. Hopefully they will be over soon and we can get on with things.
*OK, that is sort of a lie. It is. Just not one we have created.
**Yes, that was a film reference. You'd think it was on while this was written or something.