Following on from a tumultuous week dealing with large scale DDoS attacks, many of our web hosting customers have been asked excellent questions about the attack, and even how they could help us. So, I’m going to give a brief overview of the DDoS attacks we suffered, and hopefully answer a few questions along the way.
Denial of Service attacks can be mounted by individuals, organisations or governments and can take many forms. The two most common forms are flood attacks and cryptographic attacks.
The well known CryptoLocker ransomware is one example of a cryptographic DoS. In this case, the attacker infects your computer and encrypts your data, demanding a ransom to unlock it again. It’s a bit like going to the hardware store, buying a huge padlock, and then slapping it on the gates to your local park, stopping people from enjoying it (denying them service).
Anatomy of a DDoS attackThe type of attack Krystal suffered was a DoS flood attack. This is where an attacker tries to flood the victim’s network with meaningless data in the hope that no bone fide data can get through. This is like walking up to shop door and standing in the way so nobody else can get in. Of course, the people who conduct DoS attacks online aren’t quite so brave.
The team at Krystal (like most ISPs) have been dealing with individual DoS flood attacks since the day we opened. Simply put, our “shop doors” are wide enough to prevent a single malicious individual from being a threat. However, if the attacker turns up with a few thousand zombies under his command, then the doorway can still be blocked. This is the goal of a Distributed DoS (DDoS) attack.
There’s no way to know when a DDoS attack is going to happen, or how long it will last. The only certainty is that it will happen at some point. As we’ve grown as a company we’ve steadily increased our level of DDoS protection, but like sailing the high seas, a freak wave can still come along once in a while that overwhelms the ship.
This week we saw an attack which was both larger and longer than we had previously endured, disrupting the datacentre and our upstream carriers.
What’s a Zombie?
Zombies are essentially any computer or computing device which has been compromised by malware/virus software – usually without the knowledge of the owner – that accept commands from a remote command and control centre. Let’s break that down a little, because it’s a mouthful.
Imagine your computer gets infected with malicious software, which you don’t know about. This software sits there, quietly accessing the internet checking for instructions from a Command and Control (C&C) centre, run by criminals anywhere in the world. Even the company hosting the command and control centre may also not be aware of what’s going on.
Zombie ArmyThe C&C centre may even be a twitter feed or a blog entry somewhere. All the criminals have to do is tweet or blog “send some love to yourwebsite.com” and the message is read by potentially millions of infected computers and they all start to attack the victim. A quarter of a million home computers sending data at just 200 kbps (kilobits per second), a fraction of many home broadband connections, would result in a total attack of 50 Gbps (fast enough to fill up a 1Tb hard drive in a little over 2 minutes). This is around the size of the attack which was levelled at us this week.
The zombies, collectively, are called a botnet – a network of bots (short for Robot). The owners of the zombie computers are usually blissfully unaware of what’s going on because the malware they are running causes almost no disruption – apart from the fact it is often bundled with things like CryptoLocker, so when the bad guys need more money, they could disable your computer too.
How to put a stop to DDoS attacks
Like any adversity, there are two ways to tackle it – the first being to tackle the cause and the second being to tackle the effect. The cost of DDoS attacks is astronomical, particularly when compared to the trivial cost of mounting such an attack – $40’000/hr damage compared to $40/hr to hire a botnet makes it a no-brainer for anyone wanting to damage a company online.
Tackling the Cause
It would be easy to start questioning the childhood experiences of botnet controllers, but without getting into the psychological help they all clearly require, DDoS attacks depend on one thing alone – vulnerable computers that can be infected with botnet software. Unless you maintain your computer with regular security updates for your operating system, applications, websites, web browsers, and anti-virus software AND you adhere to good online security practices then you could unwittingly become part of the problem.
This is why we are always banging on about how important it is to keep your website software up to date. Website infection is an essential part of the botnet ecosystem. An infected website carried the malware that get downloaded by unsuspecting visitors to your infected webpages. That malware then becomes part of the botnet. Sometimes, even the infected code installed on your website can act as a botnet zombie itself. We have seen this happen on our own network (although it’s easily detected and removed) – a particularly good example of website born botnet activity happened to many WordPress websites last year.
While most botnet zombies are in developing countries, still a significant proportion of compromised zombies are based in the USA and Europe, including the UK – All it takes is lax security. Sometimes, law enforcement catches up with the masterminds behind the botnets, but not often enough for most people, so the best way to help is to keep your stuff updated!
Tackling the Effect
Once a DDoS attack is mounted against you, there is little other option than to find a way to mitigate the effects. This means deploying network connections to the outside world that are large enough to swallow up any attack, and still be able to filter out the good stuff so your customers can still enjoy their services. This is one of the few options open to hosting companies, and it’s exactly what we have done.
Since we implemented our changes we have absorbed 4 more attacks – and the lights stayed on for all of our customers – and we got some sleep.
What we learned
Running a hosting business is a fine balance between profitability, support, performance, availability, and security. We are genuinely most concerned with the latter four. Within each of these areas, there are a myriad options to choose from in order to improve the aggregate. We always believed we had sufficiently catered for DDoS mitigation – but clearly, we have started to attract more attention as we’ve grown. Needless to say, it has been a painful reminder of the need for constant improvement, and that you can never “arrive” at the perfect solution – because 10 minutes later, everything can (and regularly does) change.
I’d love to tell you more about what we’re up to internally this week, but the bad guys are almost certainly watching!