Mulling over DDoS

The Wikileaks fracas is escalating, with hacktivists on both sides of the divide working to take down sites that are seen to be on the wrong side of the issue. Personally, I do not have any political opinion on either the Wikileaks issue or the attacks. I leave that to less technical authors to write on. Personally, I just like mulling over the whole problem of DDoS attacks.

Any reader of this blog should be familiar with what a DDoS attack is. But to me, the problem of a DDoS attack is similar to that of an arms race – the side with the bigger gun or bullet wins. In the case of the initial attack, Wikileaks moved over to AWS and was able to hold off attacks for a while. This is because Amazon had bigger pipes and defenses than the attackers. However, the attacks have taken a turn and secondary sites such as credit cards and law-firms are being hit as well. It seems like certain parties are taking this opportunity to settle some beef.

But back to the whole problem with DDoS.

I’ve mulled over this with my former boss and some co-workers, for the purpose of a clandestine project that I was working on. The conclusion that I came to is that a DDoS is effective in its simplicity – the side with the bigger pipes win. The company that I used to work for, has one of the biggest pipes in Malaysia and we could take people down, single-handedly, if we wanted to.

However, there are strategies to mitigate a DDoS attack.

Bigger pipes
Wikileaks moving to Amazon was in effect, buying bigger pipes. This mitigated attacks for a while until they got booted off Amazon. If it is not possible to buy pipes from a single provider, it would make sense to buy pipes from multiple providers and distributing the site across the globe. This is what Wikileaks has now done – distribute its content over a list of hundreds of worldwide mirrors operated by volunteers.

Faster software
DDoS attacks exploit the fact that a computer is kept busy by servicing network packets (either at the application or networking layers). Obviously, the solution to this problem would be to have bigger machines – this means having faster processors, larger memories, lighter operating systems. Surprisingly, some people do not realise that the TCP/IP stack plays an integral role in these attacks and if the OS has a more efficient stack, it would be able to withstand more punishment. Not all web-servers are created equal and some are able to handle concurrency better.

IPv6
One method that is not often thought of, is moving infrastructure over to IPv6. Most of the internet still runs on IPv4 today. In fact, many network providers simply do not cater to IPv6 and simply drop the packets from their network. Therefore, moving a site over to IPv6-only could help mitigate DDoS attacks by limiting the available sources of attack. Compromised botnets would be less effective if most of their machines were still on IPv4 only.

Future clouds
What would have been interesting would be if the attackers had shifted to launching attacks from cloud service providers. In fact, this was one of the queries that I raised once, in a meeting with a local cloud provider – launching attacks from within the cloud provider itself. If the provider is not careful, attackers could buy network and computing resources in order to launch attacks within the cloud itself. This would be difficult to detect because most cloud providers put their metering on external traffic, not internal ones. Imagine the attackers buying EC2s to attack Wikileaks from the inside of Amazon infrastructure. The attack would only need to be bad enough to take down the Wikileaks machines, without compromising the rest of Amazon’s infrastructure. Therefore, it is important for cloud providers to think about isolating their internal machines from each other.

Of course, there are many more considerations that could fit this little blog, but these were just some thoughts that I felt like writing down.

Weird TLD in China

I was reading about a recent Gmail hack from China and they actually showed the IP used to access the account. Since I was fairly curious, I decided to take a look into the IP – 125.45.96.89 – and I was surprised with the result.

inetnum: 125.40.0.0 - 125.47.255.255
netname: UNICOM-HA
descr: China Unicom Henan province network
descr: China Unicom
country: CN
admin-c: CH1302-AP
tech-c: WW444-AP
mnt-by: APNIC-HM
mnt-lower: MAINT-CNCGROUP-HA
mnt-routes: MAINT-CNCGROUP-RR
status: ALLOCATED PORTABLE
remarks: -+-+-+-+-+-+-+-+-+-+-+-++-+-+-+-+-+-+-+-+-+-+-+-+-+-+
remarks: This object can only be updated by APNIC hostmasters.
remarks: To update this object, please contact APNIC
remarks: hostmasters and include your organisation's account
remarks: name in the subject line.
remarks: -+-+-+-+-+-+-+-+-+-+-+-++-+-+-+-+-+-+-+-+-+-+-+-+-+-+
changed: hm-changed@apnic.net 20051011
changed: hm-changed@apnic.net 20051020
changed: hm-changed@apnic.net 20090507
changed: hm-changed@apnic.net 20090508
source: APNIC

Nothing surprising here since the IP reports itself as being allocated to a Chinese ISP – China Unicom in Henan.

; <> DiG 9.7.0-P1 <> -x 125.45.96.89
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60982
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;89.96.45.125.in-addr.arpa. IN PTR

;; ANSWER SECTION:
89.96.45.125.in-addr.arpa. 85865 IN PTR hn.kd.ny.adsl.

;; Query time: 23 msec

Now, this totally caught my eye. Notice the PTR record shows that the name for that IP is hn.kd.ny.adsl – an uncommon TLD. So, I checked Wikipedia for a list of available TLDs and fair enough, the ADSL TLD does not seem to exist. If I were to try to ping hn.kd.ny.adsl, the address would not even resolve through the normal DNS system.

ping: unknown host hn.kd.ny.adsl

Now, this indicates to me that China is running its own root-servers, which does not surprise me one bit as it uses it to implement the Great Firewall of China. Since it does this, it is free to implement its own list of TLDs that nobody else uses in the rest of the world. This is all fine and dandy until ICANN decides to approve the use of an ADSL TLD in the future.

With the recent WikiLeaks fiasco, people are already talking about fragmenting the Internet. This is proof that the Internet is already fragmented – we just need to take it to the next level. Zero-One-Infinity, anyone?

Web Development

There are many work-flows for web-development and there exists many different deployment environments for web-applications. Therefore, I have found that it is important to have a suitable development environment where web-applications can be developed, validated and verified before being sent to production.

A typical web-developer might either test the application on their own development machines or they might have access to a development server for testing. In the first case, there might be subtle differences between their own machines and the production environment. Therefore, testing applications this way is not very useful. In the second case, there may not be sufficient isolation between multiple developers on the development server – be it database access or file system access. So, there is the possibility of destructive collaborations.

One solution that I think might be suitable is the use of virtualisation. VirtualBox is particularly suitable for this because it is able to access a shared folder on the host machine from within the guest virtual machine. So, code can be developed locally and tested locally on a VM running locally. The VM images themselves can be set up by the system admin to be a complete clone of the production environment. In this case, the test environment approximates the final deployment environment while there is enough isolation to avoid clobbering each others’ work.

The only catch would be the required additional computing resources. However, most modern computers are able to handle virtualisation very well. In fact, I began using virtualisation in 2004 and have no complaints with regards to web-application performance because these things are not simulating the stars in the universe. They are just doing simple jobs for the purpose of testing functionality.

So, I would recommend the use of VirtualBox for development purposes.

Server Upgrade

Not sure if anyone has noticed this – might or might not be a good thing – but I have upgraded the server hosting this blog. Externally, nothing much has changed but underneath, things have changed quite a bit. The reason for the upgrade was due to the last server, constantly choking under any slight increase in load. The new server is cheaper too.

The specs for the new server are not all impressive, but it is good enough for hosting a simple blog. There is 128MB of RAM and 256MB of swap. It runs 32-bit Debian Lenny as the OS and has MySQL with Lighttpd and PHP5. It runs smoothly with two threads of PHP managing all the connections. I have stress tested it with siege and it gracefully handles about 10 concurrent connections with a response time of under 3 seconds.

Maybe this is a good time for me to describe my methodology for tuning a web server.

Firstly, I run siege -g example.com to see if the site is working correctly. This is useful for debugging the entire web stack because it will also display the HTTP headers sent and received.

Next, I simulate a random series of connections by using siege -i -c 5 example.com to simulate a number of concurrent connections. While I am doing this, I am also running htop on the server to monitor the server load and resource consumption.

Finally, I hammer the server with increasing number of concurrent connections using siege -b -c 10 example.com while continuously monitoring the server load and the response times. I tweak the numbers until performance drops.

That gives me some statistics that I can use as a benchmark. Then, I tweak the server settings and repeat the siege to get new numbers. Rinse and repeat until the desired concurrency and response times are achieved.


Lifting the server siege... done.
Transactions: 502 hits
Availability: 100.00 %
Elapsed time: 151.08 secs
Data transferred: 6.40 MB
Response time: 1.50 secs
Transaction rate: 3.32 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 4.97
Successful transactions: 502
Failed transactions: 0
Longest transaction: 6.09
Shortest transaction: 0.76

Typical Server Setup

Since there are so many little tweaks that I do to my regular web-server setup, I felt that I should put down some of my thoughts here, for the purpose of documentation.

Operating System
I invariably go with Debian. If the VPS provider does not provide a Debian option, I will not buy a VPS from them. When choosing between Debian installs, I tend to choose 32-bit images because I do not believe in having a single large server with more than 4GB of RAM. A 32-bit installation allows me to do some wonderful optimisations. I can run a MySQL and web server with PHP5 in under 64MB of RAM on a 32-bit Debian install.

Security
Firstly, security. My firewall of choice is Shorewall because it has been around for ages and I have been using it for just as long. It has only gotten better with age. I am one of those who believes in host-based security. Every person should be vigilant and take care of themselves when going out – same with servers. I will just tweak the example configurations provided with the documentation pack.

Next, I will install Dropbear and remove OpenSSH server. The reason that I do this is largely due to resource consumption. Since the SSH server is rarely used except for administrative work, I just use a light-weight one instead, to conserve RAM. I will set it with the “-w” option to block remote root logins. Other than that, I leave things as they are.

Database
Since most Open Source web applications tend to use MySQL as their database, I need to configure this. Depending on the amount of RAM available, I will select the example configuration provided with the documentation pack. The amount of recommended RAM is stated in the example config files themselves. I will tweak it by adding the skip-networking, skip-bdb, skip-innodb flags to reduce RAM usage.

Web server
I prefer to use Lighttpd because it provides a lean web-server that is easy to configure. IMHO, Nginx is not as friendly to configure while Apache is too resource intensive. There are a number of configurations that I will do with this. I will set server.tag to emit some useful information – typically the name of the actual machine that is serving up the page. I will disable server.dir-listing. I will set the SSL certificates if required and set the algorithm to AES128-SHA to conserve CPU resources. I will also tweak the FastCGI numbers depending on the resources and expected performance of the server.

Web application
I tend to run PHP5 because that is what most Open Source stuff use. I will tweak it by turning on zlib.output_compression, reducing memory_limit and turning off expose_php for security and performance purposes. The actual values to tweak will depend on the resources available on the server.

Some points to note:

If using an opcode cache like XCache, it is important to reduce the number of FastCGI processes and increase the child threads because the cache is stored per-process. Otherwise, it is better to reduce the child threads to zero and increase the FastCGI processes to reduce resource consumption.

Miscellaneous
Other minor tweaks that I will install would be things like etckeeper that helps to keep track of maintenance issues. I may also install cron-apt to help with some update notifications. I also reduce swappiness to improve memory performance.