Prefork vs Worker

Apache2 comes with several threading/process models to handle requests. The traditional method is maintained as prefork while the newer method is worker. The main difference is that the former relies on forking processes while the latter method uses multiple threads. The result is that worker is faster than prefork under load.

I tested this by setting up two servers, one with prefork and the other with worker. I used Apache Bench to hit both systems. At low concurrency of around 200 connections, both of them worked equally as well because in both cases, the main bottleneck was CPU. After I increased the CPU from single to six-cores, the load was handled equally well until I upped the ante to 500 concurrent connections.

At this point, the worker model would still run fine while the prefork model would start to fail periodically. It is like it has trouble handling the sudden spike. Once it has been hit a few times, it will also be able to handle the load but ramping up is less graceful. This is pretty obvious once you think about it a bit and preforking processes will have a larger overhead than spawning threads.

The only advantage of using preforking is when handling non-thread safe code. However, since my Apache serves only as a reverse proxy front-end, it does not need to handle any non-thread safe scripts. These are all handled at the back-end by other servers. So, there is really no necessity for the prefork model on my setup.

Advertisements

SSL Key Size

I have been doing some Apache optimisation recently and I thought that I will note down various tweaks for documentation purposes only. These tweaks will be written down as I find them.

Using larger RSA keys slow things down.

In my benchmark system, I only get around 500-600 connections/second for a 4096-bit RSA key but get a 800-900 connections/second for a 1024-bit RSA key. Seeing that our server certificates are issued only annually, there is no reason to use a 4k RSA key. Although a 768-bit RSA key is breakable today, a 1024-bit one still stands for now and will probably do so for a couple more years.

So, stick to using 1024-bit RSA keys for SSL until it gets broken.

Network Disk Cloning

I had occasion to clone a hard-disk over the network today. I wanted to replicate a VM setup that I already had running in one machine, over onto another, which will save me time from setting it up again. So, I checked the internet about using dd with nc to clone over the network. Most places gave the following info:

Target:

# nc -l -p 9000 | dd of=/dev/sda2

Source:

# dd if=/dev/sda2 | nc serverip 9000

However, trying that set of commands would fail. So, checking the man pages for NetCat revealed that you cannot use both the -l and -p parameters together. So, the correct set of commands should be as follows:

Target:

# nc -l 9000 | dd of=/dev/sda2

Source:

# dd if=/dev/sda2 | nc serverip 9000

Which worked out just fine in the end. Thought I’d share it.

Broadcast Storm

Learning about networking is not very fun unless you actually get to see it in action, which is the opportunity that I gave my apprentice today. A few months ago, the corporate network at the place where I am doing public service and paying my dues, went down when a low-level tech accidentally connected the network in a loop.

When that happens, it becomes possible to send packets round and round in a loop. This is not so much a problem with proper network connections because the packets will be terminated at the destination. However, it is a problem for broadcast packets as they could possibly be broadcast in circles – ad infinitum.

First, I got two computers with two network ports each. Then, I bridged the ports together so that br0 on each computer bridged eth0 and eth1 together. This is the example configuration for Debian networking:

auto br0
iface br0 inet static
bridge_ports eth0 eth1
bridge_stp off
....

Next, I connected one network port from each PC together with a LAN cable. I asked my apprentice to observe the lights on the network ports. They blinked a few times when the cables were first attached due to hand-shaking of the network capabilities. Then, all activity stopped.

So, I initiated a broadcast packet by performing a ping to a random IP address. Because the PC does not know the destination machine, it will send an Address Resolution Protocol (ARP) broadcast packet asking for the destination machine for a reply. We could have just as easily done this with an arping command instead.

The lights on the network ports began to blink with activity and we managed to observe things. After my apprentice got a good look, I stopped the pinging and we confirmed that all activity stopped.

Then, we connected the other two network ports from each PC together with a second LAN cable. Now, we have got ourselves a network loop. Again the lights blinked a few times during hand-shaking and then all activity stopped. I initiated another broadcast and the lights began to blink again. So far so good.

Now, it was time for the broadcast storm. I stopped the pinging like before but this time around, the lights did not stop blinking. The broadcast packet was moving round the network in a loop. My apprentice got all excited that the network ports were blinking rapidly even when there was no application sending out any activity.

Then, I disconnected one port and the blinking stopped – magic!

Finally, I showed my apprentice how the network would work with STP enabled. I changed the settings above and set bridge_stp on instead, restarted the network stack and reconnected the network cable. With STP in place, there would be periodic network activity when the STP packets were sent out but these were easily distinguished from the crazy blinking broadcast storm.

I repeated the whole experiment again, but this time, the broadcast packets stopped after the pinging application was stopped – success!

I think that this would make networking subjects more interesting. This experiment was fairly simple to conduct and took nary five minutes. It would demonstrate networking concepts in a very visual and visceral manner. It does away with the whole idea of imagining how network packets moved through a network. Beautiful simplicity.

Polipo DNS Issues

I have been using polipo as my proxy server and recently, it has been developing some weird problems. For one thing, it sometimes refuses to connect to websites without any errors on the client side except for time-outs. I know that the sites are up because I can connect to them if I bypass polipo.

The reason that I use polipo is because of resource requirements – much less than say, squid. As an upstream web proxy, it works really well. I have gotten better results out of it than squid but I simply put that down to my lack of squid config-fu. Polipo is much easier to configure as there are less options to play. However, it is also a plain vanilla proxy and does not try to become the proxy for everything as squid does.

After some investigation, I got a clue from the polipo logs – that it was timing out as well with the following message:

Host ftp.osuosl.org lookup failed: Timeout (131072).

This confused me because it was obvious that the website was up and working. So, I dug around and found out that polipo uses its own DNS resolver by default and not the system resolver. The reason that it does this is in order to obtain the TTL information on the domain directly. However, the information still comes from the same DNS server either way.

Turns out that it was a networking problem. By default, polipo would return the AAAA record instead of the A one if both are present. It is designed to prefer IPv6 over IPv4. My home network is IPv6. This can be controlled with either of the following options:

dnsQueryIPv6 = no
dnsQueryIPv6 = reluctantly

Since I have not yet enabled IPv6 support through my gateway, I decided to just disable IPv6 for polipo entirely. I might turn it back on once I get IPv6 up at home. I really need to get down to activating my Hurricane gateway tunnel and writing a guide for it.

WordPress on PostgreSQL

PostgreSQL rocks!I thought that I should plug this WordPress plug-in. It is a PostgreSQL compatibility layer for WordPress. I have to say that it seems to work as advertised thus far – this blog runs on it.

I have been trying to find a useful compatibility layer for moving various web applications over to PostgreSQL for quite some time now. I experimented with SQLRelay and it seemed to work fine stand-alone but I had a lot of trouble trying to get it to work with FastCGI on Lighttpd. Setting the appropriate environment variables seemed to do nothing for it.

The reason for this switch is numerous, mainly due to standards compliance and stability. However, the recent takeover of Sun by Oracle did it for me. I do not see how MySQL is going to survive the exercise unscathed. I just hope that the compatibility layer works well enough to sustain this blog for a long time.