“Good show. Its effectiveness is proved. A strong case for Open Source anti-virus software!”
--Praveen Shrikhande, SVP, Asianet Dataline on ClamAV-based anti-virus solution from Linuxense

We, at Linuxense, have been experimenting with Tux for a while, and were looking for an opportunity to test it out in a real high-traffic situation. In Kerala, we could not think of a Web traffic higher than the online look-up of SSLC/HSE results. We talked to Asianet, who runs the portal Keralaonline and is an esteemed client of ours, on setting up a Web site for this purpose. They were happy and kind enough to provide the bandwidth. The following events happened in Trivandrum.

Tux is the fastest Web server in the world today for serving static files. The Board of Public Examinations provides exam results in a proprietary database format. Since Tux is good at serving static pages, we decided to pre-cook pages for every register number. This technique makes sense in a couple of ways:

1. For a given register number, the data never changes. That means a result page generated once is valid forever. If that's the case, we figured, why can't we build up the pages before opening up the Web site to the public, rather than building them as and when visitors make requests.
2. Tux is good at serving static files. With dynamic URLs, it is just as good as any other Web server. So, to get the best out of Tux, make it serve up static files.

Then, the challenge was to pre-cook ALL the pages; we had to race against time. As soon as the results come out, other Web sites would publish them on their sites using conventional techniques which won't take much time. So, we had to create a system which can build around 600,000 pages in 45 minutes or less (which, we concluded, was an acceptable time). We checked with Pariksha Bhavan and came to know that the results would be delivered either in Microsoft Access format or as Microsoft SQL server dump :(.

We started designing the program to generate all the 600,000 pages in 45 minutes. We decided to go with Java mainly because it is very versatile when it comes to database connectivity. Moreover, to make up I/O delays, we made the design a multi-threaded one (when one thread is fetching data, another thread will be making the page). Another major performance bottleneck we found was the I/O delay in writing the generated files to the disk. So our team came up with an idea of creating a zip file of all the generated pages and writing it on the disk once and for all. That saved a lot of I/O time. And that approach helped us in yet another way: there was no need to compress the generated files to copy them over to the Web server.

Of the file systems we tried, ReiserFS came out to be the best. It outscores other file systems in access time and handling of large number of small files (thanks to fast balanced trees). The number of files in a folder can be another performance-tuning factor. By hit and trial, we concluded on 10,000 files per folder.

On the morning of 15 May 2003, we got a call from our friends at Asianet saying that the results data had reached them. We rushed to their office. We set up two laptops (one was a Thinkpad (PIII 800MHz, 256MB, RH 7.2) and the other was a DELL Latitude (1.2GHz, 512MB, RH 9.0)) with the abovementioned program. We did a trial run of around 100 records. It came out very well; all of us were excited. We started generating the pages in batches of 50,000. Soon after a batch was completed, we transferred the zip file over to the Web server and unzipped it. This process continued till we were struck with the first hit: the console of our Web server threw an error saying ``Disk full''.

Soon we realized that there was a problem in our assumption regarding the size of the generated pages (which was designed by the Keralaonline.com team to match with other pages on their portal). We designed our setup for a maximum of 5 KB/page, but it was actually 25 KB/page! So we had no other option but to create a partition of the required size. By that time, all the other Web sites had gone live and were getting far more hits than they could actually handle.

Our page generation program completed all the 583,444 pages in a little more than 45 minutes and these were kept ready in zip format in two laptops. We rushed to the data center in the mid-day, negotiating the heavy road traffic at East Fort, all the while planning strategies to prepare a partition large enough to hold all the files.

There was a partition left on the disk with a size of 10GB. We decided to combine the current partition with this one for a size of 23GB. And it was done in a few minutes' time. We unzipped all the files and the site was ready to go by 1:30 PM (delayed by 3 hours!).

We then opened the site to the public (this site was linked from the Keralaonline Web site and our site was serving a splash page till then). We saw a flood of requests coming in and Tux took over the show. We didn't see the load average rising above 0.03 when we were standing by. But, by that time, the initial result traffic was subsiding and we were worried that we had missed that heavy traffic which we were longing to handle with Tux.

The next day (16 May 2003), the HSE results were coming out. We did the same thing for the HSE database too and we opened up the site at 10:03 in the morning. A list of Web sites publishing results had already been published in all the local dailies, and all these sites were getting huge numbers of hits. There were 18 of them. Later, we found that all of them, except 3, crashed, unable to handle the hits. Of the 3 sites serving, Tux was the fastest with an average connect time of 0.007 second during the peak hour and response was 2.7 times faster than the next best Web site¹.

By noon, we realized that the traffic was real high and we were experiencing what we missed yesterday! We had a look at the server log; it was logging an amazing traffic of 20 hits per second and it seemed increasing as minutes passed by. At the 50th second of 12:08 PM, Tux served 35 hits! (See the graph below).

Here are the details of the hardware we used to host the Web site:

• Compaq Desktop PC
• Pentium IV running @ 1.9 GHz
• 256 MB of RAM
• 7200 RPM IDE HDD

As you can see, it is no better than an office PC. During the peak period (see the graph), it transfered 750 MB of data and withstood nearly 800,000 hits without any problem. Detailed statistics below:

Total result views	30,310
Total unique result views	28,247
Total bytes transfered during the period shown above	~750 MB
Peak traffic	35 hits / second @ 12:08:50 PM 16 May 2003
Total page hits	1,61,269

The performance can be boosted further by using a faster HDD or the best alternative: a RAM disk. But, we found that this setup is sufficient to serve this surge traffic. With a sufficient number of network cards, it is possible to achieve direct ScatterGather DMA and hardware-based TCP/IP checksum from the Linux file data cache directly to the network; if one needs to squeeze hard to get the last bit of performance.

We thank Praveen Shrikhande, SVP, Asianet DataLine, who always appreciates R&D efforts, for providing bandwidth. Also, we extend our thanks to Anilkumar, Technical Officer, ADL, who coordinated all activities and the Keralaonline Web team members, Biju and Nitheesh, for their technical help, and Rajesh for his kind cooperation. We thank KG Kumar for his editorial help.

--
¹Performance measured using Apache Bench.
--
17 May 2003