Date: Fri, 15 Jun 2001 01:28:18 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Matt Wilbur <matt@efs.org> Cc: freebsd-advocacy@FreeBSD.ORG Subject: Re: Cheezy "SysAdmin" magazine article Message-ID: <3B29C722.5979824B@mindspring.com> References: <Pine.BSF.4.21.0106140956430.63728-100000@sargon.photon.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Matt Wilbur wrote: > > Hello, > > Sort of funny that this was the last issue I got before > I let my subscription lapse.. Any comments on this article? > > http://www.samag.com/articles/2001/0107/0107a/0107a.htm > > They test Linux vs. FreeBSD vs. Solaris vs. Windows 2000, > testing their ability to perform with large mail queues, > using different means of handling them.. And they claim > that overall, FreeBSD comes last. All OSs were run "out > of the box" save some minor tweaks to allow lots of > processes and filehandles and such. My first impression > is well, duh, Linux ext2fs with async mounts (out of the > box) probably IS faster than ffs without softupdates > (also the case out of the box)... Any other comments? The article is meaningless. Too bad they titled it "Which OS is Fastest for High- Performance Network Applications?" instead of "Which OS is Fastest for MailEngine?". The only implied caveat is the statement "Our customers frequently ask us which operating system is best for running our software" in paragraph 3 of the "Background" section. This should have been in bold type in the first paragraph. -- It makes a number of very large blunders, which are really inexcusable, given that it tries to represent itself as a fair and unbiased comparison. These blunders are in the tuning of FreeBSD, the best architecture for FreeBSD applications (one shich they did not even try to consider), in their choice of which items they could micro-benchmark would really be indicative of real-world performance, and, finally, in their experimental methodology. Here is a short list, off the top of my head: 1) The mail server they were using doesn't come with any of these systems "out of the box". 2) Threaded processes are vastly inferior to finite state automatons, when it comes to CPU utilization on single CPU systems, and even on multiple CPU systems, if there is async I/O that can be scheduled on multiple CPUs. 3) FreeBSD turns of write caching on IDE drives, by default, in FreeBSD 4.3 and above; you can set it to be like Linux, Solaris, and Windows, if you don't care about your data. On FreeBSD 4.2 and below, Soft Updates are not enabled by default. Either way, without tuning, you lose. 4) IDE drives do not support tagged command queueing, except IBM DTLA drives, which are known to fail due to overheating and due to their electronics being too slow for their radial track density for interior tracks. 5) Real servers with storage and I/O requirements use SCSI drives so they can benefit from tagged command queues, which allow I/O to be interleaved instead of serialized. 6) No well designed mail server keeps all queue files in the same directory, unless it has been designed to run on a particular system where that is not a problem; this is a design portability issue, not a performance issue. 7) Sendmail can handle 400,000 8k emails in a 24 hour period on a < 500MHz box, if it is properly set up and queue dispersal is optimally configured (e.g. with the patches from ftp://ftp.whistle.com/ ). 8) "The most efficient asynchornous architecture" for an application is OS-dependent. 9) There are more than 3 ways to skin a cat, or to architect a task. 10) Sending an RSET instead of data measures only the connection setup and teardown speeds, and does not measure real throughput, and is not representative of real world behaviour, in which mail messages, when sent, contain data, and not just trivial addressing information. 11) Mail servers which support the ESMTP PIPELINE ability have significantly higher throughput, even when just doing addressing. 12) You can not "tweak" FreeBSD's network connection limits at run time; socket structurse, inpcb's, and tcpcb's are allocated via a zone allocator, prior to the system actually being started. This zone can not be resized. Without the patch I posted to make maxfiles boot-time tunable, FreeBSD can not increase the number of sockets and files that it can simultaneously handle, without a kernel recompile. Thus the "tweaking" used was useless. 13) For each connection, there is a tcptmpl structure, which is used for keepalives. This structure will consume one mbuf per connection; in addition, the average TCP window size will be 16k; so you will need 16k/2k (8) mbufs for custer pointers plus 16k/256 (64) mbufs for the window data, plus one mbuf per connection, pplus one mbuf per connection, if you are setting options. This means that you will potentially need 74 mbufs per connection you intend to support; without patching, this also is not tunable except at compile time, and the value was not tuned. 14) The "average througput per network architecture" is extremely misleading, both because of the limited and inefficient architectures used in the test, and in using an average to "determine" which was "the best architecture for use on all OSs". Per OS numbers would have been much more meaningful, since the final architecture was chosen based on the average, and not based on what was best for the OS being tested. 15) Creation and deletion of large numbers of files in a single directory is rather meaningless; witness the moronic 'postmark' "benchmark", and the resulting discussion on several FreeBSD lists. The "benchmark" was clearly designed to put forth a political agenda (in the context of the list discussion, this agenda was the pro-ReiserFS position, despite Reiser's use of several USL/Novell patents on Delayed Ordered Writes, without the permission of the patent holders, which render a pro-ResierFS argument as meaningless as arguing about the number of angels which can dance on the head of a pin). 16) When you have an answer you want, all the evidence you gather will tend to support the conclusion you have already drawn, unless you are very, very careful to eliminate your assumed biases in your experimental design. This is very hard to do, even when trained scientists are involved. Feel free to pass this on to the magazine editors, the author, or both; at the very least, it may teach them a little bit about how to actually tune FreeBSD to get good numbers out of it, instead of frobbing meaningless knobs and merely _believing_ they are tuning the system for higher performance. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-advocacy" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B29C722.5979824B>