Date: Wed, 5 Jan 2005 21:22:46 -0800 (PST) From: stheg olloydson <stheg_olloydson@yahoo.com> To: hubert@feyrer.de, g.mcgarry@ieee.org Cc: freebsd-performance@FreeBSD.org Subject: Re: Benchmark: NetBSD 2.0 beats FreeBSD 5.3 in server performance Message-ID: <20050106052246.47980.qmail@web53902.mail.yahoo.com>
next in thread | raw e-mail | index | archive | help
it was said: Gregory McGarry asked me to host and advertize this paper for him: Abstract: ``With the recent releases of NetBSD 2.0 and FreeBSD 5.3 operating system, many new and exciting features have been implemented. Both criticism and commendation on performance, reliability and scalability have been directed towards these releases. This paper presents a suite of benchmarks and results for comparing the performance of these operating systems. The benchmarks target core operating system functionality, server scalability and thread implementation. <snip> URL: http://www.feyrer.de/NetBSD/gmcgarry/ --------------- Hello, I read the "paper" with which you trolled a couple of FreeBSD lists. Here, quoting the "paper" is what stands out to me: Abstract: "These benchmarks are useful server-based criteria for demanding applications such as loaded webservers, databases, and voice-over-IP (VoIP) media relays. The results indicate that NetBSD has surpassed FreeBSD in performance on nearly every benchmark and is poised to grab the title of the best operating system for the server environment." System-call overhead "The results in Table 1 shows that NetBSD 2.0 marginally out-performs FreeBSD 5.3." Context-switch time "The results in Table 1 shows that NetBSD 2.0 marginally outperforms FreeBSD 5.3." Process lifecycle "The results in Table 1 indicate that NetBSD 2.0 marginally outperforms FreeBSD 5.3." Forking new processes "The FreeBSD kernel has an access time which scales linearly with the number of system processes. There are also many occasions when the access-time is very fast, resembling a constant access time." Memory-mapped files "The total of both benchmarks indicate that for a single mapping and subsequent access, FreeBSD shows a 38% performance improvement over NetBSD." Socket creation scalability "Neither NetBSD nor FreeBSD shows scalability problems." Binding addresses to sockets "This result indicates the FreeBSD scales better for binding addresses to sockets." Thread creation benchmark "For less than 250 threads, the time to create a thread is better in NetBSD than FreeBSD. For more than 250 threads, the thread creation time increases as the number of threads increases. Of particular concern, the relationship is not linear for the number of threads. Although one thousand threads is ample for most multi-threaded applications, the poor scalability may be a problem for some applications." Thread lifecycle, condition variables and mutexes "The results of these benchmarks for the basic POSIX thread primitives clearly shows that the NetBSD thread implementation contains many impressive optimizations." Thread context-switch time "The creation of the first thread incurs a significant penalty on NetBSD. However, the time to complete the game is significantly higher for FreeBSD over NetBSD. This is due to increased latency in the thread lifecycle and the much longer mutex acquire time for FreeBSD over NetBSD." Conclusions "Significant performance improvements are obviously visible in the thread implementation....Microbenchmarks are not always the best indicators to make judgments on the overall performance of one operating system over another...For many applications, the results presented in the paper may never affect performance. For others, the scalability of the operating system may simply not permit the application to run suitably." Clearly, the claims in the Abstract are not supported by the tests. The author even points out the inapplicability of the benchmarks that he ran. But even if they were, the author points out that only in the case of threading performance does NetBSD do appreciably better than FreeBSD. In all other tests, FreeBSD either does better, as well as, or only "marginally" worse than NetBSD. So what this should have been called is, "Meaningless Tests Show NetBSD 2.0 is Finally About as Good as FreeBSD 5.3." I want the 20 minutes that I spent reading this "paper" and responding to it back. If you are unwilling or unable to return the piece of my life you caused me to waste, I demand to personally apologise to everyone who reads not only the post, but the "paper", as well. stheg __________________________________ Do you Yahoo!? Yahoo! Mail - Easier than ever with enhanced search. Learn more. http://info.mail.yahoo.com/mail_250
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050106052246.47980.qmail>