From owner-freebsd-questions@FreeBSD.ORG Thu Feb 15 00:32:30 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2573316A401 for ; Thu, 15 Feb 2007 00:32:30 +0000 (UTC) (envelope-from stevenb@frii.com) Received: from mail.frii.com (phobos01.frii.com [216.17.128.161]) by mx1.freebsd.org (Postfix) with ESMTP id E5E1413C461 for ; Thu, 15 Feb 2007 00:32:29 +0000 (UTC) (envelope-from stevenb@frii.com) Received: from [10.10.10.81] (fw01-e3.ftc.frii.net [216.17.230.109]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.frii.com (FRII) with ESMTP id 1BEF8AE7CA for ; Wed, 14 Feb 2007 17:10:00 -0700 (MST) Message-ID: <45D3A4D7.9000504@frii.com> Date: Wed, 14 Feb 2007 17:09:59 -0700 From: "Steven H. Baeighkley" User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: serious performance problems with 6.2 Release X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 00:32:30 -0000 Greetings, We are having some bizarre performance problems on a freshly installed 6.2 Release server. This is a supermicro superserver 6022c dual 2.0 Xeon with 2GB RAM. These CPUs do support hyperthreading. We have done significant testing with both hyperthreading turned on and off in the bios and in the OS, to no avail. The server is configured as a web server with apache 2.2.4 php 5.2.0 and ZendOptimizer. We are running proftpd 1.3.1rc1 and perl 5.8.8. We have another server running 4.11 with the same exact hardware and software versions. We have updated to the newest bios that Supermicro provides. The trouble is that the 6.2 box performs significantly worse than the 4.11 server. The load on the 6.2 server is regularly between 2.0 and 6.0. The load on the 4.11 server is between .57 and 1 despite often servicing more connections. We began this process to upgrade into the 6 tree because 4 is EOL. We kept the old 4.11 drive from this machine and when replacing it into the box performance is excellent just like our other 4.11 box. We have tired multiple tuning variables as recommended by both FreeBSD and apache and tried the recommendations in the 6.2 errata as well. The 6.2 errata states that kern.ipc.nmbclusters="0" will help the kernel memory allocator properly deal with high network traffic. We tried this and initially thought that the box was showing wonderful performance, but then we realized that the box was not allowing much network access at all. A single ssh and proftpd connection were all it would accept. Apache wouldn't even start giving a MaxClients error. Removing this option returned it to functional though poor performance mode. Are we missing something with how to use this variable? IS this expected behavior? This particular hardware does display some oddities on both machines, running either 6.2 or 4.11. We know that FreeBSD has hyperthreading turned off by default. We have done some additional testing with hyperthreading turned on in the OS, but we wish for it to remain off due to security concerns. If we disable hyperthreading in the bios and have it disabled in the OS then FreeBSD sees one physical and one logical processor (from dmesg) and only uses processor 0. If we enable hyperthreading in the bios and leave it disabled in the OS it will show 4 CPUs but only use 0 and 2. Top will show that there is 50% idle CPU despite the fact that the box is 100% loaded, CPU 1 and 3 are idle. We would expect that FreeBSD would not see logical processors when hyperthreading was disabled in either the BIOS or the OS. This may just be a communication problem between the BIOS and FreeBSD, but we don't see this behavior on other supermicro servers with hyperthreading. VMSTAT, NETSTAT, NFSSTAT and FSTAT show similar numbers between both servers, certainly nothing that would explain why a single httpd process requires 20% of a CPU on the 6.2 box and only 5-7% on the 4.11, but we could easily be missing something. We suspected NFS or disk bottlenecks, but ran IOZONE tests and found that the 6.2 box is actually having better performance on nfs and disk access. We are running a slightly customized SMP kernel with device polling enabled. The only bottleneck apears to be CPU usage, which works fine on 4.11. From what we've read we should not be seeing these performance problems with 6.2. So what are we missing? We assume its something stupid that will fix this problem quickly and easily, but so far, despite all the resources, we have been unable to find a problem with enough in common with our own to suggest possible solutions. Please Help. thanks Steve B -- --- Steven H. Baeighkley - Systems Administrator Front Range Internet, Inc. stevenb@frii.com - (970) 212-0756