From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 19:35:03 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3410816A4CE for ; Tue, 19 Oct 2004 19:35:03 +0000 (GMT) Received: from cenn.mc.mpls.visi.com (cenn.mc.mpls.visi.com [208.42.156.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1727043D2D for ; Tue, 19 Oct 2004 19:35:03 +0000 (GMT) (envelope-from krishopper@smtp-in.cybernetik.net) Received: from smtp-in.cybernetik.net (unknown [208.42.117.145]) by cenn.mc.mpls.visi.com (Postfix) with ESMTP id 820648ABE for ; Tue, 19 Oct 2004 14:35:02 -0500 (CDT) Received: by smtp-in.cybernetik.net (Postfix, from userid 1004) id 9E2121145E; Tue, 19 Oct 2004 14:35:01 -0500 (CDT) Date: Tue, 19 Oct 2004 14:35:01 -0500 From: Kristofer Pettijohn To: freebsd-performance@freebsd.org Message-ID: <20041019193501.GC78974@cybernetik.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.6i Subject: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 19:35:03 -0000 Im looking for some suggestions on I/O performance. I'm using FreeBSD 4.10-RELEASE on a Usenet transit server running Diablo for the transit software. I have 4 Seagate ST373435LC SCSI drives, 70GB each, and I am using CCD to bind them together with RAID-0 stripes. I can pull in anywhere from 30-40 MB sec and push out ~ 8-15 MB/sec.. averaging about 50 MB/sec throughput.. feeds coming in are coming in just fine, but sending stuff back out is lagging behind.. its falling about a half hour behind every hour. I've used tunefs to set the average file size to 20 MB and enabled soft-updates, as these are generally larger binary files that just get appended to, and then seeked later on to send the article out, I've played with setting the stripe size from anywhere between 8MB and 64MB, and did not see much change on performance between those. Maybe I'm just missing something small, but on these SCSI drives which have 160 MB/s transfer rates, I'm expecting a bit more than I'm getting with CCD. Can someone give me any pointers to look at or suggestions of things to try? Thanks! Kristofer From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 19:45:26 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8B81016A4CE for ; Tue, 19 Oct 2004 19:45:26 +0000 (GMT) Received: from minerva.int.gov.br (nat.int.gov.br [200.20.196.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 79EC243D31 for ; Tue, 19 Oct 2004 19:45:10 +0000 (GMT) (envelope-from jonny@jonny.eng.br) Received: from dinf-02.int.gov.br (dinf-02 [10.0.8.17]) by minerva.int.gov.br (Postfix) with ESMTP id A6BFEBE744; Tue, 19 Oct 2004 16:45:03 -0300 (BRT) Received: from 127.0.0.1 (AVG SMTP 7.0.280 [264.11.1]); Tue, 19 Oct 2004 16:45:03 -0300 Message-ID: <41756EBF.3010008@jonny.eng.br> Date: Tue, 19 Oct 2004 16:45:03 -0300 From: =?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?= User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a3) Gecko/20040817 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kristofer Pettijohn References: <20041019193501.GC78974@cybernetik.net> In-Reply-To: <20041019193501.GC78974@cybernetik.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 19:45:26 -0000 I'd say that you have to check which CCD chunk size is best for your needs. The manual for vinum recommends avoiding chunk sized to a power of two, which is probably the first big mistake of everybody. Try mounting with option noatime, if you haven't already. And use the largest block size possible when formatting. Last time I read about there was a limit of 16384, but I would expect better performance for large file with 64k blocks (and 8k frags). If you don't have a need for safety on the files, you could try mount async and measure if it suits better you need for performance than softupdates. Sometimes softupdates is faster, and it is always safer. Kristofer Pettijohn wrote: > Im looking for some suggestions on I/O performance. > > I'm using FreeBSD 4.10-RELEASE on a Usenet transit server running > Diablo for the transit software. > > I have 4 Seagate ST373435LC SCSI drives, 70GB each, and I am using > CCD to bind them together with RAID-0 stripes. > > I can pull in anywhere from 30-40 MB sec and push out ~ 8-15 MB/sec.. > averaging about 50 MB/sec throughput.. feeds coming in are coming > in just fine, but sending stuff back out is lagging behind.. its > falling about a half hour behind every hour. > > I've used tunefs to set the average file size to 20 MB and enabled > soft-updates, as these are generally larger binary files that just > get appended to, and then seeked later on to send the article out, > I've played with setting the stripe size from anywhere between 8MB > and 64MB, and did not see much change on performance between those. > > Maybe I'm just missing something small, but on these SCSI drives > which have 160 MB/s transfer rates, I'm expecting a bit more than > I'm getting with CCD. > > Can someone give me any pointers to look at or suggestions of things > to try? From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 20:52:26 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E0D9C16A4CE for ; Tue, 19 Oct 2004 20:52:26 +0000 (GMT) Received: from mail.geeks.org (jacobs.Geeks.ORG [204.153.247.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7536143D31 for ; Tue, 19 Oct 2004 20:52:26 +0000 (GMT) (envelope-from drechsau@Geeks.ORG) Received: by mail.geeks.org (Postfix, from userid 400) id 49C3B20B59; Tue, 19 Oct 2004 15:52:25 -0500 (CDT) Date: Tue, 19 Oct 2004 15:52:25 -0500 From: Mike Horwath To: =?iso-8859-1?B?Sm/jb19DYXJsb3NfTWVuZGVzX0x17XM=?= Message-ID: <20041019205225.GB39270@octanews.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <41756EBF.3010008@jonny.eng.br> X-PGP-Fingerprint: D8 24 CC E6 47 5F E4 60 BF B7 6E FA BF C7 6E C5 X-GPG-Fingerprint: 6A89 E78A B8B1 69D9 8CDB E966 4A5A C3F9 A1B0 C381 User-Agent: Mutt/1.5.6i cc: Kristofer Pettijohn cc: freebsd-performance@freebsd.org Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 20:52:27 -0000 On Tue, Oct 19, 2004 at 04:45:03PM -0300, João_Carlos_Mendes_Luís wrote: > I'd say that you have to check which CCD chunk size is best for > your needs. The manual for vinum recommends avoiding chunk sized to a > power of two, which is probably the first big mistake of everybody. > > Try mounting with option noatime, if you haven't already. And use > the largest block size possible when formatting. Last time I read about > there was a limit of 16384, but I would expect better performance for > large file with 64k blocks (and 8k frags). > > If you don't have a need for safety on the files, you could try > mount async and measure if it suits better you need for performance than > softupdates. Sometimes softupdates is faster, and it is always safer. All good ideas except the issue is bandwidth performance across the disks. I am seeing the same thing (and Kristofer and I have been working together, kinda, on this). It is as if I/O is being preferred for writing vs reading, very weird. His 5 disk stripe (well, it used to be five when I managed the machine) should not have issues, but this recently begun happening both on his systems and some of mine. Very odd stuff... -- Mike Horwath, reachable via drechsau@Geeks.ORG From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 21:01:13 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC08016A4F4 for ; Tue, 19 Oct 2004 21:01:13 +0000 (GMT) Received: from cenn.mc.mpls.visi.com (cenn.mc.mpls.visi.com [208.42.156.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id 78DB943D1F for ; Tue, 19 Oct 2004 21:01:13 +0000 (GMT) (envelope-from krishopper@smtp-in.cybernetik.net) Received: from smtp-in.cybernetik.net (unknown [208.42.117.145]) by cenn.mc.mpls.visi.com (Postfix) with ESMTP id E58DD87C1; Tue, 19 Oct 2004 16:01:12 -0500 (CDT) Received: by smtp-in.cybernetik.net (Postfix, from userid 1004) id A8BEE1144D; Tue, 19 Oct 2004 16:01:11 -0500 (CDT) Date: Tue, 19 Oct 2004 16:01:11 -0500 From: Kristofer Pettijohn To: Mike Horwath Message-ID: <20041019210111.GA80878@cybernetik.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> <20041019205225.GB39270@octanews.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041019205225.GB39270@octanews.net> User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org cc: Jo?o_Carlos_Mendes_Lu?s Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 21:01:14 -0000 > His 5 disk stripe (well, it used to be five when I managed the > machine) should not have issues, but this recently begun happening > both on his systems and some of mine. > > Very odd stuff... Still is 5.. No matter what configuration I use I average 35 MB writing, 13-14 MB reading.. its suckin. Kristofer From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 21:02:43 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 98D7316A4CF for ; Tue, 19 Oct 2004 21:02:43 +0000 (GMT) Received: from cenn.mc.mpls.visi.com (cenn.mc.mpls.visi.com [208.42.156.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C0AA43D48 for ; Tue, 19 Oct 2004 21:02:43 +0000 (GMT) (envelope-from krishopper@smtp-in.cybernetik.net) Received: from smtp-in.cybernetik.net (unknown [208.42.117.145]) by cenn.mc.mpls.visi.com (Postfix) with ESMTP id C35E18C68; Tue, 19 Oct 2004 16:02:42 -0500 (CDT) Received: by smtp-in.cybernetik.net (Postfix, from userid 1004) id EAE7511451; Tue, 19 Oct 2004 16:02:41 -0500 (CDT) Date: Tue, 19 Oct 2004 16:02:41 -0500 From: Kristofer Pettijohn To: Jo?o Carlos Mendes Lu?s Message-ID: <20041019210241.GB80878@cybernetik.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41756EBF.3010008@jonny.eng.br> User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 21:02:43 -0000 On Tue, Oct 19, 2004 at 04:45:03PM -0300, Jo?o Carlos Mendes Lu?s wrote: > I'd say that you have to check which CCD chunk size is best for > your needs. The manual for vinum recommends avoiding chunk sized to a > power of two, which is probably the first big mistake of everybody. I've tried a wide range of chunk sizes, from 128kB to 64MB.. also have been avoiding powers of 2. > Try mounting with option noatime, if you haven't already. And use > the largest block size possible when formatting. Last time I read about > there was a limit of 16384, but I would expect better performance for > large file with 64k blocks (and 8k frags). The largest block size is 64k, and I have done that as well. > If you don't have a need for safety on the files, you could try > mount async and measure if it suits better you need for performance than > softupdates. Sometimes softupdates is faster, and it is always safer. Unfortunately, the files do need a bit of safety.I did think about async, but I fear the worst to happen. Kristofer From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 21:03:13 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 54B4416A4CE for ; Tue, 19 Oct 2004 21:03:13 +0000 (GMT) Received: from mail.geeks.org (jacobs.Geeks.ORG [204.153.247.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B73143D55 for ; Tue, 19 Oct 2004 21:03:13 +0000 (GMT) (envelope-from drechsau@Geeks.ORG) Received: by mail.geeks.org (Postfix, from userid 400) id 8A5B320B60; Tue, 19 Oct 2004 16:03:12 -0500 (CDT) Date: Tue, 19 Oct 2004 16:03:12 -0500 From: Mike Horwath To: Kristofer Pettijohn Message-ID: <20041019210312.GG39270@octanews.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> <20041019205225.GB39270@octanews.net> <20041019210111.GA80878@cybernetik.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041019210111.GA80878@cybernetik.net> X-PGP-Fingerprint: D8 24 CC E6 47 5F E4 60 BF B7 6E FA BF C7 6E C5 X-GPG-Fingerprint: 6A89 E78A B8B1 69D9 8CDB E966 4A5A C3F9 A1B0 C381 User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org cc: Jo?o_Carlos_Mendes_Lu?s cc: Mike Horwath Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 21:03:13 -0000 On Tue, Oct 19, 2004 at 04:01:11PM -0500, Kristofer Pettijohn wrote: > > His 5 disk stripe (well, it used to be five when I managed the > > machine) should not have issues, but this recently begun happening > > both on his systems and some of mine. > > > > Very odd stuff... > > Still is 5.. > > No matter what configuration I use I average 35 MB writing, 13-14 MB > reading.. its suckin. I am working on some ideas, I'll drop you email once I have testing done. -- Mike Horwath, reachable via drechsau@Geeks.ORG From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 21:04:58 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2CCF616A4CE for ; Tue, 19 Oct 2004 21:04:58 +0000 (GMT) Received: from mail.geeks.org (jacobs.Geeks.ORG [204.153.247.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id D4B3843D1F for ; Tue, 19 Oct 2004 21:04:57 +0000 (GMT) (envelope-from drechsau@Geeks.ORG) Received: by mail.geeks.org (Postfix, from userid 400) id 65C6F20B60; Tue, 19 Oct 2004 16:04:57 -0500 (CDT) Date: Tue, 19 Oct 2004 16:04:57 -0500 From: Mike Horwath To: Kristofer Pettijohn Message-ID: <20041019210457.GH39270@octanews.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> <20041019210241.GB80878@cybernetik.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041019210241.GB80878@cybernetik.net> X-PGP-Fingerprint: D8 24 CC E6 47 5F E4 60 BF B7 6E FA BF C7 6E C5 X-GPG-Fingerprint: 6A89 E78A B8B1 69D9 8CDB E966 4A5A C3F9 A1B0 C381 User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org cc: Jo?o Carlos Mendes Lu?s Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 21:04:58 -0000 On Tue, Oct 19, 2004 at 04:02:41PM -0500, Kristofer Pettijohn wrote: > On Tue, Oct 19, 2004 at 04:45:03PM -0300, Jo?o Carlos Mendes Lu?s wrote: > > I'd say that you have to check which CCD chunk size is best for > > your needs. The manual for vinum recommends avoiding chunk sized to a > > power of two, which is probably the first big mistake of everybody. > > I've tried a wide range of chunk sizes, from 128kB to 64MB.. also have > been avoiding powers of 2. The old idea from someone long ago was to set the stripe size the size of the cylinder block. I found 8MB to be pretty decent, though not perfect. > > Try mounting with option noatime, if you haven't already. And use > > the largest block size possible when formatting. Last time I read about > > there was a limit of 16384, but I would expect better performance for > > large file with 64k blocks (and 8k frags). > > The largest block size is 64k, and I have done that as well. Use this for tunefs - it made a little bit of a difference: tunefs -m 8 -o time /dev/something and make sure you set up the dspool.ctl file correctly because of the change. -- Mike Horwath, reachable via drechsau@Geeks.ORG From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 21:07:12 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1FB0B16A4CE for ; Tue, 19 Oct 2004 21:07:12 +0000 (GMT) Received: from cenn.mc.mpls.visi.com (cenn.mc.mpls.visi.com [208.42.156.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id E79EB43D5D for ; Tue, 19 Oct 2004 21:07:11 +0000 (GMT) (envelope-from krishopper@smtp-in.cybernetik.net) Received: from smtp-in.cybernetik.net (unknown [208.42.117.145]) by cenn.mc.mpls.visi.com (Postfix) with ESMTP id CA0348D7C; Tue, 19 Oct 2004 16:07:10 -0500 (CDT) Received: by smtp-in.cybernetik.net (Postfix, from userid 1004) id 04D9D11453; Tue, 19 Oct 2004 16:07:10 -0500 (CDT) Date: Tue, 19 Oct 2004 16:07:09 -0500 From: Kristofer Pettijohn To: Mike Horwath Message-ID: <20041019210709.GD80878@cybernetik.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> <20041019210241.GB80878@cybernetik.net> <20041019210457.GH39270@octanews.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041019210457.GH39270@octanews.net> User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org cc: Jo?o Carlos Mendes Lu?s Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 21:07:12 -0000 > Use this for tunefs - it made a little bit of a difference: > > tunefs -m 8 -o time /dev/something > > and make sure you set up the dspool.ctl file correctly because of the > change. Already am.. Set -f, -n, played with -a a bit, -o is time, -m is 8.. Kristofer From owner-freebsd-performance@FreeBSD.ORG Tue Oct 19 21:09:05 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5741016A4CE for ; Tue, 19 Oct 2004 21:09:05 +0000 (GMT) Received: from CPE000103d44c07-CM000f9f7ae88c.cpe.net.cable.rogers.com (CPE000103d44c07-CM000f9f7ae88c.cpe.net.cable.rogers.com [69.193.222.195]) by mx1.FreeBSD.org (Postfix) with ESMTP id 18D7C43D58 for ; Tue, 19 Oct 2004 21:09:05 +0000 (GMT) (envelope-from mikej@rogers.com) Received: from 192.168.0.1 (localhost [127.0.0.1]) with ESMTP id 7C42D29542A; Tue, 19 Oct 2004 17:08:57 -0400 (EDT) Received: from 192.168.0.188 (SquirrelMail authenticated user mikej); by 192.168.0.1 with HTTP; Tue, 19 Oct 2004 17:08:57 -0400 (EDT) Message-ID: <4311.192.168.0.188.1098220137.squirrel@192.168.0.188> In-Reply-To: <20041019210312.GG39270@octanews.net> References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> <20041019205225.GB39270@octanews.net> <20041019210111.GA80878@cybernetik.net> <20041019210312.GG39270@octanews.net> Date: Tue, 19 Oct 2004 17:08:57 -0400 (EDT) From: "Mike Jakubik" To: "Mike Horwath" User-Agent: SquirrelMail/1.4.3a X-Mailer: SquirrelMail/1.4.3a MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-wettoast-MailScanner-Information: Please contact the ISP for more information X-wettoast-MailScanner: Found to be clean X-MailScanner-From: mikej@rogers.com cc: Kristofer Pettijohn cc: freebsd-performance@freebsd.org cc: Mike Horwath Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Oct 2004 21:09:05 -0000 Mike Horwath said: > On Tue, Oct 19, 2004 at 04:01:11PM -0500, Kristofer Pettijohn wrote: >> > His 5 disk stripe (well, it used to be five when I managed the >> > machine) should not have issues, but this recently begun happening >> > both on his systems and some of mine. >> > >> > Very odd stuff... >> >> Still is 5.. >> >> No matter what configuration I use I average 35 MB writing, 13-14 MB >> reading.. its suckin. > > I am working on some ideas, I'll drop you email once I have testing > done. There is a discussion on -current going on right now regarding disk performance issues. Perhaps this is related. Look for the thread "FreeBSD 5.3b7and poor ata performance". From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 12:10:20 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A106716A4CE for ; Wed, 20 Oct 2004 12:10:20 +0000 (GMT) Received: from mail.telecom.by (mail.telecom.by [213.184.225.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9358143D48 for ; Wed, 20 Oct 2004 12:09:51 +0000 (GMT) (envelope-from barzog@telecom.by) Received: from ADMIN1 (barzog-ipsec.telecom.by [172.16.2.12] (may be forged)) (authenticated bits=0) by mail.telecom.by (8.12.11/8.12.11) with ESMTP id i9KC8Z5V022876 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO) for ; Wed, 20 Oct 2004 15:08:35 +0300 (EEST) (envelope-from barzog@telecom.by) Message-ID: <004001c4b69d$80e21f40$0c0210ac@ADMIN1> From: "Oleg Gawriloff" To: Date: Wed, 20 Oct 2004 15:08:26 +0300 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="koi8-r"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Status: No, hits=-4.2 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=2.64 X-Spam-Checker-Version: SpamAssassin 2.64 (2004-01-11) on martin.telecom.by Subject: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 12:10:20 -0000 Hello! Currently we have router on FreeBSD 4.10-R on SMP 2*PIII 1Ghz. 5sec average traffic trhough fxp on Intel 82550 is about 12kpps (26Mbit/sec). CPU Load on interrupts shown in top and systat -v 1 always above 40%. We've tried to enable polling with HZ=1000, but cpu load does not change, instead round trip time increased from 1ms to 20ms. Also we've tried link0, no success. Is any other options to decrease interrupt cpu load? May be enabling fast forwarding, change fxp to em or bge NIC? Is there exist any drawbacks from enabling fast forwarding? With best regards, Oleg Gawriloff. From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 13:08:47 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9C59F16A4D0 for ; Wed, 20 Oct 2004 13:08:47 +0000 (GMT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id EB03743D49 for ; Wed, 20 Oct 2004 13:08:46 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id i9KD8k6u093437; Wed, 20 Oct 2004 08:08:46 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <41766350.4080901@centtech.com> Date: Wed, 20 Oct 2004 08:08:32 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20040912 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Oleg Gawriloff References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1> In-Reply-To: <004001c4b69d$80e21f40$0c0210ac@ADMIN1> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 13:08:47 -0000 Oleg Gawriloff wrote: > Hello! > > Currently we have router on FreeBSD 4.10-R on SMP 2*PIII 1Ghz. 5sec > average traffic trhough fxp on Intel 82550 is about 12kpps (26Mbit/sec). > CPU Load on interrupts shown in top and systat -v 1 always above 40%. > We've tried to enable polling with HZ=1000, but cpu load does not > change, instead round trip time increased from 1ms to 20ms. Also we've > tried link0, no success. Is any other options to decrease interrupt cpu > load? May be enabling fast forwarding, change fxp to em or bge NIC? Is > there exist any drawbacks from enabling fast forwarding? I've had really great performance with the em NICs, and bad experiences with bge's (the hardware is flaky). You might want to check if the interface has checksuming enabled. I believe disabling/enabling can change the performance, however I'm not certain how much, etc. In ifconfig, look for something like: options=3 If you don't need any firewalling, I would compile a custom kernel with as much removed as you can, specifically firewall related items. Also - it's possible that your bus is the bottleneck - depending on how many NICs you have, the type of bus, and the motherboard. Eric -- ------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Talk sense to a fool and he calls you foolish. ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 14:06:05 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82CBF16A4CE for ; Wed, 20 Oct 2004 14:06:05 +0000 (GMT) Received: from mail.telecom.by (mail.telecom.by [213.184.225.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id D268E43D4C for ; Wed, 20 Oct 2004 14:05:18 +0000 (GMT) (envelope-from barzog@telecom.by) Received: from ADMIN1 (barzog.telecom.by [172.16.2.12]) (authenticated bits=0) by mail.telecom.by (8.12.11/8.12.11) with ESMTP id i9KE1MrP032984 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Wed, 20 Oct 2004 17:01:22 +0300 (EEST) (envelope-from barzog@telecom.by) Message-ID: <006001c4b6ad$38a8cac0$0c0210ac@ADMIN1> From: "Oleg Gawriloff" To: "Eric Anderson" References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1> <41766350.4080901@centtech.com> Date: Wed, 20 Oct 2004 17:00:56 +0300 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Status: No, hits=-4.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=2.64 X-Spam-Checker-Version: SpamAssassin 2.64 (2004-01-11) on martin.telecom.by cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 14:06:05 -0000 Hello, Eric! You wrote to "Oleg Gawriloff" on Wed, 20 Oct 2004 08:08:32 -0500: EA> I've had really great performance with the em NICs, and bad experiences EA> with bge's (the hardware is flaky). OK, we'll try. Is there any problems with any of supported chipsets in em-driver with polling? There many negative answer about polling support in fxp on 82550, and good answers about 82558 and 82559 with fxp-driver. EA> Also - it's possible that your bus is the bottleneck - depending on how EA> many NICs you have, the type of bus, and the motherboard. Intel SCB2, integrated NIC on PCI 33Mhz bus. We use only one NIC, with five 802.1q vlan sub-interfaces configured. Is there any ideas how can I diagnose bottlenecks on bus? With best regards, Oleg Gawriloff. From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 15:15:11 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EF9F716A4CF for ; Wed, 20 Oct 2004 15:15:11 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 32B7B43D39 for ; Wed, 20 Oct 2004 15:15:11 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v7.2.0.R) with ESMTP id md50000645801.msg for ; Wed, 20 Oct 2004 16:07:36 +0100 Message-ID: <00eb01c4b6b7$7a04c5a0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Oleg Gawriloff" , "Eric Anderson" References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1><41766350.4080901@centtech.com> <006001c4b6ad$38a8cac0$0c0210ac@ADMIN1> Date: Wed, 20 Oct 2004 16:14:18 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Processed: multiplay.co.uk, Wed, 20 Oct 2004 16:07:36 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Wed, 20 Oct 2004 16:07:37 +0100 cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 15:15:12 -0000 I can confirm 82558 using fxp is very good especially with link0. Using polling an em based card in the same machine ( dual PIII 800Mhz ) will out perform the fxp but only just 16MB/s vs 12MB/s. Without polling the em is significantly slower than the fxp which appears to be mainly due to interrupt load. Note: test results from a single transfer using ftp from a win32 P2.4Ghz connected directly via Intel(R) PRO/1000. Steve ----- Original Message ----- From: "Oleg Gawriloff" > EA> I've had really great performance with the em NICs, and bad experiences > EA> with bge's (the hardware is flaky). > OK, we'll try. Is there any problems with any of supported chipsets in > em-driver with polling? There many negative answer about polling support in > fxp on 82550, and good answers about 82558 and 82559 with fxp-driver. > EA> Also - it's possible that your bus is the bottleneck - depending on how > EA> many NICs you have, the type of bus, and the motherboard. > Intel SCB2, integrated NIC on PCI 33Mhz bus. We use only one NIC, with five > 802.1q vlan sub-interfaces configured. Is there any ideas how can I diagnose > bottlenecks on bus? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 15:32:30 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C5DF416A4CE for ; Wed, 20 Oct 2004 15:32:30 +0000 (GMT) Received: from minerva.int.gov.br (nat.int.gov.br [200.20.196.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F8F643D1F for ; Wed, 20 Oct 2004 15:32:30 +0000 (GMT) (envelope-from jonny@jonny.eng.br) Received: from dinf-02.int.gov.br (dinf-02 [10.0.8.17]) by minerva.int.gov.br (Postfix) with ESMTP id CA618BE5B1; Wed, 20 Oct 2004 12:32:27 -0300 (BRT) Received: from 127.0.0.1 (AVG SMTP 7.0.280 [264.12.0]); Wed, 20 Oct 2004 12:32:23 -0300 Message-ID: <41768507.3070304@jonny.eng.br> Date: Wed, 20 Oct 2004 12:32:23 -0300 From: =?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?= User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Horwath References: <20041019193501.GC78974@cybernetik.net> <41756EBF.3010008@jonny.eng.br> <20041019205225.GB39270@octanews.net> In-Reply-To: <20041019205225.GB39270@octanews.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: Kristofer Pettijohn cc: freebsd-performance@freebsd.org Subject: Re: Disk I/O Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 15:32:30 -0000 Mike Horwath wrote: > On Tue, Oct 19, 2004 at 04:45:03PM -0300, João_Carlos_Mendes_Luís wrote: > >> I'd say that you have to check which CCD chunk size is best for >>your needs. The manual for vinum recommends avoiding chunk sized to a >>power of two, which is probably the first big mistake of everybody. >> >> Try mounting with option noatime, if you haven't already. And use >>the largest block size possible when formatting. Last time I read about >>there was a limit of 16384, but I would expect better performance for >>large file with 64k blocks (and 8k frags). >> >> If you don't have a need for safety on the files, you could try >>mount async and measure if it suits better you need for performance than >>softupdates. Sometimes softupdates is faster, and it is always safer. > > > All good ideas except the issue is bandwidth performance across the > disks. > > I am seeing the same thing (and Kristofer and I have been working > together, kinda, on this). It is as if I/O is being preferred for > writing vs reading, very weird. > > His 5 disk stripe (well, it used to be five when I managed the > machine) should not have issues, but this recently begun happening > both on his systems and some of mine. Did you make a software upgrade recently? This could point to some changes in the device driver. If this is the case, them a downgrade could help. Indeed, I would expect that, if both happened at the same time, write should have more preference than read. This would release buffers faster and keep sensible data in RAM for the minimum time possible. Of course, only after softupdates have released the data for writing, at the block device level. From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 20:16:43 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A15A616A4CE for ; Wed, 20 Oct 2004 20:16:43 +0000 (GMT) Received: from gen129.n001.c02.escapebox.net (gen129.n001.c02.escapebox.net [213.73.91.129]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5304243D41 for ; Wed, 20 Oct 2004 20:16:43 +0000 (GMT) (envelope-from gemini@geminix.org) Message-ID: <4176C7A8.6030407@geminix.org> Date: Wed, 20 Oct 2004 22:16:40 +0200 From: Uwe Doering Organization: Private UNIX Site User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20041002 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1> <41766350.4080901@centtech.com> <006001c4b6ad$38a8cac0$0c0210ac@ADMIN1> In-Reply-To: <006001c4b6ad$38a8cac0$0c0210ac@ADMIN1> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Received: from gemini by geminix.org with asmtp (TLSv1:AES256-SHA:256) (Exim 3.36 #1) id 1CKMsz-000E0N-00; Wed, 20 Oct 2004 22:16:42 +0200 Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 20:16:43 -0000 Oleg Gawriloff wrote: > Hello, Eric! > You wrote to "Oleg Gawriloff" on Wed, 20 Oct 2004 > 08:08:32 -0500: > > EA> I've had really great performance with the em NICs, and bad experiences > EA> with bge's (the hardware is flaky). > OK, we'll try. Is there any problems with any of supported chipsets in > em-driver with polling? There many negative answer about polling support > in fxp on 82550, and good answers about 82558 and 82559 with fxp-driver. Since you mentioned earlier that you run this on an SMP system, are you aware that device polling is available only for single CPU kernels, that is, not in SMP mode? This is poorly documented, unfortunately. You can find out about it by looking at the first couple of lines of 'sys/kern/kern_poll.c'. Depending on what else you do on this server you might be better off with running it in single CPU mode, with device polling enabled (kernel compiled with DEVICE_POLLING and HZ=1000). This could make a considerable difference with the fxp driver already. I suggest to try it out before you start making changes to the hardware. Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers gemini@geminix.org | http://www.escapebox.net From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 20:34:56 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E929B16A4CE for ; Wed, 20 Oct 2004 20:34:56 +0000 (GMT) Received: from dfmm.org (walter.dfmm.org [66.180.195.210]) by mx1.FreeBSD.org (Postfix) with ESMTP id A222243D55 for ; Wed, 20 Oct 2004 20:34:56 +0000 (GMT) (envelope-from freebsd-performance@dfmm.org) Received: (qmail 45490 invoked by uid 1000); 20 Oct 2004 20:34:56 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 20 Oct 2004 20:34:56 -0000 Date: Wed, 20 Oct 2004 13:34:56 -0700 (PDT) From: Jason Stone X-X-Sender: jason@walter To: freebsd-performance@freebsd.org Message-ID: <20041020133406.V79820@walter> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 20:34:57 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Since you mentioned earlier that you run this on an SMP system, are you > aware that device polling is available only for single CPU kernels, that > is, not in SMP mode? I read somewhere that there wasn't a specific reason for not allowing device polling and smp to be used at the same time, and that it was fairly safe to remove the #ifdef SMP/#error block in sys/kern/kern_poll.c and compile in both smp and device polling. I haven't done this in a production environment, but I've done it on my smp desktop box, and it seems to work okay. -Jason -------------------------------------------------------------------------- Freud himself was a bit of a cold fish, and one cannot avoid the suspicion that he was insufficiently fondled when he was an infant. -- Ashley Montagu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (FreeBSD) Comment: See https://private.idealab.com/public/jason/jason.gpg iD8DBQFBdsvwswXMWWtptckRAm63AKDzgU1l9mjbxvP8tkfJHaQcHgLOngCgjXvg N18WsVLMV/icMXyQu3xsetY= =OzO5 -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 20:38:03 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2825016A4CE for ; Wed, 20 Oct 2004 20:38:03 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 73BFF43D58 for ; Wed, 20 Oct 2004 20:38:02 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v7.2.0.R) with ESMTP id md50000646712.msg for ; Wed, 20 Oct 2004 21:31:06 +0100 Message-ID: <006701c4b6e4$ab133d20$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Uwe Doering" , References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1><41766350.4080901@centtech.com> <006001c4b6ad$38a8cac0$0c0210ac@ADMIN1> <4176C7A8.6030407@geminix.org> Date: Wed, 20 Oct 2004 21:37:47 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Processed: multiplay.co.uk, Wed, 20 Oct 2004 21:31:06 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Wed, 20 Oct 2004 21:31:09 +0100 Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 20:38:03 -0000 Actually it does work quite well on an SMP machine if you comment said lines out :) Steve ----- Original Message ----- From: "Uwe Doering" To: Sent: Wednesday, October 20, 2004 9:16 PM Subject: Re: decreasing interrupt CPU load > Since you mentioned earlier that you run this on an SMP system, are you > aware that device polling is available only for single CPU kernels, that > is, not in SMP mode? This is poorly documented, unfortunately. You can > find out about it by looking at the first couple of lines of > 'sys/kern/kern_poll.c'. > > Depending on what else you do on this server you might be better off > with running it in single CPU mode, with device polling enabled (kernel > compiled with DEVICE_POLLING and HZ=1000). This could make a > considerable difference with the fxp driver already. I suggest to try > it out before you start making changes to the hardware. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 21:08:22 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4694416A4CE for ; Wed, 20 Oct 2004 21:08:22 +0000 (GMT) Received: from gen129.n001.c02.escapebox.net (gen129.n001.c02.escapebox.net [213.73.91.129]) by mx1.FreeBSD.org (Postfix) with ESMTP id 062DE43D48 for ; Wed, 20 Oct 2004 21:08:22 +0000 (GMT) (envelope-from gemini@geminix.org) Message-ID: <4176D3C3.30007@geminix.org> Date: Wed, 20 Oct 2004 23:08:19 +0200 From: Uwe Doering Organization: Private UNIX Site User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20041002 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1><41766350.4080901@centtech.com> <006001c4b6ad$38a8cac0$0c0210ac@ADMIN1> <4176C7A8.6030407@geminix.org> <006701c4b6e4$ab133d20$b3db87d4@multiplay.co.uk> In-Reply-To: <006701c4b6e4$ab133d20$b3db87d4@multiplay.co.uk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Received: from gemini by geminix.org with asmtp (TLSv1:AES256-SHA:256) (Exim 3.36 #1) id 1CKNgy-000F7P-00; Wed, 20 Oct 2004 23:08:21 +0200 Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 21:08:22 -0000 Steven Hartland wrote: >> Since you mentioned earlier that you run this on an SMP system, are >> you aware that device polling is available only for single CPU >> kernels, that is, not in SMP mode? This is poorly documented, >> unfortunately. You can find out about it by looking at the first >> couple of lines of 'sys/kern/kern_poll.c'. > > Actually it does work quite well on an SMP machine if you comment said > lines out :) I wonder, can you define how well "quite well" actually is? I mean, it's of course everyone's own decision, but I wouldn't dare to do this on a production system. Is there a statement on this available from the author of the code? One should think that he must have had a reason for explicitly disabling device polling for SMP. Maybe locking issues (race conditions)? Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers gemini@geminix.org | http://www.escapebox.net From owner-freebsd-performance@FreeBSD.ORG Wed Oct 20 20:33:25 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2617416A4CE for ; Wed, 20 Oct 2004 20:33:25 +0000 (GMT) Received: from dfmm.org (walter.dfmm.org [66.180.195.210]) by mx1.FreeBSD.org (Postfix) with ESMTP id DE2A243D1F for ; Wed, 20 Oct 2004 20:33:24 +0000 (GMT) (envelope-from jason@shalott.net) Received: (qmail 45138 invoked by uid 1000); 20 Oct 2004 20:33:24 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 20 Oct 2004 20:33:24 -0000 Date: Wed, 20 Oct 2004 13:33:24 -0700 (PDT) From: Jason Stone X-X-Sender: jason@walter To: Uwe Doering In-Reply-To: <4176C7A8.6030407@geminix.org> Message-ID: <20041020132734.B79820@walter> References: <004001c4b69d$80e21f40$0c0210ac@ADMIN1> <41766350.4080901@centtech.com><4176C7A8.6030407@geminix.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Mailman-Approved-At: Wed, 20 Oct 2004 22:06:51 +0000 cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2004 20:33:25 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Since you mentioned earlier that you run this on an SMP system, are you > aware that device polling is available only for single CPU kernels, that > is, not in SMP mode? I read somewhere that there wasn't a specific reason for not allowing device polling and smp to be used at the same time, and that it was fairly safe to remove the #ifdef SMP/#error block in sys/kern/kern_poll.c and compile in both smp and device polling. I haven't done this in a production environment, but I've done it on my smp desktop box, and it seems to work okay. -Jason -------------------------------------------------------------------------- Freud himself was a bit of a cold fish, and one cannot avoid the suspicion that he was insufficiently fondled when he was an infant. -- Ashley Montagu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (FreeBSD) Comment: See https://private.idealab.com/public/jason/jason.gpg iD8DBQFBdsuUswXMWWtptckRAjP+AJwIVhDLkgL7FX4+wOkgOz05oD70IwCg2LdF N3+2mycBiVcR1jrAB1L41gc= =rMKb -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 04:23:03 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82AAF16A4CE for ; Thu, 21 Oct 2004 04:23:03 +0000 (GMT) Received: from note.orchestra.cse.unsw.EDU.AU (note.orchestra.cse.unsw.EDU.AU [129.94.242.24]) by mx1.FreeBSD.org (Postfix) with ESMTP id E6C1043D41 for ; Thu, 21 Oct 2004 04:23:02 +0000 (GMT) (envelope-from lukem@cse.unsw.edu.au) Received: From wagner With LocalMail ; Thu, 21 Oct 2004 14:23:01 +1000 From: lukem.freebsd@cse.unsw.edu.au Sender: lukem@cse.unsw.edu.au To: freebsd-performance@freebsd.org Date: Thu, 21 Oct 2004 14:23:01 +1000 (EST) Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: CPU utilisation cap? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2004 04:23:03 -0000 I have been doing some benchmarking as a part of some driver development work, and have encountered a phenomenon I can't explain. I am using FreeBSD 5.2.1-RELEASE with SMP and IO-APIC disabled. I am using a dual 2.8GHz xeon box, but only one CPU without hyperthreading. The box in question has three em interfaces, and one fxp. Two of the em's are 133Mhz/64bit, and one is 33MHz/32bit. I have verified these values by modifying the em driver to print out what it detects. em0: MAC type:82546 r3 Bus speed:133MHz Bus width:64bit Bus type:PCI-X em1: MAC type:82546 r3 Bus speed:133MHz Bus width:64bit Bus type:PCI-X em2: MAC type:82540 Bus speed:33MHz Bus width:32bit Bus type:PCI The particular benchmark I have been using is a UDP echo test, where I have a number of linux boxes sending UDP packets to the freebsd box, which the freebsd box echoes at user-level (think inetd udp echo, though in fact I have also used an optimised server which gets higher throughput). Throughput is measured on the boxes which generate the UDP packets. I am measuring idle time using a CPU soaker process which runs at a very low priority. Top seems to confirm the output it gives. What I see is strange. CPU utilisation always peaks (and stays) at between 80 & 85%. If I increase the amount of work done by the UDP echo program (by inserting additional packet copies), CPU utilisation does not rise, but rather, throughput declines. The 80% figure is common to both the slow and fast PCI cards as well. This is rather confusing, as I cannot tell if the system is IO bound or CPU bound. Certainly I would not have expected the 133/64 PCI bus to be saturated given that peak throughput is around 550Mbit/s with 1024-byte packets. (Such a low figure is not unexpected given there are 2 syscalls per packet). -- Luke From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 13:56:50 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ED08316A4CE for ; Thu, 21 Oct 2004 13:56:50 +0000 (GMT) Received: from adsl-68-76-19-75.dsl.klmzmi.ameritech.net (adsl-68-76-19-75.dsl.klmzmi.ameritech.net [68.76.19.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 18FA343D39 for ; Thu, 21 Oct 2004 13:56:50 +0000 (GMT) (envelope-from luke@foolishgames.com) Received: from [192.168.0.51] (24.176.8.69.kzo.mi.chartermi.net [24.176.8.69]) (authenticated bits=0)ESMTP id i9LDukJI051972; Thu, 21 Oct 2004 09:56:47 -0400 (EDT) (envelope-from luke@foolishgames.com) X-Authentication-Warning: adsl-68-76-19-75.dsl.klmzmi.ameritech.net: Host 24.176.8.69.kzo.mi.chartermi.net [24.176.8.69] claimed to be [192.168.0.51] Message-Id: <0A85B89D-2369-11D9-BD7C-000A95EFF4CA@foolishgames.com> X-Habeas-Swe-6: email in exchange for a license for this Habeas X-Habeas-Swe-3: like Habeas SWE (tm) Date: Thu, 21 Oct 2004 09:56:43 -0400 X-Habeas-Swe-8: Message (HCM) and not spam. Please report use of this From: Lucas Holt X-Habeas-Swe-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-Swe-2: brightly anticipated In-Reply-To: <20041020133406.V79820@walter> References: <20041020133406.V79820@walter> To: Jason Stone X-Habeas-Swe-7: warrant mark warrants that this is a Habeas Compliant Mime-Version: 1.0 (Apple Message framework v619) X-Habeas-Swe-4: Copyright 2002 Habeas (tm) Content-Type: text/plain; charset=US-ASCII; format=flowed X-Habeas-Swe-1: winter into spring Content-Transfer-Encoding: 7bit X-Habeas-Swe-9: mark in spam to . X-Mailer: Apple Mail (2.619) X-Virus-Scanned: ClamAV 0.80/543/Wed Oct 20 15:24:13 2004 clamav-milter version 0.80j15:24:13 2004 clamav-milter version 0.80j 15:24:13 2004 clamav-milter version 0.80j on adsl-68-76-19-75.dsl.klmzmi.amer X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2004 13:56:51 -0000 On Oct 20, 2004, at 4:34 PM, Jason Stone wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > I read somewhere that there wasn't a specific reason for not allowing > device polling and smp to be used at the same time, and that it was > fairly > safe to remove the #ifdef SMP/#error block in sys/kern/kern_poll.c and > compile in both smp and device polling. > > I haven't done this in a production environment, but I've done it on my > smp desktop box, and it seems to work okay. > > > -Jason > If you look at http://info.iet.unipi.it/~luigi/polling, the last Q & A question suggests why it is disabled for SMP. It seems that polling only runs on one thread whereas an smp box might handle concurrently interupts from different devices. Can the scheduler move the thread to another cpu or is it locked on a particular cpu? Lucas Holt Luke@FoolishGames.com ________________________________________________________ FoolishGames.com (Jewel Fan Site) JustJournal.com (Free blogging) From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 17:54:26 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 272AD16A4CE for ; Thu, 21 Oct 2004 17:54:26 +0000 (GMT) Received: from mail1.speakeasy.net (mail1.speakeasy.net [216.254.0.201]) by mx1.FreeBSD.org (Postfix) with ESMTP id ECBDB43D31 for ; Thu, 21 Oct 2004 17:54:23 +0000 (GMT) (envelope-from sean@chittenden.org) Received: (qmail 19287 invoked from network); 21 Oct 2004 17:54:23 -0000 Received: from wbar4.sjo1-4.28.216.220.sjo1.dsl-verizon.net (HELO [192.168.1.102]) (chitt@[4.28.216.220]) (envelope-sender )encrypted SMTP for ; 21 Oct 2004 17:54:23 -0000 In-Reply-To: References: Mime-Version: 1.0 (Apple Message framework v619) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <1F92711A-238A-11D9-9171-000A95C705DC@chittenden.org> Content-Transfer-Encoding: 7bit From: Sean Chittenden Date: Thu, 21 Oct 2004 10:53:32 -0700 To: lukem.freebsd@cse.unsw.edu.au X-Mailer: Apple Mail (2.619) cc: freebsd-performance@freebsd.org Subject: Re: CPU utilisation cap? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2004 17:54:26 -0000 > I am measuring idle time using a CPU soaker process which runs at a > very low priority. Top seems to confirm the output it gives. > > What I see is strange. CPU utilisation always peaks (and stays) at > between 80 & 85%. If I increase the amount of work done by the UDP > echo program (by inserting additional packet copies), CPU utilisation > does not rise, but rather, throughput declines. The 80% figure is > common to both the slow and fast PCI cards as well. > > This is rather confusing, as I cannot tell if the system is IO bound > or CPU bound. Certainly I would not have expected the 133/64 PCI bus > to be saturated given that peak throughput is around 550Mbit/s with > 1024-byte packets. (Such a low figure is not unexpected given there > are 2 syscalls per packet). There are two things that come to mind. The first being a patch that should have been applied in time for 5.2, but I forget the timing of the releases. http://lists.freebsd.org/mailman/htdig/cvs-src/2003-October/012628.html IIRC, there was another commit that made a similar change specifically in the handling of UDP packets, such that it used a TAILQ append instead of traversing a linked list. For some reason I think this happened after 5.2, but I'm not able to find high nor low of the commit and could be pulling said memory into existence. Too many commits to keep track of. Recently rwatson has been doing a bunch of testing of our UDP stack and has also committed a fair number of fixes. He likely has many words of wisdom on what's changed or not right in 5.2.1. Regardless, I wager that if you give RELENG_5_3 a shot, I bet you'd find performance to be less surprising. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Thu Oct 21 20:43:38 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D9E1316A4CF for ; Thu, 21 Oct 2004 20:43:38 +0000 (GMT) Received: from dfmm.org (walter.dfmm.org [66.180.195.210]) by mx1.FreeBSD.org (Postfix) with ESMTP id B70B043D45 for ; Thu, 21 Oct 2004 20:43:38 +0000 (GMT) (envelope-from freebsd-performance@dfmm.org) Received: (qmail 54726 invoked by uid 1000); 21 Oct 2004 20:43:38 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 21 Oct 2004 20:43:38 -0000 Date: Thu, 21 Oct 2004 13:43:35 -0700 (PDT) From: Jason Stone X-X-Sender: jason@walter To: freebsd-performance@freebsd.org In-Reply-To: <0A85B89D-2369-11D9-BD7C-000A95EFF4CA@foolishgames.com> Message-ID: <20041021133719.S79820@walter> References: <20041020133406.V79820@walter> <0A85B89D-2369-11D9-BD7C-000A95EFF4CA@foolishgames.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: luigi@iet.unipi.it Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Oct 2004 20:43:39 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > If you look at http://info.iet.unipi.it/~luigi/polling, the last Q & A > question suggests why it is disabled for SMP. It seems that polling > only runs on one thread whereas an smp box might handle concurrently > interupts from different devices. > > Can the scheduler move the thread to another cpu or is it locked on a > particular cpu? thanks for the pointer. it seems to me that the thread doing the polling could move from cpu to cpu, but that's not the issue - the issue, if I'm understanding the author, is that the polling thread will always be a single thread, whereas if we use traditional interrupts, they can be handled on multiple cpu's concurrently. so interrupts, with multiple cpu's to handle them, might give much better performance than a single polling thread. if that's the only issue, then preventing one from compiling with both SMP and DEVICE_POLLING doesn't seem necesary, since you can turn polling on and off at runtime with sysctl's. luigi: does that sound right to you? has any consideration been giving to removing the restriction preventing DEVICE_POLLING from building on an SMP system, in light of the fact that polling can just be turned off if it's giving suboptimal performance? -Jason -------------------------------------------------------------------------- Freud himself was a bit of a cold fish, and one cannot avoid the suspicion that he was insufficiently fondled when he was an infant. -- Ashley Montagu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (FreeBSD) Comment: See https://private.idealab.com/public/jason/jason.gpg iD8DBQFBeB95swXMWWtptckRAvlRAJ4806zyUOwMq89qgNKXlbbOE826LQCg6AGW qWDj3u/F4V3a51YJQAA2qpM= =sN26 -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 12:32:14 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8692016A4CE for ; Fri, 22 Oct 2004 12:32:14 +0000 (GMT) Received: from mail.telecom.by (mail.telecom.by [213.184.225.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E09043D1D for ; Fri, 22 Oct 2004 12:27:49 +0000 (GMT) (envelope-from barzog@telecom.by) Received: from ADMIN1 (barzog.telecom.by [172.16.2.12]) (authenticated bits=0) by mail.telecom.by (8.12.11/8.12.11) with ESMTP id i9MCMFLa000780 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO) for ; Fri, 22 Oct 2004 15:22:16 +0300 (EEST) (envelope-from barzog@telecom.by) Message-ID: <004e01c4b831$c317c640$0c0210ac@ADMIN1> From: "Oleg Gawriloff" To: Date: Fri, 22 Oct 2004 15:22:13 +0300 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="koi8-r"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Status: No, hits=-3.8 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=2.64 X-Spam-Checker-Version: SpamAssassin 2.64 (2004-01-11) on martin.telecom.by Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2004 12:32:14 -0000 Hello! >if that's the only issue, then preventing one from compiling with both SMP >and DEVICE_POLLING doesn't seem necesary, since you can turn polling on >and off at runtime with sysctl's. As I stated in my first letter it seems that polling on SMP w 4.10 is not working as expected. At least with i82550 chipset. With best regards, Oleg Gawriloff. From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 13:05:42 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7688A16A4CE for ; Fri, 22 Oct 2004 13:05:42 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id B13CD43D45 for ; Fri, 22 Oct 2004 13:05:41 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v7.2.0.R) with ESMTP id md50000650347.msg for ; Fri, 22 Oct 2004 13:58:01 +0100 Message-ID: <028b01c4b837$b7532c90$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Oleg Gawriloff" , References: <004e01c4b831$c317c640$0c0210ac@ADMIN1> Date: Fri, 22 Oct 2004 14:04:46 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="koi8-r"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Processed: multiplay.co.uk, Fri, 22 Oct 2004 13:58:01 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Fri, 22 Oct 2004 13:58:03 +0100 Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2004 13:05:42 -0000 When you say not working are you seeing no more that HZ interrupts per second? After compiling in you did enable it with the sysctl? Steve ----- Original Message ----- From: "Oleg Gawriloff" >>if that's the only issue, then preventing one from compiling with both SMP >>and DEVICE_POLLING doesn't seem necesary, since you can turn polling on >>and off at runtime with sysctl's. > As I stated in my first letter it seems that polling on SMP w 4.10 is not > working as expected. At least with i82550 chipset. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 14:04:23 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 114E516A4CE for ; Fri, 22 Oct 2004 14:04:23 +0000 (GMT) Received: from mail.telecom.by (mail.telecom.by [213.184.225.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 66D4A43D5A for ; Fri, 22 Oct 2004 14:04:18 +0000 (GMT) (envelope-from barzog@telecom.by) Received: from ADMIN1 (barzog.telecom.by [172.16.2.12]) (authenticated bits=0) by mail.telecom.by (8.12.11/8.12.11) with ESMTP id i9ME4FSA009372 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Fri, 22 Oct 2004 17:04:15 +0300 (EEST) (envelope-from barzog@telecom.by) Message-ID: <005301c4b840$0444ec20$0c0210ac@ADMIN1> From: "Oleg Gawriloff" To: "Steven Hartland" References: <004e01c4b831$c317c640$0c0210ac@ADMIN1> <028b01c4b837$b7532c90$7f06000a@int.mediasurface.com> Date: Fri, 22 Oct 2004 17:04:15 +0300 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="koi8-r"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Status: No, hits=-3.9 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=2.64 X-Spam-Checker-Version: SpamAssassin 2.64 (2004-01-11) on martin.telecom.by cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2004 14:04:23 -0000 Hello, Steven! You wrote to "Oleg Gawriloff" ; on Fri, 22 Oct 2004 14:04:46 +0100: >When you say not working are you seeing no more that >HZ interrupts per second? After compiling in you did >enable it with the sysctl? After recompiling kernel with options HZ=1000 options DEVICE_POLLING and kern.polling.enable=1 number of interrupts shown in systat -v 1 does not change (about 5k). CPU interrupt load does not changes as well. Also RTT in ping increases from 2ms to 20ms. After disabling polling RTT goes back to 2ms. So we decided that polling does not work on i82550 in SMP. With best regards, Oleg Gawriloff. From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 15:56:54 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EFF4216A4CE for ; Fri, 22 Oct 2004 15:56:54 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6B29843D55 for ; Fri, 22 Oct 2004 15:56:54 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v7.2.0.R) with ESMTP id md50000650753.msg for ; Fri, 22 Oct 2004 16:49:48 +0100 Message-ID: <038f01c4b84f$b63fd3e0$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Claudiu Chirita" , "Oleg Gawriloff" References: <004e01c4b831$c317c640$0c0210ac@ADMIN1> <028b01c4b837$b7532c90$7f06000a@int.mediasurface.com> <005301c4b840$0444ec20$0c0210ac@ADMIN1> <41792185.7010503@rdsnet.ro> Date: Fri, 22 Oct 2004 16:56:37 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="koi8-r"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Processed: multiplay.co.uk, Fri, 22 Oct 2004 16:49:48 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Fri, 22 Oct 2004 16:49:49 +0100 cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2004 15:56:55 -0000 Are you both on 4.X? All my testing was done on 5.X with a 82558: xp0: port 0xb000-0xb01f mem 0xea100000-0xea1fffff,0xef000000-0xef000fff irq 19 at device 4.0 on pci2 fxp0: Ethernet address ............................. miibus0: on fxp0 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: port 0xb400-0xb41f mem 0xea000000-0xea0fffff,0xef001000-0xef001fff irq 16 at device 5.0 on pci2 fxp1: Ethernet address ................................. Steve ----- Original Message ----- From: "Claudiu Chirita" > Hello, > > I have the same problem with i82550, any ideea? > I've tested on 2 systems. > > > fxp2@pci0:18:0: class=0x020000 card=0x005c1014 chip=0x12298086 rev=0x02 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' > Claudiu C. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 16:02:20 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C880216A4CE for ; Fri, 22 Oct 2004 16:02:19 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 467F943D53 for ; Fri, 22 Oct 2004 16:02:19 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v7.2.0.R) with ESMTP id md50000650769.msg for ; Fri, 22 Oct 2004 16:54:44 +0100 Message-ID: <03b701c4b850$66038e20$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: Date: Fri, 22 Oct 2004 17:01:26 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Processed: multiplay.co.uk, Fri, 22 Oct 2004 16:54:44 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Fri, 22 Oct 2004 16:54:46 +0100 Subject: Fw: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2004 16:02:20 -0000 Can someone remove this idiot from the list every time I post I get this: Steve ----- Original Message ----- From: "Kurt Jessen" To: "Steven Hartland" Sent: 22 October 2004 16:57 Subject: Re: decreasing interrupt CPU load > REMOVE PLEASE > > > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Oct 22 16:31:35 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C33F216A4CE for ; Fri, 22 Oct 2004 16:31:35 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id E751943D2F for ; Fri, 22 Oct 2004 16:31:34 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v7.2.0.R) with ESMTP id md50000650844.msg for ; Fri, 22 Oct 2004 17:23:41 +0100 Message-ID: <040701c4b854$72f0e4d0$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Claudiu Chirita" References: <004e01c4b831$c317c640$0c0210ac@ADMIN1> <028b01c4b837$b7532c90$7f06000a@int.mediasurface.com> <005301c4b840$0444ec20$0c0210ac@ADMIN1> <41792185.7010503@rdsnet.ro> <038f01c4b84f$b63fd3e0$7f06000a@int.mediasurface.com> <4179307F.3060205@rdsnet.ro> Date: Fri, 22 Oct 2004 17:30:31 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="koi8-r"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Spam-Processed: multiplay.co.uk, Fri, 22 Oct 2004 17:23:41 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Fri, 22 Oct 2004 17:23:43 +0100 cc: Oleg Gawriloff cc: freebsd-performance@freebsd.org Subject: Re: decreasing interrupt CPU load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Oct 2004 16:31:35 -0000 If you try standard packets e.g. an unlimited single ftp transfer does it limit to HZ interrupts / s? That's what I see when just looking in systat ( vm ) without this its around 4000 -> 6000 i/s depending on if link0 is active. Steve / K ----- Original Message ----- From: "Claudiu Chirita" > I've made some tests with netperf to generate small size UDP packets. > With polling enabled, I saw performance decreasing. > Claudiu C. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk.