From owner-freebsd-performance@FreeBSD.ORG Sun Jun 29 00:21:53 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E646C37B405 for ; Sun, 29 Jun 2003 00:21:53 -0700 (PDT) Received: from craig.afraid.org (h24-69-213-234.cc.shawcable.net [24.69.213.234]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0A20D43FFB for ; Sun, 29 Jun 2003 00:21:53 -0700 (PDT) (envelope-from craig@craig.afraid.org) Received: from fireball.internal.lan ([10.0.0.2] helo=fireball) by craig.afraid.org with smtp (Exim 4.20) id 19WWVU-0001Vz-I4; Sun, 29 Jun 2003 00:21:52 -0700 Message-ID: <001f01c33e0f$1f4716d0$0200000a@fireball> From: "Craig Reyenga" To: "David Gilbert" References: <20030628190036.0E06B37B405@hub.freebsd.org> <000f01c33dad$1595a0f0$e602a8c0@flatline> <16126.9805.829406.368426@canoe.velocet.net> <000901c33dd1$12268780$0200000a@fireball> <16126.19861.842507.318997@canoe.velocet.net> Date: Sun, 29 Jun 2003 00:21:56 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 cc: freebsd-performance@freebsd.org Subject: Re: Tuning Gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Craig Reyenga List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2003 07:21:54 -0000 From: "David Gilbert" > >>>>> "Craig" == Craig Reyenga writes: > > >> 300 megabit is about where 32bit 33Mhz PCI maxes out. > > Craig> Could you tell me a little more about your tests? What boards, > Craig> and what configuration? > > Well... first of all, a 33Mhz 32-bit PCI bus can transfer 33M * 32 > bits ... which is just about 1 gigabit of _total_ PCI bus bandwidth. > Consider that you're likely testing disk->RAM->NIC and you end up with > 1/3 of that as throughput (minus bus overhead) so 300 megabit is a > good number. I should have mentioned that Iperf tests only linespeed with the options I fed it. My 5400rpm disks can't even saturate a 100mbit line :( [snip] > Now some boards I've tested (like the nvidia chipset) are strangely > limited to 100megabit. I can't explain this. It seems low no matter > how you cut it. As I mentioned in a previous email, this is horrible. Does this manifest itself with disk controllers and other high-bandwidth devices? > > Our testing has been threefold: > > 1) Generating packets. We test the machines ability to generate both > large (1500, 3000 and 9000 byte) and small (64 byte) packets. The > large scale generation of packets is necessary for the other > tests. So far, some packet flood utilities from the linux hacker > camp are our most efficient small packet generators. netcat on > memory cached objects or on /dev/zero generate our big packets. > > 2) Passing packets. Primarily, we're interested in routing. Passing > packets, passing packets with 100k routes and passing packets with > 100's of ipf accounting rules are our benchmarks. We look at both > small and large packet performance. Packet passing machines have > at least two interfaces ... but sometimes 3 or 4 are tested. > Polling is a major win in the small packet passing race. > > 3) Receiving packets. netcat is our friend again here. Receiving > packets doesn't appear to be the same level of challenge as > generating or passing them. > > At any rate, we're clearly not testing file delivery. We sometimes > play with file delivery as a first test ... or for other testing > reasons. We've found several boards that corrupt packets when they > pass more than 100megabit of packets. We havn't explained that one > yet. Our tests centre on routing packets (because that's what we do > with our high performance FreeBSD boxes. All our other FreeBSD boxes > "just work" at the level of performance they have). > I look forward to seeing a paper on this; it would certainly assist people in hardware purchase decisions. [snip] -Craig