From owner-freebsd-current@FreeBSD.ORG Thu Nov 18 17:49:15 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6CE5F16A4CE; Thu, 18 Nov 2004 17:49:15 +0000 (GMT) Received: from av7-2-sn4.m-sp.skanova.net (av7-2-sn4.m-sp.skanova.net [81.228.10.109]) by mx1.FreeBSD.org (Postfix) with ESMTP id DF8BF43D58; Thu, 18 Nov 2004 17:49:14 +0000 (GMT) (envelope-from daniel_k_eriksson@telia.com) Received: by av7-2-sn4.m-sp.skanova.net (Postfix, from userid 502) id 1C7C738128; Thu, 18 Nov 2004 18:49:14 +0100 (CET) Received: from smtp2-2-sn4.m-sp.skanova.net (smtp2-2-sn4.m-sp.skanova.net [81.228.10.182]) by av7-2-sn4.m-sp.skanova.net (Postfix) with ESMTP id 0BC803811E; Thu, 18 Nov 2004 18:49:14 +0100 (CET) Received: from sentinel (h130n1fls11o822.telia.com [213.64.66.130]) by smtp2-2-sn4.m-sp.skanova.net (Postfix) with ESMTP id D471D37E49; Thu, 18 Nov 2004 18:49:13 +0100 (CET) From: "Daniel Eriksson" To: "'M. Warner Losh'" Date: Thu, 18 Nov 2004 18:49:08 +0100 Organization: Home Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 In-Reply-To: <20041118.095348.103081513.imp@bsdimp.com> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Thread-Index: AcTNj/SLjGIxsillTd6Wa8mCXkRe+QABFJuw cc: freebsd-current@freebsd.org cc: freebsd-stable@freebsd.org Subject: RE: serious networking (em) performance (ggate and NFS) problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2004 17:49:15 -0000 M. Warner Losh wrote: > Also, make sure that you aren't sharing interrupts between > GIANT-LOCKED and non-giant-locked cards. This might be exposing bugs > in the network layer that debug.mpsafenet=0 might correct. Just > noticed that our setup here has that setup, so I'll be looking into > that area of things. I have a Tyan Tiger MPX board (dual AthlonMP) that has two 64bit PCI slots. I have an Adaptec 29160 and a dual port Intel Pro/1000 MT plugged into those slots. As can be seen from the vmstat -i output below, em1 shares ithread with ahc0. This is with ACPI disabled. With ACPI enabled all devices get their own ithread (I think, not 100% sure). However, because of some hardware problem (interrupt routing?), em0 interrupts will somehow leak into atapci1+, generating a higher interrupt load. I'm not sure how expensive this is. Finally, my question. What would you recommend: 1) Run with ACPI disabled and debug.mpsafenet=1 and hope that the mix of giant-safe and giant-locked (em and ahc) doesn't trigger any bugs. This is what I currently do. 2) Run with ACPI disabled and debug.mpsafenet=0 and accept lower network performance (it is a high-traffic server, so I'm not sure this is a valid option). 3) Run with ACPI enabled and debug.mpsafenet=1 and accept that em0 interrupts "leak" to the atapci1+ ithread. This I have done in the past. # vmstat -i interrupt total rate irq1: atkbd0 2 0 irq4: sio0 710 0 irq6: fdc0 8 0 irq8: rtc 11470937 127 irq13: npx0 1 0 irq14: ata0 1744545 19 irq15: ata1 1749617 19 irq16: em0 atapci1+ 186062858 2074 irq17: em1 ahc0 27028088 301 irq18: atapci3 7393468 82 irq19: atapci4+ 7129446 79 irq0: clk 179054582 1995 Total 421634262 4699 /Daniel Eriksson