From owner-freebsd-net@FreeBSD.ORG Thu Aug 13 11:56:34 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C8D7106567C for ; Thu, 13 Aug 2009 11:56:34 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63902.mail.re1.yahoo.com (web63902.mail.re1.yahoo.com [69.147.97.117]) by mx1.freebsd.org (Postfix) with SMTP id 238688FC4A for ; Thu, 13 Aug 2009 11:56:32 +0000 (UTC) Received: (qmail 63274 invoked by uid 60001); 13 Aug 2009 11:56:32 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1250164592; bh=H349Hk3jfSjEHQWEkIeD3Ic6djCg/pwxZiW3+9WD9q4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=2fHNAAYST1XSJ7CIBljq1QPIXk328XJmaM65MOGx6iAAanI6mNpEcuvMYo8hr0WkqP9nNPcpxQcZSzdynoSvub9nm1JIuBav+W4D4Tq2+OccGqDFBHTEH+UzGRjVDZZVnct4JAW0lBcapLB+DB8A9MtHI0HesGiNFJEY6u+X5ck= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=veOvk4q8K+/pCCOspEwT9bdWaKe/H48lU2oKvBENauKnn5Jh/6mwORncfjKJwsjE6qvML+r/Aadi1TZzWmq6ulZESX0zW5ED5d7VTU1XM9XjzAIw9sZmciTp2YC5oVmICPcRpK7amj1rKKU4zNgymL7G1/pkbRdue2NW6fXT4B0=; Message-ID: <430428.63263.qm@web63902.mail.re1.yahoo.com> X-YMail-OSG: gl1OqG8VM1l9rjwEqXqTidNctAOwuPi3MnoMMMGLDNfMbT6A0IDV62w6nHM2lzckfWIWDIu2njfcc_wljExKGn2PYTeAHcR8FsBKrc1nHKpcWHYD_ybXrX_JOf23oSbi6yV8HemlIDyrj4KTbv1yqLTbfOTJ4QC_bz71L_YPDiZf6hPYRl1bAVSGNja.Lem8tefMFheGHLNfSQEos7jtzdgxQvdLYo9Q.Zw9H19KB9jZObeYYTX9xPsjLnYe6Uubatf_iIp4GuIbvZFM8Qasj5iGg5G4j53TpSpx.SaVSX0r9B50Fm8qnyitvQ-- Received: from [66.176.162.245] by web63902.mail.re1.yahoo.com via HTTP; Thu, 13 Aug 2009 04:56:32 PDT X-Mailer: YahooMailClassic/6.1.2 YahooMailWebService/0.7.338.2 Date: Thu, 13 Aug 2009 04:56:32 -0700 (PDT) From: Barney Cordoba To: Peter Steele , pyunyh@gmail.com In-Reply-To: <20090812213521.GG55129@michelle.cdnetworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: nfe taskq performance issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2009 11:56:34 -0000 =0A=0A--- On Wed, 8/12/09, Pyun YongHyeon wrote:=0A=0A> = From: Pyun YongHyeon =0A> Subject: Re: nfe taskq performa= nce issues=0A> To: "Peter Steele" =0A> Cc: f= reebsd-net@freebsd.org=0A> Date: Wednesday, August 12, 2009, 5:35 PM=0A> On= Thu, Jul 23, 2009 at 08:58:07AM=0A> -0700, Peter Steele wrote:=0A> > We've= been hitting serious nfe taskq performance=0A> issues during stress=0A> > = tests and in doing some research on the problem we=0A> came across this old= =0A> > email:=0A> > =0A> >=A0 =0A> > =0A> > From: Ivan Voras =0A> > Date: April 28, 2009 3:53:14 AM PDT=0A> > To: freebsd-threads@= freebsd.org=0A> > Cc: freebsd-net@freebsd.org,=0A> freebsd-performance@free= bsd.org=0A> > Subject: Re: FreeBSD 7.1 taskq em performance=0A> > >=0A> > >= I have been hitting some barrier with FreeBSD 7.1=0A> network performance.= =0A> > I=0A> > > have written an application which contains two=0A> kernel = threads that=0A> > takes=0A> > > mbufs directly from a network interface an= d=0A> forwards to another=0A> > network=0A> > > interface. This idea is to = simulate different=0A> network environment.=0A> > >=0A> > > I have been usi= ng FreeBSD 6.4 amd64 and tested=0A> with an Ixia box=0A> > > (specialised h= ardware firing very high packet=0A> rate). The PC was a=0A> > Core2 2.6=0A>= > > GHz with dual ports Intel PCIE Gigabit network=0A> card. It can manage= up=0A> > to 1.2=0A> > > million pps.=0A> > >=0A> > > I have a higher spec = PC with FreeBSD 7.1 amd64=0A> and Quadcore 2.3 GHz=0A> > and=0A> > > PCIE G= igabit network card. The performance can=0A> only achieve up to 600k=0A> > = pps.=0A> > > I notice the 'taskq em0' and 'taskq em1' is solid=0A> 100% CPU= but it is=0A> > not in=0A> > > FreeBSD 6.4. =0A> > =0A> >=A0 =0A> > =0A> >= In our case we are running FreeBSD 7.0, but we are=0A> seeing our boxes=0A= > > experience serious thread starvation issues as the=0A> nfe0 cpu percent= age=0A> > climbs steadily while cpu idle time drops at times to=0A> 0 perce= nt. This=0A> > email thread mentioned a patch for the em driver=0A> here:= =0A> > =0A> >=A0 =0A> > =0A> > http://people.yandex-team.ru/~wawa/ =0A> > <= http://people.yandex-team.ru/%7Ewawa/>=0A> =0A> > =0A> >=A0 =0A> > =0A> > D= oes anyone know if this patch will work with the nfe=0A> driver?=0A> > =0A>= =0A> That's for em(4).=0A> =0A> AFAIK all nfe(4) controllers lacks intelli= gent interrupts=0A> moderation so driver should be prepared to handle=0A> e= xcessive=0A> interrupt loads. I'm not sure whether NVIDIA ethernet=0A> cont= rollers=0A> really lacks efficient interrupt mitigation mechanism but=0A> i= t=0A> seems Linux also faces the same hardware problem.=0A> As you might kn= ow there is no publicly available data sheet=0A> for=0A> NVIDIA controllers= so setting it right looks very hard to=0A> me.=0A> =0A=0ATry removing the = INTR_MPSAFE flag from the bus_setup_intr() call.=0AThe entire point of usin= g filters is to reduce lock contention.=0AIt might not solve the problem bu= t its clearly an unnecessary=0Apotential bottleneck.=0A=0ABarney=0A=0A=0A =