From owner-freebsd-net@FreeBSD.ORG Fri Jul 24 21:10:57 2009 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6642A10656E7 for ; Fri, 24 Jul 2009 21:10:57 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63902.mail.re1.yahoo.com (web63902.mail.re1.yahoo.com [69.147.97.117]) by mx1.freebsd.org (Postfix) with SMTP id 203F78FC0A for ; Fri, 24 Jul 2009 21:10:57 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 58504 invoked by uid 60001); 24 Jul 2009 21:10:56 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1248469856; bh=h67Y9kDlDiqY73BZ7p54VxX0lMKh3IoJKSJUmhLau0E=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=yxw2TFTzmN4eJdp29ONVytm0EF/rOdTFeatqlEA9hc3Gf8Ouw6dntDjlzSVNylsUaW7nXHSTyoaknUSH3LbkUuljwSqcucYDMYa3G9ZNY5c4UWZcITJjI0NCFrI5V39gDAg1GIcj9/QUBVDKyrCSRJ1j6j210m0czv69u+vnWPI= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=YoG4rAtS82Rsc2vXDyWM5aNVv7FbpaL1zpnlroEUUx3ydiUX9hx/4O5T6gbM0bhagFbcA9ty5GyNJKzUckyJTvDuWlP/Veqeq/TTuBJIcM+s/LO1UJi5rjP7Dke3U3zvdXGBqer6ETrx1qC3yuNGgidiCbKM8gzp646YLFXTebw=; Message-ID: <624694.56110.qm@web63902.mail.re1.yahoo.com> X-YMail-OSG: KwVNkjcVM1mJnV6rmnH_BDA4WPlfBnMHg7itM.w2VZQ712teh2IcHx1jHEfGcRaoS6UQpJeX58gZblcxW6x548K.E9.jN9VcyrUH4nat9FvDL3f5R6utG6LYb26V29Kpopq1TTaDYpn_UYcSlP4eb3AmoLFwebOv3vsAWCGcs16VCAS5IhZ7lHrTZ7YGglidRD2oMrjZeDGhYGx72os94owVYLdxbvYZpRgucggBGn3RVlVoGeibGjJB672wRcGaDGFx67IAnFW9zZ.g2IMmqj5KErQuOvOiaN2xCQldBE6nQFX26cY3EgtnPw-- Received: from [66.176.162.245] by web63902.mail.re1.yahoo.com via HTTP; Fri, 24 Jul 2009 14:10:56 PDT X-Mailer: YahooMailClassic/5.4.17 YahooMailWebService/0.7.289.15 Date: Fri, 24 Jul 2009 14:10:56 -0700 (PDT) From: Barney Cordoba To: freebsd-net@FreeBSD.org, Peter Steele MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Subject: Re: nfe taskq performance issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jul 2009 21:10:57 -0000 --- On Thu, 7/23/09, Peter Steele wrote: > From: Peter Steele > Subject: nfe taskq performance issues > To: freebsd-net@FreeBSD.org > Date: Thursday, July 23, 2009, 11:58 AM > We've been hitting serious nfe taskq > performance issues during stress > tests and in doing some research on the problem we came > across this old > email: > > > > From: Ivan Voras > Date: April 28, 2009 3:53:14 AM PDT > To: freebsd-threads@freebsd.org > Cc: freebsd-net@freebsd.org, > freebsd-performance@freebsd.org > Subject: Re: FreeBSD 7.1 taskq em performance > > > > I have been hitting some barrier with FreeBSD 7.1 > network performance. > I > > have written an application which contains two kernel > threads that > takes > > mbufs directly from a network interface and forwards > to another > network > > interface. This idea is to simulate different network > environment. > > > > I have been using FreeBSD 6.4 amd64 and tested with an > Ixia box > > (specialised hardware firing very high packet rate). > The PC was a > Core2 2.6 > > GHz with dual ports Intel PCIE Gigabit network card. > It can manage up > to 1.2 > > million pps. > > > > I have a higher spec PC with FreeBSD 7.1 amd64 and > Quadcore 2.3 GHz > and > > PCIE Gigabit network card. The performance can only > achieve up to 600k > pps. > > I notice the 'taskq em0' and 'taskq em1' is solid 100% > CPU but it is > not in > > FreeBSD 6.4. > > > > In our case we are running FreeBSD 7.0, but we are seeing > our boxes > experience serious thread starvation issues as the nfe0 cpu > percentage > climbs steadily while cpu idle time drops at times to 0 > percent. This > email thread mentioned a patch for the em driver here: > > > > http://people.yandex-team.ru/~wawa/ > It means you're using your CPU up processing packets. There are any number of reasons for it; lock contention, poor general design, network stack contention. I'm not sure why you'd want to use a 64 bit build for a network application, but you'll have to track down the source by profiling or running focused tests to isolate your bottlenecks. Barney