Date: Wed, 12 Aug 2009 14:35:21 -0700 From: Pyun YongHyeon <pyunyh@gmail.com> To: Peter Steele <psteele@webmail.maxiscale.com> Cc: freebsd-net@freebsd.org Subject: Re: nfe taskq performance issues Message-ID: <20090812213521.GG55129@michelle.cdnetworks.com> In-Reply-To: <B8A480488C0C6849826655761349EA431F6E00@owa.webmail.maxiscale.com> References: <B8A480488C0C6849826655761349EA431F6E00@owa.webmail.maxiscale.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 23, 2009 at 08:58:07AM -0700, Peter Steele wrote: > We've been hitting serious nfe taskq performance issues during stress > tests and in doing some research on the problem we came across this old > email: > > > > From: Ivan Voras <ivoras@freebsd.org> > Date: April 28, 2009 3:53:14 AM PDT > To: freebsd-threads@freebsd.org > Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org > Subject: Re: FreeBSD 7.1 taskq em performance > > > > I have been hitting some barrier with FreeBSD 7.1 network performance. > I > > have written an application which contains two kernel threads that > takes > > mbufs directly from a network interface and forwards to another > network > > interface. This idea is to simulate different network environment. > > > > I have been using FreeBSD 6.4 amd64 and tested with an Ixia box > > (specialised hardware firing very high packet rate). The PC was a > Core2 2.6 > > GHz with dual ports Intel PCIE Gigabit network card. It can manage up > to 1.2 > > million pps. > > > > I have a higher spec PC with FreeBSD 7.1 amd64 and Quadcore 2.3 GHz > and > > PCIE Gigabit network card. The performance can only achieve up to 600k > pps. > > I notice the 'taskq em0' and 'taskq em1' is solid 100% CPU but it is > not in > > FreeBSD 6.4. > > > > In our case we are running FreeBSD 7.0, but we are seeing our boxes > experience serious thread starvation issues as the nfe0 cpu percentage > climbs steadily while cpu idle time drops at times to 0 percent. This > email thread mentioned a patch for the em driver here: > > > > http://people.yandex-team.ru/~wawa/ > <http://people.yandex-team.ru/%7Ewawa/>; > > > > Does anyone know if this patch will work with the nfe driver? > That's for em(4). AFAIK all nfe(4) controllers lacks intelligent interrupts moderation so driver should be prepared to handle excessive interrupt loads. I'm not sure whether NVIDIA ethernet controllers really lacks efficient interrupt mitigation mechanism but it seems Linux also faces the same hardware problem. As you might know there is no publicly available data sheet for NVIDIA controllers so setting it right looks very hard to me.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090812213521.GG55129>