From owner-freebsd-current@FreeBSD.ORG Mon Feb 12 15:33:23 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1129A16A407 for ; Mon, 12 Feb 2007 15:33:23 +0000 (UTC) (envelope-from freebsd-current@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 98AF913C4A8 for ; Mon, 12 Feb 2007 15:33:22 +0000 (UTC) (envelope-from freebsd-current@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HGdB1-00013K-3e for freebsd-current@freebsd.org; Mon, 12 Feb 2007 16:33:11 +0100 Received: from mulderlab.f5.com ([205.229.151.151]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 Feb 2007 16:33:11 +0100 Received: from atkin901 by mulderlab.f5.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 Feb 2007 16:33:11 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-current@freebsd.org From: Mark Atkinson Date: Mon, 12 Feb 2007 07:32:50 -0800 Lines: 123 Message-ID: References: <20070110120731.GA1515@shark.localdomain> <200701100910.13167.jhb@freebsd.org> <20070110155331.GA2762@shark.localdomain> <20070111004044.GA33964@cdnetworks.co.kr> <20070210020650.GA51110@cdnetworks.co.kr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7Bit X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: mulderlab.f5.com User-Agent: KNode/0.10.4 Sender: news Subject: Re: nve related LOR triggered by lots of small packets, and a hard hang X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 15:33:23 -0000 Pyun YongHyeon wrote: > On Fri, Feb 09, 2007 at 09:23:41AM -0800, Mark Atkinson wrote: > > Mark Atkinson wrote: > > > > > Pyun YongHyeon wrote: > > > > > >> On Wed, Jan 10, 2007 at 06:53:31PM +0300, Sergey Zaharchenko wrote: > > >> > Hello John! > > >> > > > >> > Wed, Jan 10, 2007 at 09:10:12AM -0500 you wrote: > > >> > [snip] > > >> > > Have you tried using nfe(4)? :) > > >> > > > >> > Now I have, and it works just fine, thanks (I somehow thought nfe > > >> > was specific to some platform). Why isn't it the default? Smaller > > >> > range of hardware supported? > > >> > > > >> > > >> AFAIK, nfe(4) supports more hardwares than that of nve(4). > > >> Try overhauled nfe(4) in the following URL. > > >> > > >> http://people.freebsd.org/~yongari/nfe/if_nfe.c > > >> http://people.freebsd.org/~yongari/nfe/if_nfereg.h > > >> http://people.freebsd.org/~yongari/nfe/if_nfevar.h > > >> > > >> The patch fixed serveral bugs in nfe(4) and it should perform better > > >> than nve(4). The following hardware features are supported. > > >> o TSO > > >> o Tx/Rx IP/TCP/UDP checksum offload > > >> o VLAN hardware tag insertion/stripping > > >> o Jumbo frame(up to 9100 bytes) > > >> > > >> It seems that the hardware supports MSI/MSI-X too but I don't have > > >> nForce hardwares that supports MSI/MSI-X so it's hard to implement/ > > >> experiment it. Accoring to the Shigeaki Tagashira, the author of > > >> FreeBSD nfe(4), his hardware claims to support 8 messages. I've > > >> checked Linux forcedeth driver to get hardware information for > > >> MSI/MSI-X but it I cound't understand the details. :-( > > >> > > > > > > I've been running into this hardlock LOR a lot recently on a TYAN > > > 2895 > > > (K8WE) based box. So I tried your patch to nfe on today's -current. > > > I tried a couple of small packet ping floods to a lan neighbor > > > under nfe and > > > it survived. Did fine with some large NFS over TCP transfers as > > > well. However, I'll leave it up and running to see if it keels over > > > in the future. [nfe dmesg snipped for brevity] > > After a day of running this, it became obvious the nfe driver patch has > > some > > sort of issue, at least with -current and this board. Although NFS > > speeds seemed reasonable, transfers over TCP from a webserver suffered > > some sort > > of very noticeable pause/send/pause/send... type problem that reduced > > transfers to about 6Kbyte/s. This problem went away when putting nve > > back into the kernel and retrying the same scenerio. > > > > Would you explain the scenario to reproduce it on my box? It didn't seem to be any sort of TCP induced delay (window sizes on both ends of the connection were suitable throughout the transfer), seemed more like an interrupt throttling or driver processing delay. With nve a large file transfer through nve0 over http with it acting as the server would show interrupt total rate irq21: nve0 ehci0 1254194 42 however during the same transfer with nfe I'd get only a rate of 5 Here are my kernel differences. The difference between the two is simply commenting out nve and enabling nfe or vice versa. I'm running natd with divert -- no dummynet or other ipfw functionality is enabled (it's there only for occasional traffic tests on this machine). I should possibly retry without divert/ipfw over nfe0 as well. vlan is also in kernel but not used. [root@marka-k8we conf]$ diff -u GENERIC K8WE --- GENERIC Wed Feb 7 12:14:17 2007 +++ K8WE Fri Feb 9 08:56:18 2007 @@ -18,10 +18,10 @@ # # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.461 2007/02/07 18:55:30 marcel Exp $ -cpu I486_CPU -cpu I586_CPU +#cpu I486_CPU +#cpu I586_CPU cpu I686_CPU -ident GENERIC +ident K8WE # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. @@ -295,3 +295,12 @@ device firewire # FireWire bus code device sbp # SCSI over FireWire (Requires scbus and da) device fwe # Ethernet over FireWire (non-standard!) + +#K8WE options +options IPFIREWALL +options IPDIVERT +options DUMMYNET +options IPFIREWALL_DEFAULT_TO_ACCEPT + +device vlan +device nfe -- Mark Atkinson atkin901@yahoo.com (!wired)?(coffee++):(wired);