From owner-freebsd-current@FreeBSD.ORG Tue Feb 13 00:39:34 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3AFA416A406 for ; Tue, 13 Feb 2007 00:39:34 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.183]) by mx1.freebsd.org (Postfix) with ESMTP id E78BC13C4B4 for ; Tue, 13 Feb 2007 00:39:33 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by py-out-1112.google.com with SMTP id f47so924864pye for ; Mon, 12 Feb 2007 16:39:33 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:date:from:to:cc:subject:message-id:reply-to:references:mime-version:content-type:content-disposition:in-reply-to:user-agent; b=fuu8qs52n+jJyDjsjjJhRm4uEPFczRVpiIWh/uPSZLTqrRFnbkbXnqgvGUEWfYr/U3/hQnvYWp+JY5TfL1L+w4MAIK57ywzhnQC4dp+JB0QIuSUJYKw1oclQ4c7jB3bnZpJ0dZxWluVWBb7XQaQQPI5Qux1Iu7yfJrUPOlYH2sc= Received: by 10.35.128.17 with SMTP id f17mr31585624pyn.1171327173032; Mon, 12 Feb 2007 16:39:33 -0800 (PST) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTP id 15sm40249698nzp.2007.02.12.16.39.30; Mon, 12 Feb 2007 16:39:32 -0800 (PST) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id l1D0fO3H062688 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Feb 2007 09:41:24 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id l1D0fN4D062684; Tue, 13 Feb 2007 09:41:23 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Tue, 13 Feb 2007 09:41:23 +0900 From: Pyun YongHyeon To: Mark Atkinson Message-ID: <20070213004123.GB62412@cdnetworks.co.kr> References: <20070110120731.GA1515@shark.localdomain> <200701100910.13167.jhb@freebsd.org> <20070110155331.GA2762@shark.localdomain> <20070111004044.GA33964@cdnetworks.co.kr> <20070210020650.GA51110@cdnetworks.co.kr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: freebsd-current@freebsd.org Subject: Re: nve related LOR triggered by lots of small packets, and a hard hang X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 00:39:34 -0000 On Mon, Feb 12, 2007 at 07:47:04AM -0800, Mark Atkinson wrote: > Pyun YongHyeon wrote: > > > On Fri, Feb 09, 2007 at 09:23:41AM -0800, Mark Atkinson wrote: > > > Mark Atkinson wrote: > > > > > > > Pyun YongHyeon wrote: > > > > > > > >> On Wed, Jan 10, 2007 at 06:53:31PM +0300, Sergey Zaharchenko wrote: > > > >> > Hello John! > > > >> > > > > >> > Wed, Jan 10, 2007 at 09:10:12AM -0500 you wrote: > > > >> > [snip] > > > >> > > Have you tried using nfe(4)? :) > > > >> > > > > >> > Now I have, and it works just fine, thanks (I somehow thought nfe > > > >> > was specific to some platform). Why isn't it the default? Smaller > > > >> > range of hardware supported? > > > >> > > > > >> > > > >> AFAIK, nfe(4) supports more hardwares than that of nve(4). > > > >> Try overhauled nfe(4) in the following URL. > > > >> > > > >> http://people.freebsd.org/~yongari/nfe/if_nfe.c > > > >> http://people.freebsd.org/~yongari/nfe/if_nfereg.h > > > >> http://people.freebsd.org/~yongari/nfe/if_nfevar.h > > > >> > > > >> The patch fixed serveral bugs in nfe(4) and it should perform better > > > >> than nve(4). The following hardware features are supported. > > > >> o TSO > > > >> o Tx/Rx IP/TCP/UDP checksum offload > > > >> o VLAN hardware tag insertion/stripping > > > >> o Jumbo frame(up to 9100 bytes) > > > >> > > > >> It seems that the hardware supports MSI/MSI-X too but I don't have > > > >> nForce hardwares that supports MSI/MSI-X so it's hard to implement/ > > > >> experiment it. Accoring to the Shigeaki Tagashira, the author of > > > >> FreeBSD nfe(4), his hardware claims to support 8 messages. I've > > > >> checked Linux forcedeth driver to get hardware information for > > > >> MSI/MSI-X but it I cound't understand the details. :-( > > > >> > > > > > > > > I've been running into this hardlock LOR a lot recently on a TYAN > > > > 2895 > > > > (K8WE) based box. So I tried your patch to nfe on today's -current. > > > > I tried a couple of small packet ping floods to a lan neighbor > > > > under nfe and > > > > it survived. Did fine with some large NFS over TCP transfers as > > > > well. However, I'll leave it up and running to see if it keels over > > > > in the future. > > > > > > > > pci128: on pcib6 > > > > pci128: physical bus=128 > > > > found-> vendor=0x10de, dev=0x005e, revid=0xa3 > > > > bus=128, slot=0, func=0 > > > > class=05-80-00, hdrtype=0x00, mfdev=0 > > > > cmdreg=0x0006, statreg=0x00b0, cachelnsz=0 (dwords) > > > > lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) > > > > found-> vendor=0x10de, dev=0x00d3, revid=0xa3 > > > > bus=128, slot=1, func=0 > > > > class=05-80-00, hdrtype=0x00, mfdev=1 > > > > cmdreg=0x000f, statreg=0x00a0, cachelnsz=0 (dwords) > > > > lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) > > > > map[14]: type 1, range 32, base 0xd8400000, size 12, enabled > > > > found-> vendor=0x10de, dev=0x0057, revid=0xa3 > > > > bus=128, slot=10, func=0 > > > > class=06-80-00, hdrtype=0x00, mfdev=0 > > > > cmdreg=0x0007, statreg=0x00b0, cachelnsz=0 (dwords) > > > > lattimer=0x00 (0 ns), mingnt=0x01 (250 ns), maxlat=0x14 (5000 > > > > ns) intpin=a, irq=5 > > > > powerspec 2 supports D0 D1 D2 D3 current D0 > > > > map[10]: type 1, range 32, base 0xd8401000, size 12, enabled > > > > map[14]: type 4, range 32, base 0x3000, size 3, enabled > > > > pcib6: matched entry for 128.10.INTA (src \\_SB_.PCI1.LMAC:0) > > > > pci_link22: Picked IRQ 52 with weight 0 > > > > ioapic3: Changing polarity for pin 20 to high > > > > pcib6: slot 10 INTA routed to irq 52 via \\_SB_.PCI1.LMAC > > > > found-> vendor=0x10de, dev=0x005d, revid=0xa3 > > > > bus=128, slot=14, func=0 > > > > class=06-04-00, hdrtype=0x01, mfdev=0 > > > > cmdreg=0x0107, statreg=0x0010, cachelnsz=16 (dwords) > > > > lattimer=0x00 (0 ns), mingnt=0x04 (1000 ns), maxlat=0x00 (0 > > > > ns) > > > > powerspec 2 supports D0 D3 current D0 > > > > MSI supports 2 messages, 64 bit > > > > pci128: at device 0.0 (no driver attached) > > > > pci128: at device 1.0 (no driver attached) > > > > nfe1: port > > > > 0x3000-0x3007 mem 0xd8 > > > > 401000-0xd8401fff irq 52 at device 10.0 on pci128 > > > > nfe1: Reserved 0x1000 bytes for rid 0x10 type 3 at 0xd8401000 > > > > nfe1: bpf attached > > > > e1: Ethernet address: 00:e0:81:57:d9:af > > > > miibus1: on nfe1 > > > > e1000phy1: PHY 1 on miibus1 > > > > e1000phy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > > > > 1000baseTX-FDX, auto > > > > ioapic3: routing intpin 20 (PCI IRQ 52) to vector 57 > > > > nfe1: [MPSAFE] > > > > nfe1: [FAST] > > > > > > After a day of running this, it became obvious the nfe driver patch has > > > some > > > sort of issue, at least with -current and this board. Although NFS > > > speeds seemed reasonable, transfers over TCP from a webserver suffered > > > some sort > > > of very noticeable pause/send/pause/send... type problem that reduced > > > transfers to about 6Kbyte/s. This problem went away when putting nve > > > back into the kernel and retrying the same scenerio. > > > > > > > Would you explain the scenario to reproduce it on my box? > > How about disabling checksum offload? > > After a few tests, it's all related to TSO ( segmentation offloading ) > > Turning that off, but leaving rxcsum and txcsum enabled works and performs > speedily. Thanks for the suggestion! > > Note that, for some reason NFS over tcp speeds didn't seem affected that > much only userland TCP seemed to be negatively affected. > Thanks for the report! I'll look into the TSO related part and let you know if I have some patches. > -- > Mark Atkinson > atkin901@yahoo.com > (!wired)?(coffee++):(wired); > -- Regards, Pyun YongHyeon