From owner-freebsd-net@FreeBSD.ORG Thu Aug 16 09:56:56 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7628716A417 for ; Thu, 16 Aug 2007 09:56:56 +0000 (UTC) (envelope-from ginga@ginganet.org) Received: from mx.gate01.com (pip7.gyao.ne.jp [61.122.117.245]) by mx1.freebsd.org (Postfix) with ESMTP id 3114813C478 for ; Thu, 16 Aug 2007 09:56:55 +0000 (UTC) (envelope-from ginga@ginganet.org) Received: from [202.210.243.26] (helo=athena.ginganet.org) by smtp31.isp.us-com.jp with esmtp (Mail 4.41) id 1ILbab-0003KK-Hr for freebsd-net@freebsd.org; Thu, 16 Aug 2007 18:24:25 +0900 Received: from localhost (localhost.olympus.local [127.0.0.1]) by athena.ginganet.org (Postfix) with ESMTP id 5B8BE7BB9; Thu, 16 Aug 2007 18:24:25 +0900 (JST) X-Virus-Scanned: amavisd-new at ginganet.org Received: from athena.ginganet.org ([127.0.0.1]) by localhost (athena.ginganet.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m-fovP1xKwb0; Thu, 16 Aug 2007 18:24:18 +0900 (JST) Received: by athena.ginganet.org (Postfix, from userid 5003) id 380637BB3; Thu, 16 Aug 2007 18:24:18 +0900 (JST) Date: Thu, 16 Aug 2007 18:24:18 +0900 From: KAWAGUTI Ginga To: freebsd-net@freebsd.org Message-ID: <20070816092418.GH6523@ginganet.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.14 (2007-02-12) Cc: ginga-freebsd@ginganet.org Subject: Some Broadcom GbE NIC(bge driver) suffers packet loss at receiving X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2007 09:56:56 -0000 Hi. Some revision of Broadcom GbE NIC with bge driver seems to loss 1 packet per 1000 packet at receiving on some occasion. Sending doesn't have this problem(receiving only). This loss doesn't always happen, but when happens, it seems that loss is exactly 1 per 1000, as the last 3 digits of lost packets' sequence number are all same. (i.e. example of lost packet seq-No. are: 3284, 4284, 5284, 8284, 9284, ...) I've tested with: * tcpdump capturing of SmartBits, * ports/net/rude(network traffic generator/collector software) * owping(one-way-ping) This loss happens with: * BCM5704(?) @ FreeBSD/i386 6-stable (Aug. 2007) (6-stable of some months ago also suffered with this problem) * same type hardware (I tested with 2 HP DL385 servers, and happend with both H/W.) This loss doesn't happen with: * same hardware with Linux(Fedora-Core 5) * same server with different NIC (Intel GbE with em driver) also with same version of FreeBSD/i386 6-stable * same FreeBSD/i386 6-stable with some other Broadcom cards(Broadcom BCM5752) (not same PC/server, but other PC hardware) ------------------------------------------------------- More precise hardware information is given below: This happens with BCM5704 of this revision: (on-board with HP DL385 Opteron Server) bge0: mem 0xf7df0000-0xf7dfffff irq 28 at device 6.0 on pci3 bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf7df0000 miibus0: on bge0 brgphy0: on miibus0 brgphy0: OUI 0x000818, model 0x0019, rev. 0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: bpf attached bge0: Ethernet address: 00:17:a4:8f:22:bc ioapic2: routing intpin 0 (PCI IRQ 28) to vector 53 bge0: [MPSAFE] bge1: mem 0xf7de0000-0xf7deffff irq 29 at device 6.1 on pci3 bge1: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf7de0000 miibus1: on bge1 brgphy1: on miibus1 brgphy1: OUI 0x000818, model 0x0019, rev. 0 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: bpf attached bge1: Ethernet address: 00:17:a4:8f:22:bb ioapic2: routing intpin 1 (PCI IRQ 29) to vector 54 bge1: [MPSAFE] pciconf -lvc: bge0@pci3:6:0: class=0x020000 card=0x00d00e11 chip=0x164814e4 rev=0x10 hdr=0x00 vendor = 'Broadcom Corporation' device = 'BCM5704 NetXtreme Dual Gigabit Adapter' class = network subclass = ethernet cap 07[40] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction cap 01[48] = powerspec 2 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 8 messages, 64 bit Other bge cards that doesn't suffer with this problem: (HP xw8400 Workstation on-board) bge0: mem 0xfb600000-0xfb60ffff irq 17 at device 0.0 on pci31 bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xfb600000 bge0: Disabling fastboot bge0: Disabling fastboot miibus0: on bge0 brgphy0: on miibus0 brgphy0: OUI 0x000818, model 0x0010, rev. 0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: bpf attached bge0: Ethernet address: 00:17:08:2a:a3:a3 ioapic0: routing intpin 17 (PCI IRQ 17) to vector 49 bge0: [MPSAFE] pciconf -lvc: bge0@pci31:0:0: class=0x020000 card=0x3015103c chip=0x160014e4 rev=0x01 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom NetXtreme Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 2 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 8 messages, 64 bit cap 10[d0] = PCI-Express 1 endpoint ------------------------------------------------------- ####################################################### Example usage of ports/net/rude (and crude) to test this. ####################################################### (At packet sender:) sender% cat > rude.cfg << EOFEOF START NOW 1000 0030 ON 3002 10.1.1.1:10001 CONSTANT 10000 200 11000 0030 OFF EOFEOF sender% # Above config will send 10000 packets/sec for 10 secs. # ==> 100001 packets will be generated. # (+1 comes from rude's spec(bug?)) sender% rude -s rude.cfg (Run packet generator) rude version 0.70, Copyright (C) 1999 Juha Laine and Sampo Saaristo rude comes with ABSOLUTELY NO WARRANTY! This is free software, and you are welcome to redistribute it under GNU GENERAL PUBLIC LICENSE Version 2. F_ID: F_START: F_STOP: F_SPORT: F_DADD: F_DPORT: F_err: F_suc: F_seq: F_TYPE: [+ type params] 30 1187254729.594905 1187254739.594905 3002 129.60.75.70 10001 0 100001 100001 CBR [r:10000 s:200] sender% ####################################################### (At packet receiver; which have bge cards) receiver% crude -s 30 crude version 0.70, Copyright (C) 1999 Juha Laine and Sampo Saaristo crude comes with ABSOLUTELY NO WARRANTY! This is free software, and you are welcome to redistribute it under GNU GENERAL PUBLIC LICENSE Version 2. ^C (Press Ctrl+C after rude@sender has finished) Runtime statistics results: --------------------------- Flow_ID=30 Packets: received=100001 out-of-seq=0 lost(est)=0 Total bytes received=20000200 Sequence numbers: first=0 last=100000 Delay: average = -0.001333 jitter=0.000069 seconds Absolute maximum jitter=0.008760 seconds Throughput=2.00001e+06 Bps (from first to last packet received) receiver% ------------------------------------------------------- For good GbE-cards, packet received count should be equal with sender's packet count. For some bge cards mentioned above, lost packet percentage comes ~ 0.1%, and when looking to sequence numbers, the loss seem to happen every 1000 packets.