From owner-freebsd-net@FreeBSD.ORG Mon Sep 30 09:10:31 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 93FB0B41 for ; Mon, 30 Sep 2013 09:10:31 +0000 (UTC) (envelope-from anri@polynet.lviv.ua) Received: from postal.polynet.lviv.ua (postal.polynet.lviv.ua [195.22.112.6]) by mx1.freebsd.org (Postfix) with ESMTP id 297C3257D for ; Mon, 30 Sep 2013 09:10:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=polynet.lviv.ua; s=dkimpolynet; h=Subject:MIME-Version:Content-Type:To:From:Message-ID:Date; bh=kJz1aqgZcZyPpo9U4r8XNZlthjNPIqNzLCdZ8sgn4LA=; b=YUHVO21j6l+k5hPLjvbflYRvda9bPNqavGTLZxp19QP/fqyel7UZySsqgEtVWO9IrGXT3h2bLPObmk8u1+O9asKfGo/FAVU/aPnol50J2OL6AW7yRXaB7rXzmXDsxCYl65Sfxt2lJ0Gl2mI3Q+OsUTjjCDMN53eltbBgYLFyQ9M=; Received: from [192.168.0.55] (helo=localhost) by postal.polynet.lviv.ua with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1VQZUq-0002cj-Me for freebsd-net@freebsd.org; Mon, 30 Sep 2013 12:10:28 +0300 Received: from 195.22.112.3 ([195.22.112.3]) by webmail.polynet.lviv.ua (Horde Framework) with HTTP; Mon, 30 Sep 2013 12:10:27 +0300 Date: Mon, 30 Sep 2013 12:10:27 +0300 Message-ID: <20130930121027.Horde.taTEbLTlKvRSSUADQ3_q1_A@webmail.polynet.lviv.ua> From: Andriy Kopystyansky To: freebsd-net@freebsd.org User-Agent: Internet Messaging Program (IMP) H4 (5.0.22-git) Content-Type: text/plain; charset=KOI8-U; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-SA-Exim-Connect-IP: 192.168.0.55 X-SA-Exim-Mail-From: anri@polynet.lviv.ua X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on postal.polynet.lviv.ua X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT shortcircuit=ham autolearn=disabled version=3.3.2 Subject: Intel 82580 lagg(4) problem X-SA-Exim-Version: 4.2 X-SA-Exim-Scanned: Yes (on postal.polynet.lviv.ua) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Sep 2013 09:10:31 -0000 Greetings! We've got a strange problem with lagg(4) interfaces built on Intel's 82580 chipset igb (dual port 1Gb card). We cannot send over lagg interface more than ~1Gb/s, yet receiving ~2GB over it. Looks like something indicating the problem can be seen here: anri@host:[8:13]~#ifstat -i lagg0 -i igb1 -i igb3 1 lagg0 igb1 igb3 KB/s in KB/s out KB/s in KB/s out KB/s in KB/s out 9116.50 26515.16 4147.70 28871.43 5004.86 23683.31 8423.08 26544.62 3853.22 28980.60 4594.21 23520.97 8796.48 26395.28 4248.46 28344.00 4567.42 23978.17 Note (IN) traffic - everything's ok, about 50% on each igb interfaces and total summary on lagg0. But (OUT) traffic looks weird - there is more traffic on the single igb1 than on lagg0! Tried with default driver came with the system, also the new one igb-2.3.10 from Intel site - no luck. Testing lagg interface built on other cards (em, bce) on the same machine shows expected normal behavior for both directions. --------------------------------------- anri@host:[9:02]~#uname -a FreeBSD host 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #5: Fri Sep 20 14:52:18 EEST 2013 anri@host:/usr/obj/usr/src/sys/NODEONE amd64 -------------------------------------- Interfaces: lagg0: flags=8843 metric 0 mtu 1500 options=401bb ether 90:e2:ba:36:95:1d inet 172.16.0.1 netmask 0xffffffc0 broadcast 172.16.0.63 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: igb3 flags=1c laggport: igb1 flags=1c igb1: flags=8843 metric 0 mtu 1500 options=401bb ether 90:e2:ba:36:95:1d media: Ethernet autoselect (1000baseT ) status: active igb3: flags=8843 metric 0 mtu 1500 options=401bb ether 90:e2:ba:36:95:1d media: Ethernet autoselect (1000baseT ) status: active --------------------------------------- Hardware: IBM 3550 with 2x Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (amd64) pciconf -lv: igb1@pci0:19:0:1: class=0x020000 card=0x12b28086 chip=0x15168086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82580 Gigabit Network Connection' class = network subclass = ethernet --------------------------------------- /boot/loader.conf: hw.igb.rxd=4096 hw.igb.txd=4096 hw.igb.max_interrupt_rate=32000 hw.igb.num_queues=2 net.isr.maxthreads=8 net.isr.dispatch=direct net.isr.bindthreads=1 net.link.ifqmaxlen=10240 /etc/sysctl.conf: (relevant part) dev.igb.1.rx_processing_limit=-1 dev.igb.3.rx_processing_limit=-1 net.inet.ip.intr_queue_maxlen=512 net.route.netisr_maxqlen=4096 net.inet.ip.redirect=0 net.inet.ip.fastforwarding=1 kern.maxfiles=204800 kern.maxfilesperproc=200000 kern.ipc.nmbclusters=204800 kern.ipc.maxsockets=204800 Any tips, advices? thanks in advance, with best regards, Andriy Kopystyansky