Date: Mon, 30 Sep 2013 12:10:27 +0300 From: Andriy Kopystyansky <anri@polynet.lviv.ua> To: freebsd-net@freebsd.org Subject: Intel 82580 lagg(4) problem Message-ID: <20130930121027.Horde.taTEbLTlKvRSSUADQ3_q1_A@webmail.polynet.lviv.ua>
next in thread | raw e-mail | index | archive | help
Greetings!
We've got a strange problem with lagg(4) interfaces built on Intel's
82580 chipset igb (dual port 1Gb card).
We cannot send over lagg interface more than ~1Gb/s, yet receiving
~2GB over it.
Looks like something indicating the problem can be seen here:
anri@host:[8:13]~#ifstat -i lagg0 -i igb1 -i igb3 1
lagg0 igb1 igb3
KB/s in KB/s out KB/s in KB/s out KB/s in KB/s out
9116.50 26515.16 4147.70 28871.43 5004.86 23683.31
8423.08 26544.62 3853.22 28980.60 4594.21 23520.97
8796.48 26395.28 4248.46 28344.00 4567.42 23978.17
Note (IN) traffic - everything's ok, about 50% on each igb interfaces
and total summary on lagg0.
But (OUT) traffic looks weird - there is more traffic on the single
igb1 than on lagg0!
Tried with default driver came with the system, also the new one
igb-2.3.10 from Intel site - no luck.
Testing lagg interface built on other cards (em, bce) on the same
machine shows expected normal behavior for both directions.
---------------------------------------
anri@host:[9:02]~#uname -a
FreeBSD host 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #5: Fri Sep 20
14:52:18 EEST 2013 anri@host:/usr/obj/usr/src/sys/NODEONE amd64
--------------------------------------
Interfaces:
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 90:e2:ba:36:95:1d
inet 172.16.0.1 netmask 0xffffffc0 broadcast 172.16.0.63
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2,l3,l4
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 90:e2:ba:36:95:1d
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 90:e2:ba:36:95:1d
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
---------------------------------------
Hardware: IBM 3550 with 2x Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (amd64)
pciconf -lv:
igb1@pci0:19:0:1: class=0x020000 card=0x12b28086 chip=0x15168086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82580 Gigabit Network Connection'
class = network
subclass = ethernet
---------------------------------------
/boot/loader.conf:
hw.igb.rxd=4096
hw.igb.txd=4096
hw.igb.max_interrupt_rate=32000
hw.igb.num_queues=2
net.isr.maxthreads=8
net.isr.dispatch=direct
net.isr.bindthreads=1
net.link.ifqmaxlen=10240
/etc/sysctl.conf: (relevant part)
dev.igb.1.rx_processing_limit=-1
dev.igb.3.rx_processing_limit=-1
net.inet.ip.intr_queue_maxlen=512
net.route.netisr_maxqlen=4096
net.inet.ip.redirect=0
net.inet.ip.fastforwarding=1
kern.maxfiles=204800
kern.maxfilesperproc=200000
kern.ipc.nmbclusters=204800
kern.ipc.maxsockets=204800
Any tips, advices?
thanks in advance, with best regards,
Andriy Kopystyansky
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130930121027.Horde.taTEbLTlKvRSSUADQ3_q1_A>