Date: Sun, 15 Mar 2015 12:38:42 +0200 From: Vitalii Duk <mlevel.ars@gmail.com> To: freebsd-net@freebsd.org Subject: Unbalanced LACP link Message-ID: <CAH6gFOnpQ9LrpuWOe6w2nQ42Yz%2BhKSoy4WcBOTSEOkMMxnV8sA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, guys.
I have a problem with my LACP link (2*1GbE) between FreeBSD 10 and D-Link
DGS-3610-26G switch. This link worked fine last two years on FreeBSD
9.2-STABLE, but when I've updated my FreeBSD to 10.1-RELEASE traffic
started to go mostly through one interface.
lagg0 - input interface (no IP-address, only VLANs).
lagg1 - output interface.
[router]# ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
ether 90:e2:ba:02:d0:ae
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
[#router]# ifconfig lagg1
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
ether 00:1b:21:93:52:d4
inet 172.16.204.9 netmask 0xfffffff8 broadcast 172.16.204.15
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
[router]# netstat -hw1 -I igb0
input igb0 output
packets errs idrops bytes packets errs bytes colls
27K 0 0 12M 5 0 350 0
26K 0 0 12M 7 0 693 0
29K 0 0 12M 8 0 838 0
29K 0 0 11M 10 0 1.0K 0
27K 0 0 10M 7 0 770 0
As you can see most traffic is going through igb1 (lagg0) and igb3 (lagg1)
interfaces. In peak times there are more that 1 Gbps of traffic mostly
through one 1GbE interface, and it cause an errors. So this link is
assymetric.
[router]# netstat -hw1 -I igb1
input igb1 output
packets errs idrops bytes packets errs bytes colls
22K 0 0 8.5M 63K 0 69M 0
20K 0 0 7.7M 58K 0 62M 0
20K 0 0 7.3M 57K 0 61M 0
19K 0 0 6.8M 56K 0 61M 0
18K 0 0 6.5M 57K 0 62M 0
[router]# netstat -hw1 -I igb2
input igb2 output
packets errs idrops bytes packets errs bytes colls
40K 0 0 49M 1 0 93 0
41K 0 0 51M 1 0 88 0
38K 0 0 45M 1 0 88 0
33K 0 0 39M 2 0 174 0
31K 0 0 35M 1 0 89 0
[router]# netstat -hw1 -I igb3
input igb3 output
packets errs idrops bytes packets errs bytes colls
32K 0 0 34M 44K 0 18M 0
29K 0 0 30M 45K 0 18M 0
32K 0 0 34M 48K 0 17M 0
28K 0 0 29M 42K 0 15M 0
25K 0 0 26M 40K 0 16M 0
[router]# netstat -hw1 -I lagg0
input lagg0 output
packets errs idrops bytes packets errs bytes colls
45K 0 0 17M 62K 0 69M 0
46K 0 0 17M 66K 0 75M 0
43K 0 0 17M 62K 0 69M 0
50K 0 0 19M 71K 1 82M 0
46K 0 0 18M 66K 0 75M 0
[router]# netstat -hw1 -I lagg1
input lagg1 output
packets errs idrops bytes packets errs bytes colls
70K 0 0 79M 49K 0 18M 0
71K 0 0 79M 49K 0 19M 0
79K 0 0 91M 54K 0 20M 0
65K 0 0 72M 45K 0 17M 0
65K 0 0 72M 45K 0 17M 0
All igb interfaces are the same:
igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82576 Gigabit Network Connection'
class = network
subclass = ethernet
I've tried to disable/enable strict_mode, put switch into active/passive
mode and trying to set duplex/speed manyally. Nothing helps.
Also I've changed lagghash from l2,l3,l4 to l2 only on FreeBSD because
switch supports only l2 load-balancing, but it also doesn't helps.
net.link.lagg.lacp.debug: 0
net.link.lagg.0.lacp.lacp_strict_mode: 0
net.link.lagg.0.lacp.debug.rx_test: 0
net.link.lagg.0.lacp.debug.tx_test: 0
net.link.lagg.1.lacp.lacp_strict_mode: 0
net.link.lagg.1.lacp.debug.rx_test: 0
net.link.lagg.1.lacp.debug.tx_test: 0
I've read that there were some problems with LACP on FreeBSD 10.
Information from the switch:
# show lacp summary
Aggregate port 6:
Local information:
LACP port Oper Port Port
Port Flags State Priority Key Number State
----------------------------------------------------------------------
Gi0/18 SP bndl 4096 0x6 0x12 0x3c
Gi0/22 SP bndl 4096 0x6 0x16 0x3c
Partner information:
LACP port Oper Port Port
Port Flags Priority Dev ID Key Number State
---------------------------------------------------------------------
Gi0/18 SA 32768 001b.2193.52d4 0x12b 0x3 0x3d
Gi0/22 SA 32768 001b.2193.52d4 0x12b 0x4 0x3d
Aggregate port 7:
Local information:
LACP port Oper Port Port
Port Flags State Priority Key Number State
----------------------------------------------------------------------
Gi0/19 SP bndl 4096 0x7 0x13 0x3c
Gi0/21 SP bndl 4096 0x7 0x15 0x3c
Partner information:
LACP port Oper Port Port
Port Flags Priority Dev ID Key Number State
---------------------------------------------------------------------
Gi0/19 SA 32768 90e2.ba02.d0ae 0x10b 0x1 0x3d
Gi0/21 SA 32768 90e2.ba02.d0ae 0x10b 0x2 0x3d
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAH6gFOnpQ9LrpuWOe6w2nQ42Yz%2BhKSoy4WcBOTSEOkMMxnV8sA>