Date: Mon, 16 Mar 2015 11:37:16 +0200 From: Vitalii Duk <mlevel.ars@gmail.com> To: Konstantin Kulikov <k.kulikov2@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Unbalanced LACP link Message-ID: <CAH6gFOkksMYMt8bKh6=eytb9KcbM=1CEpsztFqR%2BSZ5PvXcahw@mail.gmail.com> In-Reply-To: <CAD%2BeXGRB4BD7xRL14KkLo-7uMWWEwZx97Y01g4dywR3-KGmTuA@mail.gmail.com> References: <CAH6gFOnpQ9LrpuWOe6w2nQ42Yz%2BhKSoy4WcBOTSEOkMMxnV8sA@mail.gmail.com> <CAD%2BeXGRB4BD7xRL14KkLo-7uMWWEwZx97Y01g4dywR3-KGmTuA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I've changed use_flowid to 0 and it helped! But isn't it setting significant? In a description it says "Shift flowid bits to prevent multiqueue collisions". On 16 March 2015 at 09:50, Konstantin Kulikov <k.kulikov2@gmail.com> wrote: > Hello, Vitalii. > > Make sure following sysctl are set to zero: > sysctl net.link.lagg.default_use_flowid=0 > sysctl net.link.lagg.0.use_flowid=0 > sysctl net.link.lagg.1.use_flowid=0 > > Then adjust lagghash depending on your configuration. > For example if you freebsd machine acts as router with single, default, > route, > you should set lagghash to l3, since for l2 both src and dst MACs will > always be same and you'd get unbalanced load. > > On Sun, Mar 15, 2015 at 1:38 PM, Vitalii Duk <mlevel.ars@gmail.com> wrote: > > Hello, guys. > > I have a problem with my LACP link (2*1GbE) between FreeBSD 10 and D-Link > > DGS-3610-26G switch. This link worked fine last two years on FreeBSD > > 9.2-STABLE, but when I've updated my FreeBSD to 10.1-RELEASE traffic > > started to go mostly through one interface. > > > > lagg0 - input interface (no IP-address, only VLANs). > > lagg1 - output interface. > > > > [router]# ifconfig lagg0 > > lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > > > options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO> > > ether 90:e2:ba:02:d0:ae > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > media: Ethernet autoselect > > status: active > > laggproto lacp lagghash l2 > > laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > > > [#router]# ifconfig lagg1 > > lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > > > options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO> > > ether 00:1b:21:93:52:d4 > > inet 172.16.204.9 netmask 0xfffffff8 broadcast 172.16.204.15 > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > media: Ethernet autoselect > > status: active > > laggproto lacp lagghash l2 > > laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > > > [router]# netstat -hw1 -I igb0 > > input igb0 output > > packets errs idrops bytes packets errs bytes colls > > 27K 0 0 12M 5 0 350 0 > > 26K 0 0 12M 7 0 693 0 > > 29K 0 0 12M 8 0 838 0 > > 29K 0 0 11M 10 0 1.0K 0 > > 27K 0 0 10M 7 0 770 0 > > > > As you can see most traffic is going through igb1 (lagg0) and igb3 > (lagg1) > > interfaces. In peak times there are more that 1 Gbps of traffic mostly > > through one 1GbE interface, and it cause an errors. So this link is > > assymetric. > > > > [router]# netstat -hw1 -I igb1 > > input igb1 output > > packets errs idrops bytes packets errs bytes colls > > 22K 0 0 8.5M 63K 0 69M 0 > > 20K 0 0 7.7M 58K 0 62M 0 > > 20K 0 0 7.3M 57K 0 61M 0 > > 19K 0 0 6.8M 56K 0 61M 0 > > 18K 0 0 6.5M 57K 0 62M 0 > > > > [router]# netstat -hw1 -I igb2 > > input igb2 output > > packets errs idrops bytes packets errs bytes colls > > 40K 0 0 49M 1 0 93 0 > > 41K 0 0 51M 1 0 88 0 > > 38K 0 0 45M 1 0 88 0 > > 33K 0 0 39M 2 0 174 0 > > 31K 0 0 35M 1 0 89 0 > > > > [router]# netstat -hw1 -I igb3 > > input igb3 output > > packets errs idrops bytes packets errs bytes colls > > 32K 0 0 34M 44K 0 18M 0 > > 29K 0 0 30M 45K 0 18M 0 > > 32K 0 0 34M 48K 0 17M 0 > > 28K 0 0 29M 42K 0 15M 0 > > 25K 0 0 26M 40K 0 16M 0 > > > > [router]# netstat -hw1 -I lagg0 > > input lagg0 output > > packets errs idrops bytes packets errs bytes colls > > 45K 0 0 17M 62K 0 69M 0 > > 46K 0 0 17M 66K 0 75M 0 > > 43K 0 0 17M 62K 0 69M 0 > > 50K 0 0 19M 71K 1 82M 0 > > 46K 0 0 18M 66K 0 75M 0 > > > > [router]# netstat -hw1 -I lagg1 > > input lagg1 output > > packets errs idrops bytes packets errs bytes colls > > 70K 0 0 79M 49K 0 18M 0 > > 71K 0 0 79M 49K 0 19M 0 > > 79K 0 0 91M 54K 0 20M 0 > > 65K 0 0 72M 45K 0 17M 0 > > 65K 0 0 72M 45K 0 17M 0 > > > > All igb interfaces are the same: > > igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086 > > rev=0x01 hdr=0x00 > > vendor = 'Intel Corporation' > > device = '82576 Gigabit Network Connection' > > class = network > > subclass = ethernet > > > > I've tried to disable/enable strict_mode, put switch into active/passive > > mode and trying to set duplex/speed manyally. Nothing helps. > > Also I've changed lagghash from l2,l3,l4 to l2 only on FreeBSD because > > switch supports only l2 load-balancing, but it also doesn't helps. > > > > net.link.lagg.lacp.debug: 0 > > net.link.lagg.0.lacp.lacp_strict_mode: 0 > > net.link.lagg.0.lacp.debug.rx_test: 0 > > net.link.lagg.0.lacp.debug.tx_test: 0 > > net.link.lagg.1.lacp.lacp_strict_mode: 0 > > net.link.lagg.1.lacp.debug.rx_test: 0 > > net.link.lagg.1.lacp.debug.tx_test: 0 > > > > I've read that there were some problems with LACP on FreeBSD 10. > > > > Information from the switch: > > > > # show lacp summary > > Aggregate port 6: > > > > Local information: > > LACP port Oper Port Port > > Port Flags State Priority Key Number State > > ---------------------------------------------------------------------- > > Gi0/18 SP bndl 4096 0x6 0x12 0x3c > > Gi0/22 SP bndl 4096 0x6 0x16 0x3c > > > > Partner information: > > LACP port Oper Port Port > > Port Flags Priority Dev ID Key Number State > > --------------------------------------------------------------------- > > Gi0/18 SA 32768 001b.2193.52d4 0x12b 0x3 0x3d > > Gi0/22 SA 32768 001b.2193.52d4 0x12b 0x4 0x3d > > > > Aggregate port 7: > > > > Local information: > > LACP port Oper Port Port > > Port Flags State Priority Key Number State > > ---------------------------------------------------------------------- > > Gi0/19 SP bndl 4096 0x7 0x13 0x3c > > Gi0/21 SP bndl 4096 0x7 0x15 0x3c > > > > Partner information: > > LACP port Oper Port Port > > Port Flags Priority Dev ID Key Number State > > --------------------------------------------------------------------- > > Gi0/19 SA 32768 90e2.ba02.d0ae 0x10b 0x1 0x3d > > Gi0/21 SA 32768 90e2.ba02.d0ae 0x10b 0x2 0x3d > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAH6gFOkksMYMt8bKh6=eytb9KcbM=1CEpsztFqR%2BSZ5PvXcahw>