Date: Wed, 12 Oct 2016 14:35:11 +0200 From: Julien Charbon <jch@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info> Subject: Re: 11.0 stuck on high network load Message-ID: <62d8861c-673e-6d86-e96e-751399e505e5@freebsd.org> In-Reply-To: <20161012121322.GB57876@zxy.spb.ru> References: <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org> <20161012084045.GA57714@zxy.spb.ru> <f3c0e73a-5e6e-2190-aed3-499250c1764c@freebsd.org> <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru> <e4f1343c-636a-0970-856b-e65955f79e1a@freebsd.org> <20161012121322.GB57876@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --p0Bsep5oESfbx0NfeiJT4H3hKPE0HcIq2 Content-Type: multipart/mixed; boundary="EGjrQK3qasa9e1amPuPHTuUJUJQEj6Bc8"; protected-headers="v1" From: Julien Charbon <jch@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info> Message-ID: <62d8861c-673e-6d86-e96e-751399e505e5@freebsd.org> Subject: Re: 11.0 stuck on high network load References: <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org> <20161012084045.GA57714@zxy.spb.ru> <f3c0e73a-5e6e-2190-aed3-499250c1764c@freebsd.org> <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru> <e4f1343c-636a-0970-856b-e65955f79e1a@freebsd.org> <20161012121322.GB57876@zxy.spb.ru> In-Reply-To: <20161012121322.GB57876@zxy.spb.ru> --EGjrQK3qasa9e1amPuPHTuUJUJQEj6Bc8 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Slawa, On 10/12/16 2:13 PM, Slawa Olhovchenkov wrote: > On Wed, Oct 12, 2016 at 02:06:59PM +0200, Julien Charbon wrote: >>>>>>> sofree() call tcp_usr_detach() and in tcp_usr_detach() we have >>>>>>> unexpected INP_TIMEWAIT. >>>>>> >>>>>> I see, thus just for the context: The TCP stack in sys/dev/cxgb*= is a >>>>>> TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a >>>>>> separate/side TCP stack that is used only with TCP_OFFLOAD option.= >>>>>> >>>>>> This TOE TCP stack actually has its own set of detach()/input() >>>>>> functions and seems to check INP_DROPPED flag properly. I guess @= np >>>>>> check fixes in socket TCP stack and decides which one can also imp= act >>>>>> the Chelsio TOE TCP stack. Some bugs are only in socket TCP stack= , some >>>>>> are only in TOE TCP stack. >>>>> >>>>> I am fear about other direction -- setting INP_TIMEWAIT in Chelsio = TOE >>>>> TCP stack and impact this to >>>>> tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path. >>>> >>>> I see, I expect no problem on this side as tcp_timer_2msl() checks = the >>>> INP_TIMEWAIT flag and do not call tcp_close() if set. >>> >>> I am about case when at time of first INP_WUNLOCK() tcp_timer_2msl() >>> don't see INP_TIMEWAIT, call tcp_close(), tcp_close() do INP_WUNLOCK(= ) >>> and now Chelsio TOE take INP_WLOCK, do tcp_twstart() and set >>> INP_TIMEWAIT. After this tcp_timer_2msl resume and have unexpected >>> INP_TIMEWAIT in tcp_usr_detach(). >> >> Sure, basically the same bug that in classic TCP stack. If you think= >> it can happen, send an email describing that to np@ and he will check >> and fix that. He is a TOE TCP stack expert and I am not. In all case= s, >> if this issue is possible in TOE TCP stack context, the patch will be >> straightforward: If the INP_DROPPED flag is set do not call tcp_twsta= rt(). >> >> The current patch focuses only on the classic TCP stack. >=20 > May be current workaround (with logging) in tcp_usr_detach() is good > solutuion for preventing system lockout by similar bugs? Good question, the quick workaround in tcp_usr_detach() does not handle all the cases. If it reduces the number of crashes you can still find scenarios where it can have unexpected side effect. Long term solution is to enforce: If the inp has the INP_DROPPED flag just stop processing it and return. If you grep the INP_DROPPED flag in kernel sources, you can see that this test is already done in almost all tcp_*() processing functions but tcp_input(). I would say that even without this issue tcp_input() should check INP_DROPPED flags after INP_WLOCK anyway. Same for the TOE TCP stack, you are simply not supposed to process a inp with INP_DROPPED flag. -- Julien --EGjrQK3qasa9e1amPuPHTuUJUJQEj6Bc8-- --p0Bsep5oESfbx0NfeiJT4H3hKPE0HcIq2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX/i4HAAoJEKVlQ5Je6dhxB3MH/R4zM90cMHFNgzG2fkXaRFUz C0YVYDO6SxffHON9HVNQrTE+hV5D/n/FrtPHSqeCK+JnKi3SO2Td5fwvIb1FnhQs SF0iK640pJO5fpzrgYRX6FwmEM192XWylAiBTAzuMyPRpl4QHWEIxv7F4bt01Xtj QCV/PJt7JUipvv6Q7KCuCvLAOKOTOSjWRXwhatCeVXS9kKVVOmZrjARMxLRFnQJG a2Ww2fStHSodnmaJCIWWomU5vll+NNo0QHA6q6JCOMHFO9UD8evGHsP4vDuGQiyr 244KogxvJBU7K/F2rFhUekTR0tFcce/dGcyxP+RJO6wUL+ikSpBDpXSnB/jCQxU= =MQ14 -----END PGP SIGNATURE----- --p0Bsep5oESfbx0NfeiJT4H3hKPE0HcIq2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?62d8861c-673e-6d86-e96e-751399e505e5>