Date: Wed, 12 Oct 2016 14:06:59 +0200 From: Julien Charbon <jch@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info> Subject: Re: 11.0 stuck on high network load Message-ID: <e4f1343c-636a-0970-856b-e65955f79e1a@freebsd.org> In-Reply-To: <20161012095233.GC57714@zxy.spb.ru> References: <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> <f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org> <20161012084045.GA57714@zxy.spb.ru> <f3c0e73a-5e6e-2190-aed3-499250c1764c@freebsd.org> <20161012092945.GB57714@zxy.spb.ru> <4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org> <20161012095233.GC57714@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0
Content-Type: multipart/mixed; boundary="S2SvsBPGCh1kCtCwqdd8V9n6muFopH9rD";
protected-headers="v1"
From: Julien Charbon <jch@freebsd.org>
To: Slawa Olhovchenkov <slw@zxy.spb.ru>
Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org,
hiren panchasara <hiren@strugglingcoder.info>
Message-ID: <e4f1343c-636a-0970-856b-e65955f79e1a@freebsd.org>
Subject: Re: 11.0 stuck on high network load
References: <20161010142941.GV54003@zxy.spb.ru>
<52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org>
<20161010173531.GI6177@zxy.spb.ru>
<8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org>
<20161011121145.GJ6177@zxy.spb.ru>
<f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org>
<20161012084045.GA57714@zxy.spb.ru>
<f3c0e73a-5e6e-2190-aed3-499250c1764c@freebsd.org>
<20161012092945.GB57714@zxy.spb.ru>
<4b0d4b58-6d13-3cd5-6991-27163f27acca@freebsd.org>
<20161012095233.GC57714@zxy.spb.ru>
In-Reply-To: <20161012095233.GC57714@zxy.spb.ru>
--S2SvsBPGCh1kCtCwqdd8V9n6muFopH9rD
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Hi Slawa,
On 10/12/16 11:52 AM, Slawa Olhovchenkov wrote:
> On Wed, Oct 12, 2016 at 11:42:38AM +0200, Julien Charbon wrote:
>> On 10/12/16 11:29 AM, Slawa Olhovchenkov wrote:
>>> On Wed, Oct 12, 2016 at 11:19:48AM +0200, Julien Charbon wrote:
>>>
>>>>> if INP_WLOCK is like spinlock -- this is dead lock.
>>>>> if INP_WLOCK is like mutex -- thread1 resheduled.
>>>>
>>>> Thanks, I understand you question now. No an interrupt cannot bypa=
ss a
>>>> lock: Here INP_WLOCK is like mutex -- thread1 resheduled.
>>>
>>> Thanks, nice.
>>>
>>>>>>> As I remeber race created by call tcp_twstart() at time of end
>>>>>>> tcp_close(), at path sofree()-tcp_usr_detach() and unexpected
>>>>>>> INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in t=
cp_twstart()
>>>>>>
>>>>>> Exactly, thus the current fix is: If you already have the INP_DR=
OPPED
>>>>>> flag set you are not allowed to call tcp_twstart(), actually it is=
a
>>>>>> good candidate for a new INVARIANT. Let me add that.
>>>>>>
>>>>>>> After check source code I am found invocation of tcp_twstart() in=
>>>>>>> sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c,
>>>>>>> sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c=
=2E
>>>>>>>
>>>>>>> Invocation from sys/netinet/tcp_stacks/fastpath.c and
>>>>>>> sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and =
now
>>>>>>> will be OK.
>>>>>>>
>>>>>>> Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and
>>>>>>> sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see indepe=
nded
>>>>>>> INP_WLOCK. Is this OK?
>>>>>>>
>>>>>>> Can be thread A wants do_peer_close() directed from chelsio IRQ
>>>>>>> handler, bypass tcp_input()?
>>>>>>
>>>>>> If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and
>>>>>> t4_cpl_io.c before calling tcp_twstart().
>>>>>
>>>>> Yes, and you remeber: sys/netinet/tcp_subr.c
>>>>>
>>>>> 1535 struct tcpcb *
>>>>> 1536 tcp_close(struct tcpcb *tp)
>>>>> 1537 {
>>>>> ...
>>>>> 1569 INP_WUNLOCK(inp);
>>>>> 1570 ACCEPT_LOCK();
>>>>> 1571 SOCK_LOCK(so);
>>>>> 1572 so->so_state &=3D ~SS_PROTOREF;
>>>>> 1573 sofree(so);
>>>>> 1574 return (NULL);
>>>>>
>>>>> sofree() call tcp_usr_detach() and in tcp_usr_detach() we have
>>>>> unexpected INP_TIMEWAIT.
>>>>
>>>> I see, thus just for the context: The TCP stack in sys/dev/cxgb* i=
s a
>>>> TOE (TCP Offload Engine?) TCP stack for Chelsio NICs, it is a
>>>> separate/side TCP stack that is used only with TCP_OFFLOAD option.
>>>>
>>>> This TOE TCP stack actually has its own set of detach()/input()
>>>> functions and seems to check INP_DROPPED flag properly. I guess @np=
>>>> check fixes in socket TCP stack and decides which one can also impac=
t
>>>> the Chelsio TOE TCP stack. Some bugs are only in socket TCP stack, =
some
>>>> are only in TOE TCP stack.
>>>
>>> I am fear about other direction -- setting INP_TIMEWAIT in Chelsio TO=
E
>>> TCP stack and impact this to
>>> tcp_timer_2msl()/tcp_close()/sofree()/tcp_usr_detach() path.
>>
>> I see, I expect no problem on this side as tcp_timer_2msl() checks th=
e
>> INP_TIMEWAIT flag and do not call tcp_close() if set.
>=20
> I am about case when at time of first INP_WUNLOCK() tcp_timer_2msl()
> don't see INP_TIMEWAIT, call tcp_close(), tcp_close() do INP_WUNLOCK()
> and now Chelsio TOE take INP_WLOCK, do tcp_twstart() and set
> INP_TIMEWAIT. After this tcp_timer_2msl resume and have unexpected
> INP_TIMEWAIT in tcp_usr_detach().
Sure, basically the same bug that in classic TCP stack. If you think
it can happen, send an email describing that to np@ and he will check
and fix that. He is a TOE TCP stack expert and I am not. In all cases,
if this issue is possible in TOE TCP stack context, the patch will be
straightforward: If the INP_DROPPED flag is set do not call tcp_twstart(=
).
The current patch focuses only on the classic TCP stack.
--
Julien
--S2SvsBPGCh1kCtCwqdd8V9n6muFopH9rD--
--QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJX/idwAAoJEKVlQ5Je6dhxHcoIALKEpseITVVpDlpMkteujLge
nKwhtpaTyxAh1S7CDbvuGoHSDcvmy8ajHW928lD+Y7zIT325Zk2LCBjxfNqcMoRz
rTXrFcph7nhEhHawEXCeKHhxP3VLmy76axfm13SG9stT0vWYOFwbdiEF1BiGrIZb
7Qu3JPumAGsW8CGPlhFu1kTXx7msVCxnCBC4N84UmhU/oVL6oKKyDlvNZ7barKXM
kOknO5B2JDujT/WispMGtcKjqwLbtrAlgustfPGBzxSKpMw4DyUT+ps8yZRY/Zyn
T8RduPXnvzwvTY5CufUoc3nYySvb6U8Nbxk9fmrlUsTNFIvkHlBs5WW3ezwgJPY=
=uWUU
-----END PGP SIGNATURE-----
--QUlFkUkrLXFqeCg5Mmgu2mfi4vc2xCgp0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e4f1343c-636a-0970-856b-e65955f79e1a>