Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 12 Oct 2016 10:18:18 +0200
From:      Julien Charbon <jch@freebsd.org>
To:        Slawa Olhovchenkov <slw@zxy.spb.ru>
Cc:        Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info>
Subject:   Re: 11.0 stuck on high network load
Message-ID:  <f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org>
In-Reply-To: <20161011121145.GJ6177@zxy.spb.ru>
References:  <20160928115909.GC54003@zxy.spb.ru> <a0425aad-a421-05bc-c1a8-c6fe06b83833@freebsd.org> <20161006111043.GH54003@zxy.spb.ru> <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru>

next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts
Content-Type: multipart/mixed; boundary="LWk23kBf77cRBBkv9C82XIU038BlJnf1A";
 protected-headers="v1"
From: Julien Charbon <jch@freebsd.org>
To: Slawa Olhovchenkov <slw@zxy.spb.ru>
Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org,
 hiren panchasara <hiren@strugglingcoder.info>
Message-ID: <f1d9e34e-3d85-bd02-e660-6d647e4343fb@freebsd.org>
Subject: Re: 11.0 stuck on high network load
References: <20160928115909.GC54003@zxy.spb.ru>
 <a0425aad-a421-05bc-c1a8-c6fe06b83833@freebsd.org>
 <20161006111043.GH54003@zxy.spb.ru>
 <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org>
 <20161010133220.GU54003@zxy.spb.ru>
 <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org>
 <20161010142941.GV54003@zxy.spb.ru>
 <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org>
 <20161010173531.GI6177@zxy.spb.ru>
 <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org>
 <20161011121145.GJ6177@zxy.spb.ru>
In-Reply-To: <20161011121145.GJ6177@zxy.spb.ru>

--LWk23kBf77cRBBkv9C82XIU038BlJnf1A
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable


 Hi Slawa,

On 10/11/16 2:11 PM, Slawa Olhovchenkov wrote:
> On Tue, Oct 11, 2016 at 09:20:17AM +0200, Julien Charbon wrote:
>>  Then threads are competing for the INP_WLOCK lock.  For the example,
>> let's say the thread A wants to run tcp_input()/in_pcblookup_mbuf() an=
d
>> racing for this INP_WLOCK:
>>
>> https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/in_=
pcb.c#L1964
>>
>>  And thread B wants to run tcp_timer_2msl()/tcp_close()/in_pcbdrop() a=
nd
>> racing for this INP_WLOCK:
>>
>> https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/tcp=
_timer.c#L323
>>
>>  That leads to two cases:
>>
>>  o Thread A wins the race:
>>
>>   Thread A will continue tcp_input() as usal and INP_DROPPED flags is
>> not set and inp is still in TCP hash table.
>>   Thread B is waiting on thread A to release INP_WLOCK after finishing=

>> tcp_input() processing, and thread B will continue
>> tcp_timer_2msl()/tcp_close()/in_pcbdrop() processing.
>>
>>  o Thread B wins the race:
>>
>>   Thread B runs tcp_timer_2msl()/tcp_close()/in_pcbdrop() and inp
>> INP_DROPPED is set and inp being removed from TCP hash table.
>>   In parallel, thread A has found the inp in TCP hash before is was
>> removed, and waiting on the found inp INP_WLOCK lock.
>>   Once thread B has released the INP_WLOCK lock, thread A gets this lo=
ck
>> and sees the INP_DROPPED flag and do "goto findpcb" but here because t=
he
>> inp is not more in TCP hash table and it will not be find again by
>> in_pcblookup_mbuf().
>>
>>  Hopefully I am clear enough here.
>=20
> Thanks, very clear.
> Small qeustion: when both thread run on same CPU core, INP_WLOCK will
> be re-schedule?

 Hmm, a thread can re-scheduled but not a lock. Thus no sure I
understand your question here.  :)

> As I remeber race created by call tcp_twstart() at time of end
> tcp_close(), at path sofree()-tcp_usr_detach() and unexpected
> INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in tcp_tws=
tart()

 Exactly, thus the current fix is:  If you already have the INP_DROPPED
flag set you are not allowed to call tcp_twstart(), actually it is a
good candidate for a new INVARIANT.  Let me add that.

> After check source code I am found invocation of tcp_twstart() in
> sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c,
> sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c.
>=20
> Invocation from sys/netinet/tcp_stacks/fastpath.c and
> sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and now
> will be OK.
>=20
> Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and
> sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see independed
> INP_WLOCK. Is this OK?
>=20
> Can be thread A wants do_peer_close() directed from chelsio IRQ
> handler, bypass tcp_input()?

 If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and
t4_cpl_io.c before calling tcp_twstart().

--
Julien


--LWk23kBf77cRBBkv9C82XIU038BlJnf1A--

--lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJX/fHPAAoJEKVlQ5Je6dhxftQIAOFpZ+iF1ve1hfyXLFdI7kSx
2WU04gzNQ40+NXmMJvdZ3hQovwNbXCAu3aM7rqHbOV016Nf8x23gBFTqH+5wgE4z
VRrfWifZZk8k1UFXsx2c6u2iVAcfx4WqDnJg98EtYP0Nh6dRuKuqcvPlblchHQJy
OO4Zxc8TmIASZrOy1ce8m/fBspLkshoh7c1UdLtXlpyMyboh9dA/B+hC3lN2tVf8
7ghELfaPletstCfTQh6emgxE+IBEBbjAkgChsTCTmCMDoCK4jQ5YtbdjGJ8ZfSLi
vBciJ7jWZhRHlANW8ZHbydHXvkOBVCPakmvt+xa8NotQhUSzB4UKRFgZG9Cw9tM=
=OisL
-----END PGP SIGNATURE-----

--lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f1d9e34e-3d85-bd02-e660-6d647e4343fb>