Date: Tue, 11 Oct 2016 15:11:45 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Julien Charbon <jch@freebsd.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info> Subject: Re: 11.0 stuck on high network load Message-ID: <20161011121145.GJ6177@zxy.spb.ru> In-Reply-To: <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> References: <20160928115909.GC54003@zxy.spb.ru> <a0425aad-a421-05bc-c1a8-c6fe06b83833@freebsd.org> <20161006111043.GH54003@zxy.spb.ru> <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 11, 2016 at 09:20:17AM +0200, Julien Charbon wrote: > Then threads are competing for the INP_WLOCK lock. For the example, > let's say the thread A wants to run tcp_input()/in_pcblookup_mbuf() and > racing for this INP_WLOCK: > > https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/in_pcb.c#L1964 > > And thread B wants to run tcp_timer_2msl()/tcp_close()/in_pcbdrop() and > racing for this INP_WLOCK: > > https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/tcp_timer.c#L323 > > That leads to two cases: > > o Thread A wins the race: > > Thread A will continue tcp_input() as usal and INP_DROPPED flags is > not set and inp is still in TCP hash table. > Thread B is waiting on thread A to release INP_WLOCK after finishing > tcp_input() processing, and thread B will continue > tcp_timer_2msl()/tcp_close()/in_pcbdrop() processing. > > o Thread B wins the race: > > Thread B runs tcp_timer_2msl()/tcp_close()/in_pcbdrop() and inp > INP_DROPPED is set and inp being removed from TCP hash table. > In parallel, thread A has found the inp in TCP hash before is was > removed, and waiting on the found inp INP_WLOCK lock. > Once thread B has released the INP_WLOCK lock, thread A gets this lock > and sees the INP_DROPPED flag and do "goto findpcb" but here because the > inp is not more in TCP hash table and it will not be find again by > in_pcblookup_mbuf(). > > Hopefully I am clear enough here. Thanks, very clear. Small qeustion: when both thread run on same CPU core, INP_WLOCK will be re-schedule? As I remeber race created by call tcp_twstart() at time of end tcp_close(), at path sofree()-tcp_usr_detach() and unexpected INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in tcp_twstart() After check source code I am found invocation of tcp_twstart() in sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c, sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c. Invocation from sys/netinet/tcp_stacks/fastpath.c and sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and now will be OK. Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see independed INP_WLOCK. Is this OK? Can be thread A wants do_peer_close() directed from chelsio IRQ handler, bypass tcp_input()?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161011121145.GJ6177>