From owner-freebsd-stable@freebsd.org Wed Oct 12 08:18:34 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66591C0D3F1 for ; Wed, 12 Oct 2016 08:18:34 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE35FD34 for ; Wed, 12 Oct 2016 08:18:33 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-wm0-f52.google.com with SMTP id f193so16144297wmg.0 for ; Wed, 12 Oct 2016 01:18:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=X1bGVMaN2YuDZiyrnVxIRhKVSUV5+xvXsp4Mz3C5TKk=; b=TT02wam364AZOGoUk55WIpp3zzxXrbwhe9sxhog53WfuyOiuptvXbMMLx7iJ3m3KyL DIusOkpPD4R6TEc0mFcUN0sSEr8HTyTt2ZrfXYhzi/xH+LKLV/KRVdPHFLkZ23D4cjQY Olyk5c08iN4Wr8rvJsLOwPQ2+KBs3G6YyUeQOMgQ+L4enM3ceMBeTySXCVlUTH1DY7XW LETdjzCfDcTeHKQthg3XPiXOAmHKAWnOL/OQoDFmxIBt6P1Q6rpeHFpZEsNo/Ng6+eRK 4UOha+qruVi8ymrqKYSYcQqp8nNXwubA/IDiRGiWZFAQNwm5SCDjYrO7W32T8oRSTobk 5/qA== X-Gm-Message-State: AA6/9RmIxuBw9aYwC45NtfrmgNBk5+CZiVihib7PUlJ3Z9dhovjSv77Vq1G8i1Pb1PBKsw== X-Received: by 10.28.88.134 with SMTP id m128mr469141wmb.39.1476260305912; Wed, 12 Oct 2016 01:18:25 -0700 (PDT) Received: from [10.100.64.21] ([217.30.88.7]) by smtp.gmail.com with ESMTPSA id p78sm1609112wma.0.2016.10.12.01.18.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Oct 2016 01:18:25 -0700 (PDT) Subject: Re: 11.0 stuck on high network load To: Slawa Olhovchenkov References: <20160928115909.GC54003@zxy.spb.ru> <20161006111043.GH54003@zxy.spb.ru> <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara From: Julien Charbon Message-ID: Date: Wed, 12 Oct 2016 10:18:18 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161011121145.GJ6177@zxy.spb.ru> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 08:18:34 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts Content-Type: multipart/mixed; boundary="LWk23kBf77cRBBkv9C82XIU038BlJnf1A"; protected-headers="v1" From: Julien Charbon To: Slawa Olhovchenkov Cc: Konstantin Belousov , freebsd-stable@FreeBSD.org, hiren panchasara Message-ID: Subject: Re: 11.0 stuck on high network load References: <20160928115909.GC54003@zxy.spb.ru> <20161006111043.GH54003@zxy.spb.ru> <1431484c-c00e-24c5-bd76-714be8ae5ed5@freebsd.org> <20161010133220.GU54003@zxy.spb.ru> <23f1200e-383e-befb-b76d-c88b3e1287b0@freebsd.org> <20161010142941.GV54003@zxy.spb.ru> <52d634aa-639c-bef7-1f10-c46dbadc4d85@freebsd.org> <20161010173531.GI6177@zxy.spb.ru> <8143cd8f-c007-2378-b004-b2b037402d03@freebsd.org> <20161011121145.GJ6177@zxy.spb.ru> In-Reply-To: <20161011121145.GJ6177@zxy.spb.ru> --LWk23kBf77cRBBkv9C82XIU038BlJnf1A Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Slawa, On 10/11/16 2:11 PM, Slawa Olhovchenkov wrote: > On Tue, Oct 11, 2016 at 09:20:17AM +0200, Julien Charbon wrote: >> Then threads are competing for the INP_WLOCK lock. For the example, >> let's say the thread A wants to run tcp_input()/in_pcblookup_mbuf() an= d >> racing for this INP_WLOCK: >> >> https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/in_= pcb.c#L1964 >> >> And thread B wants to run tcp_timer_2msl()/tcp_close()/in_pcbdrop() a= nd >> racing for this INP_WLOCK: >> >> https://github.com/freebsd/freebsd/blob/release/11.0.0/sys/netinet/tcp= _timer.c#L323 >> >> That leads to two cases: >> >> o Thread A wins the race: >> >> Thread A will continue tcp_input() as usal and INP_DROPPED flags is >> not set and inp is still in TCP hash table. >> Thread B is waiting on thread A to release INP_WLOCK after finishing= >> tcp_input() processing, and thread B will continue >> tcp_timer_2msl()/tcp_close()/in_pcbdrop() processing. >> >> o Thread B wins the race: >> >> Thread B runs tcp_timer_2msl()/tcp_close()/in_pcbdrop() and inp >> INP_DROPPED is set and inp being removed from TCP hash table. >> In parallel, thread A has found the inp in TCP hash before is was >> removed, and waiting on the found inp INP_WLOCK lock. >> Once thread B has released the INP_WLOCK lock, thread A gets this lo= ck >> and sees the INP_DROPPED flag and do "goto findpcb" but here because t= he >> inp is not more in TCP hash table and it will not be find again by >> in_pcblookup_mbuf(). >> >> Hopefully I am clear enough here. >=20 > Thanks, very clear. > Small qeustion: when both thread run on same CPU core, INP_WLOCK will > be re-schedule? Hmm, a thread can re-scheduled but not a lock. Thus no sure I understand your question here. :) > As I remeber race created by call tcp_twstart() at time of end > tcp_close(), at path sofree()-tcp_usr_detach() and unexpected > INP_TIMEWAIT state in the tcp_usr_detach(). INP_TIMEWAIT set in tcp_tws= tart() Exactly, thus the current fix is: If you already have the INP_DROPPED flag set you are not allowed to call tcp_twstart(), actually it is a good candidate for a new INVARIANT. Let me add that. > After check source code I am found invocation of tcp_twstart() in > sys/netinet/tcp_stacks/fastpath.c, sys/netinet/tcp_input.c, > sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c, sys/dev/cxgbe/tom/t4_cpl_io.c. >=20 > Invocation from sys/netinet/tcp_stacks/fastpath.c and > sys/netinet/tcp_input.c guarded by INP_WLOCK in tcp_input(), and now > will be OK. >=20 > Invocation from sys/dev/cxgb/ulp/tom/cxgb_cpl_io.c and > sys/dev/cxgbe/tom/t4_cpl_io.c is not clear to me, I am see independed > INP_WLOCK. Is this OK? >=20 > Can be thread A wants do_peer_close() directed from chelsio IRQ > handler, bypass tcp_input()? If you look carefully INP_WLOCK is used in cxgb_cpl_io.c and t4_cpl_io.c before calling tcp_twstart(). -- Julien --LWk23kBf77cRBBkv9C82XIU038BlJnf1A-- --lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJX/fHPAAoJEKVlQ5Je6dhxftQIAOFpZ+iF1ve1hfyXLFdI7kSx 2WU04gzNQ40+NXmMJvdZ3hQovwNbXCAu3aM7rqHbOV016Nf8x23gBFTqH+5wgE4z VRrfWifZZk8k1UFXsx2c6u2iVAcfx4WqDnJg98EtYP0Nh6dRuKuqcvPlblchHQJy OO4Zxc8TmIASZrOy1ce8m/fBspLkshoh7c1UdLtXlpyMyboh9dA/B+hC3lN2tVf8 7ghELfaPletstCfTQh6emgxE+IBEBbjAkgChsTCTmCMDoCK4jQ5YtbdjGJ8ZfSLi vBciJ7jWZhRHlANW8ZHbydHXvkOBVCPakmvt+xa8NotQhUSzB4UKRFgZG9Cw9tM= =OisL -----END PGP SIGNATURE----- --lo6WsNmCMpKpkEu6WS6lHh4jDTQdQVNts--