From owner-svn-src-all@FreeBSD.ORG Tue Jan 20 19:02:45 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BCD65F3; Tue, 20 Jan 2015 19:02:45 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 11333BCE; Tue, 20 Jan 2015 19:02:45 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id F3C1DB990; Tue, 20 Jan 2015 14:02:43 -0500 (EST) From: John Baldwin To: Hans Petter Selasky Subject: Re: svn commit: r277213 - in head: share/man/man9 sys/kern sys/ofed/include/linux sys/sys Date: Tue, 20 Jan 2015 11:07:53 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <201501151532.t0FFWV2Y037455@svn.freebsd.org> <20150120090057.GD42409@kib.kiev.ua> <54BE21F0.6010602@selasky.org> In-Reply-To: <54BE21F0.6010602@selasky.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="windows-1252" Content-Transfer-Encoding: 7bit Message-Id: <201501201107.53553.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 20 Jan 2015 14:02:44 -0500 (EST) Cc: Konstantin Belousov , "svn-src-head@freebsd.org" , Adrian Chadd , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jan 2015 19:02:45 -0000 On Tuesday, January 20, 2015 4:37:52 am Hans Petter Selasky wrote: > On 01/20/15 10:00, Konstantin Belousov wrote: > > On Tue, Jan 20, 2015 at 08:58:34AM +0100, Hans Petter Selasky wrote: > >> On 01/20/15 08:51, Konstantin Belousov wrote: > >>> On Tue, Jan 20, 2015 at 05:30:25AM +0100, Hans Petter Selasky wrote: > >>>> On 01/19/15 22:59, Adrian Chadd wrote: > >>>>> Hi, > >>>>> > >>>>> Would you please check what the results of this are with CPU specific > >>>>> callwheels? > >>>>> > >>>>> I'm doing some 10+ gig traffic testing on -HEAD with RSS enabled (on > >>>>> ixgbe) and with this setup, the per-CPU TCP callwheel stuff is > >>>>> enabled. But all the callwheels are now back on clock(0) and so is the > >>>>> lock contention. :( > >>>>> > >>>>> Thanks, > >>>>> > >>>> > >>>> Hi, > >>>> > >>>> Like stated in the manual page, callout_reset_curcpu/on() does not work > >>>> with MPSAFE callouts any more! > >>> I.e. you 'fixed' some undeterminate bugs in callout migration by not > >>> doing migration at all anymore. > >>> > >>>> > >>>> You need to use callout_init_{mtx,rm,rw} and remove the custom locking > >>>> inside the callback in the TCP stack to get it working like before! > >>> > >>> No, you need to do this, if you think that whole callout KPI must be > >>> rototiled. It is up to the person who modifies the KPI, to ensure that > >>> existing code is not broken. > > Hi, > > It is not very hard to update existing callout clients and you can do it > too, if you need the extra bits of performance. > > Are there more API's than the TCP stack which you think needs an update > and are performance critical? > > >>> > >>> As I understand, currently we are back to the one-cpu callouts. > >>> Do other people consider this situation acceptable ? > > For the TCP stack - yes, but not for other clients like cv_timedwait() > and such. > > If you think you have a better way to solve the callout problems, please > tell me! In order for a callout to change its CPU you need a lock to > protect which CPU the callout is on. Instead of introducing a third lock > in the callout path, which will be a congestion point, to protect > against changing the CPU number, I decided that we will use the client's > mutex and the MPSAFE implies the client doesn't have any mutex. So it > won't work with callout clients which use the CALLOUT_MPSAFE flag. > Honestly CALLOUT_MPSAFE should not be used, because it leads to extra > complexity in the clients catching the race when tearing down the > callouts and any pending callbacks. > > >> > >> Please read the callout 9 manual page first. > > > > Assume I read it. How this changes any of my points above ? > > """ > > A change in the CPU selection cannot happen if this function is > > re-scheduled inside a callout function. Else the callback function given > > by the func argument will be executed on the same CPU like previously > > done. > > """ > > You cannot do this without fixing consumers. > > > > The code simply needs an update. It is not broken in any ways - right? > If it is not broken, fixing it is not that urgent. This is not at all acceptable. TCP callouts were the largest potential user of multi-cpu callouts and you've just broken them. Your proposed change to handle inp locks is not necessarily correct either since dropping the inp lock inside a callout introduces new races (now callout_stop doesn't have quite the same semantics as it does for other callout_init_*()). Given this, it seems that your fix just mostly disabled multi-CPU callouts, so it is not at all clear that you've actually fixed anything. :( -- John Baldwin