From owner-svn-src-all@FreeBSD.ORG Wed Jan 21 00:50:00 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C49C4BE; Wed, 21 Jan 2015 00:50:00 +0000 (UTC) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D1AF07E6; Wed, 21 Jan 2015 00:49:59 +0000 (UTC) Received: by mail-ig0-f181.google.com with SMTP id hn18so6095450igb.2; Tue, 20 Jan 2015 16:49:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=quqiDEo6lbjiQU3p59cMdu3xkbwGtlVCKg33kJKiBXI=; b=XIVsc0kP68F80uzjgKxKv8yLvvgoR/So3T3KsGauRqWWdEP39yy1wA/LCGZtCmT4i2 5vg0xwjyelVCeFIN5vOY3U5Ok/1WrmN/3i9xFN3EXoHClvGDljB1zLwVu8C5AZ60EflQ NYNLXgQxiqbsmYbP+s+MDZpSGP1IA71l52fNGzWm9UG6D8WyFzhlBOUGCneM9R8Va1E0 08w2DCdsr6JzD7kriQcaQ/vcqYFC2/AAgqD9MgWv647pN2phuMBC6uUqp2xSIt7U1Gyq BqECdn/Gkaidffvr5bMoZLvyEGUZDlUDuaq1MGYvjX+P6LuVApMJUpACuazAdDP0HdzJ CutQ== MIME-Version: 1.0 X-Received: by 10.50.66.171 with SMTP id g11mr517250igt.49.1421801399255; Tue, 20 Jan 2015 16:49:59 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.36.78.14 with HTTP; Tue, 20 Jan 2015 16:49:59 -0800 (PST) In-Reply-To: <54BED6FB.8060401@selasky.org> References: <201501151532.t0FFWV2Y037455@svn.freebsd.org> <54BDD9E1.6090505@selasky.org> <20150120075126.GA42409@kib.kiev.ua> <20150120211137.GY15484@FreeBSD.org> <54BED6FB.8060401@selasky.org> Date: Tue, 20 Jan 2015 16:49:59 -0800 X-Google-Sender-Auth: NXKq1hKRrZ9-zvF905vKlDFggLY Message-ID: Subject: Re: svn commit: r277213 - in head: share/man/man9 sys/kern sys/ofed/include/linux sys/sys From: Adrian Chadd To: Hans Petter Selasky Content-Type: text/plain; charset=UTF-8 Cc: "src-committers@freebsd.org" , Jason Wolfe , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" , Gleb Smirnoff , Konstantin Belousov X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jan 2015 00:50:00 -0000 On 20 January 2015 at 14:30, Hans Petter Selasky wrote: > Backing out my callout API patch means we will for sure re-introduce an > unknown callout spinlock hang, as noted to me by several people. What do you > think about that? > > Maybe "Jason Wolfe" CC'ed can add to 10-stable w/o my patches: > > int > callout_reset_sbt_on(struct callout *c, sbintime_t sbt, sbintime_t > precision, > void (*ftn)(void *), void *arg, int cpu, int flags) > { > sbintime_t to_sbt, pr; > struct callout_cpu *cc; > int cancelled, direct; > > + cpu = timeout_cpu; /* XXX test code XXX */ > > cancelled = 0; > > And see if he observes a callout spinlock hang or not on his test setup. The > patch above should force all callouts to the same thread basically. Then we > could maybe see if single threading the callouts has anything to do with > solving the spinlock hang. > > The "rewritten" callout API still has all the features and capabilities the > old one had, when used as described in "man 9 callout". > > At the present moment I'm not technically convinced a backout is correct. > > Gleb: I think we would see far better results with high speed internet links > using TCP if we could extend the LRO (large receive offload) code to > accumulate more than 64KBytes worth of data per call to the TCP stack > instead of complaining about some callouts ending up on the same thread! > Actually I have a patch for that. You should totally try say, 100,000 active TCP connections on a box. See what happens to swi0 (clock). TL;DR - the lock contention sucks and it takes a chunk of the core up. The lock contention is highly not good. That's why I'd like to see both the callout stuff in its slightly-better-defined-and-sane state from you /and/ make it so TCP can use it. I'll have to double-check to see if the RSS stuff is all lined up correctly so we can use it when we create the callouts (well, at inpcb creation time, right), rather than when we first schedule them. Then we can experiment with having the initial CPU be specified at callout create time rather than expecting to be able to move it when we first schedule it. Or, hm, maybe have it so we don't have a CPU chosen until the first time we schedule the timeout, and if it hasn't been scheduled before, allow the CPU to be set? Because at that point we aren't migrating it off f timeout_cpu - it's never been added to it in the first place. -a