From owner-svn-src-head@FreeBSD.ORG  Wed Jan 21 14:01:12 2015
Return-Path: <owner-svn-src-head@FreeBSD.ORG>
Delivered-To: svn-src-head@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 46EE5C0C;
 Wed, 21 Jan 2015 14:01:12 +0000 (UTC)
Received: from lauren.room52.net (lauren.room52.net [210.50.193.198])
 by mx1.freebsd.org (Postfix) with ESMTP id 0705A684;
 Wed, 21 Jan 2015 14:01:11 +0000 (UTC)
Received: from lgwl-lstewart2.corp.netflix.com
 (c110-22-60-167.eburwd6.vic.optusnet.com.au [110.22.60.167])
 by lauren.room52.net (Postfix) with ESMTPSA id D9FBA7E820;
 Thu, 22 Jan 2015 01:01:09 +1100 (EST)
Message-ID: <54BFB10C.9080004@freebsd.org>
Date: Thu, 22 Jan 2015 01:00:44 +1100
From: Lawrence Stewart <lstewart@freebsd.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:31.0) Gecko/20100101 Thunderbird/31.2.0
MIME-Version: 1.0
To: Adrian Chadd <adrian@freebsd.org>, 
 Hans Petter Selasky <hselasky@freebsd.org>
Subject: Re: svn commit: r277213 - in head: share/man/man9 sys/kern
 sys/ofed/include/linux sys/sys
References: <201501151532.t0FFWV2Y037455@svn.freebsd.org>	<CAJ-Vmok0GXZoojyi=jE=b5D-d338APztaf3Pw0_AAQ-173XSWw@mail.gmail.com>
 <CAJ-VmomOE6AzbEFw45VOtVbLTtDMbnixkYDVK951pSCkcMrQjQ@mail.gmail.com>
In-Reply-To: <CAJ-VmomOE6AzbEFw45VOtVbLTtDMbnixkYDVK951pSCkcMrQjQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Spam-Status: No, score=2.4 required=5.0 tests=DNS_FROM_AHBL_RHSBL,
 UNPARSEABLE_RELAY autolearn=no version=3.3.2
X-Spam-Level: **
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net
Cc: "svn-src-head@freebsd.org" <svn-src-head@freebsd.org>,
 "svn-src-all@freebsd.org" <svn-src-all@freebsd.org>,
 "src-committers@freebsd.org" <src-committers@freebsd.org>
X-BeenThere: svn-src-head@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: SVN commit messages for the src tree for head/-current
 <svn-src-head.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-head>,
 <mailto:svn-src-head-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-head/>
List-Post: <mailto:svn-src-head@freebsd.org>
List-Help: <mailto:svn-src-head-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-head>,
 <mailto:svn-src-head-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Jan 2015 14:01:12 -0000

On 01/20/15 09:22, Adrian Chadd wrote:
> Yeah, it looks like you set c_cpu to timeout_cpu in
> _callout_init_locked(), but then you only handle the case of the CPU
> being changed in certain circumstances. You aren't handling the CPU
> being initialised when the callout is first added.
> 
> And, callout_restart_async() calls callout_lock(), which calls
> CC_LOCK() on the callout CPU, which initially is zero. Then, it never
> seems to be 'moved' into the correct CPU, even though it's being
> called with a CPU id.
> 
> So, if I am reading this all correctly, you aren't really handling
> multi CPU callwheels at all. ;)
> 
> In my instance, I'm seeing quite a lot of lock contention between the
> userland threads, the network RX threads and the clock thread, all
> contending on a single callout wheel being used for TCP timers. One of
> the early goals for fixing up the RSS stuff in -HEAD was to make
> per-CPU (Well, per-RSS-bucket really) TCP timers not contend with the
> NIC RX processing path, and now it does. :(

To clarify, you're seeing this with net.inet.tcp.per_cpu_timers=1?

Cheers,
Lawrence