From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:35:43 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97CC7106566B; Thu, 15 Dec 2011 21:35:43 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id 4311C8FC12; Thu, 15 Dec 2011 21:35:42 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email2.allantgroup.com (8.14.4/8.14.4) with ESMTP id pBFLZenw043742 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Dec 2011 15:35:40 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.5/8.14.5) with ESMTP id pBFLZeLW064284 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Dec 2011 15:35:40 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.5/8.14.5/Submit) id pBFLZd3w064283; Thu, 15 Dec 2011 15:35:39 -0600 (CST) (envelope-from dan) Date: Thu, 15 Dec 2011 15:35:39 -0600 From: Dan Nelson To: Jeremy Chadwick Message-ID: <20111215213539.GJ53453@dan.emsphone.com> References: <4EEA5DD0.1040009@FreeBSD.org> <20111215210421.GA33083@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111215210421.GA33083@icarus.home.lan> X-OS: FreeBSD 8.2-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.97.2 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (email2.allantgroup.com [199.67.51.78]); Thu, 15 Dec 2011 15:35:40 -0600 (CST) X-Scanned-By: MIMEDefang 2.68 on 199.67.51.78 Cc: Doug Barton , freebsd-stable@freebsd.org Subject: Re: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:35:43 -0000 In the last episode (Dec 15), Jeremy Chadwick said: > On Thu, Dec 15, 2011 at 12:51:28PM -0800, Doug Barton wrote: > > Web server under heavy'ish load (7 on a 2 cpu system) running > > 8.2-RELEASE-p4 i386 I'm seeing this: > > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} > > > > Any ideas why the clock should be taking so much cpu? HZ=100 if that > > makes a difference ... > > Could be wrong, but I believe this correlates with IRQ 4. What does > vmstat -i show for a total and rate for irq4 if you run it, wait a few > seconds, then run it again? Does the number greatly/rapidly increase? That would be "irq4" in that case, though. "swi4" is just a software interrupt thread, and "clock" is the softclock callout handler. There are both KTR and DTrace logging functions in kern_timeout.c, so you could use either one to get a handle on what's eating your CPU. Busy-looping "procstat -k 12" for a few seconds might get you some useful stacks, as well. > Shot in the dark here, but the only thing I can think of that might > cause this is software being extremely aggressive with calls to things > like gettimeofday(2) or clock_gettime(2). Really not sure. ntpd maybe > (unlikely but possible)? Sort of grasping at straws here. -- Dan Nelson dnelson@allantgroup.com