From owner-freebsd-stable@FreeBSD.ORG Tue Mar 20 05:26:53 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C394106566C for ; Tue, 20 Mar 2012 05:26:53 +0000 (UTC) (envelope-from matt.thyer@gmail.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id 107858FC17 for ; Tue, 20 Mar 2012 05:26:52 +0000 (UTC) Received: by wibhq7 with SMTP id hq7so3746522wib.13 for ; Mon, 19 Mar 2012 22:26:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=18p9rfDwXVml3/GBfqxe4kj10e3/P+IdvfUd8+QluHs=; b=oRO8pAhtLwMSUb0SGK+hb36SSWlCt39wihkguN2wBtBysbiLxuPyFWchRk2Zduy5Nt Btd8qwD6HZivF9RkrP2eBn19q/0aen6xlLB0cBS8LefDgdPDIF3Wwp/Dp4ReEJxhvgIs 8MrYJ3v1YN1Wn774mlv5sjzxdUi6WEa42PBUeG4kKnFFOda1INB4FgWhujSVPKpy2SP+ enc6KOu5acW1vDQI7C2uARL/V7dAIlCMnvQuDTUFae5I1QYWre3hKsDXG1rRLb9lxNlg rvm4myH7eu6vn7LiPzrzYQVjJDTRwBedZV1kp3i8OEuAlPeqWDwivnUcxpfyIaFch16q kLDA== MIME-Version: 1.0 Received: by 10.216.132.8 with SMTP id n8mr8397445wei.36.1332221212275; Mon, 19 Mar 2012 22:26:52 -0700 (PDT) Received: by 10.216.229.10 with HTTP; Mon, 19 Mar 2012 22:26:52 -0700 (PDT) Date: Tue, 20 Mar 2012 15:56:52 +1030 Message-ID: From: Matt Thyer To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: 157k interrupts per second causing 60% CPU load on idle system X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Mar 2012 05:26:53 -0000 I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to r232477 (4th Mar 2012) and am finding that a system process called "intr" is now constantly using about 60% of 1 CPU starting a short time after reboot (possibly triggered by use of the samba server). When this starts, systat -vm 1 says that the system is 85% idle and 14% interrupt handling. It says that there's around 157k interrupts per second. After a reboot the system is back to it's normal state doing between 3 and 250 or so interrupts per second. The hardware is an Intel Core i3-530 (dual core @ 2.93 GHz with Hyperthreading) with 8 GB RAM (2x4GB) on a Gigabyte H55M-D2H rev 1.3 motherboard running the latest BIOS (F4). The system runs a GENERIC kernel with the following significant items in /boot/loader.conf: zfs_load="YES" aio_load="YES" ahci_load="YES" geom_mirror_load="YES" vfs.root.mountfrom="zfs:zroot" vboxdrv_load="YES" It has 2 x 300 GB disks for the system with GPT partitioning and zmirror for the OS ala http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror I have swap on a gmirror as I want swap to survive the loss of one system disk. The NAS data is on a raidz2 pool of 8 disks connected to a SuperMicro AOC-USAS2-L8i (flashed to behave as an AOC-USAS2-L8e). The system is basically a CIFS NAS with ports/net/samba36 built with AIO_SUPPORT and configured like: socket options = SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY min receivefile size=16384 use sendfile=true aio read size = 16384 aio write size = 16384 aio write behind = true The only other interesting workload on the box is a java Minecraft server using ports/java/jdk16. I'm going to try to reproduce the problem in a VM and binary search down to the revision where it started as soon as I can work out a reliable way to trigger the behaviour (as it doesn't start at boot time). Any idea what could be the cause ?