From owner-freebsd-stable@FreeBSD.ORG Mon Jul 19 02:34:21 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 149D81065670 for ; Mon, 19 Jul 2010 02:34:21 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.emeryville.ca.mail.comcast.net (qmta09.emeryville.ca.mail.comcast.net [76.96.30.96]) by mx1.freebsd.org (Postfix) with ESMTP id F02738FC0C for ; Mon, 19 Jul 2010 02:34:20 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta09.emeryville.ca.mail.comcast.net with comcast id jdno1e00117UAYkA9eaLFX; Mon, 19 Jul 2010 02:34:20 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta13.emeryville.ca.mail.comcast.net with comcast id jeaK1e0053LrwQ28ZeaKtc; Mon, 19 Jul 2010 02:34:20 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 32D5F9B425; Sun, 18 Jul 2010 19:34:19 -0700 (PDT) Date: Sun, 18 Jul 2010 19:34:19 -0700 From: Jeremy Chadwick To: Mike Tancsa Message-ID: <20100719023419.GA91006@icarus.home.lan> References: <201007182108.o6IL88eG043887@lava.sentex.ca> <20100718211415.GA84127@icarus.home.lan> <201007182142.o6ILgDQW044046@lava.sentex.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201007182142.o6ILgDQW044046@lava.sentex.ca> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org Subject: Re: deadlock or bad disk ? RELENG_8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2010 02:34:21 -0000 On Sun, Jul 18, 2010 at 05:42:14PM -0400, Mike Tancsa wrote: > At 05:14 PM 7/18/2010, Jeremy Chadwick wrote: > > >Where exactly is your swap partition? > > On one of the areca raidsets. > > # swapctl -l > Device: 1024-blocks Used: > /dev/da0s1b 10485760 108 So is da0 actually a RAID volume "behind the scenes" on the Areca controller? How many disks are involved in that set? > >If you Google for "swap_pager: indefinite wait buffer: bufobj" you'll > >find this is a pretty well-established problem, but the situation varies > >per person. A common one is here (read the entire thread): > > > >http://www.mail-archive.com/freebsd-questions@freebsd.org/msg192481.html > > > >I have no advice as far as how to solve this problem. > > If feels like a disk issue, but SMART values all seem ok Well, the thread I linked you stated that the problem has to do with a controller or disk "taking too long". I have no idea what the threshold is. I suppose it could also indicate that your system is (possibly) running low on resources (RAM); I would imagine swap_pager would get called if a processes needed to be offloaded to swap. So maybe this is a system tuning thing more than a hardware thing. You should probably set up a series of monitoring scripts that monitor things like interrupt rate on devices, I/O statistics, and some general memory statistics to determine if processes are being swapped out excessively. vmstat and iostat would help here; see man page for relevant options (for swap stuff, vmstat -s). There's also systat with the -vmstat flag. > CLI> disk smart drv=1 > [...] Unrelated to the problem, but important to note: The SMART output from the Areca CLI is hardly useful (bordering on worthless); it only shows the adjusted/calculated values and not the actual raw values. Even if the CLI lets you print this information, I would still strongly suggest using smartctl. There's no indication the Areca CLI has a quirks database for each drive model/type. I'm also not sure if the Areca CLI can provide the SMART error log, self-test log, or the selective self-test log. > smartctl -a -d 3ware,1 /dev/twa0 Now I'm confused -- this indicates twa(4) is involved, not arcmsr(4). Can you please provide a verbose explanation of the configuration of the disks and controllers in this machine, including device and disk names and what they're associated with, plus if they're RAIDed in any way? Thanks. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |