From owner-freebsd-current@FreeBSD.ORG Tue Sep 13 05:43:21 2005 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0E98A16A41F; Tue, 13 Sep 2005 05:43:21 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from fileserver.fields.utoronto.ca (fileserver.fields.utoronto.ca [128.100.216.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9033F43D45; Tue, 13 Sep 2005 05:43:20 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from fields.fields.utoronto.ca (fields.localdomain [192.168.216.11]) by fileserver.fields.utoronto.ca (8.12.8/8.12.8/Fields 6.0) with ESMTP id j8D5hJ0r015831 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Sep 2005 01:43:20 -0400 Received: from obsecurity.dyndns.org (localhost.localdomain [127.0.0.1]) by fields.fields.utoronto.ca (8.12.8/8.12.8/Fields WS 6.0) with ESMTP id j8D5hJ6P027311; Tue, 13 Sep 2005 01:43:19 -0400 Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 90A1151215; Tue, 13 Sep 2005 01:43:18 -0400 (EDT) Date: Tue, 13 Sep 2005 01:43:18 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20050913054318.GA28259@xor.obsecurity.org> References: <20050911075157.GA93947@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="TB36FDmn/VVEgNH/" Content-Disposition: inline In-Reply-To: <20050911075157.GA93947@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: current@FreeBSD.org Subject: Re: 'swap_pager: indefinite wait buffer' with swapfile X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Sep 2005 05:43:21 -0000 --TB36FDmn/VVEgNH/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Sep 11, 2005 at 03:51:57AM -0400, Kris Kennaway wrote: > I configured a vnode-backed md and enabled swapping on it. A few > hours later after moderate swap use the console showed: >=20 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 889347, size: 8192 > [...repeated...] >=20 > The backing store was a sparse file, but there was ample space: >=20 > # ls -l /data2/swapfile > -rw-r--r-- 1 root wheel 17179869184 Sep 11 16:50 /data2/swapfile > # df /data2 > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/stripe/data 51666218 27042730 20490192 57% /data2 > # swapinfo > Device 1K-blocks Used Avail Capacity > /dev/da0b 6297480 949304 6297480 15% > /dev/md41 16777216 842544 16777216 5% > Total 23074696 1791848 21282848 8% I think these messages happen when a ton of stuff gets paged out at once, since the vnode backing store is going to be a significant bottleneck. I got a few dozen of these in groups of 5 or 6 over the past 24 hours, and the system seemed to be fine (i.e. no immediate panics or filesystem corruption from lost transactions). However, the system has now deadlocked. 0 c04188e0 0 0 0 0000200 [SLPQ vmwait 0xc05695e0][SLP] swapp= er db> wh 0 Tracing pid 0 tid 0 td 0xc0418c30 mi_switch() at mi_switch+0x2b0 sleepq_switch() at sleepq_switch+0xf4 sleepq_wait() at sleepq_wait+0x3c msleep() at msleep+0x378 vm_wait() at vm_wait+0xa8 scheduler() at scheduler+0x58 mi_startup() at mi_startup+0x12c btext() at btext+0x34 This is the md that I'm swapping onto: 9151 fffff8010fbf5a80 0 0 0 0000204 [SLPQ vmwait 0xc05695e0][SL= P] md41 db> wh 9151 Tracing pid 9151 tid 100359 td 0xfffff801047a30a0 mi_switch() at mi_switch+0x2b0 sleepq_switch() at sleepq_switch+0xf4 sleepq_wait() at sleepq_wait+0x3c msleep() at msleep+0x378 vm_wait() at vm_wait+0xa8 allocbuf() at allocbuf+0x614 getblk() at getblk+0x598 breadn() at breadn+0x58 bread() at bread+0x20 ffs_balloc_ufs2() at ffs_balloc_ufs2+0xcf0 ffs_write() at ffs_write+0x2a4 VOP_WRITE_APV() at VOP_WRITE_APV+0x120 mdstart_vnode() at mdstart_vnode+0x16c md_kthread() at md_kthread+0x1f8 fork_exit() at fork_exit+0x94 fork_trampoline() at fork_trampoline+0x8 db> Most other processes on the system are sleeping in various states and/or trying to swap, e.g.: 30588 fffff8010051ba80 0 30586 44971 0004000 [SLPQ vmwait 0xc05695e0][SL= P] bsdtar 30587 fffff800b05169f0 0 30586 44971 0004000 [SLPQ vmwait 0xc05695e0][SL= P] bsdtar 30586 fffff800d48d73e0 0 28045 44971 0004000 [SLPQ wait 0xfffff800d48d73= e0][SLP][SWAP] sh 30585 fffff80017de73e0 0 30583 45059 0004000 [SLPQ vmwait 0xc05695e0][SL= P] bsdtar 30584 fffff800425d6000 0 30583 45059 0004000 [SLPQ pipdwt 0xfffff8005088= e780][SLP] bsdtar 30583 fffff800b0517730 0 28564 45059 0004000 [SLPQ wait 0xfffff800b05177= 30][SLP][SWAP] sh Looks like swapfiles are broken. Kris --TB36FDmn/VVEgNH/ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFDJmb2Wry0BWjoQKURAiLOAKD+5yluKU8nWL5tf81zvLC3Dcc4iACfc+eF eW5Vo0/0ZtY09Ue7YjBuHEI= =Kknd -----END PGP SIGNATURE----- --TB36FDmn/VVEgNH/--