Date: Tue, 11 Jan 2005 16:38:41 -0500 From: Craig Reyenga <craig@puresimplicity.net> To: Matt Reimer <mreimer@vpop.net> Cc: freebsd-current@freebsd.org Subject: Re: Reproducible filesystem deadlock on RELENG_5 Message-ID: <20050111213841.GA6373@burnout.lan.bluemidnight.ca> In-Reply-To: <200501101547.18892.mreimer@vpop.net> References: <200501101547.18892.mreimer@vpop.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jan 10, 2005 at 03:47:18PM -0800, Matt Reimer wrote: > On a UP machine (P4, 128M RAM) running RELENG_5 (as of Friday), I am seeing > what looks like a hang or deadlock on a filesystem with 10 snapshots. Our > problems began when we ran out of disk space, resulting in a series of > these log messages: > > kernel: pid 39 (bufdaemon), uid 0 inumber 7277783 on /backup: filesystem > full > kernel: initiate_write_filepage: already started > > So I tried to delete a snapshot to free up some space, but then the kernel > began panicking. In my effort to workaround the panic, I disabled > softupdates. Then I came across the identical panic in a post by Kris > Kennaway > (http://lists.freebsd.org/pipermail/freebsd-current/2004-September/036946.html), > which he fixed by increasing KSTACK_PAGES. After increasing it to 31, the > kernel no longer panics, but instead filesystem access seems to deadlock: > if I try to even touch a file into existence on that partition, the touch > command hangs in state 'wdrain', and other attempts to access that > filesystem hang as well. This problem is 100% reproducible. > > How to proceed? Serial console access is available if someone wants to > tackle it. > > Matt > [sniiip] Hi, I am seeing this too, and it would appear that I've been beaten to sending a message about it. :) This is what my /var/log/kernel had to say, after rebooting from a (live|dead)lock. Jan 11 00:23:08 burnout kernel: >pid 44 (pagedaemon), uid 0 inumber 8298 on /var: filesystem full Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: I/O error 28 Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: residual I/O 65536 at 21872 Jan 11 00:23:08 burnout kernel: pid 44 (pagedaemon), uid 0 inumber 8298 on /var: filesystem full Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: I/O error 28 Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: residual I/O 65536 at 21872 Over and over and over. Of course, this log is On the /var FS itself. FreeBSD burnout 5.3-RELEASE-p2 FreeBSD 5.3-RELEASE-p2 #1: Wed Jan 5 18:44:27 EST 2005 craig@burnout:/usr/obj/usr/src/sys/BURNOUT5 i386 I'm not sure what other info to paste, my vfs.* sysctls are all defaults, except for vfs.usermount=1. -Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050111213841.GA6373>