From owner-freebsd-fs@FreeBSD.ORG Sun Jun 1 17:12:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33C3A106564A for ; Sun, 1 Jun 2008 17:12:11 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.237]) by mx1.freebsd.org (Postfix) with ESMTP id F049C8FC17 for ; Sun, 1 Jun 2008 17:12:10 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so685254rvf.43 for ; Sun, 01 Jun 2008 10:12:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=VALIKSf9fimpPwda2LVINne13kvxJPSS9j3KTo+GYOE=; b=mOzWGE6XUypAzPQwE+KxCEkBOH6JWn5Pxa8IXItZl3onBKKcmFW00IaBxABmcXfXxVmljJRXMUxN21CjVF5CHyCQhOI9sxAOaRS1EwCr3QCtUM4TCkg/d/H9keSkGRbrNG8ptNXDzBA+UhwB5HxOBa+rBtnHzAbldYeFnKBfZDU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=tcbVYvReepsRnlpFvmTeRh1TebjsZonNLXVLFlCe50rGh1fhn8MZjlKLQfIhaQ50P+BnpupFE7Ojl2TspZXoFLa6Obp90DMOEElS48DJuVnj6DtC/gn2XnYr3pP1qpu6S9LFu3Q9MXGdFtrreCvHPBTWV1M/5NFxiSmGDLror20= Received: by 10.140.157.1 with SMTP id f1mr4399960rve.220.1212338658810; Sun, 01 Jun 2008 09:44:18 -0700 (PDT) Received: by 10.140.207.1 with HTTP; Sun, 1 Jun 2008 09:44:18 -0700 (PDT) Message-ID: <2e77fc10806010944p5fde4782nffa437230ff36362@mail.gmail.com> Date: Sun, 1 Jun 2008 12:44:18 -0400 From: "Niki Denev" Sender: ndenev@gmail.com To: "Pawel Jakub Dawidek" In-Reply-To: <20080530065439.GB3596@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <2e77fc10802120304n32fd1c42m52e6bc617ba07c35@mail.gmail.com> <20080530065439.GB3596@garage.freebsd.pl> X-Google-Sender-Auth: 82e85300968cde0a Cc: freebsd-fs@freebsd.org Subject: Re: ZFS panic when changing zfs dataset mountpoint X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jun 2008 17:12:11 -0000 On Fri, May 30, 2008 at 2:54 AM, Pawel Jakub Dawidek wrote: > On Tue, Feb 12, 2008 at 01:04:38PM +0200, Niki Denev wrote: >> Hi, >> >> I got the following panic when trying to set/change the mountpoint >> property of a dataset. >> >> I did : >> # zfs set mountpoint=/usr/ports zfs2/ports >> and the machine crashed. >> >> The datased had one snapshot taken. >> >> Here is what i was able to extract from the dump : > [...] >> #8 0xffffffff804800d5 in _sx_xlock (sx=0xa0, opts=0, >> file=0xffffffff80c697f0 >> "/usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c", >> line=1069) at atomic.h:142 >> #9 0xffffffff80c50b2a in zfsctl_umount_snapshots (vfsp=Variable >> "vfsp" is not available. >> ) at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c:1069 >> #10 0xffffffff80c57978 in zfs_umount (vfsp=0xffffff00014f5650, >> fflag=0, td=0xffffff001483b6a0) >> at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:692 > [...] > > I tried to reproduce your problem, but I can't. This is clearly related > to unmounting snapshots. Was your snapshot mounted at the time of > calling 'zfs set mountpoint='? I tried both scenarious (having mounted > and unmounted snapshot) and no panic. Is there anything else you did? > > -- > Pawel Jakub Dawidek http://www.wheel.pl > pjd@FreeBSD.org http://www.FreeBSD.org > FreeBSD committer Am I Evil? Yes, I Am! > Hi Pawel, Sorry for my late reply. I cannot test right now because the machine suffered multiple disk failures, and is currently offline, but as far as I remember the machine had several snapshots mounted at the time of the panic. Regards, Niki From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 05:35:44 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3880C1065765 for ; Mon, 2 Jun 2008 05:35:44 +0000 (UTC) (envelope-from ighighi@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.237]) by mx1.freebsd.org (Postfix) with ESMTP id D25878FC17 for ; Mon, 2 Jun 2008 05:35:43 +0000 (UTC) (envelope-from ighighi@gmail.com) Received: by wx-out-0506.google.com with SMTP id h27so708972wxd.7 for ; Sun, 01 Jun 2008 22:35:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:content-type; bh=HZPZLdzt+6dBZr/djnr2pS3x58zpwtxx3YuZuWbxSt4=; b=dJbuMMBUc/QWgJKbTHJtr1VoZeLBCJCxBzw4LhOZwswdMgAdaOiZVwTWz/o/mSmAO7zPaYTrisuvG+xsVNGUqBpiUdWU7mKn+Eo0UQcqelFI81n4wkxWzk0OAtP+lUGKwMswJH6Any/ByKezBMt00Y0mfsQ9xzvXdc1HRJJzsho= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject:content-type; b=cJkW9RqKPqDikKyOPbwaQJYKeifVxBUs6KJ5e31dkwUd5Xakl9rYgHRJ9V1WD9ONt8N4NRJ7ZuxRGHxuaWdmlMCTlUKPXDLKaPc+KJF69ZkVSUp3PSBMn78aCOjvp8Pf+Zq3rJTNVAuHYFjrETEMYx0q+CkFeywAqY1flCQJmvs= Received: by 10.90.71.16 with SMTP id t16mr10803333aga.94.1212384942746; Sun, 01 Jun 2008 22:35:42 -0700 (PDT) Received: from orion.nebula.mil ( [200.82.219.99]) by mx.google.com with ESMTPS id 6sm4049623agd.31.2008.06.01.22.35.38 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 01 Jun 2008 22:35:41 -0700 (PDT) Message-ID: <48438687.1080606@gmail.com> Date: Mon, 02 Jun 2008 01:05:03 -0430 From: Ighighi User-Agent: Thunderbird 2.0.0.14 (X11/20080504) MIME-Version: 1.0 To: bug-followup@freebsd.org Content-Type: multipart/mixed; boundary="------------050007060101030401040504" Cc: freebsd-fs@freebsd.org Subject: Re: kern/122047: [ext2fs] incorrect handling of UF_IMMUTABLE / UF_APPEND, flag on EXT2FS (maybe others) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 05:35:44 -0000 This is a multi-part message in MIME format. --------------050007060101030401040504 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On Linux, only the root user may set/clear the immutable/append flags on ext2 filesystems... Shouldn't FreeBSD do this too, as a POLA? Anyway the attached patch extends the previous one by making it possible to follow the current Linux convention by setting the sysctl to 0. Setting it to 1, allows normal users to set them as well, and setting it to -1 preserves current (though erroneous) FreeBSD behavior. --------------050007060101030401040504 Content-Type: text/plain; name="ext2fs.patch.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ext2fs.patch.txt" # # (!c) 2008 by Ighighi # # See http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/122047 # # This patch adds a vfs.e2fs.userflags sysctl to allow/prevent normal users # to set/clear the append/immutable filesystem flags on EXT2 filesystems. # If set to 0, only the superuser may set/clear these flags. This is the # default behavior on Linux, which FreeBSD should mimick (POLA). # If set to 1, users are also permitted to set/clear them on files they own. # If set to -1 (default), maintain the current (erroneus) behavior. # # As a bonus, this patch sets st_birthtime to zero. # # Built and tested on FreeBSD 6.3-STABLE (RELENG_6). # Known to patch on -CURRENT # # To install, run as root: # /sbin/umount -v -t ext2fs -a # /sbin/kldunload -v ext2fs # /usr/bin/patch -d /usr < /path/to/ext2fs.patch # cd /sys/modules/ext2fs/ # make clean obj depend && make && make install # /sbin/kldload -v ext2fs # /sbin/sysctl vfs.e2fs.userflags=1 # /sbin/mount -v -t ext2fs -a # --- src/sys/gnu/fs/ext2fs/ext2_inode_cnv.c.orig 2005-06-14 22:36:10.000000000 -0400 +++ src/sys/gnu/fs/ext2fs/ext2_inode_cnv.c 2008-06-02 00:35:34.658524358 -0430 @@ -30,11 +30,19 @@ #include #include #include +#include +#include #include #include #include +SYSCTL_DECL(_vfs_e2fs); + +static int userflags = -1; +SYSCTL_INT(_vfs_e2fs, OID_AUTO, userflags, CTLFLAG_RW, + &userflags, 0, "Users may set/clear filesystem flags"); + void ext2_print_inode( in ) struct inode *in; @@ -83,8 +91,37 @@ ext2_ei2i(ei, ip) ip->i_mtime = ei->i_mtime; ip->i_ctime = ei->i_ctime; ip->i_flags = 0; - ip->i_flags |= (ei->i_flags & EXT2_APPEND_FL) ? APPEND : 0; - ip->i_flags |= (ei->i_flags & EXT2_IMMUTABLE_FL) ? IMMUTABLE : 0; + switch (userflags) { + case 0: + /* + * Only the superuser may set/clear these flags. + * This is the current behavior on Linux. + */ + if (ei->i_flags & EXT2_APPEND_FL) + ip->i_flags |= SF_APPEND; + if (ei->i_flags & EXT2_IMMUTABLE_FL) + ip->i_flags |= SF_IMMUTABLE; + break; + case 1: + /* + * Users may set/clear these flags on files they own. + */ + if (ei->i_flags & EXT2_APPEND_FL) + ip->i_flags |= UF_APPEND; + if (ei->i_flags & EXT2_IMMUTABLE_FL) + ip->i_flags |= UF_IMMUTABLE; + break; + case -1: + default: + /* + * Default behavior on FreeBSD + */ + if (ei->i_flags & EXT2_APPEND_FL) + ip->i_flags |= APPEND; + if (ei->i_flags & EXT2_IMMUTABLE_FL) + ip->i_flags |= IMMUTABLE; + break; + } ip->i_blocks = ei->i_blocks; ip->i_gen = ei->i_generation; ip->i_uid = ei->i_uid; @@ -121,8 +158,37 @@ ext2_i2ei(ip, ei) ei->i_ctime = ip->i_ctime; ei->i_flags = ip->i_flags; ei->i_flags = 0; - ei->i_flags |= (ip->i_flags & APPEND) ? EXT2_APPEND_FL: 0; - ei->i_flags |= (ip->i_flags & IMMUTABLE) ? EXT2_IMMUTABLE_FL: 0; + switch (userflags) { + case 0: + /* + * Only the superuser may set/clear these flags. + * This is the current behavior on Linux. + */ + if (ip->i_flags & SF_APPEND) + ei->i_flags |= EXT2_APPEND_FL; + if (ip->i_flags & SF_IMMUTABLE) + ei->i_flags |= EXT2_IMMUTABLE_FL; + break; + case 1: + /* + * Users may set/clear these flags on files they own. + */ + if (ip->i_flags & UF_APPEND) + ei->i_flags |= EXT2_APPEND_FL; + if (ip->i_flags & UF_IMMUTABLE) + ei->i_flags |= EXT2_IMMUTABLE_FL; + break; + case -1: + default: + /* + * Default behavior on FreeBSD + */ + if (ip->i_flags & APPEND) + ei->i_flags |= EXT2_APPEND_FL; + if (ip->i_flags & IMMUTABLE) + ei->i_flags |= EXT2_IMMUTABLE_FL; + break; + } ei->i_blocks = ip->i_blocks; ei->i_generation = ip->i_gen; ei->i_uid = ip->i_uid; --- src/sys/gnu/fs/ext2fs/ext2_lookup.c.orig 2006-01-04 15:32:00.000000000 -0400 +++ src/sys/gnu/fs/ext2fs/ext2_lookup.c 2008-06-01 05:38:42.363332933 -0430 @@ -66,7 +66,7 @@ static int dirchk = 1; static int dirchk = 0; #endif -static SYSCTL_NODE(_vfs, OID_AUTO, e2fs, CTLFLAG_RD, 0, "EXT2FS filesystem"); +SYSCTL_NODE(_vfs, OID_AUTO, e2fs, CTLFLAG_RD, 0, "EXT2FS filesystem"); SYSCTL_INT(_vfs_e2fs, OID_AUTO, dircheck, CTLFLAG_RW, &dirchk, 0, ""); /* --- src/sys/gnu/fs/ext2fs/ext2_vnops.c.orig 2006-02-19 20:53:14.000000000 -0400 +++ src/sys/gnu/fs/ext2fs/ext2_vnops.c 2008-05-28 07:58:02.189157441 -0430 @@ -358,6 +358,8 @@ ext2_getattr(ap) vap->va_mtime.tv_nsec = ip->i_mtimensec; vap->va_ctime.tv_sec = ip->i_ctime; vap->va_ctime.tv_nsec = ip->i_ctimensec; + vap->va_birthtime.tv_sec = 0; + vap->va_birthtime.tv_nsec = 0; vap->va_flags = ip->i_flags; vap->va_gen = ip->i_gen; vap->va_blocksize = vp->v_mount->mnt_stat.f_iosize; --------------050007060101030401040504-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 06:04:14 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B7AE1065675 for ; Mon, 2 Jun 2008 06:04:14 +0000 (UTC) (envelope-from andrew@thefrog.net) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.156]) by mx1.freebsd.org (Postfix) with ESMTP id D1E428FC19 for ; Mon, 2 Jun 2008 06:04:13 +0000 (UTC) (envelope-from andrew@thefrog.net) Received: by fg-out-1718.google.com with SMTP id l26so789772fgb.35 for ; Sun, 01 Jun 2008 23:04:12 -0700 (PDT) Received: by 10.82.112.3 with SMTP id k3mr800814buc.56.1212386652417; Sun, 01 Jun 2008 23:04:12 -0700 (PDT) Received: by 10.82.149.3 with HTTP; Sun, 1 Jun 2008 23:04:12 -0700 (PDT) Message-ID: <16a6ef710806012304m48b63161oee1bc6d11e54436a@mail.gmail.com> Date: Mon, 2 Jun 2008 16:04:12 +1000 From: "Andrew Hill" Sender: andrew@thefrog.net To: freebsd-fs@freebsd.org In-Reply-To: <93F07874-8D5F-44AE-945F-803FFC3B9279@thefrog.net> MIME-Version: 1.0 References: <683A6ED2-0E54-42D7-8212-898221C05150@thefrog.net> <20080518124217.GA16222@eos.sc1.parodius.com> <93F07874-8D5F-44AE-945F-803FFC3B9279@thefrog.net> X-Google-Sender-Auth: c517165170c0559e Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFS lockup in "zfs" state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 06:04:14 -0000 some more info... On Mon, May 19, 2008 at 1:11 AM, Andrew Hill wrote: > i tend to find that the timeouts occur on one or two disks at once - e.g. > ad0 and 2 will complain of timeouts, and the system locks up shortly > thereafter... after spitting out the usual errors from ad0 and ad2 (in this case) with TIMEOUTs and subsequent FAILUREs on READ_DMA[48] and WRITE_DMA[48]... i got the following panic vm_fault: pager read error, pid 1552 (tlsmgr) ad0: FAILURE - READ_DMA48 timed out LBA=352903900 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 437, size: 4096 ad2: FAILURE - WRITE_DMA timed out LBA=239717693 panic: ZFS: I/O failure (write on off 0: zio 0xffffff001d47c810 [L0 ZIL intent log] b000L/b000P DVA[0]=<0:c807795000:d000> zilog uncompressed LE contiguous birth=750230 fill=0 cksum=69f76525a84e1816:f6d86fe1d94cd68c:39:8af): error 5 KDB: enter: panic [thread pid 72 tid 100071 ] Stopped at kdb_enter_why+0x3d: movq $0,0x39b248(%rip) db> generally the lockups don't result in a panic (at least not in the short term of 5-10 minutes), so i can't be sure that this panic is necessarily caused by the same problem, but thought it might be worth posting in case it gives an indication of the location/cause of the deadlock unfortunately i couldn't get a backtrace or core dump for 'political' reasons (the system was required for use by others) but i'll see if i can get a panic happening after-hours to get some more info... From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 06:40:23 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E1171065671 for ; Mon, 2 Jun 2008 06:40:23 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 2B5698FC30 for ; Mon, 2 Jun 2008 06:40:23 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 20DF01CC031; Sun, 1 Jun 2008 23:40:23 -0700 (PDT) Date: Sun, 1 Jun 2008 23:40:23 -0700 From: Jeremy Chadwick To: Andrew Hill Message-ID: <20080602064023.GA95247@eos.sc1.parodius.com> References: <683A6ED2-0E54-42D7-8212-898221C05150@thefrog.net> <20080518124217.GA16222@eos.sc1.parodius.com> <93F07874-8D5F-44AE-945F-803FFC3B9279@thefrog.net> <16a6ef710806012304m48b63161oee1bc6d11e54436a@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <16a6ef710806012304m48b63161oee1bc6d11e54436a@mail.gmail.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS lockup in "zfs" state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 06:40:23 -0000 On Mon, Jun 02, 2008 at 04:04:12PM +1000, Andrew Hill wrote: > On Mon, May 19, 2008 at 1:11 AM, Andrew Hill wrote: > > > i tend to find that the timeouts occur on one or two disks at once - e.g. > > ad0 and 2 will complain of timeouts, and the system locks up shortly > > thereafter... > > after spitting out the usual errors from ad0 and ad2 (in this case) with > TIMEOUTs and subsequent FAILUREs on READ_DMA[48] and WRITE_DMA[48]... > > i got the following panic > > vm_fault: pager read error, pid 1552 (tlsmgr) > ad0: FAILURE - READ_DMA48 timed out LBA=352903900 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 437, size: 4096 > ad2: FAILURE - WRITE_DMA timed out LBA=239717693 > panic: ZFS: I/O failure (write on off 0: zio 0xffffff001d47c810 > [L0 ZIL intent log] b000L/b000P DVA[0]=<0:c807795000:d000> zilog > uncompressed LE contiguous birth=750230 fill=0 > cksum=69f76525a84e1816:f6d86fe1d94cd68c:39:8af): error 5 > KDB: enter: panic > [thread pid 72 tid 100071 ] > Stopped at kdb_enter_why+0x3d: movq $0,0x39b248(%rip) > db> I would say the ZFS crash is a result of the ad0/ad2 timeouts. The ZIL log shows a hard checksum failure in the ZIL, which indicates a serious problem -- very likely hardware-related (or rather, at a lower level than ZFS). You've read this already, but maybe you missed the DMA error part: http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues The DMA errors can actually be legitimate too -- it's very hard to troubleshoot if they're superfluous (e.g. a FreeBSD bug) or if they're real. If the problem is reproducable, then this is convenient with regards to providing you additional help. I really need to sit down and write a huge HOWTO doc for people on how to diagnose whether or not their disks or cables are bad, etc... It's a very hard thing to document, because everyone's situation is different. The first piece to start with is simplest, though: install ports/sysutils/smartmontools and provide the output of "smartctl -a /dev/ad0" and /dev/ad2. Actual disk errors will very likely show up there in one of the counters, or in the SMART log. I'd personally like to see the output from smartctl, because it's something you can do while the system is up/working. The next step would involve replacing your cables. If the problem continues, you've at least removed one piece of the puzzle. Next, replace the disks -- especially if they were bought at the same time, and are from the same vendor. Hard disk vendors are known to have bad batches of disks. For sake of example, I just had two Western Digital disks (which I bought at the same time) fail a short I/O test, returning errors at different LBAs (blocks). The 2nd one only started showing problems a few weeks after the first. I obviously got both of them RMA'd. Finally, replace the controller or motherboard. Some people have reported success with this. > generally the lockups don't result in a panic (at least not in the short > term of 5-10 minutes), so i can't be sure that this panic is necessarily > caused by the same problem, but thought it might be worth posting in case it > gives an indication of the location/cause of the deadlock The DMA timeout errors you've seen, others have seen as well -- including me -- even when the hardware, disks, cabling, and controllers are in a 100% working state. (Even switching OSes results in no errors, indicating there is a problem with FreeBSD in some way.) If the problem is reproducable, you should get in contact with Scott Long and let him poke at things. (I mentioned this last time. :-) ) I myself am not familiar with the FreeBSD kernel, the device drivers, or working with the kernel at such a low level to debug things of this nature. > unfortunately i couldn't get a backtrace or core dump for 'political' > reasons (the system was required for use by others) but i'll see if i can > get a panic happening after-hours to get some more info... I can't tell you what to do or how to do your job, but honestly you should be pulling this system out of production and replacing it with a different one, or a different implementation, or a different OS. Your users/employees are probably getting ticked off at the crashes, and it probably irritates you too. The added benefit is that you could get Scott access to the box. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 07:27:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E7F81065670 for ; Mon, 2 Jun 2008 07:27:45 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outZ.internet-mail-service.net (outz.internet-mail-service.net [216.240.47.249]) by mx1.freebsd.org (Postfix) with ESMTP id 4EB2E8FC20 for ; Mon, 2 Jun 2008 07:27:45 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 1EF5C23EA; Mon, 2 Jun 2008 00:14:47 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id CAC882D601A; Mon, 2 Jun 2008 00:14:46 -0700 (PDT) Message-ID: <48439DE6.50505@elischer.org> Date: Mon, 02 Jun 2008 00:14:46 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Ighighi References: <48438687.1080606@gmail.com> In-Reply-To: <48438687.1080606@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/122047: [ext2fs] incorrect handling of UF_IMMUTABLE / UF_APPEND, flag on EXT2FS (maybe others) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 07:27:45 -0000 Ighighi wrote: > On Linux, only the root user may set/clear the immutable/append flags > on ext2 filesystems... Shouldn't FreeBSD do this too, as a POLA? No I think it should preserver the BSD scheme where being able to change the immutable bits is controlled by the system secure level. (and your UID of course). At least I think that is what I would expect. (All file systems to behave about the same for a particular OS. > > Anyway the attached patch extends the previous one by making it possible > to follow the current Linux convention by setting the sysctl to 0. > Setting it to 1, allows normal users to set them as well, and setting it > to -1 preserves current (though erroneous) FreeBSD behavior. > > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 10:30:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32E0F106564A; Mon, 2 Jun 2008 10:30:11 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail11.syd.optusnet.com.au (mail11.syd.optusnet.com.au [211.29.132.192]) by mx1.freebsd.org (Postfix) with ESMTP id 958978FC18; Mon, 2 Jun 2008 10:30:10 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c220-239-252-11.carlnfd3.nsw.optusnet.com.au (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail11.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m52AU2Ak000559 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 2 Jun 2008 20:30:04 +1000 Date: Mon, 2 Jun 2008 20:30:02 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Julian Elischer In-Reply-To: <48439DE6.50505@elischer.org> Message-ID: <20080602202420.N3083@delplex.bde.org> References: <48438687.1080606@gmail.com> <48439DE6.50505@elischer.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, Ighighi , bug-followup@freebsd.org Subject: Re: kern/122047: [ext2fs] incorrect handling of UF_IMMUTABLE / UF_APPEND, flag on EXT2FS (maybe others) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 10:30:11 -0000 On Mon, 2 Jun 2008, Julian Elischer wrote: > Ighighi wrote: >> On Linux, only the root user may set/clear the immutable/append flags >> on ext2 filesystems... Shouldn't FreeBSD do this too, as a POLA? > > No I think it should preserver the BSD scheme where being able to > change the immutable bits is controlled by the system secure level. > (and your UID of course). At least I think that is what I would > expect. (All file systems to behave about the same for a > particular OS. No, the securelevel already controls things, and the BSD scheme reduces to only allowing root (strictly, processes with appropriate privilege, as restricted by securelevel and jails etc, but never mere users), to change immutable bits, because ext2fs doesn't have any user immutable bits to change (except phantom bits due to bugs in the current FreeBSD implementation). Bruce From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 11:06:52 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8FB081065680 for ; Mon, 2 Jun 2008 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8553D8FC0C for ; Mon, 2 Jun 2008 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m52B6qmH093148 for ; Mon, 2 Jun 2008 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m52B6lwF093143 for freebsd-fs@FreeBSD.org; Mon, 2 Jun 2008 11:06:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 2 Jun 2008 11:06:47 GMT Message-Id: <200806021106.m52B6lwF093143@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 11:06:52 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t 6 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o bin/118249 fs mv(1): moving a directory changes its mtime 6 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 14:56:04 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B9E31065671 for ; Mon, 2 Jun 2008 14:56:04 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id 92BD68FC26 for ; Mon, 2 Jun 2008 14:56:03 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail07.syd.optusnet.com.au (mail07.syd.optusnet.com.au [211.29.132.188]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m527opq6008585 for ; Mon, 2 Jun 2008 17:50:56 +1000 Received: from c220-239-252-11.carlnfd3.nsw.optusnet.com.au (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m527ojOk011104 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 2 Jun 2008 17:50:47 +1000 Date: Mon, 2 Jun 2008 17:50:45 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Ighighi In-Reply-To: <48438687.1080606@gmail.com> Message-ID: <20080602163212.G2551@delplex.bde.org> References: <48438687.1080606@gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org, bug-followup@FreeBSD.org Subject: Re: kern/122047: [ext2fs] incorrect handling of UF_IMMUTABLE / UF_APPEND, flag on EXT2FS (maybe others) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 14:56:04 -0000 On Mon, 2 Jun 2008, Ighighi wrote: > On Linux, only the root user may set/clear the immutable/append flags > on ext2 filesystems... Shouldn't FreeBSD do this too, as a POLA? I think it should do just that... > Anyway the attached patch extends the previous one by making it possible > to follow the current Linux convention by setting the sysctl to 0. > Setting it to 1, allows normal users to set them as well, and setting it > to -1 preserves current (though erroneous) FreeBSD behavior. ... but nothing more. Why have complications provide more control over Linux file systems than Linux does? The current behaviour seems to be just a bug from not understanding that Linux has no user immutable/append flags. % --- src/sys/gnu/fs/ext2fs/ext2_inode_cnv.c.orig 2005-06-14 22:36:10.000000000 -0400 % +++ src/sys/gnu/fs/ext2fs/ext2_inode_cnv.c 2008-06-02 00:35:34.658524358 -0430 % @@ -30,11 +30,19 @@ % #include % #include % #include % +#include % +#include % % #include % #include % #include % % +SYSCTL_DECL(_vfs_e2fs); % + % +static int userflags = -1; % +SYSCTL_INT(_vfs_e2fs, OID_AUTO, userflags, CTLFLAG_RW, % + &userflags, 0, "Users may set/clear filesystem flags"); % + % void % ext2_print_inode( in ) % struct inode *in; The existence of vfs sysctls is another bug. They should be mount options, or perhaps system-wide, or not exist at all. ext2fs has only one vfs sysctl now: - vfs.e2fs.dircheck. This sysctl is less broken than in ffs, where the corresponding sysctl is spelled debug.dircheck and a comment still says that the corresponding static kernel variable `dirchk' is changed by "patching". The kernel variable is spelled differently to the sysctl to confuse me when grepping for dircheck. - the debug.doasyncfree sysctl is in dead code (under the non-option FANCY_REALLOC. Block reallocation is also dead in ext2fs). This sysctl is less broken in ffs. There it is named vfs.ffs.doasyncfree. OTOH, perhaps these sysctls really did belong under debug or vfs.debug. It is not very useful to restrict them to just ffs or ext2fs when have many mounted file systems. This bug should not be extended. % @@ -83,8 +91,37 @@ ext2_ei2i(ei, ip) % ip->i_mtime = ei->i_mtime; % ip->i_ctime = ei->i_ctime; % ip->i_flags = 0; % - ip->i_flags |= (ei->i_flags & EXT2_APPEND_FL) ? APPEND : 0; % - ip->i_flags |= (ei->i_flags & EXT2_IMMUTABLE_FL) ? IMMUTABLE : 0; I think it would work to just map EXT2*_FL to SF_*. % + switch (userflags) { % + case 0: % + /* % + * Only the superuser may set/clear these flags. % + * This is the current behavior on Linux. % + */ % + if (ei->i_flags & EXT2_APPEND_FL) % + ip->i_flags |= SF_APPEND; % + if (ei->i_flags & EXT2_IMMUTABLE_FL) % + ip->i_flags |= SF_IMMUTABLE; % + break; % + case 1: % + /* % + * Users may set/clear these flags on files they own. % + */ % + if (ei->i_flags & EXT2_APPEND_FL) % + ip->i_flags |= UF_APPEND; % + if (ei->i_flags & EXT2_IMMUTABLE_FL) % + ip->i_flags |= UF_IMMUTABLE; % + break; For administration it can be convenient for the file system to behave a little differently to native mode, but I letting root change the (system) immutable flags is enough. % + case -1: % + default: % + /* % + * Default behavior on FreeBSD % + */ % + if (ei->i_flags & EXT2_APPEND_FL) % + ip->i_flags |= APPEND; % + if (ei->i_flags & EXT2_IMMUTABLE_FL) % + ip->i_flags |= IMMUTABLE; % + break; % + } I think the current behaviour is too buggy to keep. (Your original PR describes the bugs -- FreeBSD makes a mess by setting 2 flags in the in-core inode and allowing these flags to be changed independently, then cannot merge the flags properly when writing to the on-disk inode.) [conversion to the on-disk inode] Similarly. There is a problem in ext2_vnops.c that is not touched by these patches. Even in the simple version that only support SF_*, ext2_setattr() needs changes to disallow setting of UF_* -- otherwise FreeBSD still makes a mess, with stat() showing changes to UF_* succeeding while the inode is in-core but going away when the inode is flushed from the inode/vnode cache. The fix is simply to remove the code that supports UF_*. (We could also globably replace IMMUTABLE and APPEND by SF_IMMUTABLE and SF_APPEND. This would be clearer but would increase divergence from ffs.) The fix to support all 3 sysctl modes is not so simple: - case 0: dynamically disallow attempts to set UF_*. - case 1: dynamically disallow attempts to set SF_*. - case -1: dynamically convert attempts to set SF_* or UF*_ into attempts to set both, and somehow relax the checks for setting SF_* so that this has a chance of succeeding for non-root. I don't want these complications. Note that the corresponding code in ffs is poorly organized and buggy (it doesn't allow preservation of flags in the right way). ext2_setattr() was once almost identical to ufs_settatr() but now has the following bitrot: - missing support for setting atimes on exec. - different comments about privilege though the code is the same. - different comments about truncate() on r/o file systems. Missing a critical fix for truncate(). - missing the comment expansion and cleanup for utimes(). I think there was a minor security-related fix in there too, but this is now null. % --- src/sys/gnu/fs/ext2fs/ext2_vnops.c.orig 2006-02-19 20:53:14.000000000 -0400 % +++ src/sys/gnu/fs/ext2fs/ext2_vnops.c 2008-05-28 07:58:02.189157441 -0430 % @@ -358,6 +358,8 @@ ext2_getattr(ap) % vap->va_mtime.tv_nsec = ip->i_mtimensec; % vap->va_ctime.tv_sec = ip->i_ctime; % vap->va_ctime.tv_nsec = ip->i_ctimensec; % + vap->va_birthtime.tv_sec = 0; % + vap->va_birthtime.tv_nsec = 0; % vap->va_flags = ip->i_flags; % vap->va_gen = ip->i_gen; % vap->va_blocksize = vp->v_mount->mnt_stat.f_iosize; This is unrelated and should be handled centrally. Almost all file systems get this wrong. Most fail to set va_birthtime, so stat() returns kernel stack garbage for st_birthtime. ffs1 does the same as the above. msdosfs does the above correctly, by setting tv_sec to (time_t)-1 in unsupported cases. Bruce From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 19:31:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C06A31065673 for ; Mon, 2 Jun 2008 19:31:54 +0000 (UTC) (envelope-from yalur@mail.ru) Received: from mx39.mail.ru (mx39.mail.ru [194.67.23.35]) by mx1.freebsd.org (Postfix) with ESMTP id 3032A8FC1B for ; Mon, 2 Jun 2008 19:31:54 +0000 (UTC) (envelope-from yalur@mail.ru) Received: from [77.122.142.19] (port=43565 helo=dive.liberties.volia.net) by mx39.mail.ru with asmtp id 1K3Fl1-000KRg-00; Mon, 02 Jun 2008 23:31:51 +0400 From: Ruslan Kovtun Organization: Home To: freebsd-fs@freebsd.org Date: Mon, 2 Jun 2008 22:31:45 +0300 User-Agent: KMail/1.9.7 References: <683A6ED2-0E54-42D7-8212-898221C05150@thefrog.net> <16a6ef710806012304m48b63161oee1bc6d11e54436a@mail.gmail.com> <20080602064023.GA95247@eos.sc1.parodius.com> In-Reply-To: <20080602064023.GA95247@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200806022231.46079.yalur@mail.ru> Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable X-Spam: Not detected X-Mras: OK Cc: Andrew Hill Subject: Re: ZFS lockup in "zfs" state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: yalur@mail.ru List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 19:31:54 -0000 Hi. I have the same problem very often with HDD (READ_DMA UDMA ICRC error) whi= ch=20 is in zfs pool. Before, this HDD was in mirror ar0 but not in ZFS pool and= =20 this hard disk sometimes have failed but with no any panic only detached fr= om=20 mirror. After I included this HDD to ZFS pool problem have apeared. I am= =20 sure that this is problem with hard disk.=20 Smartmontools notified me by mail that UDMA_CRC_Error_Count have increased= =20 after HDD failure and acording smartctl I can see that HDD have hardware=20 problem. I replased cable, tried to connect this HDD to another port - but= =20 no result: 100% hard disk problem. I can not create kernel coredump during panic: savecore: no dumps found :( Only logs are available: In log file: Jun 1 10:43:11 yalur kernel: ad16: WARNING - READ_DMA UDMA ICRC error=20 (retrying request) LBA=3D233909187 Jun 1 10:43:20 yalur kernel: ad16: WARNING - SETFEATURES SET TRANSFER MODE= =20 taskqueue timeout - completing request directly Jun 1 10:43:36 yalur kernel: ad16: WARNING - SETFEATURES SET TRANSFER MODE= =20 taskqueue timeout - completing request directly Jun 1 10:43:36 yalur kernel: ad16: WARNING - SETFEATURES ENABLE RCACHE=20 taskqueue timeout - completing request directly Jun 1 10:43:36 yalur kernel: ad16: WARNING - SETFEATURES ENABLE WCACHE=20 taskqueue timeout - completing request directly Jun 1 10:43:36 yalur kernel: ad16: WARNING - SET_MULTI taskqueue timeout -= =20 completing request directly Jun 1 10:43:36 yalur kernel: ad16: TIMEOUT - READ_DMA retrying (0 retries= =20 left) LBA=3D233909187 Jun 1 11:07:50 yalur syslogd: restart Jun 1 11:07:50 yalur syslogd: kernel boot file is /boot/kernel/kernel Jun 1 11:07:50 yalur kernel: ad16: FAILURE - device detached Jun 1 11:07:50 yalur kernel: subdisk16: detached Jun 1 11:07:50 yalur kernel: ad16: detached Jun 1 11:07:50 yalur kernel: Jun 1 11:07:50 yalur kernel: Jun 1 11:07:50 yalur kernel: Fatal trap 12: page fault while in kernel mode Jun 1 11:07:50 yalur kernel: cpuid =3D 0; apic id =3D 00 Jun 1 11:07:50 yalur kernel: fault virtual address =3D 0x2c Jun 1 11:07:50 yalur kernel: fault code =3D supervisor writ= e,=20 page not present Jun 1 11:07:50 yalur kernel: instruction pointer =3D 0x20:0x805aab85 Jun 1 11:07:50 yalur kernel: stack pointer =3D 0x28:0xed71ac5c Jun 1 11:07:50 yalur kernel: frame pointer =3D 0x28:0xed71ac70 Jun 1 11:07:50 yalur kernel: code segment =3D base 0x0, limit= =20 0xfffff, type 0x1b Jun 1 11:07:50 yalur kernel: =3D DPL 0, pres 1, def32 1, gran 1 Jun 1 11:07:50 yalur kernel: processor eflags =3D interrupt enabled, resu= me,=20 IOPL =3D 0 Jun 1 11:07:50 yalur kernel: current process =3D 3 (g_up) Jun 1 11:07:50 yalur kernel: trap number =3D 12 Jun 1 11:07:50 yalur kernel: panic: page fault Jun 1 11:07:50 yalur kernel: cpuid =3D 0 [root@yalur /home/ruslan]# zpool status pool: data state: ONLINE scrub: scrub completed with 0 errors on Mon Jun 2 12:05:52 2008 config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad8 ONLINE 0 0 0 ad10 ONLINE 0 0 0 ad4 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad12 ONLINE 0 0 0 ad14 ONLINE 0 0 0 ad16 ONLINE 0 0 0 ad20 ONLINE 0 0 0 spares ad26 AVAIL errors: No known data errors =F7 =D3=CF=CF=C2=DD=C5=CE=C9=C9 =CF=D4 =F0=CF=CE=C5=C4=C5=CC=D8=CE=C9=CB 02= =C9=C0=CE=D1 2008 Jeremy Chadwick =CE=C1=D0=C9=D3=C1=CC(a): > On Mon, Jun 02, 2008 at 04:04:12PM +1000, Andrew Hill wrote: > > On Mon, May 19, 2008 at 1:11 AM, Andrew Hill wrote: > > > i tend to find that the timeouts occur on one or two disks at once - > > > e.g. ad0 and 2 will complain of timeouts, and the system locks up > > > shortly thereafter... > > > > after spitting out the usual errors from ad0 and ad2 (in this case) with > > TIMEOUTs and subsequent FAILUREs on READ_DMA[48] and WRITE_DMA[48]... > > > > i got the following panic > > > > vm_fault: pager read error, pid 1552 (tlsmgr) > > ad0: FAILURE - READ_DMA48 timed out LBA=3D352903900 > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 437, size: 4096 > > ad2: FAILURE - WRITE_DMA timed out LBA=3D239717693 > > panic: ZFS: I/O failure (write on off 0: zio 0xffffff001d47c8= 10 > > [L0 ZIL intent log] b000L/b000P DVA[0]=3D<0:c807795000:d000> zilog > > uncompressed LE contiguous birth=3D750230 fill=3D0 > > cksum=3D69f76525a84e1816:f6d86fe1d94cd68c:39:8af): error 5 > > KDB: enter: panic > > [thread pid 72 tid 100071 ] > > Stopped at kdb_enter_why+0x3d: movq $0,0x39b248(%rip) > > db> > > I would say the ZFS crash is a result of the ad0/ad2 timeouts. The ZIL > log shows a hard checksum failure in the ZIL, which indicates a serious > problem -- very likely hardware-related (or rather, at a lower level > than ZFS). > > You've read this already, but maybe you missed the DMA error part: > > http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues > > The DMA errors can actually be legitimate too -- it's very hard to > troubleshoot if they're superfluous (e.g. a FreeBSD bug) or if they're > real. If the problem is reproducable, then this is convenient with > regards to providing you additional help. > > I really need to sit down and write a huge HOWTO doc for people on how > to diagnose whether or not their disks or cables are bad, etc... It's a > very hard thing to document, because everyone's situation is different. > > The first piece to start with is simplest, though: install > ports/sysutils/smartmontools and provide the output of "smartctl -a > /dev/ad0" and /dev/ad2. Actual disk errors will very likely show up > there in one of the counters, or in the SMART log. I'd personally like > to see the output from smartctl, because it's something you can do while > the system is up/working. > > The next step would involve replacing your cables. If the problem > continues, you've at least removed one piece of the puzzle. > > Next, replace the disks -- especially if they were bought at the same > time, and are from the same vendor. Hard disk vendors are known to have > bad batches of disks. For sake of example, I just had two Western > Digital disks (which I bought at the same time) fail a short I/O test, > returning errors at different LBAs (blocks). The 2nd one only started > showing problems a few weeks after the first. I obviously got both of > them RMA'd. > > Finally, replace the controller or motherboard. Some people have > reported success with this. > > > generally the lockups don't result in a panic (at least not in the short > > term of 5-10 minutes), so i can't be sure that this panic is necessarily > > caused by the same problem, but thought it might be worth posting in ca= se > > it gives an indication of the location/cause of the deadlock > > The DMA timeout errors you've seen, others have seen as well -- > including me -- even when the hardware, disks, cabling, and controllers > are in a 100% working state. (Even switching OSes results in no errors, > indicating there is a problem with FreeBSD in some way.) > > If the problem is reproducable, you should get in contact with Scott > Long and let him poke at things. (I mentioned this last time. :-) ) > I myself am not familiar with the FreeBSD kernel, the device drivers, or > working with the kernel at such a low level to debug things of this > nature. > > > unfortunately i couldn't get a backtrace or core dump for 'political' > > reasons (the system was required for use by others) but i'll see if i c= an > > get a panic happening after-hours to get some more info... > > I can't tell you what to do or how to do your job, but honestly you > should be pulling this system out of production and replacing it with a > different one, or a different implementation, or a different OS. Your > users/employees are probably getting ticked off at the crashes, and it > probably irritates you too. The added benefit is that you could get > Scott access to the box. =2D-=20 ________________ =F3 =D5=D7=C1=D6=C5=CE=C9=C5=CD =EB=CF=D7=D4=D5=CE =F2=D5=D3=CC=C1=CE mailto From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 22:26:17 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ED2A1065679; Mon, 2 Jun 2008 22:26:17 +0000 (UTC) (envelope-from fbsd-fs@mawer.org) Received: from outbound.icp-qv1-irony-out4.iinet.net.au (outbound.icp-qv1-irony-out4.iinet.net.au [203.59.1.150]) by mx1.freebsd.org (Postfix) with ESMTP id A00138FC12; Mon, 2 Jun 2008 22:26:16 +0000 (UTC) (envelope-from fbsd-fs@mawer.org) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ah8BANcIREjLzq3r/2dsb2JhbAAIrw8 X-IronPort-AV: E=Sophos;i="4.27,579,1204470000"; d="scan'208";a="229934898" Received: from unknown (HELO [10.24.1.1]) ([203.206.173.235]) by outbound.icp-qv1-irony-out4.iinet.net.au with ESMTP; 03 Jun 2008 05:57:06 +0800 Message-ID: <48446C42.4070208@mawer.org> Date: Tue, 03 Jun 2008 07:55:14 +1000 From: Antony Mawer User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Jeremy Chadwick References: <683A6ED2-0E54-42D7-8212-898221C05150@thefrog.net> <20080518124217.GA16222@eos.sc1.parodius.com> <93F07874-8D5F-44AE-945F-803FFC3B9279@thefrog.net> <16a6ef710806012304m48b63161oee1bc6d11e54436a@mail.gmail.com> <20080602064023.GA95247@eos.sc1.parodius.com> In-Reply-To: <20080602064023.GA95247@eos.sc1.parodius.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Andrew Hill Subject: Re: ZFS lockup in "zfs" state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2008 22:26:17 -0000 Jeremy Chadwick wrote: > On Mon, Jun 02, 2008 at 04:04:12PM +1000, Andrew Hill wrote: ... >> unfortunately i couldn't get a backtrace or core dump for 'political' >> reasons (the system was required for use by others) but i'll see if i can >> get a panic happening after-hours to get some more info... > > I can't tell you what to do or how to do your job, but honestly you > should be pulling this system out of production and replacing it with a > different one, or a different implementation, or a different OS. Your > users/employees are probably getting ticked off at the crashes, and it > probably irritates you too. The added benefit is that you could get > Scott access to the box. It's a home fileserver rather than a production "work" system, so the challenge is finding another system with an equivalent amount of storage.. :-) As one knows these things are often hard enough to procure out of a company budget, let alone out of ones own pocket! --Antony From owner-freebsd-fs@FreeBSD.ORG Tue Jun 3 07:46:52 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9144D1065676 for ; Tue, 3 Jun 2008 07:46:52 +0000 (UTC) (envelope-from freebsd-fs@mindstep.com) Received: from postfix1-g20.free.fr (postfix1-g20.free.fr [212.27.60.42]) by mx1.freebsd.org (Postfix) with ESMTP id 2AE7A8FC1A for ; Tue, 3 Jun 2008 07:46:51 +0000 (UTC) (envelope-from freebsd-fs@mindstep.com) Received: from smtp8-g19.free.fr (smtp8-g19.free.fr [212.27.42.65]) by postfix1-g20.free.fr (Postfix) with ESMTP id 1DB7826CAB88 for ; Tue, 3 Jun 2008 09:23:33 +0200 (CEST) Received: from smtp8-g19.free.fr (localhost [127.0.0.1]) by smtp8-g19.free.fr (Postfix) with ESMTP id 593CF17F575 for ; Tue, 3 Jun 2008 09:23:31 +0200 (CEST) Received: from crest.mindstep.com (crest.mindstep.com [88.167.204.204]) by smtp8-g19.free.fr (Postfix) with ESMTP id 3CC1A17F53A for ; Tue, 3 Jun 2008 09:23:31 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by kawa.mindstep.fr (Postfix) with ESMTP id C68CC1FFCD for ; Tue, 3 Jun 2008 09:23:30 +0200 (CEST) (envelope-from freebsd-fs@mindstep.com) X-Virus-Scanned: by amavisd-new on ZunoBox at kawa.mindstep.fr Received: from kawa.mindstep.fr ([127.0.0.1]) by localhost (kawa.mindstep.fr [127.0.0.1]) (amavisd-new, port 10024) with LMTP id Io3RRA1K76oA for ; Tue, 3 Jun 2008 09:23:30 +0200 (CEST) Received: from [192.168.25.13] (pickwick.mindstep.fr [192.168.25.13]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by kawa.mindstep.fr (Postfix) with ESMTP id 644811FEE3 for ; Tue, 3 Jun 2008 09:23:30 +0200 (CEST) (envelope-from freebsd-fs@mindstep.com) Message-ID: <4844F171.3000801@mindstep.com> Date: Tue, 03 Jun 2008 09:23:29 +0200 From: Patrick Bihan-Faou User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Booting a disk with gmirror and gjournal X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@mindstep.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 07:46:52 -0000 Hi, I am trying to setup a system with one gjournal'ed partition mirrored on two disks using FreeBSD 6.3, and it does not work... Here is what I have routinely done so far: one partition, no journal, mirrored disks => I can have a working system. one partition, no mirror, journal => everything is ok but as soon as I try combine both gjournal and gmirror, I get stuck: once the system boots I get stuck at the F1 prompt and things do go any further. Could anybody help me with this ? Patrick. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 3 08:11:41 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F20B106564A for ; Tue, 3 Jun 2008 08:11:41 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id E8D208FC17 for ; Tue, 3 Jun 2008 08:11:40 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id BF0561CC033; Tue, 3 Jun 2008 01:11:40 -0700 (PDT) Date: Tue, 3 Jun 2008 01:11:40 -0700 From: Jeremy Chadwick To: Patrick Bihan-Faou Message-ID: <20080603081140.GA62607@eos.sc1.parodius.com> References: <4844F171.3000801@mindstep.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4844F171.3000801@mindstep.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-fs@freebsd.org Subject: Re: Booting a disk with gmirror and gjournal X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 08:11:41 -0000 On Tue, Jun 03, 2008 at 09:23:29AM +0200, Patrick Bihan-Faou wrote: > Hi, > > I am trying to setup a system with one gjournal'ed partition mirrored on > two disks using FreeBSD 6.3, and it does not work... > > Here is what I have routinely done so far: > > one partition, no journal, mirrored disks => I can have a working system. > one partition, no mirror, journal => everything is ok > > but as soon as I try combine both gjournal and gmirror, I get stuck: once > the system boots I get stuck at the F1 prompt and things do go any further. > > > Could anybody help me with this ? > > Patrick. As far as I know, FreeBSD's bootloader doesn't understand how to handle such things. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Jun 3 09:13:08 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3C03106564A for ; Tue, 3 Jun 2008 09:13:08 +0000 (UTC) (envelope-from freebsd-fs@mindstep.com) Received: from smtp8-g19.free.fr (smtp8-g19.free.fr [212.27.42.65]) by mx1.freebsd.org (Postfix) with ESMTP id 71D288FC15 for ; Tue, 3 Jun 2008 09:13:08 +0000 (UTC) (envelope-from freebsd-fs@mindstep.com) Received: from smtp8-g19.free.fr (localhost [127.0.0.1]) by smtp8-g19.free.fr (Postfix) with ESMTP id 8F2A917F58C for ; Tue, 3 Jun 2008 11:13:07 +0200 (CEST) Received: from crest.mindstep.com (crest.mindstep.com [88.167.204.204]) by smtp8-g19.free.fr (Postfix) with ESMTP id 6F0F717F583 for ; Tue, 3 Jun 2008 11:13:07 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by kawa.mindstep.fr (Postfix) with ESMTP id 2C2161FFCF for ; Tue, 3 Jun 2008 10:53:59 +0200 (CEST) (envelope-from freebsd-fs@mindstep.com) X-Virus-Scanned: by amavisd-new on ZunoBox at kawa.mindstep.fr Received: from kawa.mindstep.fr ([127.0.0.1]) by localhost (kawa.mindstep.fr [127.0.0.1]) (amavisd-new, port 10024) with LMTP id lBVGZEY41DUv for ; Tue, 3 Jun 2008 10:53:58 +0200 (CEST) Received: from [192.168.25.13] (pickwick.mindstep.fr [192.168.25.13]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by kawa.mindstep.fr (Postfix) with ESMTP id 0F1021FF4C for ; Tue, 3 Jun 2008 10:53:58 +0200 (CEST) (envelope-from freebsd-fs@mindstep.com) Message-ID: <48450B21.3080700@mindstep.com> Date: Tue, 03 Jun 2008 11:13:05 +0200 From: Patrick Bihan-Faou User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4844F171.3000801@mindstep.com> <20080603081140.GA62607@eos.sc1.parodius.com> In-Reply-To: <20080603081140.GA62607@eos.sc1.parodius.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: Booting a disk with gmirror and gjournal X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@mindstep.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 09:13:09 -0000 Jeremy Chadwick a écrit : > On Tue, Jun 03, 2008 at 09:23:29AM +0200, Patrick Bihan-Faou wrote: > >> Hi, >> >> I am trying to setup a system with one gjournal'ed partition mirrored on >> two disks using FreeBSD 6.3, and it does not work... >> >> Here is what I have routinely done so far: >> >> one partition, no journal, mirrored disks => I can have a working system. >> one partition, no mirror, journal => everything is ok >> >> but as soon as I try combine both gjournal and gmirror, I get stuck: once >> the system boots I get stuck at the F1 prompt and things do go any further. >> >> >> Could anybody help me with this ? >> >> Patrick. >> > > As far as I know, FreeBSD's bootloader doesn't understand how to handle > such things. > > Actually, the bootloader copes just fine with such a setup. I just didn't do things in the right order. Here is how I managed to get it going: # gmirror label -h gm0 ad0 # fdisk -I -B -v /dev/mirror/gm0 # bsdlabel -r -w /dev/mirror/gm0s1 auto # bsdlabel /dev/mirror/gm0s1 > bsdlabel.file ... edit the label to create one single a partition (and/or whatever ones need) # bsdlabel -R -B /dev/mirror/gm0s1 bsdlabel.file # gjournal label -f /dev/mirror/gm0s1a # newfs -L 'boothd' -J /dev/mirror/gm0s1a.journal ... on now install freebsd on /dev/mirror/gm0s1a.journal or /dev/ufs/boothd Sorry for the noise, my problem was due to not doing things in the right order. Patrick. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 3 09:33:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47DC51065671 for ; Tue, 3 Jun 2008 09:33:28 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [91.103.162.4]) by mx1.freebsd.org (Postfix) with ESMTP id 046C88FC1E for ; Tue, 3 Jun 2008 09:33:28 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id AB62719E023; Tue, 3 Jun 2008 11:18:14 +0200 (CEST) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 2E7DF19E019; Tue, 3 Jun 2008 11:18:12 +0200 (CEST) Message-ID: <48450C67.7080907@quip.cz> Date: Tue, 03 Jun 2008 11:18:31 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: freebsd-fs@mindstep.com References: <4844F171.3000801@mindstep.com> In-Reply-To: <4844F171.3000801@mindstep.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Booting a disk with gmirror and gjournal X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 09:33:28 -0000 Patrick Bihan-Faou wrote: > Hi, > > I am trying to setup a system with one gjournal'ed partition mirrored on > two disks using FreeBSD 6.3, and it does not work... > > Here is what I have routinely done so far: > > one partition, no journal, mirrored disks => I can have a working system. > one partition, no mirror, journal => everything is ok > > but as soon as I try combine both gjournal and gmirror, I get stuck: > once the system boots I get stuck at the F1 prompt and things do go any > further. > > > Could anybody help me with this ? Can you discribe it more closely? Do you want to have 2 physical disks in gmirror with only 1 slice covering whole disk with only 1 partition (gjournaled) used as root (/) of your system? Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Tue Jun 3 12:55:31 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF476106568E for ; Tue, 3 Jun 2008 12:55:31 +0000 (UTC) (envelope-from lopez.on.the.lists@yellowspace.net) Received: from mail.yellowspace.net (neruda.yellowspace.net [80.190.200.164]) by mx1.freebsd.org (Postfix) with ESMTP id 884EE8FC1E for ; Tue, 3 Jun 2008 12:55:31 +0000 (UTC) (envelope-from lopez.on.the.lists@yellowspace.net) Received: from five.intranet ([88.217.69.39]) (AUTH: LOGIN lopez.on.the.lists@yellowspace.net) by mail.yellowspace.net with esmtp; Tue, 03 Jun 2008 14:45:26 +0200 id 0033C120.0000000048453CE6.00002C8B Message-Id: <38DAE942-319A-4A44-A8F6-491D4269A8E7@yellowspace.net> From: Lorenzo Perone To: freebsd-fs@freebsd.org In-Reply-To: <48446C42.4070208@mawer.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v924) Date: Tue, 3 Jun 2008 14:45:26 +0200 References: <683A6ED2-0E54-42D7-8212-898221C05150@thefrog.net> <20080518124217.GA16222@eos.sc1.parodius.com> <93F07874-8D5F-44AE-945F-803FFC3B9279@thefrog.net> <16a6ef710806012304m48b63161oee1bc6d11e54436a@mail.gmail.com> <20080602064023.GA95247@eos.sc1.parodius.com> <48446C42.4070208@mawer.org> X-Mailer: Apple Mail (2.924) Subject: Re: ZFS lockup in "zfs" state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2008 12:55:32 -0000 Hello, just to add one more voice to the issue: I'm experiencing the lockups with zfs too. Environment: development test machine, amd64, 3GHz AMD, 2GB ram, running FreeBSD/amd64 7.0-STABLE #8, Sat Apr 26 10:10:53 CEST 2008, with one 400GB SATA disk devoted completely to a zpool (no raid of any kind). This disk has 5 filesystems which get rsynced on a daily basis from different other development hosts. Some of the filesystems are nfs-exported. /boot/loader.conf contains: vm.kmem_size=900M vm.kmem_size_max=900M vfs.zfs.arc_max=300M vfs.zfs.prefetch_disable=1 The disk itself has no known hw problems. A script controlled by cron makes a daily or weekly snapshot of the filesystems (at 2:30 AM). Before that, a "housekeeping" script checks for available space, and if the space is getting below a certain threshold, it destroys older snapshots (at 1:30 am). The rsyncs to the pool all happen a few hours later (4:30 am). I've seen lockups periodically, where I could not do anything else but hard-reboot the machine to unstuck it. It was possible to use other filesystems, but any process trying to access the zpool would hang. Now the very first hang was about 3 months after 7.0-BETA4, which was when I first setup the pools. I then csupped and rebuilt world and kernel periodically, the last time being end of april. After that I got those lockups more often, that is, after a maximum of 2 weeks. I noticed that now that I lowered the threshold of the "housekeeping" script, it hasn't locked up for about 3 weeks. That seems to point at a problem with zfs destroy fs@snapshot - or to anything my script does, so here's a link to it: http://lorenzo.yellowspace.net/zfs_housekeeping.sh.txt haven't seen any adX- timeouts or any other suspicious console messages so far. If there is anything I can provide to help nail down zfs problems please refer to it and I'll do my best... Thanx to everyone working on this great OS and on this cute file/volsystem :) Regards, Lorenzo On 02.06.2008, at 23:55, Antony Mawer wrote: > Jeremy Chadwick wrote: >> On Mon, Jun 02, 2008 at 04:04:12PM +1000, Andrew Hill wrote: > ... >>> unfortunately i couldn't get a backtrace or core dump for >>> 'political' >>> reasons (the system was required for use by others) but i'll see >>> if i can >>> get a panic happening after-hours to get some more info... >> I can't tell you what to do or how to do your job, but honestly you >> should be pulling this system out of production and replacing it >> with a >> different one, or a different implementation, or a different OS. >> Your >> users/employees are probably getting ticked off at the crashes, and >> it >> probably irritates you too. The added benefit is that you could get >> Scott access to the box. > > It's a home fileserver rather than a production "work" system, so > the challenge is finding another system with an equivalent amount of > storage.. :-) As one knows these things are often hard enough to > procure out of a company budget, let alone out of ones own pocket! > > --Antony > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Jun 5 07:45:10 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49ABA1065674; Thu, 5 Jun 2008 07:45:10 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id ACF618FC14; Thu, 5 Jun 2008 07:45:09 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1K4A9j-00099Z-Fx; Thu, 05 Jun 2008 10:45:07 +0300 Message-ID: <4847997D.4060404@icyb.net.ua> Date: Thu, 05 Jun 2008 10:45:01 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: Kostik Belousov References: <4846AFC3.3050101@icyb.net.ua> <20080604152332.GE63348@deviant.kiev.zoral.com.ua> <4846B5D9.1050903@icyb.net.ua> <20080604161001.GF63348@deviant.kiev.zoral.com.ua> In-Reply-To: <20080604161001.GF63348@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, freebsd-geom@freebsd.org Subject: Re: mystery: lock up after fs dump X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2008 07:45:10 -0000 on 04/06/2008 19:10 Kostik Belousov said the following: > SU are irrelevant to the problem I am thinking of. > > vfs_write_suspend() returns 0 when the filesystem being suspended is already > in suspend state. vfs_write_resume() clears the suspend state. > > vfs_write_suspend/vfs_write_resume are used both by snapshot code and > the gjournal. If two users of these interfaces interleave, then you could > get: > > thread1 thread2 > > vfs_write_suspend() > <- fs is suspended there > vfs_write_suspend() <- returns 0 > vfs_write_resume() > <- fs is no more suspended > thread2 is burned in flame > > Snapshots are protected against this because they are created through > the mount(2). The mount(2) locks the covered vnode and thus serializes > snapshot creation (I think there are further serialization points that > prevent simultaneous snapshotting of the same fs). > > There is nothing I can see that protects snapshots/gjournal interaction. Looks like something to be quite concerned about. Thank you for the analysis. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Thu Jun 5 10:48:40 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9FEA1065677 for ; Thu, 5 Jun 2008 10:48:40 +0000 (UTC) (envelope-from jh@saunalahti.fi) Received: from emh01.mail.saunalahti.fi (emh01.mail.saunalahti.fi [62.142.5.107]) by mx1.freebsd.org (Postfix) with ESMTP id 6E07C8FC13 for ; Thu, 5 Jun 2008 10:48:40 +0000 (UTC) (envelope-from jh@saunalahti.fi) Received: from saunalahti-vams (vs3-10.mail.saunalahti.fi [62.142.5.94]) by emh01-2.mail.saunalahti.fi (Postfix) with SMTP id 9F0931ACF0 for ; Thu, 5 Jun 2008 13:29:01 +0300 (EEST) Received: from emh06.mail.saunalahti.fi ([62.142.5.116]) by vs3-10.mail.saunalahti.fi ([62.142.5.94]) with SMTP (gateway) id A04E0E9BDA3; Thu, 05 Jun 2008 13:29:01 +0300 Received: from a91-153-120-204.elisa-laajakaista.fi (a91-153-120-204.elisa-laajakaista.fi [91.153.120.204]) by emh06.mail.saunalahti.fi (Postfix) with SMTP id 8DD9EE51A8 for ; Thu, 5 Jun 2008 13:29:00 +0300 (EEST) Date: Thu, 5 Jun 2008 13:29:00 +0300 From: Jaakko Heinonen To: freebsd-fs@freebsd.org Message-ID: <20080605102900.GA1971@a91-153-120-204.elisa-laajakaista.fi> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) X-Antivirus: VAMS Subject: =?utf-8?b?W3BhdGNoXcKgYnVn?= in cd9660 readdir code X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2008 10:48:40 -0000 Hi, There's a bug in cd9660_readdir() (src/sys/fs/cd9660/cd9660_vnops.c) which may change the directory position (offset) to an invalid value. The problem is that if all directory entries has been read and idp->curroff >= endsearch the code doesn't enter to while (idp->curroff < endsearch) { loop which initializes idp->uio_off. Later in code uio->uio_offset is changed to idp->uio_off (which may be uninitialized). The PR 122925 (http://www.freebsd.org/cgi/query-pr.cgi?pr=122925) has a real life example of a problem caused by this bug. There's also a stripped down test program attached to the PR. Problems include readdir(3) restarting from random position and geom errors caused by read attempts from bogus offsets. Does following patch look good? Few people have tested the patch and it has fixed problems for them. If the patch looks good could someone consider committing it? Index: cd9660_vnops.c =================================================================== RCS file: /home/ncvs/src/sys/fs/cd9660/cd9660_vnops.c,v retrieving revision 1.113 diff -p -u -r1.113 cd9660_vnops.c --- cd9660_vnops.c 15 Feb 2007 22:08:34 -0000 1.113 +++ cd9660_vnops.c 20 May 2008 06:45:20 -0000 @@ -495,6 +495,7 @@ cd9660_readdir(ap) } idp->eofflag = 1; idp->curroff = uio->uio_offset; + idp->uio_off = uio->uio_offset; if ((entryoffsetinblock = idp->curroff & bmask) && (error = cd9660_blkatoff(vdp, (off_t)idp->curroff, NULL, &bp))) { -- Jaakko From owner-freebsd-fs@FreeBSD.ORG Sat Jun 7 14:39:51 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7477A10656A8 for ; Sat, 7 Jun 2008 14:39:51 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206046210.chello.pl [87.206.46.210]) by mx1.freebsd.org (Postfix) with ESMTP id E7BEA8FC25 for ; Sat, 7 Jun 2008 14:39:50 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id DEFF245C99; Sat, 7 Jun 2008 16:39:48 +0200 (CEST) Received: from localhost (chello087206046210.chello.pl [87.206.46.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 0B02E45B26; Sat, 7 Jun 2008 16:39:44 +0200 (CEST) Date: Sat, 7 Jun 2008 16:39:44 +0200 From: Pawel Jakub Dawidek To: Andrei Kolu Message-ID: <20080607143944.GC1344@garage.freebsd.pl> References: <200712071131.31773.antik@bsd.ee> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kfjH4zxOES6UT95V" Content-Disposition: inline In-Reply-To: <200712071131.31773.antik@bsd.ee> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: raidtest: Cannot open 'raidtest.data' device: Operation not permitted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Jun 2008 14:39:51 -0000 --kfjH4zxOES6UT95V Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Dec 07, 2007 at 11:31:31AM +0200, Andrei Kolu wrote: > # uname -a > FreeBSD test.demo 7.0-BETA4 FreeBSD 7.0-BETA4 #0: Sun Dec 2 16:34:41 UTC= 2007 =20 > root@myers.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >=20 > 3ware device driver for 9000 series storage controllers, version: 3.70.05= .001 > twa0: <3ware 9000 series Storage Controller> port 0x3000-0x30ff mem=20 > 0xd8000000-0xd9ffffff,0xda300000-0xda300fff irq 16 at device 0.0 on pci7 > twa0: [ITHREAD] > twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: > twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8 po= rts,=20 > Firmware FE9X 3.08.02.007, BIOS BE9X 3.08.00.002 >=20 > raidtest-1.1 =3D up-to-date with port >=20 > # set mediasize=3D`diskinfo /dev/da0 | awk '{print $3}'` > # set sectorsize=3D`diskinfo /dev/da0 | awk '{print $2}'` > # raidtest genfile -s $mediasize -S $sectorsize -n 50000 > # raidtest test -d /dev/da0 -n 10 > raidtest: Cannot open 'raidtest.data' device: Operation not permitted >=20 > # echo $mediasize > 1919932170240 > # echo $sectorsize > 512 >=20 >=20 > Or anyone can recommend other raid performance testing utility? A bit late, but... There was a bug in raidtest that it printed wrong file name. What it wanted to say was: raidtest: Cannot open '/dev/da0' device: Operation not permitted Which means it cannot open /dev/da0 for reading and writing, because it is already open for writting. If you only want to test reading performance try 'raidtest test -r'. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --kfjH4zxOES6UT95V Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFISp2vForvXbEpPzQRAi5EAJ9JTxOhdAiRxnD6GCX3BeeAlIAjiACgsIrL XtzrorG/+XTCIkFaShujdxA= =4Bm5 -----END PGP SIGNATURE----- --kfjH4zxOES6UT95V--