From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 01:52:36 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5CE2106566B; Sun, 10 Jan 2010 01:52:36 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8C73A8FC0A; Sun, 10 Jan 2010 01:52:36 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0A1qaY9000815; Sun, 10 Jan 2010 01:52:36 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0A1qa4h000811; Sun, 10 Jan 2010 01:52:36 GMT (envelope-from linimon) Date: Sun, 10 Jan 2010 01:52:36 GMT Message-Id: <201001100152.o0A1qa4h000811@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/142558: [msdosfs] patch] Minor updates to fs/msdosfs headers (from NetBSD) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 01:52:36 -0000 Old Synopsis: Minor updates to fs/msdosfs headers (from NetBSD) New Synopsis: [msdosfs] patch] Minor updates to fs/msdosfs headers (from NetBSD) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sun Jan 10 01:52:08 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=142558 From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 09:32:15 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 167161065695 for ; Sun, 10 Jan 2010 09:32:15 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id 73FEE8FC17 for ; Sun, 10 Jan 2010 09:32:14 +0000 (UTC) Received: by fxm27 with SMTP id 27so5844265fxm.3 for ; Sun, 10 Jan 2010 01:32:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:to:subject:organization:from :date:message-id:user-agent:mime-version:content-type :content-transfer-encoding; bh=HE2BbMiZpEGRQqVU0HGCzDe+MmX0xscxPP8DNbpDRCY=; b=IPwRgaeQh3+JvgAN4sWCM/g+0TasZrrITvg0qTvFaJdInZMcaThsGV4dJLzy1DQmgz 4qQ1Q/NozpYVovwtVgOlK+PDV2mQg6p6FSJkAziKTzwqETTJM1eObIsbfDP01lGconer 47haJ3iAP+gIvVosjxu82yrkHYfpleNZKBo7s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=to:subject:organization:from:date:message-id:user-agent :mime-version:content-type:content-transfer-encoding; b=oYUX+ZM91E8Xb9R9VjErfN1VyISE+aboJRNXRDJhY0fmk8BayQUlsqRNpwAt7Ct4q3 28JCFV93LRLw8B4gUbZGZ0mLxQqU+aMPrZf85rQrLWcASQFgtvkpxP/fVD52HuVaQLVa g+rATRs3RogUA7v38DOei7rVMvV+FMlgR86cc= Received: by 10.223.74.144 with SMTP id u16mr6735627faj.21.1263114238381; Sun, 10 Jan 2010 01:03:58 -0800 (PST) Received: from localhost ([95.69.171.121]) by mx.google.com with ESMTPS id 16sm9040162fxm.8.2010.01.10.01.03.57 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 10 Jan 2010 01:03:57 -0800 (PST) To: freebsd-fs@FreeBSD.org Organization: TOA Ukraine From: Mikolaj Golub Date: Sun, 10 Jan 2010 11:03:56 +0200 Message-ID: <86ocl272mb.fsf@kopusha.onet> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Cc: Subject: FreeBSD NFS client/Linux NFS server issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 09:32:15 -0000 Hi, I wrote about this problem some times ago to freebsd-stable http://lists.freebsd.org/pipermail/freebsd-stable/2009-December/053746.html but there have not been any response there so far and also we have collected some additional information since then so I am trying again here :-) We have CentOS 5.3 NFS server and many FreeBSD clients. The problem have been observed since we started upgrade FreeBSD host from 6.3 to 7.1 so we thought this is 7.1 related bu recently we had the similar issue with 6.3 host and actually it looks like the problem is rather on NFS server side, but I would like to describe the problem here and ask some advise. So the "problem" FreeBSD clients have several NFS mounts from CentOS 5.3 server. Mount options: rw,-3,-T,-s,-i,-r=32768,-w=32768,-o=noinet6 At some point on one of the clients some of the mounts might get stuck -- we observe one or two our php applications hanged on writing to NFS folder in nfs_flush()->bufobj_wwait() and all other processes that try to access this NFS mount hang acquiring nfs vn_lock hold by one of the above php processes. For one of the incident we were tcpdumping "problem" NFS connection for about 1 hour and during this hour an activity was observed only once: 08:20:38.281422 IP (tos 0x0, ttl 64, id 56110, offset 0, flags [DF], proto TCP (6), length 140) 172.30.10.27.344496259 > 172.30.10.121.2049: 88 access fh[1:9300:10df8001] 003f 08:20:38.281554 IP (tos 0x0, ttl 64, id 26624, offset 0, flags [DF], proto TCP (6), length 52) 172.30.10.121.2049 > 172.30.10.27.971: ., cksum 0xca5e (correct), 89408667:89408667(0) ack 1517941890 win 46 The client sent rpc ACCESS request for root exported inode, received tcp ack response (so tcp connection was ok) but did not receive any RPC reply from the server. So it looks like the problem on NFS server side. But for me it looks a bit strange that freebsd client is sending rpc packets so rarely. Shouldn't it retransmit them more frequently? For another incident we monitored tcp connection for 4 minutes and did not see any packets then. Unfortunately we can't run tcpdumping long time as these are production servers and we need to reboot hosts to restore normal operations. When the client was in such state we made it crush dump so I have a core file for the investigation. Below is some relevant data from the crush dump. Could you please advise what other things would be interesting to look at in the core for better understanding what is going on? Backtraces of two writing php processes hanged in nfs_flush(): #0 sched_switch (td=0xc6befaf0, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f460c in sleepq_catch_signals (wchan=0xc9cb21f8) at /usr/src/sys/kern/subr_sleepqueue.c:417 #4 0xc07f4ebd in sleepq_wait_sig (wchan=0xc9cb21f8) at /usr/src/sys/kern/subr_sleepqueue.c:594 #5 0xc07cb047 in _sleep (ident=0xc9cb21f8, lock=0xc9cb219c, priority=333, wmesg=0xc0b731ed "bo_wwait", timo=0) at /usr/src/sys/kern/kern_synch.c:224 #6 0xc0827295 in bufobj_wwait (bo=0xc9cb21d4, slpflag=256, timeo=0) at /usr/src/sys/kern/vfs_bio.c:3870 #7 0xc0966307 in nfs_flush (vp=0xc9cb2114, waitfor=1, td=0xc6befaf0, commit=1) at /usr/src/sys/nfsclient/nfs_vnops.c:2989 #8 0xc09667ce in nfs_fsync (ap=0xed2258ec) at /usr/src/sys/nfsclient/nfs_vnops.c:2725 #9 0xc0aee5d2 in VOP_FSYNC_APV (vop=0xc0c2b920, a=0xed2258ec) at vnode_if.c:1007 #10 0xc0827864 in bufsync (bo=0xc9cb21d4, waitfor=1, td=0xc6befaf0) at vnode_if.h:538 #11 0xc083f354 in bufobj_invalbuf (bo=0xc9cb21d4, flags=1, td=0xc6befaf0, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1066 #12 0xc083f6e2 in vinvalbuf (vp=0xc9cb2114, flags=1, td=0xc6befaf0, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1142 #13 0xc094f216 in nfs_vinvalbuf (vp=0xc9cb2114, flags=Variable "flags" is not available. ) at /usr/src/sys/nfsclient/nfs_bio.c:1326 #14 0xc0951825 in nfs_write (ap=0xed225bc4) at /usr/src/sys/nfsclient/nfs_bio.c:918 #15 0xc0aef956 in VOP_WRITE_APV (vop=0xc0c2b920, a=0xed225bc4) at vnode_if.c:691 #16 0xc0850097 in vn_write (fp=0xc902aafc, uio=0xed225c60, active_cred=0xcb27b900, flags=0, td=0xc6befaf0) at vnode_if.h:373 #17 0xc07f9d17 in dofilewrite (td=0xc6befaf0, fd=6, fp=0xc902aafc, auio=0xed225c60, offset=-1, flags=0) at file.h:256 #18 0xc07f9ff8 in kern_writev (td=0xc6befaf0, fd=6, auio=0xed225c60) at /usr/src/sys/kern/sys_generic.c:401 #19 0xc07fa06f in write (td=0xc6befaf0, uap=0xed225cfc) at /usr/src/sys/kern/sys_generic.c:317 #20 0xc0ad9c75 in syscall (frame=0xed225d38) at /usr/src/sys/i386/i386/trap.c:1090 #21 0xc0ac01b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) #0 sched_switch (td=0xc731c8c0, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f460c in sleepq_catch_signals (wchan=0xcafa375c) at /usr/src/sys/kern/subr_sleepqueue.c:417 #4 0xc07f4ebd in sleepq_wait_sig (wchan=0xcafa375c) at /usr/src/sys/kern/subr_sleepqueue.c:594 #5 0xc07cb047 in _sleep (ident=0xcafa375c, lock=0xcafa3700, priority=333, wmesg=0xc0b731ed "bo_wwait", timo=0) at /usr/src/sys/kern/kern_synch.c:224 #6 0xc0827295 in bufobj_wwait (bo=0xcafa3738, slpflag=256, timeo=0) at /usr/src/sys/kern/vfs_bio.c:3870 #7 0xc0966307 in nfs_flush (vp=0xcafa3678, waitfor=1, td=0xc731c8c0, commit=1) at /usr/src/sys/nfsclient/nfs_vnops.c:2989 #8 0xc09667ce in nfs_fsync (ap=0xed2578ec) at /usr/src/sys/nfsclient/nfs_vnops.c:2725 #9 0xc0aee5d2 in VOP_FSYNC_APV (vop=0xc0c2b920, a=0xed2578ec) at vnode_if.c:1007 #10 0xc0827864 in bufsync (bo=0xcafa3738, waitfor=1, td=0xc731c8c0) at vnode_if.h:538 #11 0xc083f354 in bufobj_invalbuf (bo=0xcafa3738, flags=1, td=0xc731c8c0, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1066 #12 0xc083f6e2 in vinvalbuf (vp=0xcafa3678, flags=1, td=0xc731c8c0, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1142 #13 0xc094f216 in nfs_vinvalbuf (vp=0xcafa3678, flags=Variable "flags" is not available. ) at /usr/src/sys/nfsclient/nfs_bio.c:1326 #14 0xc0951825 in nfs_write (ap=0xed257bc4) at /usr/src/sys/nfsclient/nfs_bio.c:918 #15 0xc0aef956 in VOP_WRITE_APV (vop=0xc0c2b920, a=0xed257bc4) at vnode_if.c:691 #16 0xc0850097 in vn_write (fp=0xc8c232f8, uio=0xed257c60, active_cred=0xc89ee600, flags=0, td=0xc731c8c0) at vnode_if.h:373 #17 0xc07f9d17 in dofilewrite (td=0xc731c8c0, fd=6, fp=0xc8c232f8, auio=0xed257c60, offset=-1, flags=0) at file.h:256 #18 0xc07f9ff8 in kern_writev (td=0xc731c8c0, fd=6, auio=0xed257c60) at /usr/src/sys/kern/sys_generic.c:401 #19 0xc07fa06f in write (td=0xc731c8c0, uap=0xed257cfc) at /usr/src/sys/kern/sys_generic.c:317 #20 0xc0ad9c75 in syscall (frame=0xed257d38) at /usr/src/sys/i386/i386/trap.c:1090 #21 0xc0ac01b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) fr 6 #6 0xc0827295 in bufobj_wwait (bo=0xcafa3738, slpflag=256, timeo=0) at /usr/src/sys/kern/vfs_bio.c:3870 3870 error = msleep(&bo->bo_numoutput, BO_MTX(bo), (kgdb) list 3865 KASSERT(bo != NULL, ("NULL bo in bufobj_wwait")); 3866 ASSERT_BO_LOCKED(bo); 3867 error = 0; 3868 while (bo->bo_numoutput) { 3869 bo->bo_flag |= BO_WWAIT; 3870 error = msleep(&bo->bo_numoutput, BO_MTX(bo), 3871 slpflag | (PRIBIO + 1), "bo_wwait", timeo); 3872 if (error) 3873 break; 3874 } (kgdb) p *bo $1 = {bo_mtx = 0xcafa3700, bo_clean = {bv_hd = {tqh_first = 0xda869ec0, tqh_last = 0xda869ef8}, bv_root = 0xda869ec0, bv_cnt = 1}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xcafa374c}, bv_root = 0x0, bv_cnt = 0}, bo_numoutput = 1, bo_flag = 2, bo_ops = 0xc0c2bae0, bo_bsize = 32768, bo_object = 0xca637744, bo_synclist = {le_next = 0xcb64d510, le_prev = 0xca129aac}, bo_private = 0xcafa3678, __bo_vnode = 0xcafa3678} The stack of the some of the processes gotten stuck acquiring _vn_lock: #0 sched_switch (td=0xc8bab460, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f4946 in sleepq_wait (wchan=0xcafa36d0) at /usr/src/sys/kern/subr_sleepqueue.c:580 #4 0xc07cb056 in _sleep (ident=0xcafa36d0, lock=0xc0c77880, priority=80, wmesg=0xc0b80b92 "nfs", timo=0) at /usr/src/sys/kern/kern_synch.c:226 #5 0xc07adf5a in acquire (lkpp=0xed5fd7f0, extflags=Variable "extflags" is not available. ) at /usr/src/sys/kern/kern_lock.c:151 #6 0xc07ae84c in _lockmgr (lkp=0xcafa36d0, flags=8194, interlkp=0xcafa3700, td=0xc8bab460, file=0xc0b74aeb "/usr/src/sys/kern/vfs_subr.c", line=2061) at /usr/src/sys/kern/kern_lock.c:384 #7 0xc0832470 in vop_stdlock (ap=0xed5fd840) at /usr/src/sys/kern/vfs_default.c:305 #8 0xc0aef4f6 in VOP_LOCK1_APV (vop=0xc0c1d5c0, a=0xed5fd840) at vnode_if.c:1618 #9 0xc084ed86 in _vn_lock (vp=0xcafa3678, flags=8194, td=0xc8bab460, file=0xc0b74aeb "/usr/src/sys/kern/vfs_subr.c", line=2061) at vnode_if.h:851 #10 0xc0841d84 in vget (vp=0xcafa3678, flags=8194, td=0xc8bab460) at /usr/src/sys/kern/vfs_subr.c:2061 #11 0xc08355b3 in vfs_hash_get (mp=0xc7232000, hash=3326873010, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/vfs_hash.c:81 #12 0xc09534d4 in nfs_nget (mntp=0xc7232000, fhp=0xcd5bd878, fhsize=20, npp=0xed5fd9f0, flags=2) at /usr/src/sys/nfsclient/nfs_node.c:120 #13 0xc0964a05 in nfs_lookup (ap=0xed5fda84) at /usr/src/sys/nfsclient/nfs_vnops.c:947 #14 0xc0aefbe6 in VOP_LOOKUP_APV (vop=0xc0c2b920, a=0xed5fda84) at vnode_if.c:99 #15 0xc0836841 in lookup (ndp=0xed5fdb48) at vnode_if.h:57 #16 0xc083756f in namei (ndp=0xed5fdb48) at /usr/src/sys/kern/vfs_lookup.c:219 #17 0xc0844fef in kern_lstat (td=0xc8bab460, path=0x486112b0
, pathseg=UIO_USERSPACE, sbp=0xed5fdc18) at /usr/src/sys/kern/vfs_syscalls.c:2169 #18 0xc08451af in lstat (td=0xc8bab460, uap=0xed5fdcfc) at /usr/src/sys/kern/vfs_syscalls.c:2152 #19 0xc0ad9c75 in syscall (frame=0xed5fdd38) at /usr/src/sys/i386/i386/trap.c:1090 #20 0xc0ac01b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #21 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) p *lkp $2 = {lk_object = {lo_name = 0xc0b80b92 "nfs", lo_type = 0xc0b80b92 "nfs", lo_flags = 70844416, lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, lk_interlock = 0xc0c77880, lk_flags = 33816640, lk_sharecount = 0, lk_waitcount = 1, lk_exclusivecount = 1, lk_prio = 80, lk_timo = 51, lk_lockholder = 0xc731c8c0, lk_newlock = 0x0} Note lk_lockholder here is one of two php processes above. nfsmount structure of the "problem" mount: $4 = {nm_mtx = {lock_object = {lo_name = 0xc0b808ee "NFSmount lock", lo_type = 0xc0b808ee "NFSmount lock", lo_flags = 16973824, lo_witness_data = {lod_list = { stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, nm_flag = 35399, nm_state = 1310720, nm_mountp = 0xc7232000, nm_numgrps = 16, nm_fh = "\001\000\000\000\000\223\000\000\001@\003\n", '\0' , nm_fhsize = 12, nm_rpcclnt = {rc_flag = 0, rc_wsize = 0, rc_rsize = 0, rc_name = 0x0, rc_so = 0x0, rc_sotype = 0, rc_soproto = 0, rc_soflags = 0, rc_timeo = 0, rc_retry = 0, rc_srtt = {0, 0, 0, 0}, rc_sdrtt = {0, 0, 0, 0}, rc_sent = 0, rc_cwnd = 0, rc_timeouts = 0, rc_deadthresh = 0, rc_authtype = 0, rc_auth = 0x0, rc_prog = 0x0, rc_proctlen = 0, rc_proct = 0x0}, nm_so = 0xc722e340, nm_sotype = 1, nm_soproto = 0, nm_soflags = 44, nm_nam = 0xc68eca00, nm_timeo = 6000, nm_retry = 2, nm_srtt = {15, 15, 17, 16}, nm_sdrtt = {3, 3, 4, 3}, nm_sent = 0, nm_cwnd = 4096, nm_timeouts = 0, nm_deadthresh = 9, nm_rsize = 32768, nm_wsize = 32768, nm_readdirsize = 4096, nm_readahead = 1, nm_wcommitsize = 1177026, nm_acdirmin = 30, nm_acdirmax = 60, nm_acregmin = 3, nm_acregmax = 60, nm_verf = "Jë¾W\000\004oí", nm_bufq = {tqh_first = 0xda93cc68, tqh_last = 0xda869f0c}, nm_bufqlen = 2, nm_bufqwant = 0, nm_bufqiods = 1, nm_maxfilesize = 1099511627775, nm_rpcops = 0xc0c2b5bc, nm_tprintf_initial_delay = 12, nm_tprintf_delay = 30, nm_nfstcpstate = {rpcresid = 0, flags = 3, sock_send_inprog = 0}, nm_hostname = "172.30.10.92\000/var/www/app31", '\0' , nm_clientid = 0, nm_fsid = {val = {0, 0}}, nm_lease_time = 0, nm_last_renewal = 0} (kgdb) p *nmp->nm_bufq.tqh_first $9 = {b_bufobj = 0xc9cb21d4, b_bcount = 3545, b_caller1 = 0x0, b_data = 0xdd74d000 " 1641\n\n Warning summary:\n Empty Service Category:", ' ' , "10\n\nUpload finished at 2010-01-08 20:17:10 GMT; Run time is 8.5791680812836 sec.\n\n\n\nUpdate upload started at 2010-01-08"..., b_error = 0, b_iocmd = 2 '\002', b_ioflags = 0 '\0', b_iooffset = 163840, b_resid = 0, b_iodone = 0, b_blkno = 320, b_offset = 163840, b_bobufs = { tqe_next = 0x0, tqe_prev = 0xc9cb21d8}, b_left = 0x0, b_right = 0x0, b_vflags = 0, b_freelist = { tqe_next = 0xda869ec0, tqe_prev = 0xc722a3c0}, b_qindex = 0, b_flags = 536870948, b_xflags = 2 '\002', b_lock = {lk_object = {lo_name = 0xc0b73635 "bufwait", lo_type = 0xc0b73635 "bufwait", lo_flags = 70844416, lo_witness_data = {lod_list = { stqe_next = 0x0}, lod_witness = 0x0}}, lk_interlock = 0xc0c77fa0, lk_flags = 262144, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 1, lk_prio = 80, lk_timo = 0, lk_lockholder = 0xfffffffe, lk_newlock = 0x0}, b_bufsize = 3584, b_runningbufspace = 0, b_kvabase = 0xdd74d000 " 1641\n\n Warning summary:\n Empty Service Category:", ' ' , "10\n\nUpload finished at 2010-01-08 20:17:10 GMT; Run time is 8.5791680812836 sec.\n\n\n\nUpdate upload started at 2010-01-08"..., b_kvasize = 32768, b_lblkno = 5, b_vp = 0xc9cb2114, b_dirtyoff = 3492, b_dirtyend = 3545, b_rcred = 0x0, b_wcred = 0xcb27b900, b_saveaddr = 0xdd74d000, b_pager = {pg_reqpage = 0}, b_cluster = {cluster_head = {tqh_first = 0xda8f2e60, tqh_last = 0xda95fc34}, cluster_entry = {tqe_next = 0xda8f2e60, tqe_prev = 0xda95fc34}}, b_pages = {0xc2e543a0, 0x0 }, b_npages = 1, b_dep = {lh_first = 0x0}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, b_fsprivate3 = 0x0, b_pin_count = 0} (some bytes of the log php process was trying to write on nfs share :-) When the incident occurs we usally have 1-3 nfsiod processes serving other "good" shares. In this particular case when crush dump was generated we had 1 nfsiod process: (kgdb) bt #0 sched_switch (td=0xc724cd20, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f460c in sleepq_catch_signals (wchan=0xc0c89f00) at /usr/src/sys/kern/subr_sleepqueue.c:417 #4 0xc07f4db9 in sleepq_timedwait_sig (wchan=0xc0c89f00) at /usr/src/sys/kern/subr_sleepqueue.c:631 #5 0xc07cb021 in _sleep (ident=0xc0c89f00, lock=0xc0c89ed0, priority=348, wmesg=0xc0b69679 "-", timo=120000) at /usr/src/sys/kern/kern_synch.c:220 #6 0xc095b46e in nfssvc_iod (instance=0xc0c89880) at /usr/src/sys/nfsclient/nfs_nfsiod.c:244 #7 0xc079e6c9 in fork_exit (callout=0xc095b380 , arg=0xc0c89880, frame=0xed27fd38) at /usr/src/sys/kern/kern_fork.c:804 #8 0xc0ac01c0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:264 (kgdb) fr 6 #6 0xc095b46e in nfssvc_iod (instance=0xc0c89880) at /usr/src/sys/nfsclient/nfs_nfsiod.c:244 244 error = msleep(&nfs_iodwant[myiod], &nfs_iod_mtx, PWAIT | PCATCH, (kgdb) list 239 nfs_iodmount[myiod] = NULL; 240 /* 241 * Always keep at least nfs_iodmin kthreads. 242 */ 243 timo = (myiod < nfs_iodmin) ? 0 : nfs_iodmaxidle * hz; 244 error = msleep(&nfs_iodwant[myiod], &nfs_iod_mtx, PWAIT | PCATCH, 245 "-", timo); 246 if (error) { 247 nmp = nfs_iodmount[myiod]; 248 /* (kgdb) p *nmp $5 = {nm_mtx = {lock_object = {lo_name = 0xc0b808ee "NFSmount lock", lo_type = 0xc0b808ee "NFSmount lock", lo_flags = 16973824, lo_witness_data = {lod_list = { stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, nm_flag = 35399, nm_state = 1310720, nm_mountp = 0xc6b46864, nm_numgrps = 16, nm_fh = "\001\000\000\000\000\223\000\000\001\200n\005", '\0' , nm_fhsize = 12, nm_rpcclnt = {rc_flag = 0, rc_wsize = 0, rc_rsize = 0, rc_name = 0x0, rc_so = 0x0, rc_sotype = 0, rc_soproto = 0, rc_soflags = 0, rc_timeo = 0, rc_retry = 0, rc_srtt = {0, 0, 0, 0}, rc_sdrtt = {0, 0, 0, 0}, rc_sent = 0, rc_cwnd = 0, rc_timeouts = 0, rc_deadthresh = 0, rc_authtype = 0, rc_auth = 0x0, rc_prog = 0x0, rc_proctlen = 0, rc_proct = 0x0}, nm_so = 0xc722e000, nm_sotype = 1, nm_soproto = 0, nm_soflags = 44, nm_nam = 0xc68da280, nm_timeo = 6000, nm_retry = 2, nm_srtt = {15, 15, 15, 33}, nm_sdrtt = {3, 3, 3, 21}, nm_sent = 0, nm_cwnd = 4360, nm_timeouts = 0, nm_deadthresh = 9, nm_rsize = 32768, nm_wsize = 32768, nm_readdirsize = 4096, nm_readahead = 1, nm_wcommitsize = 1177026, nm_acdirmin = 30, nm_acdirmax = 60, nm_acregmin = 3, nm_acregmax = 60, nm_verf = "Jë¾W\000\004oí", nm_bufq = {tqh_first = 0x0, tqh_last = 0xc722aa50}, nm_bufqlen = 0, nm_bufqwant = 0, nm_bufqiods = 1, nm_maxfilesize = 1099511627775, nm_rpcops = 0xc0c2b5bc, nm_tprintf_initial_delay = 12, nm_tprintf_delay = 30, nm_nfstcpstate = {rpcresid = 0, flags = 1, sock_send_inprog = 0}, nm_hostname = "172.30.10.75\000/var/www/app20", '\0' , nm_clientid = 0, nm_fsid = {val = {0, 0}}, nm_lease_time = 0, nm_last_renewal = 0} I have another crush dump and it looks similarly. Also could someone please explain me the following thing that is unclear for me due to the lack of necessary knowledge of freebsd internals. Here we have writing processes spinning in bufobj_wwait. As I understand in this state they are just waiting checking bo->bo_numoutput until some other kernel thread finishes I/O on the buffer and clear bo->bo_numoutput. But what is this thread that should do actual IO? I can't find nothing from running threads in back trace that would do this. nfsiod process? Then how is it triggered? Understanding how the things go on would be very helpful for me :-) It would be nice to ensure that the problem is not in a client to concentrate the investigation on the server side. Thanks in advance. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 15:08:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DDE4106566B; Sun, 10 Jan 2010 15:08:39 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f218.google.com (mail-gx0-f218.google.com [209.85.217.218]) by mx1.freebsd.org (Postfix) with ESMTP id 498F78FC0C; Sun, 10 Jan 2010 15:08:39 +0000 (UTC) Received: by gxk10 with SMTP id 10so19755743gxk.3 for ; Sun, 10 Jan 2010 07:08:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=H0SKTpX3MAzf78YJXdcG+7YOk9/khvqbwWSKqz1uwvA=; b=R6DILnfT6LIWkv9lhXGPDSU61gu1TAbXt/fBV+li9W9N/R1mv/a8flAhzLroT2UIKz I437l1JrioqVfTmCWt6YOxMzpOo0z9cICvdd6tme0AFufqRHEkt9v3aQk1IcyPoiJrGb q3Jkp4RctHzbyjX7hUQzBQTQ6YdXrieRLiZkU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=ShMxoGOy4AD0W4ltHxTZEB/W0yqmTzahVYukcWvwm5n9q3WbyiuDupo22cTG8AilPf fIJqeyZP6HvlqRvRrHqpgT/0KDxYgsXpsZGYtyjg3byddelO5yTXjuFnFyJ75qO0Pr2C 020HkR1K0ojO8yLubEQNsMIWO4F6E4bvfi4Ys= MIME-Version: 1.0 Received: by 10.101.132.25 with SMTP id j25mr4982969ann.79.1263136110033; Sun, 10 Jan 2010 07:08:30 -0800 (PST) Date: Sun, 10 Jan 2010 17:08:29 +0200 Message-ID: From: Dan Naumov To: FreeBSD-STABLE Mailing List , freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 15:08:39 -0000 Hello list. I am evaluating options for my new upcoming storage system, where for various reasons the data will be stored on 2 x 2tb SATA disk in a mirror and has to be encrypted (a 40gb Intel SSD will be used for the system disk). Right now I am considering the options of FreeBSD with GELI+ZFS and Debian Linux with MDRAID and cryptofs. Has anyone here made any benchmarks regarding how much of a performance hit is caused by using 2 geli devices as vdevs for a ZFS mirror pool in FreeBSD (a similar configuration is described here: http://blog.experimentalworks.net/2008/03/setting-up-an-encrypted-zfs-with-freebsd/)? Some direct comparisons using bonnie++ or similar, showing the number differences of "this is read/write/IOPS on top of a ZFS mirror and this is read/write/IOPS on top of a ZFS mirror using GELI" would be nice. I am mostly interested in benchmarks on lower end hardware, the system is an Atom 330 which is currently using Windows 2008 server with TrueCrypt in a non-raid configuration and with that setup, I am getting roughly 55mb/s reads and writes when using TrueCrypt (nonencrypted it's around 115mb/s). Thanks. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 17:45:02 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6AF010656BB for ; Sun, 10 Jan 2010 17:45:02 +0000 (UTC) (envelope-from rsmith@xs4all.nl) Received: from smtp-vbr1.xs4all.nl (smtp-vbr1.xs4all.nl [194.109.24.21]) by mx1.freebsd.org (Postfix) with ESMTP id 024408FC35 for ; Sun, 10 Jan 2010 17:45:01 +0000 (UTC) Received: from slackbox.xs4all.nl (slackbox.xs4all.nl [213.84.242.160]) by smtp-vbr1.xs4all.nl (8.13.8/8.13.8) with ESMTP id o0AHVRZ3058700; Sun, 10 Jan 2010 18:31:27 +0100 (CET) (envelope-from rsmith@xs4all.nl) Received: by slackbox.xs4all.nl (Postfix, from userid 1001) id 9688ABA8E; Sun, 10 Jan 2010 18:31:27 +0100 (CET) Date: Sun, 10 Jan 2010 18:31:27 +0100 From: Roland Smith To: Dan Naumov Message-ID: <20100110173127.GA52730@slackbox.xs4all.nl> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="HlL+5n6rz5pIUxbD" Content-Disposition: inline In-Reply-To: X-GPG-Fingerprint: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 X-GPG-Key: http://www.xs4all.nl/~rsmith/pubkey.txt X-GPG-Notice: If this message is not signed, don't assume I sent it! User-Agent: Mutt/1.5.20 (2009-06-14) X-Virus-Scanned: by XS4ALL Virus Scanner Cc: freebsd-fs@freebsd.org, FreeBSD-STABLE Mailing List Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 17:45:02 -0000 --HlL+5n6rz5pIUxbD Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 10, 2010 at 05:08:29PM +0200, Dan Naumov wrote: > Hello list. >=20 > I am evaluating options for my new upcoming storage system, where for > various reasons the data will be stored on 2 x 2tb SATA disk in a > mirror and has to be encrypted (a 40gb Intel SSD will be used for the > system disk). Right now I am considering the options of FreeBSD with > GELI+ZFS and Debian Linux with MDRAID and cryptofs. Has anyone here > made any benchmarks regarding how much of a performance hit is caused > by using 2 geli devices as vdevs for a ZFS mirror pool in FreeBSD (a > similar configuration is described here: > http://blog.experimentalworks.net/2008/03/setting-up-an-encrypted-zfs-wit= h-freebsd/)? > Some direct comparisons using bonnie++ or similar, showing the number > differences of "this is read/write/IOPS on top of a ZFS mirror and > this is read/write/IOPS on top of a ZFS mirror using GELI" would be > nice. >=20 > I am mostly interested in benchmarks on lower end hardware, the system > is an Atom 330 which is currently using Windows 2008 server with > TrueCrypt in a non-raid configuration and with that setup, I am > getting roughly 55mb/s reads and writes when using TrueCrypt > (nonencrypted it's around 115mb/s). Although I cannot comment on ZFS, my $HOME partition is UFS2+geli. Reads (w= ith dd) of uncached big[1] files are ~70MB/s. Reading an unchached big file fro= m a non-encrypted UFS2 partition is ~120MB/s. Note that the vfs cache has a huge influence here; Repeating the same read will be 4 =E2=80=93 7 times faster! The sysctls for ZFS chaching will probably have a big impact too. Roland [1] several 100s of MiB. --=20 R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) --HlL+5n6rz5pIUxbD Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAktKDu8ACgkQEnfvsMMhpyWdWwCeNcyvtNzeYvJIo8ObiJMjrIfF 7GgAoKjQ80Hx8SIgL6QuB8f61Zr/KKay =et7A -----END PGP SIGNATURE----- --HlL+5n6rz5pIUxbD-- From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 18:19:14 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 549E3106566C; Sun, 10 Jan 2010 18:19:14 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-yw0-f197.google.com (mail-yw0-f197.google.com [209.85.211.197]) by mx1.freebsd.org (Postfix) with ESMTP id F20218FC0C; Sun, 10 Jan 2010 18:19:13 +0000 (UTC) Received: by ywh35 with SMTP id 35so11746504ywh.7 for ; Sun, 10 Jan 2010 10:19:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=5jFEha+NBhRM4oBl9F0mD5JdJeVZq6ocYW1/Nn4eQ6o=; b=NRSwrMZas38FdHj20tKP4WUXqU1HnkCQD5ZUOFfTFrpA8WOcoPTrTqw6auPrDFs8g3 47Kgmik0BXhp51bcUtLIKgIr90bw25pJNtUSFid3qaEXAUCCKnHtlmfhW91JPJWHAXec 0Wrad8/g6GRMiXcyTEy98Vw5R6S33OcigQdzM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=tfhFe60NdB/aIHjdaYF5z3QjJ+RE7LlUwrRzscoZZBJsHhhvdofIsn7UcKNwvJp3Md ZHiFQBw/KWVo3Zg9WsPx+qnMHe8v+X1WrZZxSOqp9tE1TuBmtwF3CWCljVcTwsx8AFYw jvGbGBoOwE2qohDIXlO+SpCmv559iw02Bs+J0= MIME-Version: 1.0 Received: by 10.101.3.8 with SMTP id f8mr8763013ani.149.1263147545575; Sun, 10 Jan 2010 10:19:05 -0800 (PST) In-Reply-To: <20100110161206.GA86684@plebeian.afflictions.org> References: <20100110161206.GA86684@plebeian.afflictions.org> Date: Sun, 10 Jan 2010 20:19:05 +0200 Message-ID: From: Dan Naumov To: Damian Gerow , FreeBSD-STABLE Mailing List , freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 18:19:14 -0000 On Sun, Jan 10, 2010 at 6:12 PM, Damian Gerow wrot= e: > Dan Naumov wrote: > : I am mostly interested in benchmarks on lower end hardware, the system > : is an Atom 330 which is currently using Windows 2008 server with > : TrueCrypt in a non-raid configuration and with that setup, I am > : getting roughly 55mb/s reads and writes when using TrueCrypt > : (nonencrypted it's around 115mb/s). > > I've been using GELI-backed vdevs for some time now -- since 7.2-ish > timeframes. =A0I've never benchmarked it, but I was running on relatively > low-end hardware. =A0A few things to take into consideration: > > 1) Make sure the individual drives are encrypted -- especially if they're > =A0 >=3D1TB. =A0This is less a performance thing and more a "make sure yo= ur > =A0 encryption actually encrypts properly" thing. > 2) Seriously consider using the new AHCI driver. =A0I've been using it in= a > =A0 few places, and it's quite stable, and there is a marked performance > =A0 improvement - 10-15% on the hardware I've got. > 3) Take a look at the VIA platform, as a replacement for the Atom. =A0I w= as > =A0 running on an EPIA-SN 1800 (1.8GHz), and didn't have any real trouble= s > =A0 with the encryption aspect of the rig (4x1TB drives). =A0Actually, if= you > =A0 get performance numbers privately comparing the Atom to a VIA (Nano o= r > =A0 otherwise), can you post them to the list? =A0I'm curious to see if t= he > =A0 on-chip encryption actually makes a difference. > 4) Since you're asking for benchmarks, probably best if you post the > =A0 specific bonnie command you want run -- that way, it's tailored to yo= ur > =A0 use-case, and you'll get consistant, comparable results. Yes, this is what I was basically considering: new AHCI driver =3D> 40gb Intel SSD =3D> UFS2 with Softupdates for the system installation new AHCI driver =3D> 2 x 2tb disks, each fully encrypted with geli =3D> 2 geli vdevs for a ZFS mirror for important data The reason I am considering the new AHCI driver is to get NCQ support now and TRIM support for the SSD later when it gets implemented, although if the performance difference right now is already 10-15%, that's a reason good enough on it's own. On a semi-related note, is it still recommended to use softupdates or is GJournal a better choice today? - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 19:28:52 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7927B106566C; Sun, 10 Jan 2010 19:28:52 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f218.google.com (mail-gx0-f218.google.com [209.85.217.218]) by mx1.freebsd.org (Postfix) with ESMTP id 211BB8FC12; Sun, 10 Jan 2010 19:28:51 +0000 (UTC) Received: by gxk10 with SMTP id 10so19856593gxk.3 for ; Sun, 10 Jan 2010 11:28:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=0hSRGY6+kL3bjd+kOmylcIzZ3qtHB1QS2EFsxm5ff8I=; b=t/ny8x6u4Kw1m9WFu4eQDnKncq+KvaGLK6XYDYUeTb8O7prqiBGq5HFRu++4kXYY1X eAtw/3TfTNGvpM91nPjderD9/BRmmpHM/ZnJLDBVvDOd+x8O4aTBSdyW7oTQp9/m8agn CBkGbB8CsmVMfUKjOQS/Gbf6i/opQFg3q2200= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=Ur/TX1mNPK3bpTH2g313bCZeZAb13FcIrqWYHcgSiZgw7mEfMaoR0dBOGWKw+QTAti Hz+Mcm/n8G42fNs63vDx4AFOgpeE1aMsz7Em+dWmwDasLooCzgQJsH8Ri3oC0EACjlkO 09cdQIX5hJn7sZg5Hdti7UiHouGPre+99nZCE= MIME-Version: 1.0 Received: by 10.101.10.15 with SMTP id n15mr24949753ani.82.1263151728952; Sun, 10 Jan 2010 11:28:48 -0800 (PST) In-Reply-To: <20100110184612.GC86684@plebeian.afflictions.org> References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> Date: Sun, 10 Jan 2010 21:28:48 +0200 Message-ID: From: Dan Naumov To: Damian Gerow , freebsd-fs@freebsd.org, FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 19:28:52 -0000 On Sun, Jan 10, 2010 at 8:46 PM, Damian Gerow wrot= e: > Dan Naumov wrote: > : Yes, this is what I was basically considering: > : > : new AHCI driver =3D> 40gb Intel SSD =3D> UFS2 with Softupdates for the > : system installation > : new AHCI driver =3D> 2 x 2tb disks, each fully encrypted with geli =3D>= 2 > : geli vdevs for a ZFS mirror for important data > > If performance is an issue, you may want to consider carving off a partit= ion > on that SSD, geli-fying it, and using it as a ZIL device. =A0You'll proba= bly > see a marked performance improvement with such a setup. That is true, but using a single device for a dedicated ZIL is a huge no-no, considering it's an intent log, it's used to reconstruct the pool in case of a power failure for example, should such an event occur at the same time as a ZIL provider dies, you lose the entire pool because there is no way to recover it, so if ZIL gets put "elsewhere", that elsewhere really should be a mirror and sadly I don't see myself affording to use 2 SSDs for my setup :) - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 21:13:54 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F6651065670; Sun, 10 Jan 2010 21:13:54 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 57DA18FC15; Sun, 10 Jan 2010 21:13:54 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0ALDsqj046509; Sun, 10 Jan 2010 21:13:54 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0ALDs73046505; Sun, 10 Jan 2010 21:13:54 GMT (envelope-from linimon) Date: Sun, 10 Jan 2010 21:13:54 GMT Message-Id: <201001102113.o0ALDs73046505@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/142594: [zfs] Modification time reset to 1 Jan 1970 after fsync+crash on ZFS root volume ('legacy' mount) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 21:13:54 -0000 Old Synopsis: Modification time reset to 1 Jan 1970 after fsync+crash on ZFS root volume ('legacy' mount) New Synopsis: [zfs] Modification time reset to 1 Jan 1970 after fsync+crash on ZFS root volume ('legacy' mount) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sun Jan 10 21:13:34 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=142594 From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 21:26:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61D02106566B for ; Sun, 10 Jan 2010 21:26:17 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 18F028FC18 for ; Sun, 10 Jan 2010 21:26:16 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApoEAPzUSUuDaFvH/2dsb2JhbADReYIhgg4E X-IronPort-AV: E=Sophos;i="4.49,251,1262581200"; d="scan'208";a="60717293" Received: from danube.cs.uoguelph.ca ([131.104.91.199]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 10 Jan 2010 16:26:15 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 4DD3A1084454; Sun, 10 Jan 2010 16:26:14 -0500 (EST) X-Virus-Scanned: amavisd-new at danube.cs.uoguelph.ca Received: from danube.cs.uoguelph.ca ([127.0.0.1]) by localhost (danube.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dalOjxewKhYe; Sun, 10 Jan 2010 16:26:13 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 971CD108440B; Sun, 10 Jan 2010 16:26:12 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o0ALaIg09132; Sun, 10 Jan 2010 16:36:18 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Sun, 10 Jan 2010 16:36:18 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Mikolaj Golub In-Reply-To: <86ocl272mb.fsf@kopusha.onet> Message-ID: References: <86ocl272mb.fsf@kopusha.onet> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org Subject: Re: FreeBSD NFS client/Linux NFS server issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 21:26:17 -0000 On Sun, 10 Jan 2010, Mikolaj Golub wrote: > > For one of the incident we were tcpdumping "problem" NFS connection for about > 1 hour and during this hour an activity was observed only once: > > 08:20:38.281422 IP (tos 0x0, ttl 64, id 56110, offset 0, flags [DF], proto TCP (6), length 140) 172.30.10.27.344496259 > 172.30.10.121.2049: 88 access fh[1:9300:10df8001] 003f > 08:20:38.281554 IP (tos 0x0, ttl 64, id 26624, offset 0, flags [DF], proto TCP (6), length 52) 172.30.10.121.2049 > 172.30.10.27.971: ., cksum 0xca5e (correct), 89408667:89408667(0) ack 1517941890 win 46 > > The client sent rpc ACCESS request for root exported inode, received tcp ack > response (so tcp connection was ok) but did not receive any RPC reply from the > server. > > So it looks like the problem on NFS server side. But for me it looks a bit > strange that freebsd client is sending rpc packets so rarely. Shouldn't it > retransmit them more frequently? For another incident we monitored tcp > connection for 4 minutes and did not see any packets then. Unfortunately we > can't run tcpdumping long time as these are production servers and we need to > reboot hosts to restore normal operations. > For NFSv3 over TCP, there was no RFC specification, so client behaviour when the server failed to reply to an RPC was essentially undefined. (For NFSv4, a client isn't allowed to retry a non-NULL RPC on the same TCP connection and a server is expected to reply to all RPCs received on the connection or do a disconnect, but that's NFSv4 not NFSv3.) I think the new krpc in FreeBSD8 does to a slow timeout on RPCs over TCP for NFSv3 and eventually does a retry, but I didn't write the code, so I'm not absolutely sure. (I'll try and remember to take a look, or maybe dfr can comment?) However, this krpc code isn't used for FreeBSD7. Bottom line is I don't think the client does a retry until it sees the TCP connection break and if the server isn't replying to the RPC nor disconnecting the TCP connection, it'll be stuck as you describe. I think you have three choices: 1 - Fix the NFS server so that it does reply or disconnects, if that is possible. (I have no idea if the Linux NFS server can be reconfigured?) 2 - Switch to using UDP (which will retry RPCs when no reply is received). 3 - Try a FreeBSD8 system and see if it works ok, then upgrade if that's practical? rick ps: As an historical note, I think I implemented NFS over TCP before anyone else and assumed that a server would reply to all RPC requests, so retries at the RPC level wouldn't be necessary. Others, like Sun, implemented NFS over TCP with RPC timeout/retries and then slowly came over to my way of thinking, but it wasn't spelled out until NFSv4. From owner-freebsd-fs@FreeBSD.ORG Sun Jan 10 21:43:27 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 175B11065670; Sun, 10 Jan 2010 21:43:27 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E3A2F8FC17; Sun, 10 Jan 2010 21:43:26 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0ALhQdV073168; Sun, 10 Jan 2010 21:43:26 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0ALhQNG073164; Sun, 10 Jan 2010 21:43:26 GMT (envelope-from linimon) Date: Sun, 10 Jan 2010 21:43:26 GMT Message-Id: <201001102143.o0ALhQNG073164@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/142597: [ext2fs] ext2fs does not work on filesystems with really big directories X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 21:43:27 -0000 Old Synopsis: ext2fs does not work on filesystems with really big directories New Synopsis: [ext2fs] ext2fs does not work on filesystems with really big directories Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sun Jan 10 21:43:16 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=142597 From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 11:06:58 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D583F1065696 for ; Mon, 11 Jan 2010 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AB76A8FC23 for ; Mon, 11 Jan 2010 11:06:58 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0BB6wS7034645 for ; Mon, 11 Jan 2010 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0BB6vQW034643 for freebsd-fs@FreeBSD.org; Mon, 11 Jan 2010 11:06:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 11 Jan 2010 11:06:57 GMT Message-Id: <201001111106.o0BB6vQW034643@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 11:06:58 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142594 fs [zfs] Modification time reset to 1 Jan 1970 after fsyn o kern/142558 fs [msdosfs] patch] Minor updates to fs/msdosfs headers ( o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142271 fs [zfs] [patch] race condition on zpool create o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs(/ufs) o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141718 fs [zfs] [panic] kernel panic when 'zfs rename' is used o o kern/141685 fs [zfs] zfs corruption on adaptec 5805 raid controller o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141257 fs [gvinum] No puedo crear RAID5 por SW con gvinum o kern/141235 fs [disklabel] 8.0 no longer provides /dev entries for al o kern/141177 fs [zfs] fsync() on FIFO causes panic() on zfs o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140682 fs [netgraph] [panic] random panic in netgraph o kern/140661 fs [zfs] /boot/loader fails to work on a GPT/ZFS-only sys o kern/140640 fs [zfs] snapshot crash o kern/140433 fs [zfs] [panic] panic while replaying ZIL after crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138524 fs [msdosfs] disks and usb flashes/cards with Russian lab o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138367 fs [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/138109 fs [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 160 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 17:30:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1190106566C; Mon, 11 Jan 2010 17:30:26 +0000 (UTC) (envelope-from petefrench@ticketswitch.com) Received: from constantine.ticketswitch.com (unknown [IPv6:2002:57e0:1d4e:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id B861B8FC16; Mon, 11 Jan 2010 17:30:26 +0000 (UTC) Received: from dilbert.rattatosk ([10.64.50.6] helo=dilbert.ticketswitch.com) by constantine.ticketswitch.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NUO5m-0004JW-Vd; Mon, 11 Jan 2010 17:30:14 +0000 Received: from petefrench by dilbert.ticketswitch.com with local (Exim 4.71 (FreeBSD)) (envelope-from ) id 1NUO5m-0001aK-Uk; Mon, 11 Jan 2010 17:30:14 +0000 Date: Mon, 11 Jan 2010 17:30:14 +0000 Message-Id: To: dan.naumov@gmail.com, freebsd-fs@freebsd.org, freebsd-stable@freebsd.org In-Reply-To: From: Pete French Cc: Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 17:30:27 -0000 > GELI+ZFS and Debian Linux with MDRAID and cryptofs. Has anyone here > made any benchmarks regarding how much of a performance hit is caused > by using 2 geli devices as vdevs for a ZFS mirror pool in FreeBSD (a I havent done it directly on the same boxes, but I have two systems with idenitical drives, each with a ZFS mirror pool, one wth GELI, and one without. Simple read test shows no overhead in using GELI at all. I would recommend using the new AHCI driver though - greatly improves throughput. -pete. From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 18:39:40 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23EB91065693; Mon, 11 Jan 2010 18:39:40 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-yw0-f197.google.com (mail-yw0-f197.google.com [209.85.211.197]) by mx1.freebsd.org (Postfix) with ESMTP id 8389D8FC1A; Mon, 11 Jan 2010 18:39:39 +0000 (UTC) Received: by ywh35 with SMTP id 35so12515763ywh.7 for ; Mon, 11 Jan 2010 10:39:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=hGN1wnc/smunzvwI8T8azBDCghURWRkVTZJvSTwS/lA=; b=SsZXrHdk+y+6igAHAHDSEw63Mcsxi90fDXYo00uukY+pJU/9yoAuqE89dK6RS/mkn6 ZoedUhROnbAZ0cIWFR38CPJc2tAx9p2VA0vJFyTLdCpc9f8iUJOQ8Rqj7KgXDChC0Lo0 gvgmzBJCmj5c/WgNmQ/DhvEHHuNJmKRgKMcqI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=b8u+nLOktDsPK5WKxvQfsGvvnRB4V7m0Ezwmc1JHUL1aXyPiUBAas2ahT6plNiqax4 NrT/ZxKBXufpIEJK9XI8MPb+j3x91fQsoIA14LwAJVoPW7NZ1HaGmTra8g/EikCrIxdG 4JsNXGnPJy33GjqMbfq8fHzngGYnM0LrwtNjM= MIME-Version: 1.0 Received: by 10.101.63.18 with SMTP id q18mr11866272ank.110.1263235170968; Mon, 11 Jan 2010 10:39:30 -0800 (PST) In-Reply-To: References: Date: Mon, 11 Jan 2010 20:39:30 +0200 Message-ID: From: Dan Naumov To: Pete French Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 18:39:40 -0000 On Mon, Jan 11, 2010 at 7:30 PM, Pete French wrote: >> GELI+ZFS and Debian Linux with MDRAID and cryptofs. Has anyone here >> made any benchmarks regarding how much of a performance hit is caused >> by using 2 geli devices as vdevs for a ZFS mirror pool in FreeBSD (a > > I havent done it directly on the same boxes, but I have two systems > with idenitical drives, each with a ZFS mirror pool, one wth GELI, and > one without. Simple read test shows no overhead in using GELI at all. > > I would recommend using the new AHCI driver though - greatly > improves throughput. How fast is the CPU in the system showing no overhead? Having no noticable overhead whatsoever sounds extremely unlikely unless you are actually using it on something like a very modern dualcore or better. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 18:42:41 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A0DE10656D5; Mon, 11 Jan 2010 18:42:41 +0000 (UTC) (envelope-from petefrench@ticketswitch.com) Received: from constantine.ticketswitch.com (unknown [IPv6:2002:57e0:1d4e:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id 2F4EC8FC2A; Mon, 11 Jan 2010 18:42:41 +0000 (UTC) Received: from dilbert.rattatosk ([10.64.50.6] helo=dilbert.ticketswitch.com) by constantine.ticketswitch.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NUPDd-0004s9-Df; Mon, 11 Jan 2010 18:42:25 +0000 Received: from petefrench by dilbert.ticketswitch.com with local (Exim 4.71 (FreeBSD)) (envelope-from ) id 1NUPDd-0000dg-Cp; Mon, 11 Jan 2010 18:42:25 +0000 Date: Mon, 11 Jan 2010 18:42:25 +0000 Message-Id: To: dan.naumov@gmail.com In-Reply-To: From: Pete French Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 18:42:41 -0000 > How fast is the CPU in the system showing no overhead? Having no > noticable overhead whatsoever sounds extremely unlikely unless you are > actually using it on something like a very modern dualcore or better. It's a very modern dual core :-) Phenom 550 - the other machine is an old Opteron 252. -pete. From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 19:09:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1E30106566B for ; Mon, 11 Jan 2010 19:09:24 +0000 (UTC) (envelope-from benschumacher@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9193B8FC1E for ; Mon, 11 Jan 2010 19:09:24 +0000 (UTC) Received: by pxi12 with SMTP id 12so14718826pxi.3 for ; Mon, 11 Jan 2010 11:09:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=66/gcKaGQH6bl6JP5zQLOWTSuN+rAAzwNgfNywIX73k=; b=UtunBnxHrgUkiRPewFXLdBm/CsyAJXoJrxc6dzyuWhDf0TLQoW0MwKP8tRNIqlKZMP AljALtstP5sDK1CUBxCWA06xx3pGnr8TxeCI9cCMZdmnhOfhXEDaL2dssDuj9wRkLwxS cKhxwVInolrEBvk+iipYchhMwAHaEnR7yu1Qw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=poIfBXSJqb3ZRIytKBZkRCXenCMiH7Vr4Kr+mwLesdmO7cXdqbiJ1Hgka9tPNbZU4q 630+NbAO56F+XFwMrqinAXjWQo9hS340pnYRQ405QwTzeb3yCv/uUOlIg+b41hKpIe05 SXvapfHV1YhXVfJJkmWSWNcSF1r0KQwbVi7vE= MIME-Version: 1.0 Received: by 10.142.120.1 with SMTP id s1mr2852739wfc.245.1263236955844; Mon, 11 Jan 2010 11:09:15 -0800 (PST) In-Reply-To: References: Date: Mon, 11 Jan 2010 12:09:15 -0700 Message-ID: <9859143f1001111109m7156d92fk9443b347ddb9441a@mail.gmail.com> From: Ben Schumacher To: Dan Naumov Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, Pete French Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 19:09:25 -0000 On Mon, Jan 11, 2010 at 11:39 AM, Dan Naumov wrote: > On Mon, Jan 11, 2010 at 7:30 PM, Pete French > How fast is the CPU in the system showing no overhead? Having no > noticable overhead whatsoever sounds extremely unlikely unless you are > actually using it on something like a very modern dualcore or better. IIRC, GELI can take advantage of hardware acceleration for encryption, so I'd bet that a slower CPU with hardware crypto (Via Nano, for example) would probably be fast enough too. I've actually got one of these at home, I might have to check this out and see how it runs. Cheers, Ben From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 21:41:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD9C5106566B for ; Mon, 11 Jan 2010 21:41:48 +0000 (UTC) (envelope-from yuri.pankov@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id 4E1A78FC08 for ; Mon, 11 Jan 2010 21:41:48 +0000 (UTC) Received: by fxm27 with SMTP id 27so7269976fxm.3 for ; Mon, 11 Jan 2010 13:41:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received :x-authentication-warning:date:from:to:subject:message-id :mime-version:content-type:content-disposition:user-agent; bh=WLmoKqiEedb/IxGZ+Ofc3J/KIlYJvzXu7OOiAbD8f1Y=; b=f50IhDGRq7c/PJWTLFqceohKN/Pe6OVNLpTHlJVR5y+v17Zh/8ViL76+AM+MSt1cfx 11kNbS85hrMl7KHTYdDKkihhdnZ52VPJWPwHxkQzxvtMwlZhdR+x112qjXkIIfERv7mU ZNxpc0+NdiXB/yDet7tz/a0W4H0EaRxWEcXHg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=x-authentication-warning:date:from:to:subject:message-id :mime-version:content-type:content-disposition:user-agent; b=CFiMAULureGLGxN9eWRmmsgw2ZF7+s0V7SOScpj0WAAU7W7tB8cyIMz9W9jRoQyIRX 36rfnmZQBg9WZSg/b4qvQAoozgeXBVIaDUdL+hWzd0cNYOVqkdld/WXHRa+FhzjFpF7W jjcsjg1ECE5uQ+J6oN2nTHxNUBfVRH9+SgVW4= Received: by 10.87.64.15 with SMTP id r15mr6999890fgk.35.1263244210129; Mon, 11 Jan 2010 13:10:10 -0800 (PST) Received: from darklight.org.ru ([213.132.76.16]) by mx.google.com with ESMTPS id e11sm58069327fga.9.2010.01.11.13.10.08 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 11 Jan 2010 13:10:09 -0800 (PST) Received: from darklight.org.ru (yuri@darklight.org.ru [127.0.0.1]) by darklight.org.ru (8.14.3/8.14.3) with ESMTP id o0BLA6tF038210 for ; Tue, 12 Jan 2010 00:10:06 +0300 (MSK) (envelope-from yuri.pankov@gmail.com) Received: (from yuri@localhost) by darklight.org.ru (8.14.3/8.14.3/Submit) id o0BLA61O038208 for freebsd-fs@FreeBSD.org; Tue, 12 Jan 2010 00:10:06 +0300 (MSK) (envelope-from yuri.pankov@gmail.com) X-Authentication-Warning: darklight.org.ru: yuri set sender to yuri.pankov@gmail.com using -f Date: Tue, 12 Jan 2010 00:10:05 +0300 From: Yuri Pankov To: freebsd-fs@FreeBSD.org Message-ID: <20100111211005.GC1264@darklight.org.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Subject: ZFS filesystem version confusion X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 21:41:48 -0000 Hi, I was somewhat confused by the output of `zfs get version ` being 3, when dmesg reads 'ZFS filesystem version 14'. Shouldn't we use ZPL_VERSION_STRING in sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c instead of SPA_VERSION_STRING to report ZFS filesystem version? TIA, Yuri From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 21:46:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB13D1065694 for ; Mon, 11 Jan 2010 21:46:39 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 715738FC0C for ; Mon, 11 Jan 2010 21:46:39 +0000 (UTC) Received: by pxi12 with SMTP id 12so14819357pxi.3 for ; Mon, 11 Jan 2010 13:46:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=2MSTIzeXeDQy2TjqqAAz6nLZUVNjozMY3+o/JJpj5NU=; b=n8EAiMHzBRztBhhRBypdl8D9saDSneSQz46pQgsHtdKzfZVPsHmSVqj44tMGha52bx YWWXpgmvzVtzxdg8klcr/JozSFe8h/dG3ohYZsAJndg9ThcSXFtuYvTxpoV9avYdXltd TMw8bTfpTczeaNSQfmikNuQy9s9kW9wFbrxto= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=LKlUFmgI19G/m+9jTabc1QUPgA9dywoeNelTViHNQjT2fhudOiOQh3dz4kkqG4aZEF scfeXkqyGitQuYXMHsL1JIzk2xAywJyi8CzQr6k/lksH62Qg/jSm8bsWbTeoxjoflLv9 OQT2as0yO977KDIGR1XgMJ08YndfppXsKd7sY= MIME-Version: 1.0 Received: by 10.115.114.18 with SMTP id r18mr5961339wam.24.1263246399011; Mon, 11 Jan 2010 13:46:39 -0800 (PST) In-Reply-To: <20100111211005.GC1264@darklight.org.ru> References: <20100111211005.GC1264@darklight.org.ru> Date: Mon, 11 Jan 2010 13:46:38 -0800 Message-ID: From: Xin LI To: Yuri Pankov Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS filesystem version confusion X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 21:46:39 -0000 On Mon, Jan 11, 2010 at 1:10 PM, Yuri Pankov wrote: > Hi, > > I was somewhat confused by the output of `zfs get version ` being 3, > when dmesg reads 'ZFS filesystem version 14'. Shouldn't we use > ZPL_VERSION_STRING in sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c > instead of SPA_VERSION_STRING to report ZFS filesystem version? I think it should be "ZFS Pool Version 14"... Cheers, -- Xin LI http://www.delphij.net From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 22:16:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E69811065670 for ; Mon, 11 Jan 2010 22:16:01 +0000 (UTC) (envelope-from lucas.james@ldjcs.com.au) Received: from mail.ldjcs.com.au (gw.ldjcs.com.au [58.6.71.89]) by mx1.freebsd.org (Postfix) with ESMTP id 5E7288FC0A for ; Mon, 11 Jan 2010 22:16:01 +0000 (UTC) Received: from bella.ldjcs.com.au (bella.ldjcs.com.au [203.17.30.179]) (authenticated bits=0) by mail.ldjcs.com.au (8.14.3/8.14.3) with ESMTP id o0BLxdRG014527 for ; Tue, 12 Jan 2010 08:59:39 +1100 (EST) (envelope-from lucas.james@ldjcs.com.au) From: Lucas James Organization: LDJ Computer Services To: freebsd-fs@freebsd.org Date: Tue, 12 Jan 2010 08:59:30 +1100 User-Agent: KMail/1.9.10 References: <20100111211005.GC1264@darklight.org.ru> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201001120859.31778.lucas.james@ldjcs.com.au> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (mail.ldjcs.com.au [203.17.30.35]); Tue, 12 Jan 2010 08:59:40 +1100 (EST) X-Spam-Status: No, score=0.4 required=5.0 tests=AWL autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.ldjcs.com.au Subject: Re: ZFS filesystem version confusion X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 22:16:02 -0000 On Tue, 12 Jan 2010 08:46:38 am Xin LI wrote: > On Mon, Jan 11, 2010 at 1:10 PM, Yuri Pankov wrote: > > Hi, > > > > I was somewhat confused by the output of `zfs get version ` being 3, > > when dmesg reads 'ZFS filesystem version 14'. Shouldn't we use > > ZPL_VERSION_STRING in > > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c instead of > > SPA_VERSION_STRING to report ZFS filesystem version? > > I think it should be "ZFS Pool Version 14"... I haven't upgraded to 14 yet, but: > zfs get version zroot NAME PROPERTY VALUE SOURCE zroot version 3 - > zpool get version zroot NAME PROPERTY VALUE SOURCE zroot version 13 default > dmesg|grep ZFS ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, ZFS filesystem version 13 ZFS storage pool version 13 Note the version in the output of dmesg for filesystem version. > zfs upgrade -v The following filesystem versions are supported: VER DESCRIPTION --- -------------------------------------------------------- 1 Initial ZFS filesystem version 2 Enhanced directory entries 3 Case insensitive and File system unique identifer (FUID) [...] regards, Lucas -- Filtering is what YOU do to your internet. Censorship is what THEY do to your internet. From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 22:25:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F0A71065676; Mon, 11 Jan 2010 22:25:37 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2DCEF8FC1D; Mon, 11 Jan 2010 22:25:36 +0000 (UTC) Received: by pxi12 with SMTP id 12so14843190pxi.3 for ; Mon, 11 Jan 2010 14:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=Mte3MGMDjDE5IYXcje9U3rxGKKaS5hmMiasjhwaMQ/k=; b=RbdqdMhzoLrhoD1YoP8hepc8yWeu27DwubFzbLt4DgckMVfzboY9ezlDu3ETv4oSGF 2Yol8SvCQ/dPekfzj9T9gJo73sC33PDsP0briWzmfbLFxvytzhFxn8w6UVl8YWcan/7Z gcmec7oG2O0dP+ftPbUgOXs/YQNBLeBt+5mz0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=QKuvXK/TgdRNIOmPqhPIVHKJxAi7FfF/6UUzdoR00/CJG3aWFQPFzRDZdIYIzpie1M JcrfDk2GBWpA7P+vZSOS+PksK6fPeC52ZsmR7mibqKPrm4bCaKLV8qXfuFqLvW6p0unx C1TS84vFfhVOG/fE0+Y91kuXhtx5e8kvZNw70= MIME-Version: 1.0 Received: by 10.114.6.2 with SMTP id 2mr3136311waf.124.1263248729783; Mon, 11 Jan 2010 14:25:29 -0800 (PST) In-Reply-To: <201001120859.31778.lucas.james@ldjcs.com.au> References: <20100111211005.GC1264@darklight.org.ru> <201001120859.31778.lucas.james@ldjcs.com.au> Date: Mon, 11 Jan 2010 14:25:29 -0800 Message-ID: From: Xin LI To: Pawel Jakub Dawidek Content-Type: multipart/mixed; boundary=0016e6480ebecd3d72047ceb0383 Cc: freebsd-fs Subject: Re: ZFS filesystem version confusion X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 22:25:37 -0000 --0016e6480ebecd3d72047ceb0383 Content-Type: text/plain; charset=UTF-8 Hey guys, I think I have misunderstood Yuri's original post. Here is a patch to fix the problem. Pawel, may I commit this patch against -HEAD? For me it would give the following during boot: ZFS filesystem version 3 ZFS storage pool version 14 Instead of the previous: ZFS filesystem version 14 ZFS storage pool version 14 Cheers, -- Xin LI http://www.delphij.net --0016e6480ebecd3d72047ceb0383 Content-Type: application/octet-stream; name="zfs-version.diff" Content-Disposition: attachment; filename="zfs-version.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_g4btn1240 SW5kZXg6IHN5cy9jZGRsL2NvbnRyaWIvb3BlbnNvbGFyaXMvdXRzL2NvbW1vbi9mcy96ZnMvemZz X3Zmc29wcy5jCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT0KLS0tIHN5cy9jZGRsL2NvbnRyaWIvb3BlbnNvbGFyaXMvdXRz L2NvbW1vbi9mcy96ZnMvemZzX3Zmc29wcy5jCShyZXZpc2lvbiAyMDIwOTQpCisrKyBzeXMvY2Rk bC9jb250cmliL29wZW5zb2xhcmlzL3V0cy9jb21tb24vZnMvemZzL3pmc192ZnNvcHMuYwkod29y a2luZyBjb3B5KQpAQCAtMTM4OCw3ICsxMzg4LDcgQEAKIHpmc19pbml0KHZvaWQpCiB7CiAKLQlw cmludGYoIlpGUyBmaWxlc3lzdGVtIHZlcnNpb24gIiBTUEFfVkVSU0lPTl9TVFJJTkcgIlxuIik7 CisJcHJpbnRmKCJaRlMgZmlsZXN5c3RlbSB2ZXJzaW9uICIgWlBMX1ZFUlNJT05fU1RSSU5HICJc biIpOwogCiAJLyoKIAkgKiBJbml0aWFsaXplIHpub2RlIGNhY2hlLCB2bm9kZSBvcHMsIGV0Yy4u Lgo= --0016e6480ebecd3d72047ceb0383-- From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 22:28:52 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D79610657B2 for ; Mon, 11 Jan 2010 22:28:52 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 12B688FC08 for ; Mon, 11 Jan 2010 22:28:51 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 23F8F45C99; Mon, 11 Jan 2010 23:28:49 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id D2BE8456B1; Mon, 11 Jan 2010 23:28:43 +0100 (CET) Date: Mon, 11 Jan 2010 23:28:53 +0100 From: Pawel Jakub Dawidek To: Xin LI Message-ID: <20100111222853.GB1650@garage.freebsd.pl> References: <20100111211005.GC1264@darklight.org.ru> <201001120859.31778.lucas.james@ldjcs.com.au> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="b5gNqxB1S1yM7hjW" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs Subject: Re: ZFS filesystem version confusion X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 22:28:52 -0000 --b5gNqxB1S1yM7hjW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 11, 2010 at 02:25:29PM -0800, Xin LI wrote: > Hey guys, >=20 > I think I have misunderstood Yuri's original post. Here is a patch to > fix the problem. >=20 > Pawel, may I commit this patch against -HEAD? For me it would give > the following during boot: >=20 > ZFS filesystem version 3 > ZFS storage pool version 14 >=20 > Instead of the previous: >=20 > ZFS filesystem version 14 > ZFS storage pool version 14 Patch looks correct, please go ahead. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --b5gNqxB1S1yM7hjW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFLS6YkForvXbEpPzQRAkqOAKDFWm9Bh/gd3VJOMNUiJc+3H8rHowCg5mki w7kFXFEfBkI3+fbd3x0DhV4= =RmfB -----END PGP SIGNATURE----- --b5gNqxB1S1yM7hjW-- From owner-freebsd-fs@FreeBSD.ORG Mon Jan 11 23:29:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B5C0106566B; Mon, 11 Jan 2010 23:29:55 +0000 (UTC) (envelope-from kmatthew.macy@gmail.com) Received: from mail-pz0-f202.google.com (mail-pz0-f202.google.com [209.85.222.202]) by mx1.freebsd.org (Postfix) with ESMTP id 054258FC19; Mon, 11 Jan 2010 23:29:54 +0000 (UTC) Received: by pzk40 with SMTP id 40so827134pzk.7 for ; Mon, 11 Jan 2010 15:29:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=WWCC5qLzjoPtt0FDCKXcR2Gs47AL1T5f4gLtiwgmq8c=; b=rmyGkw2RXckqZyBE2/MhDcb6nWaliRHFbkYgv6x3uT5YPD4wvWwrqKGNszwfeXY0GP jAykegYkkUEtjQMXjDbfWxSa6Ja6ajewE5TLCAHeEiWpryBz3rScCx4wDtsBYyD33/9Z CK4+3pHDulzh57PZbSOT4n7K88yJRvT6nncjU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=BUfG6hvr12LCiqOWN0fuJyLKYTh5tOohiBOxG+wJ/IeN/5sf+j+BNai7uiO/C9VZsQ vQeK8FgVolZ7wVjdLVAo1fdH+yjNtS/FMHtTPF8FF0SyNkiNANpJ1siu/O7/jNM4qpSx 7eNfZp6iCqXFL7wyqh3/hn5I9JU0Ennxa9WCY= MIME-Version: 1.0 Sender: kmatthew.macy@gmail.com Received: by 10.115.26.7 with SMTP id d7mr4343810waj.12.1263252594642; Mon, 11 Jan 2010 15:29:54 -0800 (PST) In-Reply-To: References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> Date: Mon, 11 Jan 2010 15:29:54 -0800 X-Google-Sender-Auth: a1a2555c6bcea339 Message-ID: <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> From: "K. Macy" To: Dan Naumov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Damian Gerow , FreeBSD-STABLE Mailing List Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2010 23:29:55 -0000 >> >> If performance is an issue, you may want to consider carving off a parti= tion >> on that SSD, geli-fying it, and using it as a ZIL device. =A0You'll prob= ably >> see a marked performance improvement with such a setup. > > That is true, but using a single device for a dedicated ZIL is a huge > no-no, considering it's an intent log, it's used to reconstruct the > pool in case of a power failure for example, should such an event > occur at the same time as a ZIL provider dies, you lose the entire > pool because there is no way to recover it, so if ZIL gets put > "elsewhere", that elsewhere really should be a mirror and sadly I > don't see myself affording to use 2 SSDs for my setup :) > This is false. The ZIL is used for journalling synchronous writes. If your ZIL is lost you will lose the data that was written to the ZIL, but not yet written to the file system proper. Barring disk corruption, the file system is always consistent. -Kip From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 00:15:59 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8DC8A106566B; Tue, 12 Jan 2010 00:15:59 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f209.google.com (mail-gx0-f209.google.com [209.85.217.209]) by mx1.freebsd.org (Postfix) with ESMTP id 3675C8FC14; Tue, 12 Jan 2010 00:15:59 +0000 (UTC) Received: by gxk1 with SMTP id 1so9907441gxk.14 for ; Mon, 11 Jan 2010 16:15:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=2queXapYqRrEg06ZaveaXHl/TPh9H9C7KFSkOBRHX50=; b=WKh+l04253gz8bS0080FDySyAxx1ydqN77qX8YhR+mMe3nknRcdonSK3moer/vwPu+ lphV+LQea6ZnGCp94c9U2NR6f/bAqMB0hFoBDUA4BzV3+5vl6BliFKJDDkKwnpsYyaRy 7mlhhy3j0y/0m7kxD3Sbzm4MiHqfShpqgy2Q4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=ocip7gWcSe560kdU0VLDk2Hvx/I3wFJIl6STUjGeTC1P43Zv/fMAmPdx3/g8DTVfUc Ol26BQWr8Mn2Gtm+9NNtw1jx4noN8yFHMNZCjxDAyYbXDFa5Wemdbhq9WhqZvJpOhN5N PszGr+7Erymh6AGNMOC1KvDHr21960bm/m/UQ= MIME-Version: 1.0 Received: by 10.101.132.14 with SMTP id j14mr8999450ann.58.1263255352721; Mon, 11 Jan 2010 16:15:52 -0800 (PST) In-Reply-To: <201001120100.16631.freebsd@o2.pl> References: <20100110173127.GA52730@slackbox.xs4all.nl> <201001120100.16631.freebsd@o2.pl> Date: Tue, 12 Jan 2010 02:15:52 +0200 Message-ID: From: Dan Naumov To: freebsd@o2.pl Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Roland Smith , freebsd-stable@freebsd.org Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 00:15:59 -0000 2010/1/12 Rafa=C5=82 Jackiewicz : > Two hdd Seagate ES2,Intel Atom 330 (2x1.6GHz), 2GB RAM: > > geli: > =C2=A0 geli init -s 4096 -K /etc/keys/ad4s2.key /dev/ad4s2 > =C2=A0 geli init -s 4096 -K /etc/keys/ad6s2.key /dev/ad6s2 > > zfs: > =C2=A0 zpool create data01 ad4s2.eli > > df -h: > =C2=A0 dev/ad6s2.eli.journal =C2=A0 =C2=A0857G =C2=A0 =C2=A08.0K =C2=A0 = =C2=A0788G =C2=A0 =C2=A0 0% =C2=A0 =C2=A0/data02 > =C2=A0 data01 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 850G =C2=A0 =C2=A0128K =C2=A0 =C2=A0850G = =C2=A0 =C2=A0 0% =C2=A0 =C2=A0/data01 > > srebrny# dd if=3D/dev/zero of=3D/data01/test bs=3D1M count=3D500 > 500+0 records in > 500+0 records out > 524288000 bytes transferred in 8.802691 secs (59559969 bytes/sec) > srebrny# dd if=3D/dev/zero of=3D/data02/test bs=3D1M count=3D500 > 500+0 records in > 500+0 records out > 524288000 bytes transferred in 20.090274 secs (26096608 bytes/sec) > > Rafal Jackiewicz Thanks, could you do the same, but using 2 .eli vdevs mirrorred together in a zfs mirror? - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 00:24:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A8B8106566C; Tue, 12 Jan 2010 00:24:30 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f209.google.com (mail-gx0-f209.google.com [209.85.217.209]) by mx1.freebsd.org (Postfix) with ESMTP id 29DE58FC0A; Tue, 12 Jan 2010 00:24:29 +0000 (UTC) Received: by gxk1 with SMTP id 1so9913762gxk.14 for ; Mon, 11 Jan 2010 16:24:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=OrDNjw9HNTxiX2bMJ1aaAekaHCccA8AT9WrlQfWucfw=; b=sMl1Qr/iRtz3jiq5SvAqMpUMa3tIenzUe+z6h5j9yoo+wVDWZGJw7sJj/srVPLVdFM o2upG6/vNtZPGxmN6nuplELCH7aFDzE5L2ZXcG70tsTwgfL3gspf8A0eGaMES9dgc7Ce 7oL4oriV+4NKc6dFG+AuuAUj0/1/6OYKO9UfE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=ScW0VdSh3w7l4+/D7WPG32ekqH+2gie/UQO7bse07mpmbYzuVtzcJpVourpWmHgUfX H1VzTA72O0vesH17Hs82E5LT/8nKYT/GmR33yV4cxmrxN5E6wJ6rlYkLYAIxot6E7TzU +XWTL8FmpHaCG61gCw2wBOcXmb3jzIsorm+as= MIME-Version: 1.0 Received: by 10.101.7.8 with SMTP id k8mr5287880ani.23.1263255867204; Mon, 11 Jan 2010 16:24:27 -0800 (PST) In-Reply-To: <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> Date: Tue, 12 Jan 2010 02:24:27 +0200 Message-ID: From: Dan Naumov To: kmacy@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Damian Gerow , FreeBSD-STABLE Mailing List Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 00:24:30 -0000 On Tue, Jan 12, 2010 at 1:29 AM, K. Macy wrote: >>> >>> If performance is an issue, you may want to consider carving off a part= ition >>> on that SSD, geli-fying it, and using it as a ZIL device. =A0You'll pro= bably >>> see a marked performance improvement with such a setup. >> >> That is true, but using a single device for a dedicated ZIL is a huge >> no-no, considering it's an intent log, it's used to reconstruct the >> pool in case of a power failure for example, should such an event >> occur at the same time as a ZIL provider dies, you lose the entire >> pool because there is no way to recover it, so if ZIL gets put >> "elsewhere", that elsewhere really should be a mirror and sadly I >> don't see myself affording to use 2 SSDs for my setup :) >> > > This is =A0false. The ZIL is used for journalling synchronous writes. If > your ZIL is lost you will lose the data that was written to the ZIL, > but not yet written to the file system proper. Barring disk > corruption, the file system is always consistent. > > -Kip Ok, lets assume we have a dedicated ZIL on a single non-redundant disk. This disk dies. How do you remove the dedicated ZIL from the pool or replace it with a new one? Solaris ZFS documentation indicates that this is possible for dedicated L2ARC - you can remove a dedicated l2arc from a pool at any time you wish and should some IO fail on the l2arc, the system will gracefully continue to run, reverting said IO to be processed by the actual default built-in ZIL on the disks of the pool. However the capability to remove dedicated ZIL or gracefully handle the death of a non-redundant dedicated ZIL vdev does not currently exist in Solaris/OpenSolaris at all. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 00:25:53 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0556F106568F for ; Tue, 12 Jan 2010 00:25:53 +0000 (UTC) (envelope-from freebsd@o2.pl) Received: from tur.go2.pl (tur.go2.pl [193.17.41.50]) by mx1.freebsd.org (Postfix) with ESMTP id 9FF1F8FC13 for ; Tue, 12 Jan 2010 00:25:52 +0000 (UTC) Received: from rekin26.go2.pl (rekin26.go2.pl [193.17.41.76]) by tur.go2.pl (o2.pl Mailer 2.0.1) with ESMTP id D5990230263 for ; Tue, 12 Jan 2010 01:00:26 +0100 (CET) Received: from rekin26.go2.pl (rekin26.go2.pl [127.0.0.1]) by rekin26.go2.pl (Postfix) with ESMTP id 0D0D435D7B6 for ; Tue, 12 Jan 2010 01:00:17 +0100 (CET) Received: from unknown (unknown [10.0.0.142]) by rekin26.go2.pl (Postfix) with SMTP for ; Tue, 12 Jan 2010 01:00:17 +0100 (CET) Received: from staticline40844.toya.net.pl [85.89.178.34] by poczta.o2.pl with ESMTP id KppvGS; Tue, 12 Jan 2010 01:00:16 +0100 From: =?utf-8?q?Rafa=C5=82_Jackiewicz?= Organization: =?utf-8?q?Rafa=C5=82?= Jackiewicz To: freebsd-stable@freebsd.org Date: Tue, 12 Jan 2010 01:00:16 +0100 User-Agent: KMail/1.12.4 (Linux/2.6.30-2-amd64; KDE/4.3.4; x86_64; ; ) References: <20100110173127.GA52730@slackbox.xs4all.nl> In-Reply-To: <20100110173127.GA52730@slackbox.xs4all.nl> MIME-Version: 1.0 Message-Id: <201001120100.16631.freebsd@o2.pl> Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-O2-Trust: 2, 61 Cc: freebsd-fs@freebsd.org, Roland Smith , Dan Naumov Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@o2.pl List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 00:25:53 -0000 Two hdd Seagate ES2,Intel Atom 330 (2x1.6GHz), 2GB RAM: geli: geli init -s 4096 -K /etc/keys/ad4s2.key /dev/ad4s2 geli init -s 4096 -K /etc/keys/ad6s2.key /dev/ad6s2 zfs: zpool create data01 ad4s2.eli df -h: dev/ad6s2.eli.journal 857G 8.0K 788G 0% /data02 data01 850G 128K 850G 0% /data01 srebrny# dd if=/dev/zero of=/data01/test bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 8.802691 secs (59559969 bytes/sec) srebrny# dd if=/dev/zero of=/data02/test bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 20.090274 secs (26096608 bytes/sec) Rafal Jackiewicz From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 00:32:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53FA0106566C; Tue, 12 Jan 2010 00:32:36 +0000 (UTC) (envelope-from kmatthew.macy@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1B6C28FC15; Tue, 12 Jan 2010 00:32:35 +0000 (UTC) Received: by pxi12 with SMTP id 12so14911784pxi.3 for ; Mon, 11 Jan 2010 16:32:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=mEuzhGHEwBNI5abi7azyt4rfezwppbC2dHx5jEpldl4=; b=Eld15rTVsvL91HrN/GtXZc5lUujSOgHt/JxuFMNbGsMX6wpLG9gj6AGW2nh9kLIHHD 5kTpfePK5HlMsYkYGdo5XDbt6UnLUNqDF13QHPY5euHKuUMFHGb2pR6yFsZB4xaPzZnM RwUMt7PwDjguMMXvx/D/A5xVuIbI7zlvlmvLw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=QI9sd7RvulTvrL+Gkmpp4UJennX6xh8j1M3nlkMIOPtsdEOWpxwVSSXXjx80LZQoew MC2+WL11UzeUkz57SJvdTu5ysee/FY1TZ3Xj6PqpO3oKm4j4Hn8GdXHkvyyvqbrTGrR+ 9DMC8nKABfDRV5y/MjN9Em9a7gf5nRYlMH2h8= MIME-Version: 1.0 Sender: kmatthew.macy@gmail.com Received: by 10.115.115.31 with SMTP id s31mr238225wam.7.1263256347443; Mon, 11 Jan 2010 16:32:27 -0800 (PST) In-Reply-To: References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> Date: Mon, 11 Jan 2010 16:32:27 -0800 X-Google-Sender-Auth: f4f1718a8bcf3056 Message-ID: <82c4140e1001111632x2cbafbf3rf4c386184e9ba06f@mail.gmail.com> From: "K. Macy" To: Dan Naumov Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org, Damian Gerow , FreeBSD-STABLE Mailing List Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 00:32:36 -0000 > Ok, lets assume we have a dedicated ZIL on a single non-redundant > disk. This disk dies. How do you remove the dedicated ZIL from the > pool or replace it with a new one? Solaris ZFS documentation indicates > that this is possible for dedicated L2ARC - you can remove a dedicated > l2arc from a pool at any time you wish and should some IO fail on the > l2arc, the system will gracefully continue to run, reverting said IO > to be processed by the actual default built-in ZIL on the disks of the > pool. However the capability to remove dedicated ZIL or gracefully > handle the death of a non-redundant dedicated ZIL vdev does not > currently exist in Solaris/OpenSolaris at all. Ahh - you're describing an implementation flaw as opposed to a design flaw. Your initial statement could be interpreted as meaning that the ZIL is required for file system consistency. I hope they fix that. -Kip From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 01:12:05 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6148106568B for ; Tue, 12 Jan 2010 01:12:05 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com [209.85.160.44]) by mx1.freebsd.org (Postfix) with ESMTP id A65AD8FC0A for ; Tue, 12 Jan 2010 01:12:05 +0000 (UTC) Received: by pwi15 with SMTP id 15so1943463pwi.3 for ; Mon, 11 Jan 2010 17:12:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=0s9pbQx5eOQUPvEEITbCaWkzE0BCzqLDksD+Bd4O3iE=; b=vJCgZymiOrRfdgkXArBq0jkmQgDe4xpNg4QNN5aUQcs6dVdScbElwSnxDVePMo43CD PR8h8zjHn3Pd2ul+DbxnpTRmTBTNns8+PfxWJdB07uHCjlftOKMuzLHbhd5oWBEsvcOB fWkJsk1T87I3DbZEWQ1C7lx4vJVoJx3W7PTLE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=szfuqBedGCSQBssdHZXvUX1IHjHW5l6rZj0NpgERQHCYDyeybLGCErNeTbbCmBeYKF kyfdWapC+L0T9EsNusHdP712KvC1aj6fcKPO9+pX2BeNKGNp0OxPkSkN95WFMirpVZvg HwSuWpU0RK1nfArgzWwjJ3MQd6FVdeeq5flmo= MIME-Version: 1.0 Received: by 10.142.67.39 with SMTP id p39mr5346463wfa.204.1263257107155; Mon, 11 Jan 2010 16:45:07 -0800 (PST) In-Reply-To: References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> Date: Mon, 11 Jan 2010 16:45:07 -0800 Message-ID: From: Freddie Cash To: freebsd-fs@freebsd.org, FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 01:12:05 -0000 On Mon, Jan 11, 2010 at 4:24 PM, Dan Naumov wrote: > On Tue, Jan 12, 2010 at 1:29 AM, K. Macy wrote: > >>> > >>> If performance is an issue, you may want to consider carving off a > partition > >>> on that SSD, geli-fying it, and using it as a ZIL device. You'll > probably > >>> see a marked performance improvement with such a setup. > >> > >> That is true, but using a single device for a dedicated ZIL is a huge > >> no-no, considering it's an intent log, it's used to reconstruct the > >> pool in case of a power failure for example, should such an event > >> occur at the same time as a ZIL provider dies, you lose the entire > >> pool because there is no way to recover it, so if ZIL gets put > >> "elsewhere", that elsewhere really should be a mirror and sadly I > >> don't see myself affording to use 2 SSDs for my setup :) > >> > > > > This is false. The ZIL is used for journalling synchronous writes. If > > your ZIL is lost you will lose the data that was written to the ZIL, > > but not yet written to the file system proper. Barring disk > > corruption, the file system is always consistent. > > > > -Kip > > Ok, lets assume we have a dedicated ZIL on a single non-redundant > disk. This disk dies. How do you remove the dedicated ZIL from the > pool or replace it with a new one? Solaris ZFS documentation indicates > that this is possible for dedicated L2ARC - you can remove a dedicated > l2arc from a pool at any time you wish and should some IO fail on the > l2arc, the system will gracefully continue to run, reverting said IO > to be processed by the actual default built-in ZIL on the disks of the > pool. However the capability to remove dedicated ZIL or gracefully > handle the death of a non-redundant dedicated ZIL vdev does not > currently exist in Solaris/OpenSolaris at all. > > That has been implemented in OpenSolaris, do a search for "slog removal". It's in a much newer zpool version than 13, though. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 04:00:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2EE01065670; Tue, 12 Jan 2010 04:00:45 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f177.google.com (mail-iw0-f177.google.com [209.85.223.177]) by mx1.freebsd.org (Postfix) with ESMTP id 5D5428FC15; Tue, 12 Jan 2010 04:00:45 +0000 (UTC) Received: by iwn7 with SMTP id 7so6899522iwn.7 for ; Mon, 11 Jan 2010 20:00:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:cc:subject :in-reply-to:message-id:references:user-agent:x-openpgp-key-id :x-openpgp-key-fingerprint:mime-version:content-type; bh=MVi9V8OJjrPEwHVfDl2wE1SqMX9NCS9FyZaQPvMrIyw=; b=GmuSqU9TH4IZhgkRAI5DdwDcH3oAYGzc60a9AVN1Dm0EH7YuU2RwdkVwVpQvk0iBjJ /Si4OlZb4gN0Dk4IyQenIFPqcbMw7A9aip21iOEXiAS8uvn8n+izsYm4PyP0QLaGfrqM kBq+QfpStdkoXl6pWz07kPx4rpxPoZulnAyQQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=WfoFkPbHl9YdcOE+eo3CrZ+uqm7xld5SUZRNUSowko+iloE5HZ4Ffk39JrD/Y2dXXe rELl/sn3DKzir54bBl8i/F0H3tTLbyRw4js2MMZgGT+pysfGKyNDYPAR07QwbGQYAfaK tpR8Qe8vOT7GdTdwnpq2oC6haHbOrqOvv6Hx4= Received: by 10.231.162.83 with SMTP id u19mr44733ibx.25.1263268839837; Mon, 11 Jan 2010 20:00:39 -0800 (PST) Received: from centel.dataix.local (ppp-21.49.dialinfree.com [209.172.21.49]) by mx.google.com with ESMTPS id 22sm2879656iwn.4.2010.01.11.20.00.35 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 11 Jan 2010 20:00:38 -0800 (PST) Sender: "J. Hellenthal" Date: Mon, 11 Jan 2010 23:00:09 -0500 From: jhell In-Reply-To: Message-ID: References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, FreeBSD-STABLE Mailing List Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 04:00:45 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 11 Jan 2010 19:45, fjwcash@ wrote: > On Mon, Jan 11, 2010 at 4:24 PM, Dan Naumov wrote: > >> On Tue, Jan 12, 2010 at 1:29 AM, K. Macy wrote: >>>>> >>>>> If performance is an issue, you may want to consider carving off a >> partition >>>>> on that SSD, geli-fying it, and using it as a ZIL device. You'll >> probably >>>>> see a marked performance improvement with such a setup. >>>> >>>> That is true, but using a single device for a dedicated ZIL is a huge >>>> no-no, considering it's an intent log, it's used to reconstruct the >>>> pool in case of a power failure for example, should such an event >>>> occur at the same time as a ZIL provider dies, you lose the entire >>>> pool because there is no way to recover it, so if ZIL gets put >>>> "elsewhere", that elsewhere really should be a mirror and sadly I >>>> don't see myself affording to use 2 SSDs for my setup :) >>>> >>> >>> This is false. The ZIL is used for journalling synchronous writes. If >>> your ZIL is lost you will lose the data that was written to the ZIL, >>> but not yet written to the file system proper. Barring disk >>> corruption, the file system is always consistent. >>> >>> -Kip >> >> Ok, lets assume we have a dedicated ZIL on a single non-redundant >> disk. This disk dies. How do you remove the dedicated ZIL from the >> pool or replace it with a new one? Solaris ZFS documentation indicates >> that this is possible for dedicated L2ARC - you can remove a dedicated >> l2arc from a pool at any time you wish and should some IO fail on the >> l2arc, the system will gracefully continue to run, reverting said IO >> to be processed by the actual default built-in ZIL on the disks of the >> pool. However the capability to remove dedicated ZIL or gracefully >> handle the death of a non-redundant dedicated ZIL vdev does not >> currently exist in Solaris/OpenSolaris at all. >> >> That has been implemented in OpenSolaris, do a search for "slog removal". > It's in a much newer zpool version than 13, though. > > What I have seen more often by users is taking the usage of slog/ZIL wrong. For instance dedicating a whole SSD or another HDD as the slog. Your slog/ZIL only has to be big enough to handle 10 seconds of synchronous writes before it flushes. A recommended ZIL from Sun Micro is 128MB but you may not even see that fully used for general purpose cases. I had is to dedicate a partition on the same disk that the pool is on and adding another ZIL vdev from another disk in the system. Results of this imply that if the off-disk ZIL dies for some stupid reason it falls back to the one that rests on the same disk as the pool and allows to replace the off-disk ZIL with something else. PS: Save your disk space and use 256MB thumb drives. you can easily get 16 of those at your local Walmart and have a priceless light show for a romantic dinner with the wife. :) - -- Mon Jan 11 22:31:17 2010 jhell -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iQEcBAEBAgAGBQJLS/PZAAoJEJBXh4mJ2FR+CTUH/RIqmRPE8SdKZYY7WIC/K9Yk HThaiYHs6a15ZY58q2nHG0x5J85TOBMN4yvC1rI1DGcjX9SXlyjxY+jJ0sdIAbHz N2+nT95X3SbNCPXtA3qo6uTplIiZnu9xgcAnFmjBh96Aq7qzcIvtFe2QMuxTp/lI Na8K4t7udDFQ9xIoyptk/PukKvV/EtzDx449w6VPxu0fkXK812uWWl+jFy3XchrW QfExuNIhVadcnxOB5/BQaAyd6daaI9tZNyARo43ww7bEKxaFP2Awre/IYfeuKZtm /n4TOdOoookyIIO0fMWDQ4WyLLsQD6eHug0B0Ef7LYcrUkPEYFJQVxujhx/cyhI= =cjuO -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 05:10:04 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9BF81065670 for ; Tue, 12 Jan 2010 05:10:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BF80B8FC0C for ; Tue, 12 Jan 2010 05:10:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0C5A3Nf080752 for ; Tue, 12 Jan 2010 05:10:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0C5A3EY080751; Tue, 12 Jan 2010 05:10:03 GMT (envelope-from gnats) Date: Tue, 12 Jan 2010 05:10:03 GMT Message-Id: <201001120510.o0C5A3EY080751@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Alexey Dokuchaev Cc: Subject: Re: kern/142271: [zfs] [patch] race condition on zpool create X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Alexey Dokuchaev List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 05:10:05 -0000 The following reply was made to PR kern/142271; it has been noted by GNATS. From: Alexey Dokuchaev To: bug-followup@FreeBSD.org, torindel@gmail.com Cc: Subject: Re: kern/142271: [zfs] [patch] race condition on zpool create Date: Tue, 12 Jan 2010 10:00:46 +0600 I've had the same problem and confirm that provided patch fixes it on recent -CURRENT. From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 08:20:46 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D485D1065670 for ; Tue, 12 Jan 2010 08:20:46 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 5F9528FC21 for ; Tue, 12 Jan 2010 08:20:45 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id o0C8KhFk018547; Tue, 12 Jan 2010 09:20:44 +0100 Received: from pmp.uni-hannover.de (arc.pmp.uni-hannover.de [130.75.117.1]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 27F4824; Tue, 12 Jan 2010 09:20:43 +0100 (CET) Date: Tue, 12 Jan 2010 09:20:43 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: kmacy@freebsd.org Message-Id: <20100112092043.d1370d55.gerrit@pmp.uni-hannover.de> In-Reply-To: <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> References: <20100110161206.GA86684@plebeian.afflictions.org> <20100110184612.GC86684@plebeian.afflictions.org> <82c4140e1001111529u7d7ac409h63dda4cc10b76522@mail.gmail.com> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.7.1 (GTK+ 2.18.4; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.388399, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2010.1.12.80923 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 08:20:46 -0000 On Mon, 11 Jan 2010 15:29:54 -0800 "K. Macy" wrote about Re: ZFS on top of GELI: KM> > That is true, but using a single device for a dedicated ZIL is a huge KM> > no-no, considering it's an intent log, it's used to reconstruct the KM> > pool in case of a power failure for example, should such an event KM> > occur at the same time as a ZIL provider dies, you lose the entire KM> > pool because there is no way to recover it, so if ZIL gets put KM> > "elsewhere", that elsewhere really should be a mirror and sadly I KM> > don't see myself affording to use 2 SSDs for my setup :) KM> This is false. The ZIL is used for journalling synchronous writes. If KM> your ZIL is lost you will lose the data that was written to the ZIL, KM> but not yet written to the file system proper. Barring disk KM> corruption, the file system is always consistent. So how would ZFS behave if ZIL is sitting on a single disk that is then lost? Throw away the intented writes and continue happily from there on? I still have some broken volume sitting around that panics the kernel when trying to replay the (obviously somehow corrupted) ZIL (see ) and I wonder if situations like this could be handled by taking out the ZIL drive (only if there is a dedicated one, of course). cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 08:45:34 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85D81106566B; Tue, 12 Jan 2010 08:45:34 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-yx0-f171.google.com (mail-yx0-f171.google.com [209.85.210.171]) by mx1.freebsd.org (Postfix) with ESMTP id 1EF018FC2A; Tue, 12 Jan 2010 08:45:33 +0000 (UTC) Received: by yxe1 with SMTP id 1so20661912yxe.3 for ; Tue, 12 Jan 2010 00:45:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=0Qh9fU99LQ6Y6PztQoFGer6tmJYDrqgoWEcZBk3Z0o0=; b=fnYHbBkyL+FVXbsJWd6rjRjGzttoKwgEYCKsQjKA3ny2HbNOObB9fJikejZTXH7+Fo kqlGBrV/GfQDg4u6tPWUrlHAhAsg+f2WSnUgnk4clMlqgLriyoswtkjTcLshRKJBV6xO RKdfFp6DKDg0G5nX3/7+fDOYBYDrd50RkH7hk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=woyxqk/KeirAyP6oMEhfz477UKCyyd5p+ORJq3fCiB0itRpvM7HkI7YzQbnH+fNYIv 2ZpVsncUsKNqNq+MaEt9FkP/3nJ5HQDXnZx0IIm/1Deo7vy4CHwCd2rkBvh5o2j5h+sT AIkdBan+lur/PDNKxJonu33k3fAuenn3F9P/0= MIME-Version: 1.0 Received: by 10.101.132.14 with SMTP id j14mr9600223ann.58.1263285927478; Tue, 12 Jan 2010 00:45:27 -0800 (PST) Date: Tue, 12 Jan 2010 10:45:27 +0200 Message-ID: From: Dan Naumov To: freebsd-questions@freebsd.org, freebsd-fs@freebsd.org, FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: installing FreeBSD 8 on SSDs and UFS2 - partition alignment, block sizes, what does one need to know? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 08:45:34 -0000 For my upcoming storage system, the OS install is going to be on a 80gb Intel SSD disk and for various reasons, I am now pretty convinced to stick with UFS2 for the root partition (the actual data pool will be ZFS using traditional SATA disks). I am probably going to use GPT partitioning and have the SSD host the swap, boot, root and a few other partitions. What do I need to know in regards to partition alignment and filesystem block sizes to get the best performance out of the Intel SSDs? Thanks. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 20:28:13 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBB8C1065676 for ; Tue, 12 Jan 2010 20:28:13 +0000 (UTC) (envelope-from freebsd@o2.pl) Received: from rekin26.go2.pl (rekin26.go2.pl [193.17.41.76]) by mx1.freebsd.org (Postfix) with ESMTP id 702DC8FC19 for ; Tue, 12 Jan 2010 20:28:13 +0000 (UTC) Received: from rekin26.go2.pl (rekin26.go2.pl [127.0.0.1]) by rekin26.go2.pl (Postfix) with ESMTP id 517F835D871 for ; Tue, 12 Jan 2010 21:27:58 +0100 (CET) Received: from unknown (unknown [10.0.0.74]) by rekin26.go2.pl (Postfix) with SMTP for ; Tue, 12 Jan 2010 21:27:58 +0100 (CET) Received: from staticline40844.toya.net.pl [85.89.178.34] by poczta.o2.pl with ESMTP id fbtWhS; Tue, 12 Jan 2010 21:27:57 +0100 From: =?utf-8?q?Rafa=C5=82_Jackiewicz?= Organization: =?utf-8?q?Rafa=C5=82?= Jackiewicz To: freebsd-stable@freebsd.org Date: Tue, 12 Jan 2010 21:27:56 +0100 User-Agent: KMail/1.12.4 (Linux/2.6.30-2-amd64; KDE/4.3.4; x86_64; ; ) References: <201001120100.16631.freebsd@o2.pl> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201001122127.56716.freebsd@o2.pl> X-O2-Trust: 2, 60 Cc: freebsd-fs@freebsd.org, Roland Smith , Dan Naumov Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@o2.pl List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 20:28:14 -0000 >Thanks, could you do the same, but using 2 .eli vdevs mirrorred >together in a zfs mirror? > >- Sincerely, >Dan Naumov Hi, Proc: Intell Atom 330 (2x1.6Ghz) - 1 package(s) x 2 core(s) x 2 HTT threads Chipset: Intel 82945G Sys: 8.0-RELEASE FreeBSD 8.0-RELEASE #0 empty file: /boot/loader.conf Hdd: ad4: 953869MB at ata2-master SATA150 ad6: 953869MB at ata3-master SATA150 Geli: geli init -s 4096 -K /etc/keys/ad4s2.key /dev/ad4s2 geli init -s 4096 -K /etc/keys/ad6s2.key /dev/ad6s2 Results: **************************************************** *** single drive write MB/s read MB/s eli.journal.ufs2 23 14 eli.zfs 19 36 *** mirror write MB/s read MB/s mirror.eli.journal.ufs2 23 16 eli.zfs 31 40 zfs 83 79 *** degraded mirror write MB/s read MB/s mirror.eli.journal.ufs2 16 9 eli.zfs 56 40 zfs 86 71 **************************************************** ***** Single drive: ****** Mount: data01 on /data01 (zfs, local) /dev/ad6s2.eli.journal on /data02 (ufs, local, gjournal) *** (single hdd) UFS2, gjournal, eli. srebrny# time dd if=/dev/zero of=/data02/test01 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 92.451346 secs (22683845 bytes/sec) 0.068u 10.386s 1:32.46 11.2% 26+1497k 63+16066io 0pf+0w ** umount / mount, and: srebrny# time dd if=/data02/test01 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 147.219379 secs (14245081 bytes/sec) 0.008u 4.456s 2:27.22 3.0% 23+1324k 16066+0io 0pf+0w *** (single hdd) zfs: srebrny# time dd if=/dev/zero of=/data01/test01 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 113.049629 secs (18550720 bytes/sec) 0.014u 5.480s 1:53.05 4.8% 26+1516k 0+0io 0pf+0w ** umount / mount, and: srebrny# time dd if=/data01/test01 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 59.012860 secs (35537203 bytes/sec) 0.000u 3.135s 0:59.01 5.3% 24+1397k 0+0io 0pf+0w ***** Mirror: ***** *** (mirror hdd) UFS2, gjournal, eli. srebrny# gmirror list Geom name: data02 State: COMPLETE Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ** srebrny# time dd if=/dev/zero of=/data02/test01 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 89.441874 secs (23447094 bytes/sec) 0.022u 11.110s 1:29.45 12.4% 26+1515k 64+16066io 0pf+0w ** umount / mount, and: srebrny# time dd if=/data02/test01 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 134.567914 secs (15584339 bytes/sec) 0.007u 4.333s 2:14.62 3.2% 26+1498k 16067+0io 0pf+0w *** (mirror hdd, eli) zfs: srebrny# time dd if=/dev/zero of=/data01/test01 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 67.255574 secs (31181832 bytes/sec) 0.029u 6.422s 1:07.25 9.5% 26+1531k 0+0io 0pf+0w ** (eli) umount / mount, and: srebrny# time dd if=/data01/test01 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 52.307404 secs (40092833 bytes/sec) 0.036u 3.405s 0:52.31 6.5% 27+1543k 0+0io 0pf+0w *** (mirror hdd, no eli!) zfs: srebrny# time dd if=/dev/zero of=/data01/test01 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 25.185164 secs (83269341 bytes/sec) 0.000u 5.506s 0:25.18 21.8% 26+1513k 0+0io 0pf+0w ** (no eli!) umount / mount, and: srebrny# time dd if=/data01/test01 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 26.457374 secs (79265312 bytes/sec) 0.000u 3.011s 0:26.45 11.3% 24+1396k 0+0io 0pf+0w ***** *** (mirror !!!degraded!!!, single drive ad4s2) UFS2, gjournal, eli. df -h /dev/mirror/data02.eli.journal 857G 8.0K 788G 0% /data02 ** srebrny# time dd if=/dev/zero of=/data02/test01 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 131.554958 secs (15941262 bytes/sec) 0.029u 10.057s 2:11.58 7.6% 26+1528k 64+16066io 0pf+0w ** (mirror !!!degraded!!!, single drive ad4s2) umount / mount, and: srebrny# time dd if=/data02/test01 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 226.056204 secs (9277127 bytes/sec) 0.029u 4.226s 3:46.08 1.8% 26+1529k 16066+0io 0pf+0w *** (mirror !!!degraded!!!, single drive ad4s2; eli) zfs: srebrny# time dd if=/dev/zero of=/data01/test011 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 37.548232 secs (55852217 bytes/sec) 0.007u 5.480s 0:37.55 14.5% 26+1513k 0+0io 0pf+0w ** (mirror !!!degraded!!!, single drive ad4s2; eli) umount / mount, and: srebrny# time dd if=/data01/test011 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 51.266119 secs (40907173 bytes/sec) 0.036u 3.238s 0:51.28 6.3% 28+1549k 0+0io 0pf+0w *** (mirror !!!degraded!!!, single drive ad4s2; no eli!) zfs: srebrny# time dd if=/dev/zero of=/data01/test011 bs=1m count=2000 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 24.296032 secs (86316646 bytes/sec) 0.000u 5.463s 0:24.29 22.4% 26+1512k 0+0io 0pf+0w ** (mirror !!!degraded!!!, single drive ad4s2; no eli!) umount / mount, and: srebrny# time dd if=/data01/test011 of=/dev/null bs=1m 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 29.512372 secs (71060096 bytes/sec) 0.036u 3.275s 0:29.51 11.1% 27+1563k 0+0io 0pf+0w Best regards, Rafal Jackiewicz From owner-freebsd-fs@FreeBSD.ORG Tue Jan 12 20:49:54 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BB62106568D; Tue, 12 Jan 2010 20:49:54 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f209.google.com (mail-gx0-f209.google.com [209.85.217.209]) by mx1.freebsd.org (Postfix) with ESMTP id 90D248FC26; Tue, 12 Jan 2010 20:49:53 +0000 (UTC) Received: by gxk1 with SMTP id 1so10836664gxk.14 for ; Tue, 12 Jan 2010 12:49:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=RQo+tMLSHoZmsRGH7RPaf/BsGx44obwXIScAEjxDxVQ=; b=sg9JEyAtt00OHphC3ARFnSHWKtYSH19FUQ6inGd1ijIier6YOr+CEvl0HLegNfL5Bc aBv3ySdV52AgbPpVBLt3eA7vqrXVfUqoyrp09RZ4oOuy/cr+IpECIjIA27Uevfh7wZvI 5MUGlxqqdPqCO0e0ryXAKbxIuAaOR7JYjcTMU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=rYkhcpb1YGZBwhK1CmQ4uzd5sb141vVL8wXIUbk2t87AdXaVYMZNO21NjO4VUxYgHT T0zH/RgRJrTCdY2UKkB9EVqozGXxhAV5uh/RZmMODniWYc65ggseFwGehj+jYHJL0Wej VG6ExaRLMCYj2RYdGP8pFo2SIxiySG2gwol0Q= MIME-Version: 1.0 Received: by 10.100.63.16 with SMTP id l16mr789170ana.47.1263329386146; Tue, 12 Jan 2010 12:49:46 -0800 (PST) In-Reply-To: <201001122127.56716.freebsd@o2.pl> References: <201001120100.16631.freebsd@o2.pl> <201001122127.56716.freebsd@o2.pl> Date: Tue, 12 Jan 2010 22:49:46 +0200 Message-ID: From: Dan Naumov To: freebsd@o2.pl, FreeBSD-STABLE Mailing List , freebsd-questions@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: ZFS on top of GELI X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2010 20:49:54 -0000 2010/1/12 Rafa=C5=82 Jackiewicz : >>Thanks, could you do the same, but using 2 .eli vdevs mirrorred >>together in a zfs mirror? >> >>- Sincerely, >>Dan Naumov > > Hi, > > Proc: Intell Atom 330 (2x1.6Ghz) - 1 package(s) x 2 core(s) x 2 HTT threa= ds > Chipset: Intel 82945G > Sys: 8.0-RELEASE FreeBSD 8.0-RELEASE #0 > empty file: /boot/loader.conf > Hdd: > =C2=A0 ad4: 953869MB at ata2-master SATA150 > =C2=A0 ad6: 953869MB at ata3-master SATA150 > Geli: > =C2=A0 geli init -s 4096 -K /etc/keys/ad4s2.key /dev/ad4s2 > =C2=A0 geli init -s 4096 -K /etc/keys/ad6s2.key /dev/ad6s2 > > > Results: > **************************************************** > > *** single drive =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0write MB/s =C2=A0 =C2=A0 =C2=A0read =C2=A0MB/s > eli.journal.ufs2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A023 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A014 > eli.zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 19 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A036 > > > *** mirror =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0write MB/s =C2=A0 =C2=A0 =C2=A0re= ad =C2=A0MB/s > mirror.eli.journal.ufs2 23 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A016 > eli.zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 31 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040 > zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 83 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A079 > > > *** degraded mirror =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 write MB/s = =C2=A0 =C2=A0 =C2=A0read MB/s > mirror.eli.journal.ufs2 16 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A09 > eli.zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 56 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040 > zfs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 86 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A071 > > **************************************************** Thanks a lot for your numbers, the relevant part for me was this: *** mirror write MB/s read MB/s eli.zfs 31 40 zfs 83 79 *** degraded mirror write MB/s read MB/s eli.zfs 56 40 zfs 86 71 31 mb/s writes and 40 mb/s reads is something that I guess I could potentially live with. I am guessing the main problem of stacking ZFS on top of geli like this is the fact that writing to a mirror requires double the CPU use, because we have to encrypt all written data twice (once to each disk) instead of encrypting first and then writing the encrypted data to 2 disks as would be the case if we had crypto sitting on top of ZFS instead of ZFS sitting on top of crypto. I now have to reevaluate my planned use of an SSD though, I was planning to use a 40gb partition on an Intel 80GB X25-M G2 as a dedicated L2ARC device for a ZFS mirror of 2 x 2tb disks. However these numbers make it quite obvious that I would already be CPU-starved at 40-50mb/s throughput on the encrypted ZFS mirror, so adding an l2arc SSD, while improving latency, would do really nothing for actual disk read speeds, considering the l2arc itself would too, have to sit on top of a GELI device. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Wed Jan 13 09:13:25 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 141F1106568B for ; Wed, 13 Jan 2010 09:13:25 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f213.google.com (mail-bw0-f213.google.com [209.85.218.213]) by mx1.freebsd.org (Postfix) with ESMTP id 3BE228FC19 for ; Wed, 13 Jan 2010 09:13:23 +0000 (UTC) Received: by bwz5 with SMTP id 5so15244126bwz.3 for ; Wed, 13 Jan 2010 01:13:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:to:subject:references :organization:from:date:in-reply-to:message-id:user-agent :mime-version:content-type:content-transfer-encoding; bh=kEqvGLdQ9k/rRUL907VYeWF3u3B8i2rZPjqAiHLqzGE=; b=JNv4JBhUFlFQItr/7oTc0fQrJGXKusfxyPyVy+uVuieoQ8uXg3hdDAslxDA+/UP8Ld Kz2Ai7nijXyJtSRbt29Q64Lri8kFTE7xAdQRNzMoYeuzdFfNWXzRw6xHjRhm/LWGtHma m2XvIwCN5PjYv4zOVfQ44QrNJAG+bQCS5Faw0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=to:subject:references:organization:from:date:in-reply-to:message-id :user-agent:mime-version:content-type:content-transfer-encoding; b=vMQehf31fNdQB7zGTbbuS8HHxuvRPuVUBLIDAo6s4hBD5cm0JaD7I+YyZzLABe7ivy w166Mj3Veku6uSU6A+DEJdC2w7AvHSIEI3do10GKO6s/pmE0myroUYb769Svj9mtSj2s s6+SFNtw8lbCMXfM2N3MErmEzBfM8sVGaPiX0= Received: by 10.204.6.203 with SMTP id a11mr3203117bka.50.1263373997434; Wed, 13 Jan 2010 01:13:17 -0800 (PST) Received: from localhost (ms.singlescrowd.net [80.85.90.67]) by mx.google.com with ESMTPS id 15sm9768253bwz.0.2010.01.13.01.13.15 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 13 Jan 2010 01:13:16 -0800 (PST) To: freebsd-fs@FreeBSD.org References: <86ocl272mb.fsf@kopusha.onet> Organization: TOA Ukraine From: Mikolaj Golub Date: Wed, 13 Jan 2010 11:13:14 +0200 In-Reply-To: <86ocl272mb.fsf@kopusha.onet> (Mikolaj Golub's message of "Sun\, 10 Jan 2010 11\:03\:56 +0200") Message-ID: <86tyuqnz9x.fsf@zhuzha.ua1> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Cc: Subject: Re: FreeBSD NFS client/Linux NFS server issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 09:13:25 -0000 On Sun, 10 Jan 2010 11:03:56 +0200 Mikolaj Golub wrote: > Hi, > > I wrote about this problem some times ago to freebsd-stable > > http://lists.freebsd.org/pipermail/freebsd-stable/2009-December/053746.html > > but there have not been any response there so far and also we have collected > some additional information since then so I am trying again here :-) > > We have CentOS 5.3 NFS server and many FreeBSD clients. The problem have been > observed since we started upgrade FreeBSD host from 6.3 to 7.1 so we thought > this is 7.1 related bu recently we had the similar issue with 6.3 host and > actually it looks like the problem is rather on NFS server side, but I would > like to describe the problem here and ask some advise. > > So the "problem" FreeBSD clients have several NFS mounts from CentOS 5.3 > server. Mount options: rw,-3,-T,-s,-i,-r=32768,-w=32768,-o=noinet6 > > At some point on one of the clients some of the mounts might get stuck -- we > observe one or two our php applications hanged on writing to NFS folder in > nfs_flush()->bufobj_wwait() and all other processes that try to access this > NFS mount hang acquiring nfs vn_lock hold by one of the above php processes. > > For one of the incident we were tcpdumping "problem" NFS connection for about > 1 hour and during this hour an activity was observed only once: > > 08:20:38.281422 IP (tos 0x0, ttl 64, id 56110, offset 0, flags [DF], proto TCP (6), length 140) 172.30.10.27.344496259 > 172.30.10.121.2049: 88 access fh[1:9300:10df8001] 003f > 08:20:38.281554 IP (tos 0x0, ttl 64, id 26624, offset 0, flags [DF], proto TCP (6), length 52) 172.30.10.121.2049 > 172.30.10.27.971: ., cksum 0xca5e (correct), 89408667:89408667(0) ack 1517941890 win 46 > > The client sent rpc ACCESS request for root exported inode, received tcp ack > response (so tcp connection was ok) but did not receive any RPC reply from the > server. > > So it looks like the problem on NFS server side. But for me it looks a bit > strange that freebsd client is sending rpc packets so rarely. Shouldn't it > retransmit them more frequently? For another incident we monitored tcp > connection for 4 minutes and did not see any packets then. Unfortunately we > can't run tcpdumping long time as these are production servers and we need to > reboot hosts to restore normal operations. There are some new details and now it looks like rather client side problem. We set up tcpdump monitoring of 2049 port on the "problem" NFS clients and now have full tcpdump log + kernel crush dump for the latest incident. So this time again the first gotten stuck processes were php scripts writing log files to NFS folder. (kgdb) thr 152 [Switching to thread 152 (Thread 100073)]#0 sched_switch (td=0xc6bf4690, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 1944 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xc6bf4690, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f460c in sleepq_catch_signals (wchan=0xcc2f2ee8) at /usr/src/sys/kern/subr_sleepqueue.c:417 #4 0xc07f4ebd in sleepq_wait_sig (wchan=0xcc2f2ee8) at /usr/src/sys/kern/subr_sleepqueue.c:594 #5 0xc07cb047 in _sleep (ident=0xcc2f2ee8, lock=0xcc2f2e8c, priority=333, wmesg=0xc0b731ed "bo_wwait", timo=0) at /usr/src/sys/kern/kern_synch.c:224 #6 0xc0827295 in bufobj_wwait (bo=0xcc2f2ec4, slpflag=256, timeo=0) at /usr/src/sys/kern/vfs_bio.c:3870 #7 0xc0966307 in nfs_flush (vp=0xcc2f2e04, waitfor=1, td=0xc6bf4690, commit=1) at /usr/src/sys/nfsclient/nfs_vnops.c:2989 #8 0xc09667ce in nfs_fsync (ap=0xed1928ec) at /usr/src/sys/nfsclient/nfs_vnops.c:2725 #9 0xc0aee5d2 in VOP_FSYNC_APV (vop=0xc0c2b920, a=0xed1928ec) at vnode_if.c:1007 #10 0xc0827864 in bufsync (bo=0xcc2f2ec4, waitfor=1, td=0xc6bf4690) at vnode_if.h:538 #11 0xc083f354 in bufobj_invalbuf (bo=0xcc2f2ec4, flags=1, td=0xc6bf4690, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1066 #12 0xc083f6e2 in vinvalbuf (vp=0xcc2f2e04, flags=1, td=0xc6bf4690, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1142 #13 0xc094f216 in nfs_vinvalbuf (vp=0xcc2f2e04, flags=Variable "flags" is not available. ) at /usr/src/sys/nfsclient/nfs_bio.c:1326 #14 0xc0951825 in nfs_write (ap=0xed192bc4) at /usr/src/sys/nfsclient/nfs_bio.c:918 #15 0xc0aef956 in VOP_WRITE_APV (vop=0xc0c2b920, a=0xed192bc4) at vnode_if.c:691 #16 0xc0850097 in vn_write (fp=0xc95087b8, uio=0xed192c60, active_cred=0xc84db000, flags=0, td=0xc6bf4690) at vnode_if.h:373 #17 0xc07f9d17 in dofilewrite (td=0xc6bf4690, fd=6, fp=0xc95087b8, auio=0xed192c60, offset=-1, flags=0) at file.h:256 #18 0xc07f9ff8 in kern_writev (td=0xc6bf4690, fd=6, auio=0xed192c60) at /usr/src/sys/kern/sys_generic.c:401 #19 0xc07fa06f in write (td=0xc6bf4690, uap=0xed192cfc) at /usr/src/sys/kern/sys_generic.c:317 #20 0xc0ad9c75 in syscall (frame=0xed192d38) at /usr/src/sys/i386/i386/trap.c:1090 #21 0xc0ac01b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) fr 13 #13 0xc094f216 in nfs_vinvalbuf (vp=0xcc2f2e04, flags=Variable "flags" is not available. ) at /usr/src/sys/nfsclient/nfs_bio.c:1326 1326 error = vinvalbuf(vp, flags, td, slpflag, 0); (kgdb) p * (struct nfsnode *) vp->v_data $5 = {n_mtx = {lock_object = {lo_name = 0xc0b80584 "NFSnode lock", lo_type = 0xc0b80584 "NFSnode lock", lo_flags = 16973824, lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, n_size = 133758, n_brev = 0, n_lrev = 0, n_vattr = {va_type = VREG, va_mode = 420, va_nlink = 1, va_uid = 4002, va_gid = 4002, va_fsid = 50396930, va_fileid = 213581825, va_size = 133758, va_blocksize = 512, va_atime = {tv_sec = 1263232925, tv_nsec = 0}, va_mtime = {tv_sec = 1263232924, tv_nsec = 0}, va_ctime = {tv_sec = 1263232924, tv_nsec = 0}, va_birthtime = {tv_sec = 0, tv_nsec = 0}, va_gen = 0, va_flags = 0, va_rdev = 0, va_bytes = 143360, va_filerev = 0, va_vaflags = 0, va_spare = 0}, n_attrstamp = 0, n_mode = 13, n_modeuid = 4002, n_modestamp = 1263232925, n_mtime = {tv_sec = 1263232029, tv_nsec = 0}, n_ctime = 1263232029, n_expiry = 0, n_fhp = 0xce9002a4, n_vnode = 0xcc2f2e04, n_dvp = 0x0, n_error = 0, n_un1 = {nf_atim = {tv_sec = 0, tv_nsec = 0}, nd_cookieverf = {nfsuquad = {0, 0}}, nd4_cookieverf = "\000\000\000\000\000\000\000"}, n_un2 = {nf_mtim = {tv_sec = 0, tv_nsec = 0}, nd_direof = 0}, n_un3 = {nf_silly = 0x0, nd_cook = {lh_first = 0x0}}, n_fhsize = 20, n_flag = 4, n_fh = { fh_generic = {fh_fsid = {val = {16777217, 37632}}, fh_fid = {fid_len = 16385, fid_reserved = 4277, fid_data = "\001\000»\fRüÿ¯\000\000\000\000\000\000\000"}}, fh_bytes = "\001\000\000\001\000\223\000\000\001@µ\020\001\000»\fRüÿ¯", '\0' }, n_rfc = {next = {tqe_next = 0x0, tqe_prev = 0x0}, refcnt = 0, lop = 0x0, np = 0x0, stateid = '\0' }, n_wfc = {next = {tqe_next = 0x0, tqe_prev = 0x0}, refcnt = 0, lop = 0x0, np = 0x0, stateid = '\0' }, n_name = 0x0, n_namelen = 0, n_directio_opens = 0, n_directio_asyncwr = 0, n_ac_ts = {nfs_ac_ts_tid = 100073, nfs_ac_ts_pid = 64011, nfs_ac_ts_syscalls = 1660222}} (kgdb) thr 153 [Switching to thread 153 (Thread 100535)]#0 sched_switch (td=0xcc1d2d20, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 1944 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xcc1d2d20, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f460c in sleepq_catch_signals (wchan=0xc7a42420) at /usr/src/sys/kern/subr_sleepqueue.c:417 #4 0xc07f4ebd in sleepq_wait_sig (wchan=0xc7a42420) at /usr/src/sys/kern/subr_sleepqueue.c:594 #5 0xc07cb047 in _sleep (ident=0xc7a42420, lock=0xc7a423c4, priority=333, wmesg=0xc0b731ed "bo_wwait", timo=0) at /usr/src/sys/kern/kern_synch.c:224 #6 0xc0827295 in bufobj_wwait (bo=0xc7a423fc, slpflag=256, timeo=0) at /usr/src/sys/kern/vfs_bio.c:3870 #7 0xc0966307 in nfs_flush (vp=0xc7a4233c, waitfor=1, td=0xcc1d2d20, commit=1) at /usr/src/sys/nfsclient/nfs_vnops.c:2989 #8 0xc09667ce in nfs_fsync (ap=0xed8978ec) at /usr/src/sys/nfsclient/nfs_vnops.c:2725 #9 0xc0aee5d2 in VOP_FSYNC_APV (vop=0xc0c2b920, a=0xed8978ec) at vnode_if.c:1007 #10 0xc0827864 in bufsync (bo=0xc7a423fc, waitfor=1, td=0xcc1d2d20) at vnode_if.h:538 #11 0xc083f354 in bufobj_invalbuf (bo=0xc7a423fc, flags=1, td=0xcc1d2d20, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1066 #12 0xc083f6e2 in vinvalbuf (vp=0xc7a4233c, flags=1, td=0xcc1d2d20, slpflag=256, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1142 #13 0xc094f216 in nfs_vinvalbuf (vp=0xc7a4233c, flags=Variable "flags" is not available. ) at /usr/src/sys/nfsclient/nfs_bio.c:1326 #14 0xc0951825 in nfs_write (ap=0xed897bc4) at /usr/src/sys/nfsclient/nfs_bio.c:918 #15 0xc0aef956 in VOP_WRITE_APV (vop=0xc0c2b920, a=0xed897bc4) at vnode_if.c:691 #16 0xc0850097 in vn_write (fp=0xcdd5a000, uio=0xed897c60, active_cred=0xc8d5c200, flags=0, td=0xcc1d2d20) at vnode_if.h:373 #17 0xc07f9d17 in dofilewrite (td=0xcc1d2d20, fd=6, fp=0xcdd5a000, auio=0xed897c60, offset=-1, flags=0) at file.h:256 #18 0xc07f9ff8 in kern_writev (td=0xcc1d2d20, fd=6, auio=0xed897c60) at /usr/src/sys/kern/sys_generic.c:401 #19 0xc07fa06f in write (td=0xcc1d2d20, uap=0xed897cfc) at /usr/src/sys/kern/sys_generic.c:317 #20 0xc0ad9c75 in syscall (frame=0xed897d38) at /usr/src/sys/i386/i386/trap.c:1090 #21 0xc0ac01b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) fr 13 #13 0xc094f216 in nfs_vinvalbuf (vp=0xc7a4233c, flags=Variable "flags" is not available. ) at /usr/src/sys/nfsclient/nfs_bio.c:1326 1326 error = vinvalbuf(vp, flags, td, slpflag, 0); (kgdb) p * (struct nfsnode *) vp->v_data $6 = {n_mtx = {lock_object = {lo_name = 0xc0b80584 "NFSnode lock", lo_type = 0xc0b80584 "NFSnode lock", lo_flags = 16973824, lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, n_size = 14566, n_brev = 0, n_lrev = 0, n_vattr = {va_type = VREG, va_mode = 420, va_nlink = 1, va_uid = 4002, va_gid = 4002, va_fsid = 50396930, va_fileid = 116310018, va_size = 14566, va_blocksize = 512, va_atime = {tv_sec = 1263232925, tv_nsec = 0}, va_mtime = {tv_sec = 1263232924, tv_nsec = 0}, va_ctime = {tv_sec = 1263232924, tv_nsec = 0}, va_birthtime = {tv_sec = 0, tv_nsec = 0}, va_gen = 0, va_flags = 0, va_rdev = 0, va_bytes = 20480, va_filerev = 0, va_vaflags = 0, va_spare = 0}, n_attrstamp = 0, n_mode = 13, n_modeuid = 4002, n_modestamp = 1263232925, n_mtime = {tv_sec = 1263232029, tv_nsec = 0}, n_ctime = 1263232029, n_expiry = 0, n_fhp = 0xce2ee2a4, n_vnode = 0xc7a4233c, n_dvp = 0x0, n_error = 0, n_un1 = {nf_atim = {tv_sec = 0, tv_nsec = 0}, nd_cookieverf = {nfsuquad = {0, 0}}, nd4_cookieverf = "\000\000\000\000\000\000\000"}, n_un2 = {nf_mtim = {tv_sec = 0, tv_nsec = 0}, nd_direof = 0}, n_un3 = {nf_silly = 0x0, nd_cook = {lh_first = 0x0}}, n_fhsize = 20, n_flag = 4, n_fh = { fh_generic = {fh_fsid = {val = {16777217, 37632}}, fh_fid = {fid_len = 16385, fid_reserved = 4277, fid_data = "\002Àî\006+Ë\223\v\000\000\000\000\000\000\000"}}, fh_bytes = "\001\000\000\001\000\223\000\000\001@µ\020\002Àî\006+Ë\223\v", '\0' }, n_rfc = {next = {tqe_next = 0x0, tqe_prev = 0x0}, refcnt = 0, lop = 0x0, np = 0x0, stateid = '\0' }, n_wfc = {next = {tqe_next = 0x0, tqe_prev = 0x0}, refcnt = 0, lop = 0x0, np = 0x0, stateid = '\0' }, n_name = 0x0, n_namelen = 0, n_directio_opens = 0, n_directio_asyncwr = 0, n_ac_ts = {nfs_ac_ts_tid = 100535, nfs_ac_ts_pid = 64012, nfs_ac_ts_syscalls = 1926475}} So inodes for this files are known (50396930 and 50396930) and I can track nfs operations in tcpdump for these files. These are log files -- php scripts write there some summary information. One of the script was managed to write the following to the file before the hang: Update upload started at 2010-01-11 18:02:01 GMT: filename: [XXXXXX_UPDATE_011110_094706.ZIP] date: [2010-01-11] FSR equipment file, secs: 0.63525295257568. Upload summary: inserted: 3 updated: 3 up-to-date: 1495 And rpc requests in wireshark for this file: 26756 18:02:00.648590 172.30.10.83 172.30.10.54 NFS V3 LOOKUP Reply (Call In 26728), FH:0x29d8e1c4 26756 18:02:00.648590 172.30.10.83 172.30.10.54 NFS V3 LOOKUP Reply (Call In 26728), FH:0x29d8e1c4 26757 18:02:00.648675 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 26759), FH:0x29d8e1c4 26759 18:02:00.648988 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 26757) 28216 18:02:01.342201 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 28217), FH:0x29d8e1c4 28217 18:02:01.342433 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 28216) 28218 18:02:01.343514 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 28239), FH:0x29d8e1c4 Offset:0 Len:14282 28239 18:02:01.356049 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 28218) Len:14167 28938 18:02:01.995767 172.30.10.54 172.30.10.83 NFS V3 WRITE Call (Reply In 28939), FH:0x29d8e1c4 Offset:14167 Len:115 UNSTABLE 28939 18:02:02.014548 172.30.10.83 172.30.10.54 NFS V3 WRITE Reply (Call In 28938) Len:115 UNSTABLE 28940 18:02:02.014626 172.30.10.54 172.30.10.83 NFS V3 COMMIT Call (Reply In 29569), FH:0x29d8e1c4 29569 18:02:02.223967 172.30.10.83 172.30.10.54 NFS V3 COMMIT Reply (Call In 28940) 29572 18:02:02.224055 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 29575), FH:0x29d8e1c4 29575 18:02:02.224474 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 29572) 29578 18:02:02.224552 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 29607), FH:0x29d8e1c4 Offset:0 Len:14331 29607 18:02:02.418815 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 29578) Len:14282 29612 18:02:02.418936 172.30.10.54 172.30.10.83 NFS V3 WRITE Call (Reply In 29613), FH:0x29d8e1c4 Offset:14282 Len:49 UNSTABLE 29613 18:02:02.430813 172.30.10.83 172.30.10.54 NFS V3 WRITE Reply (Call In 29612) Len:49 UNSTABLE 29614 18:02:02.430909 172.30.10.54 172.30.10.83 NFS V3 COMMIT Call (Reply In 29634), FH:0x29d8e1c4 29634 18:02:02.731175 172.30.10.83 172.30.10.54 NFS V3 COMMIT Reply (Call In 29614) 29635 18:02:02.731296 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 29640), FH:0x29d8e1c4 29640 18:02:02.732761 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 29635) 29641 18:02:02.732925 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 29667), FH:0x29d8e1c4 Offset:0 Len:14351 29667 18:02:02.736009 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 29641) Len:14331 29669 18:02:02.736172 172.30.10.54 172.30.10.83 NFS V3 WRITE Call (Reply In 29675), FH:0x29d8e1c4 Offset:14331 Len:20 UNSTABLE 29675 18:02:02.799350 172.30.10.83 172.30.10.54 NFS V3 WRITE Reply (Call In 29669) Len:20 UNSTABLE 29676 18:02:02.799439 172.30.10.54 172.30.10.83 NFS V3 COMMIT Call (Reply In 29764), FH:0x29d8e1c4 29764 18:02:03.267882 172.30.10.83 172.30.10.54 NFS V3 COMMIT Reply (Call In 29676) 29767 18:02:03.268003 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 29770), FH:0x29d8e1c4 29770 18:02:03.268902 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 29767) 29771 18:02:03.268980 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 29787), FH:0x29d8e1c4 Offset:0 Len:14404 29787 18:02:03.274837 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 29771) Len:14351 29788 18:02:03.275013 172.30.10.54 172.30.10.83 NFS V3 WRITE Call (Reply In 29791), FH:0x29d8e1c4 Offset:14351 Len:53 UNSTABLE 29791 18:02:03.373167 172.30.10.83 172.30.10.54 NFS V3 WRITE Reply (Call In 29788) Len:53 UNSTABLE 29792 18:02:03.373252 172.30.10.54 172.30.10.83 NFS V3 COMMIT Call (Reply In 29839), FH:0x29d8e1c4 29839 18:02:03.989291 172.30.10.83 172.30.10.54 NFS V3 COMMIT Reply (Call In 29792) 29840 18:02:03.989382 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 29842), FH:0x29d8e1c4 29842 18:02:03.989563 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 29840) 29843 18:02:03.989627 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 29859), FH:0x29d8e1c4 Offset:0 Len:14457 29859 18:02:03.991310 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 29843) Len:14404 29860 18:02:03.991477 172.30.10.54 172.30.10.83 NFS V3 WRITE Call (Reply In 29861), FH:0x29d8e1c4 Offset:14404 Len:53 UNSTABLE 29861 18:02:03.994289 172.30.10.83 172.30.10.54 NFS V3 WRITE Reply (Call In 29860) Len:53 UNSTABLE 29862 18:02:03.994360 172.30.10.54 172.30.10.83 NFS V3 COMMIT Call (Reply In 30299), FH:0x29d8e1c4 30299 18:02:04.911587 172.30.10.83 172.30.10.54 NFS [TCP Previous segment lost] V3 COMMIT Reply (Call In 29862) 30301 18:02:04.911675 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 30356), FH:0x29d8e1c4 30356 18:02:04.917405 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 30301) 30360 18:02:04.917482 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 30539), FH:0x29d8e1c4 Offset:0 Len:14513 30539 18:02:04.929902 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 30360) Len:14457 30551 18:02:04.930074 172.30.10.54 172.30.10.83 NFS V3 WRITE Call (Reply In 30593), FH:0x29d8e1c4 Offset:14457 Len:56 UNSTABLE 30593 18:02:04.932002 172.30.10.83 172.30.10.54 NFS V3 WRITE Reply (Call In 30551) Len:56 UNSTABLE 30594 18:02:04.932104 172.30.10.54 172.30.10.83 NFS V3 COMMIT Call (Reply In 30907), FH:0x29d8e1c4 30907 18:02:05.037990 172.30.10.83 172.30.10.54 NFS V3 COMMIT Reply (Call In 30594) 30912 18:02:05.038141 172.30.10.54 172.30.10.83 NFS V3 ACCESS Call (Reply In 30982), FH:0x29d8e1c4 30982 18:02:05.049686 172.30.10.83 172.30.10.54 NFS V3 ACCESS Reply (Call In 30912) 30991 18:02:05.050141 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 31178), FH:0x29d8e1c4 Offset:0 Len:14566 31178 18:02:05.077695 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 30991) Len:14513 So because it was appending to the file every php write call caused the sequence of the following rpc: ACCESS - READ - WRITE - COMMIT. And trying to flush the next line of the log it got stuck after READ call (the next should be WRITE call but client never did it). The same thing is for other log file written by othe php process. The last rpc for this file: 30990 18:02:05.050063 172.30.10.54 172.30.10.83 NFS V3 READ Call (Reply In 31068), FH:0x532fa29d Offset:131072 Len:2686 31068 18:02:05.062801 172.30.10.83 172.30.10.54 NFS V3 READ Reply (Call In 30990) Len:2685 A bit later there were several successful COMMIT calls (when php processes were closing other files I think). And other NFS activity was observed -- our nagios checks and other applications, which was just looking for presence and status of certain files, were running successfully and in tcpdump there are successful readdir/access/lookup/fstat calls. df utility did not hanged then too. Later when our engineer tried to access the mounted folder with mc the process locked acquiring nfs vn_lock held by php script (td=0xc6bf4690): (kgdb) thr 155 [Switching to thread 155 (Thread 100252)]#0 sched_switch (td=0xc70d98c0, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 1944 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xc70d98c0, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f4946 in sleepq_wait (wchan=0xcc2f2e5c) at /usr/src/sys/kern/subr_sleepqueue.c:580 #4 0xc07cb056 in _sleep (ident=0xcc2f2e5c, lock=0xc0c77730, priority=80, wmesg=0xc0b80b92 "nfs", timo=0) at /usr/src/sys/kern/kern_synch.c:226 #5 0xc07adf5a in acquire (lkpp=0xed46e7f0, extflags=Variable "extflags" is not available. ) at /usr/src/sys/kern/kern_lock.c:151 #6 0xc07ae84c in _lockmgr (lkp=0xcc2f2e5c, flags=8194, interlkp=0xcc2f2e8c, td=0xc70d98c0, file=0xc0b74aeb "/usr/src/sys/kern/vfs_subr.c", line=2061) at /usr/src/sys/kern/kern_lock.c:384 #7 0xc0832470 in vop_stdlock (ap=0xed46e840) at /usr/src/sys/kern/vfs_default.c:305 #8 0xc0aef4f6 in VOP_LOCK1_APV (vop=0xc0c1d5c0, a=0xed46e840) at vnode_if.c:1618 #9 0xc084ed86 in _vn_lock (vp=0xcc2f2e04, flags=8194, td=0xc70d98c0, file=0xc0b74aeb "/usr/src/sys/kern/vfs_subr.c", line=2061) at vnode_if.h:851 #10 0xc0841d84 in vget (vp=0xcc2f2e04, flags=8194, td=0xc70d98c0) at /usr/src/sys/kern/vfs_subr.c:2061 #11 0xc08355b3 in vfs_hash_get (mp=0xc7047b30, hash=2492309256, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/vfs_hash.c:81 #12 0xc09534d4 in nfs_nget (mntp=0xc7047b30, fhp=0xcb8d0868, fhsize=20, npp=0xed46e9f0, flags=2) at /usr/src/sys/nfsclient/nfs_node.c:120 #13 0xc0964a05 in nfs_lookup (ap=0xed46ea84) at /usr/src/sys/nfsclient/nfs_vnops.c:947 #14 0xc0aefbe6 in VOP_LOOKUP_APV (vop=0xc0c2b920, a=0xed46ea84) at vnode_if.c:99 #15 0xc0836841 in lookup (ndp=0xed46eb48) at vnode_if.h:57 #16 0xc083756f in namei (ndp=0xed46eb48) at /usr/src/sys/kern/vfs_lookup.c:219 #17 0xc0844fef in kern_lstat (td=0xc70d98c0, path=0x48705400
, pathseg=UIO_USERSPACE, sbp=0xed46ec18) at /usr/src/sys/kern/vfs_syscalls.c:2169 #18 0xc08451af in lstat (td=0xc70d98c0, uap=0xed46ecfc) at /usr/src/sys/kern/vfs_syscalls.c:2152 #19 0xc0ad9c75 in syscall (frame=0xed46ed38) at /usr/src/sys/i386/i386/trap.c:1090 #20 0xc0ac01b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #21 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) fr 6 #6 0xc07ae84c in _lockmgr (lkp=0xcc2f2e5c, flags=8194, interlkp=0xcc2f2e8c, td=0xc70d98c0, file=0xc0b74aeb "/usr/src/sys/kern/vfs_subr.c", line=2061) at /usr/src/sys/kern/kern_lock.c:384 384 error = acquire(&lkp, extflags, (LK_HAVE_EXCL | LK_WANT_EXCL), &contested, &waitstart); (kgdb) p *lkp $8 = { lk_object = { lo_name = 0xc0b80b92 "nfs", lo_type = 0xc0b80b92 "nfs", lo_flags = 70844416, lo_witness_data = { lod_list = { stqe_next = 0x0 }, lod_witness = 0x0 } }, lk_interlock = 0xc0c77730, lk_flags = 33816640, lk_sharecount = 0, lk_waitcount = 1, lk_exclusivecount = 1, lk_prio = 80, lk_timo = 51, lk_lockholder = 0xc6bf4690, lk_newlock = 0x0 } After this all processes that tried to access the share and df utility got stuck and no any traffic in tcpdump was observed. This time in crush dump I see 19 nfsiod processes but according to *nmp all were working with another mounts (which did not get stuck), threads look similar to this one: (kgdb) bt #0 sched_switch (td=0xcc179230, newtd=Variable "newtd" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0xc07cabe6 in mi_switch (flags=Variable "flags" is not available. ) at /usr/src/sys/kern/kern_synch.c:440 #2 0xc07f42fb in sleepq_switch (wchan=Variable "wchan" is not available. ) at /usr/src/sys/kern/subr_sleepqueue.c:497 #3 0xc07f460c in sleepq_catch_signals (wchan=0xc0c89f00) at /usr/src/sys/kern/subr_sleepqueue.c:417 #4 0xc07f4db9 in sleepq_timedwait_sig (wchan=0xc0c89f00) at /usr/src/sys/kern/subr_sleepqueue.c:631 #5 0xc07cb021 in _sleep (ident=0xc0c89f00, lock=0xc0c89ed0, priority=348, wmesg=0xc0b69679 "-", timo=120000) at /usr/src/sys/kern/kern_synch.c:220 #6 0xc095b46e in nfssvc_iod (instance=0xc0c89880) at /usr/src/sys/nfsclient/nfs_nfsiod.c:244 #7 0xc079e6c9 in fork_exit (callout=0xc095b380 , arg=0xc0c89880, frame=0xed981d38) at /usr/src/sys/kern/kern_fork.c:804 #8 0xc0ac01c0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:264 (kgdb) fr 6 #6 0xc095b46e in nfssvc_iod (instance=0xc0c89880) at /usr/src/sys/nfsclient/nfs_nfsiod.c:244 244 error = msleep(&nfs_iodwant[myiod], &nfs_iod_mtx, PWAIT | PCATCH, (kgdb) list 239 nfs_iodmount[myiod] = NULL; 240 /* 241 * Always keep at least nfs_iodmin kthreads. 242 */ 243 timo = (myiod < nfs_iodmin) ? 0 : nfs_iodmaxidle * hz; 244 error = msleep(&nfs_iodwant[myiod], &nfs_iod_mtx, PWAIT | PCATCH, 245 "-", timo); 246 if (error) { 247 nmp = nfs_iodmount[myiod]; 248 /* (kgdb) p *nmp $2 = {nm_mtx = {lock_object = {lo_name = 0xc0b808ee "NFSmount lock", lo_type = 0xc0b808ee "NFSmount lock", lo_flags = 16973824, lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}, nm_flag = 35399, nm_state = 1310720, nm_mountp = 0xc6d98b30, nm_numgrps = 16, nm_fh = "\001\000\000\000\000\223\000\000\001\000Í\a", '\0' , nm_fhsize = 12, nm_rpcclnt = {rc_flag = 0, rc_wsize = 0, rc_rsize = 0, rc_name = 0x0, rc_so = 0x0, rc_sotype = 0, rc_soproto = 0, rc_soflags = 0, rc_timeo = 0, rc_retry = 0, rc_srtt = {0, 0, 0, 0}, rc_sdrtt = {0, 0, 0, 0}, rc_sent = 0, rc_cwnd = 0, rc_timeouts = 0, rc_deadthresh = 0, rc_authtype = 0, rc_auth = 0x0, rc_prog = 0x0, rc_proctlen = 0, rc_proct = 0x0}, nm_so = 0xc95201a0, nm_sotype = 1, nm_soproto = 0, nm_soflags = 44, nm_nam = 0xc69a4ac0, nm_timeo = 6000, nm_retry = 2, nm_srtt = {15, 15, 15, 15}, nm_sdrtt = { 3, 3, 3, 3}, nm_sent = -256, nm_cwnd = 4192, nm_timeouts = 0, nm_deadthresh = 9, nm_rsize = 32768, nm_wsize = 32768, nm_readdirsize = 4096, nm_readahead = 1, nm_wcommitsize = 1177026, nm_acdirmin = 30, nm_acdirmax = 60, nm_acregmin = 3, nm_acregmax = 60, nm_verf = "KJ¡~\000\n\207é", nm_bufq = { tqh_first = 0x0, tqh_last = 0xc70493c0}, nm_bufqlen = 0, nm_bufqwant = 0, nm_bufqiods = 0, nm_maxfilesize = 1099511627775, nm_rpcops = 0xc0c2b5bc, nm_tprintf_initial_delay = 12, nm_tprintf_delay = 30, nm_nfstcpstate = {rpcresid = 0, flags = 1, sock_send_inprog = 0}, nm_hostname = "172.30.10.50\000/var/www/app10", '\0' , nm_clientid = 0, nm_fsid = {val = { 0, 0}}, nm_lease_time = 0, nm_last_renewal = 0} -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Wed Jan 13 10:50:02 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8EF51065672 for ; Wed, 13 Jan 2010 10:50:02 +0000 (UTC) (envelope-from peppe.maniscalco@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id 649038FC0A for ; Wed, 13 Jan 2010 10:50:02 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 22so123091fge.13 for ; Wed, 13 Jan 2010 02:49:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=leBwzeVgO2BMQ8KFXPZDyWjYuC1OjqgVQkXCLT4PjKk=; b=I8vfY4PWFhnxt0lSZWulbgtKGZLUyQtzzFGuTXbTY8y8RWxnBJLnWZT5yOMKFFvooo Pfh1srHKF5sGyN8ETnnG2VYGokg+qX1ZkDy5hK3e5ROXMIs+WeZPCHwU8bp9hHJN6gt9 ZBmP2+Vj0pC4FRLcwgMoJPzzeEdvZd9fER8Xs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=JPz9zOhAXokV4B9iwLdEax9fu++UG9rH0NKuzkI5lKmjPdtSZj7YPGKZb5xXvhQ0Vd V5sGqov6wWtc5RannYqJEyHWBo/bf7njb108imMla+ngOqf08KRcHH+HXseqFNzfiGgE NF+1HH+2h9+CplCbwFf3a3a37WuxfREOJ8KiA= MIME-Version: 1.0 Received: by 10.87.73.4 with SMTP id a4mr2068916fgl.76.1263377917142; Wed, 13 Jan 2010 02:18:37 -0800 (PST) Date: Wed, 13 Jan 2010 11:18:37 +0100 Message-ID: From: Giuseppe Maniscalco To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: A simple (?) question about fs... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 10:50:02 -0000 Ciao, I've a question that leave me a little bit confused... On a FreeBSD 6.1 server, I can't find a way to free up space on the / file system... root@mysql:/# df -h Filesystem Size Used Avail Capacity Mounted on /dev/mirror/gm0s1a 496M 400M 57M 88% / devfs 1.0K 1.0K 0B 100% /dev /dev/mirror/gm0s1e 989M 218K 910M 0% /tmp /dev/mirror/gm0s1f 60G 30G 25G 54% /usr /dev/mirror/gm0s1d 9.7G 181M 8.7G 2% /var devfs 1.0K 1.0K 0B 100% /usr/jail/sol/dev /usr/ports 60G 30G 25G 54% /usr/jail/sol/usr/ports Well, where are this 400M on /??? root@mysql:/# du -hd 1 2.0K ./.snap 2.0K ./dev 218K ./tmp 31G ./usr 174M ./var 1.6M ./etc 2.0K ./cdrom 2.0K ./dist 922K ./bin 33M ./boot 3.2M ./lib 274K ./libexec 2.0K ./mnt 2.0K ./proc 6.2M ./rescue 54K ./root 3.9M ./sbin 31G . Maybe temporary file? or system cache? how to clean it? Excuse me if this is a real newbie question, but I'm really confused.. :) Thank you! Peppe From owner-freebsd-fs@FreeBSD.ORG Wed Jan 13 11:26:07 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EF08106566B for ; Wed, 13 Jan 2010 11:26:07 +0000 (UTC) (envelope-from numisemis@yahoo.com) Received: from web112406.mail.gq1.yahoo.com (web112406.mail.gq1.yahoo.com [98.137.26.141]) by mx1.freebsd.org (Postfix) with SMTP id 3F6D18FC0C for ; Wed, 13 Jan 2010 11:26:07 +0000 (UTC) Received: (qmail 82056 invoked by uid 60001); 13 Jan 2010 11:26:05 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1263381965; bh=1A6lBeH2P73vDvlxnNUdYKGHxhF6gdJkjXFBK0BOHzg=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding; b=NjWnkdVLD3DHb4TPWQQa+Dm82JfDC7iiiEBeVSJIhPGRtIoBYwnXMQ3ArkBCkj495RUHt00BE+WAWpjH07aKp63rMNHGB0YUx6uiZp3sPbIUhVqz4MMUvwRS21wG/HtX7T1mQ+vKcQ85bZ/mWoOZiLrvgbYylem5/cT9w/juiGM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding; b=swq1EysSjr2o0pi7pXbJ2S+XeN5nhZMWzQW38TRC/rU1MNCGtsGoB4KXD4KhBmZnpNH5fh0TTHF+9U1IA8X8NTnyM2UOfw+wNCGCTe2q2RVuzhU/xkjKF1deYWuD8CkYOjUQylWasLPbZ7BE8DdYeInWdCFNt/16fa6VuZqyDm0=; Message-ID: <2688.81411.qm@web112406.mail.gq1.yahoo.com> X-YMail-OSG: sBcbpEcVM1m9UAQ5r5A0OAHZi3fnhYj.6P0j5PUDuuS_4xjqOWhgwjNJIXMk5sRRHnOAYn3sYhST716Fzrrgfr9vaFYr8u3PU0K9a8F0zeeC4YIF9MGPWVgQPGOSKZjeMnWHlc9Go12KriwRC4BK5kPnYxhwTrKOSs2b2PFaJuwsf2mVjm_H7usAMyO4L_sRNBXKHChMDPgw1Tx6sNMNPrODZoo3hRgDwc2bT5ZuFsZ_MQS8Y5pjlnI6vlUTIKwT8Pa.PrgyF5bqsHPLhwYoZ7qKHw1WtLQPhCZDtdFQ6zmAojZVqK.bFarEGbgrnTl4XRSDWb.8j4KgiHIQrfj98t1RVajoWBeHwLBvE7t3.qkTRz1qifddR2Gyrim97hh0OwUqu6yP_jei Received: from [87.248.121.165] by web112406.mail.gq1.yahoo.com via HTTP; Wed, 13 Jan 2010 03:26:04 PST X-Mailer: YahooMailWebService/0.8.100.260964 Date: Wed, 13 Jan 2010 03:26:04 -0800 (PST) From: =?utf-8?B?xaBpbXVuIE1pa2VjaW4=?= To: Giuseppe Maniscalco MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-fs@freebsd.org" Subject: Re: A simple (?) question about fs... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 11:26:07 -0000 13. sij. 2010., u 11:18, Giuseppe Maniscalco n= apisao:=0A=0ACiao,=0AI've a question that leave me a little bit confused...= =0AOn a FreeBSD 6.1 server, I can't find a way to free up space on the /=0A= file system...=0A=0Aroot@mysql:/# df -h=0AFilesystem Size Use= d Avail Capacity Mounted on=0A/dev/mirror/gm0s1a 496M 400M 57M= 88% /=0Adevfs 1.0K 1.0K 0B 100% /dev=0A= /dev/mirror/gm0s1e 989M 218K 910M 0% /tmp=0A/dev/mirror/gm0= s1f 60G 30G 25G 54% /usr=0A/dev/mirror/gm0s1d 9.7G = 181M 8.7G 2% /var=0Adevfs 1.0K 1.0K 0B = 100% /usr/jail/sol/dev=0A/usr/ports 60G 30G 25G 5= 4% /usr/jail/sol/usr/ports=0A=0AWell, where are this 400M on /???=0A=0Ar= oot@mysql:/# du -hd 1=0A2.0K ./.snap=0A2.0K ./dev=0A218K ./tmp=0A3= 1G ./usr=0A174M ./var=0A1.6M ./etc=0A2.0K ./cdrom=0A2.0K ./d= ist=0A922K ./bin=0A33M ./boot=0A3.2M ./lib=0A274K ./libexec=0A2= .0K ./mnt=0A2.0K ./proc=0A6.2M ./rescue=0A54K ./root=0A3.9M = ./sbin=0A31G .=0A=0AMaybe temporary file? or system cache? how to clean = it?=0AExcuse me if this is a real newbie question, but I'm really confused.= . :)=0A=0A=0AGo to single user mode, unmount all fillesystems except / and = try 'du' again. There is a chance that some directory in / filesystem has d= ata but you don't see it because it is used as a mounpoint for some other f= ilesystem.=0AIf that is not the case, try fsck-ing it.=0A=0A=0A From owner-freebsd-fs@FreeBSD.ORG Wed Jan 13 12:50:51 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5ECBA106566C for ; Wed, 13 Jan 2010 12:50:51 +0000 (UTC) (envelope-from wonslung@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id DDFBE8FC12 for ; Wed, 13 Jan 2010 12:50:50 +0000 (UTC) Received: by fxm27 with SMTP id 27so507086fxm.3 for ; Wed, 13 Jan 2010 04:50:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=zj6MYdGbsM5N/Wxeg83IW0+gePEbplOrYjx5lkxo95A=; b=GvA1rjIPxdqc/RHhWmN3gKAIX5Lwo3nyCjqMLu+aOLClt3rIk26uLAoL/U5+4UDU14 o9oakpCMiSQblYtT3RCFol2E0pphCLRtgJXlAPPvmaCGF6+kDdqI2sbwcboCfALX5Xx0 ovB01VDSv+Sm9lqkD3CZNaiuXIs3qEoZqdYCY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=JaSamQXc/Fa/VIl+w/Vc5eI9VZyB5JHb02GjlltN7okNV1jlv6NYVeFGqyL4xcFEO7 J+nD9V20g56OdkizcwXi/nlWxz7zv5eFz/Y3uT0alJHEkFmQDbdsOiXYCQNZ/fBJY32Y PrVjas85tQ7sLo6OfibtZV9YpHc4+zDS1hTS8= MIME-Version: 1.0 Received: by 10.103.84.19 with SMTP id m19mr6577671mul.66.1263387043705; Wed, 13 Jan 2010 04:50:43 -0800 (PST) In-Reply-To: <2688.81411.qm@web112406.mail.gq1.yahoo.com> References: <2688.81411.qm@web112406.mail.gq1.yahoo.com> Date: Wed, 13 Jan 2010 07:50:43 -0500 Message-ID: From: Thomas Burgess To: =?UTF-8?Q?=C5=A0imun_Mikecin?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-fs@freebsd.org" Subject: Re: A simple (?) question about fs... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 12:50:51 -0000 did you build a custom kernel? if so it's likely /boot/kernel.old 2010/1/13 =C5=A0imun Mikecin > 13. sij. 2010., u 11:18, Giuseppe Maniscalco > napisao: > > Ciao, > I've a question that leave me a little bit confused... > On a FreeBSD 6.1 server, I can't find a way to free up space on the / > file system... > > root@mysql:/# df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/mirror/gm0s1a 496M 400M 57M 88% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/mirror/gm0s1e 989M 218K 910M 0% /tmp > /dev/mirror/gm0s1f 60G 30G 25G 54% /usr > /dev/mirror/gm0s1d 9.7G 181M 8.7G 2% /var > devfs 1.0K 1.0K 0B 100% /usr/jail/sol/dev > /usr/ports 60G 30G 25G 54% > /usr/jail/sol/usr/ports > > Well, where are this 400M on /??? > > root@mysql:/# du -hd 1 > 2.0K ./.snap > 2.0K ./dev > 218K ./tmp > 31G ./usr > 174M ./var > 1.6M ./etc > 2.0K ./cdrom > 2.0K ./dist > 922K ./bin > 33M ./boot > 3.2M ./lib > 274K ./libexec > 2.0K ./mnt > 2.0K ./proc > 6.2M ./rescue > 54K ./root > 3.9M ./sbin > 31G . > > Maybe temporary file? or system cache? how to clean it? > Excuse me if this is a real newbie question, but I'm really confused.. :) > > > Go to single user mode, unmount all fillesystems except / and try 'du' > again. There is a chance that some directory in / filesystem has data but > you don't see it because it is used as a mounpoint for some other > filesystem. > If that is not the case, try fsck-ing it. > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Jan 13 12:57:20 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DCA5106566B for ; Wed, 13 Jan 2010 12:57:20 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id E63E38FC08 for ; Wed, 13 Jan 2010 12:57:19 +0000 (UTC) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id o0DCv1IO094653; Wed, 13 Jan 2010 13:57:16 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id o0DCv1D2094652; Wed, 13 Jan 2010 13:57:01 +0100 (CET) (envelope-from olli) Date: Wed, 13 Jan 2010 13:57:01 +0100 (CET) Message-Id: <201001131257.o0DCv1D2094652@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, peppe.maniscalco@gmail.com In-Reply-To: X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.4-PRERELEASE-20080904 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Wed, 13 Jan 2010 13:57:17 +0100 (CET) Cc: Subject: Re: A simple (?) question about fs... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, peppe.maniscalco@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 12:57:20 -0000 Giuseppe Maniscalco wrote: > On a FreeBSD 6.1 server, I can't find a way to free up space on the / > file system... Please install lsof (/usr/ports/sysutils/lsof) and execute this command: lsof +aL1 / It will list all processes that keep files (inodes) open in the root file system with a link count of zero. Killing or restarting those processes will free the space occupied by the files. Explanation: When you delete a file, but the file is still opened by a process, then the space of the file is still kept allocated in the file system. df(1) will see it, because it lists the block allocations, but du(1) doesn't see it because the directory entry of the file was already removed. That's why you see such a difference in the output between df(1) and du(1). As soon as the files are closed (which happens when the processes are killed or restarted), the space will be freed in the file system. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd Perl is worse than Python because people wanted it worse. -- Larry Wall From owner-freebsd-fs@FreeBSD.ORG Wed Jan 13 14:32:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 745A2106566C for ; Wed, 13 Jan 2010 14:32:30 +0000 (UTC) (envelope-from peppe.maniscalco@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id 0929F8FC13 for ; Wed, 13 Jan 2010 14:32:29 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 22so207822fge.13 for ; Wed, 13 Jan 2010 06:32:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=TCG2Ai7S1277ymh+GhRDIN8JkOkKhiNjMArs4DGQGKI=; b=q04K7efyGqKjjy71NMG+M/FaeFSY/hd75s2yRQ692h1pKv7USwmwsbjfZuGswLyvMk KIPhew3j964XON3EoaxCiSS/vp6QC4ZACpoOYkfKXhBDdJkwQXCdNrzWOvD0c/LzK7Pj jbQrlw19NWrgxybP1LHD8WibZNzFgpNiBD6wA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=KxAixWmAvSaEKDK04IfJy9jMTtftaBSphV72qOsTIioYN/JHehQoBNnzMrG2GvAsO3 KpO8GLBJ9e1G4eQZzpMDbAFJb6qtbAjjurYZ0TQGjClc71iOteZAPAMwGFNb+jajzLdw mDBN6dZ9aeF5kRYN8wg6phe/KS9jNbHALtZ8I= MIME-Version: 1.0 Received: by 10.86.239.17 with SMTP id m17mr1367098fgh.16.1263393143243; Wed, 13 Jan 2010 06:32:23 -0800 (PST) In-Reply-To: <201001131257.o0DCv1D2094652@lurza.secnetix.de> References: <201001131257.o0DCv1D2094652@lurza.secnetix.de> Date: Wed, 13 Jan 2010 15:32:23 +0100 Message-ID: From: Giuseppe Maniscalco To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: A simple (?) question about fs... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 14:32:30 -0000 Yes! Great hint Oliver! Thank you so much! root@mysql:/usr/ports/sysutils/lsof# lsof +aL1 / COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME ssmtp 9839 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 19898 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 29204 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 39567 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 50005 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 60433 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 80982 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 89855 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) ssmtp 99693 root 4w VREG 0,70 366968101 0 49363 / (/dev/mirror/gm0s1a) Filesystem Size Used Avail Capacity Mounted on /dev/mirror/gm0s1a 496M 50M 407M 11% / :-) From owner-freebsd-fs@FreeBSD.ORG Thu Jan 14 11:21:05 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0229F1065670 for ; Thu, 14 Jan 2010 11:21:05 +0000 (UTC) (envelope-from artis.caune@gmail.com) Received: from mail-bw0-f213.google.com (mail-bw0-f213.google.com [209.85.218.213]) by mx1.freebsd.org (Postfix) with ESMTP id 8DFA48FC0C for ; Thu, 14 Jan 2010 11:21:03 +0000 (UTC) Received: by bwz5 with SMTP id 5so16227131bwz.3 for ; Thu, 14 Jan 2010 03:20:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=jvkBFB4Wgc5ybcad4G0hHL9EcaBnPDtPm2gMo8rn/Ts=; b=x3iTwcPQ7lROwOzjta1RtIKoy0egVL48FB062DP86B5dFlM7LtCn9VMKt+r/ZhpprJ YSm0O68iGkCALXd71c/y4hvHrdiFBL3j3qv+8MqhZORHQbfWtQyhb0OcJboYsufQkJwA jMZHTtbp8nnho3V9yxtt1ARg3iDZlOOfgmIjU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=urH1RoQiVhA6xcVTyHnVECP36KnBmGa4Tr+dLXoFReG7lqiUj25E0Dw4e7T/IvhjoQ y5RXCrNTiTmrdSC+MQiCVD1FJBAwxmCaLYEtkmikP1pprNtg8qpWoXhMPA63W8+UEC7/ lAvEOC31OgqfZdkjBncQKEjaPdq9U2Bvjq+l8= MIME-Version: 1.0 Received: by 10.204.23.20 with SMTP id p20mr339432bkb.54.1263466479809; Thu, 14 Jan 2010 02:54:39 -0800 (PST) Date: Thu, 14 Jan 2010 12:54:39 +0200 Message-ID: <9e20d71e1001140254r619d3439xa05ff3a6d0733dfd@mail.gmail.com> From: Artis Caune To: freebsd-fs Content-Type: text/plain; charset=UTF-8 Subject: [patch] background fsck on ZFS only X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2010 11:21:05 -0000 Hi, I'm using only ZFS file system and when box boots up it always shows this message: "Starting background file system checks in 60 seconds." How about checking for ufs and continue only if one is found? patch: http://www.nic.lv/sofq/patch-bgfsck.txt -- Artis Caune Everything should be made as simple as possible, but not simpler. From owner-freebsd-fs@FreeBSD.ORG Thu Jan 14 15:48:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DFF01065693; Thu, 14 Jan 2010 15:48:17 +0000 (UTC) (envelope-from nekoexmachina@gmail.com) Received: from gv-out-0910.google.com (gv-out-0910.google.com [216.239.58.188]) by mx1.freebsd.org (Postfix) with ESMTP id 739238FC21; Thu, 14 Jan 2010 15:48:16 +0000 (UTC) Received: by gv-out-0910.google.com with SMTP id o2so16920gve.39 for ; Thu, 14 Jan 2010 07:48:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=vlqNPZNmYqKxf5+DCWvdWfsLYVQoW48+pfqzpl1EHdc=; b=t2/iatwLPgXs98XAQN5dnIcgfduD1mUbTgbXUqxfGFQGTOte/ssVwDCEXKRkGegomm +VjA3VghR8fq3ptZ0UeODmKJEv3FfdrwZd7NOJeYAfBQeAagGFYPt99hAP9DqBsgDWxw 4loVH785xiq1eGEDJdd7thrEbNfG7IGiTXpKc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=jkaYWbB1F6jeWCPbAzsyHvOEKUVREjFaSftqBDAUevWvFcCjqXvi+FlrNVlgGQEfz1 y8iXK4OnWfooRdayK2nGkGs3FN1qnB11qQuY7RWuPmzkCDmzNOl4iWdu7/H3zQD8qunt YINh6ZdqbyD8Wow0nsbbOUmfbgbGfaAn62Bxs= MIME-Version: 1.0 Received: by 10.103.122.14 with SMTP id z14mr447389mum.10.1263484092648; Thu, 14 Jan 2010 07:48:12 -0800 (PST) In-Reply-To: <884554e61001021740h40294fabn95c5a7c1f0bca788@mail.gmail.com> References: <884554e60912090842x1bf1e8e4u842cbce4647aa63@mail.gmail.com> <4b21a7b9.5744f10a.3378.6447@mx.google.com> <884554e60912110938n79439fc1p4a3d41f81bb88991@mail.gmail.com> <5da0588e0912112203v52653fb2ic71d97970b615547@mail.gmail.com> <884554e60912120149t6cd0df5me349b1fdf485b0c@mail.gmail.com> <884554e60912120547i1ccca7acob8714fd5121a5508@mail.gmail.com> <4b2a6cad.9713f30a.573f.ffffad80@mx.google.com> <20091217180445.GA4115@a91-153-117-195.elisa-laajakaista.fi> <884554e60912180038w3442dc67of9a1ed755f399df0@mail.gmail.com> <884554e61001021740h40294fabn95c5a7c1f0bca788@mail.gmail.com> Date: Thu, 14 Jan 2010 18:48:12 +0300 Message-ID: <884554e61001140748n7f7fd3b7ic085bdee5ca4695a@mail.gmail.com> From: Mikle Krutov To: freebsd-fs@freebsd.org, jh@freebsd.org, stas@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: [8.0-RELEASE] ext2fs mount fails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2010 15:48:17 -0000 If anyone is still interested, i've figured out why my fs's could not be mounted (same thing went on one more of my machines). Disk inside of it was gpt, just like both on first pc, and after recreating partition table with mbr - everything worked flawlessly. 2010/1/3 Mikle Krutov : > Sorry for such a long time no-write. > I've got to the PC with freebsd installed, and managed to copy all of > data from ext2fs to zfs via linux-livecd. > What data about this two partitions should be sent to fbsd-bugs? > Also, not even 1 version of fbsd could mount them, tried 7.2 (livefs > cd), 8.0-RELEASE and STABLE, so it may be not the fbsd bug. > > 2009/12/18 Mikle Krutov : >> Well, i could not verify that under freebsd. Every fs-tool (except >> testdisk - tried to use it to restore superblock after message 'wrong >> magic number' appeared first time) tells me that i've got wrong magic >> number for this filesystem. >> I can not give the gpart show output right now; will send it in couple o= f days. >> >> 2009/12/17 Jaakko Heinonen : >>> On 2009-12-17, Aditya Sarawgi wrote: >>>> > Yes, it's 100% ext2fs. =C2=A0I could not create dump with fbsd - 'wr= ong >>>> > magic number', so i used linux livecd. >>> >>>> Mikle, file a pr. jh@, stas@ any ideas about this. >>> >>> How did you verify that /dev/ad8p1 contains a valid ext2fs (on FreeBSD)= ? >>> AFAIK, recent dumpe2fs should work in any case if the partition contain= s >>> a valid file system. My guess is that there is something wrong with the >>> partition or how FreeBSD sees the partition. >>> >>> "gpart show" output might be useful for starters and maybe similar >>> output from Linux. >>> >>> -- >>> Jaakko >>> >> > From owner-freebsd-fs@FreeBSD.ORG Thu Jan 14 15:50:03 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DEEB1065670 for ; Thu, 14 Jan 2010 15:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 12AC38FC19 for ; Thu, 14 Jan 2010 15:50:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0EFo28B048242 for ; Thu, 14 Jan 2010 15:50:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0EFo22i048241; Thu, 14 Jan 2010 15:50:02 GMT (envelope-from gnats) Date: Thu, 14 Jan 2010 15:50:02 GMT Message-Id: <201001141550.o0EFo22i048241@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: "Pedro F. Giffuni" Cc: Subject: Re: kern/138109: [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2fs based on BSD Lite2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Pedro F. Giffuni" List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2010 15:50:03 -0000 The following reply was made to PR kern/138109; it has been noted by GNATS. From: "Pedro F. Giffuni" To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/138109: [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2fs based on BSD Lite2 Date: Thu, 14 Jan 2010 07:14:51 -0800 (PST) Since the new BSDL ext2fs cleanup/rewrite (with these changes and much more) has been committed to head I think we can close this PR. Thanks to everyone involved! From owner-freebsd-fs@FreeBSD.ORG Thu Jan 14 17:07:32 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6EED106566C for ; Thu, 14 Jan 2010 17:07:32 +0000 (UTC) (envelope-from djp@polands.org) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.124]) by mx1.freebsd.org (Postfix) with ESMTP id 839598FC0C for ; Thu, 14 Jan 2010 17:07:32 +0000 (UTC) Received: from hrndva-omtalb.mail.rr.com ([10.128.143.54]) by hrndva-qmta03.mail.rr.com with ESMTP id <20100114165147888.YBHC8345@hrndva-qmta03.mail.rr.com> for ; Thu, 14 Jan 2010 16:51:47 +0000 X-Authority-Analysis: v=1.0 c=1 a=sVhNVL3m-NYA:10 a=6I5d2MoRAAAA:8 a=bqq2Vc5EAAAA:8 a=LS4BNEoFbNd2vE0K8A4A:9 a=_WeRSUFL9el-dtXa90wA:7 a=pTbTa6b-dium-lvMrPdQpwZWhs4A:4 a=MZUFvVIdZ-cA:10 a=5ERLOmoKdHQA:10 X-Cloudmark-Score: 0 X-Originating-IP: 75.87.219.217 Received: from [75.87.219.217] ([75.87.219.217:57004] helo=haran.polands.org) by hrndva-oedge04.mail.rr.com (envelope-from ) (ecelerity 2.2.2.39 r()) with ESMTP id C2/01-10402-56B4F4B4; Thu, 14 Jan 2010 16:50:45 +0000 Received: from ammon.polands.org (ammon.polands.org [172.16.1.7]) by haran.polands.org (8.14.3/8.14.3) with ESMTP id o0EGoiI5092454; Thu, 14 Jan 2010 10:50:44 -0600 (CST) (envelope-from djp@polands.org) Received: from ammon.polands.org (localhost [127.0.0.1]) by ammon.polands.org (8.14.3/8.14.3) with ESMTP id o0EGoikO099185; Thu, 14 Jan 2010 10:50:44 -0600 (CST) (envelope-from djp@ammon.polands.org) Received: (from djp@localhost) by ammon.polands.org (8.14.3/8.14.3/Submit) id o0EGoi37099184; Thu, 14 Jan 2010 10:50:44 -0600 (CST) (envelope-from djp) Date: Thu, 14 Jan 2010 10:50:44 -0600 From: Doug Poland To: Ivan Voras Message-ID: <20100114165034.GA99127@polands.org> References: <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> <3aa09fd8723749d1fa65f1b9a6faac60.squirrel@email.polands.org> <27117211dd662bcf93055f4351243396.squirrel@email.polands.org> <9bbcef731001140650h5d887843ubc6d555da993e8b6@mail.gmail.com> <9bbcef731001140815h5ee1d672je58c8ec91382e8d4@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9bbcef731001140815h5ee1d672je58c8ec91382e8d4@mail.gmail.com> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2010 17:07:32 -0000 Hello, I am posting the end of this thread http://lists.freebsd.org/pipermail/freebsd-questions/2010-January/210963.html concering ZFS panics on an amd64 box with 4GB RAM and 6 SCSI drives in RAIDZ1. Ivan Voras suggested I forward to this list for further analysis. On Thu, Jan 14, 2010 at 05:15:47PM +0100, Ivan Voras wrote: > 2010/1/14 Doug Poland : > >>>>> > >>>>> kstat.zfs.misc.arcstats.size > >>>>> > >>>>> seemed to fluctuate between about 164,000,00 and 180,000,000 > >>>>> bytes during this last run > >>>> > >>>> Is that with or without panicking? > >>>> > >>> with a panic > >>> > >>> > >>>> If the system did panic then it looks like the problem is a > >>>> memory leak somewhere else in the kernel, which you could confirm > >>>> by monitoring vmstat -z. > >>>> > >>> I'll give that a try. Am I looking for specific items in vmstat > >>> -z arc*, zil*, zfs*, zio*. Please advise. > >> > >> You should look for whatever is allocating all your memory between > >> 180 MB (which is your ARC size) and 1.2 GB (which is your kmem > >> size). > >> > > > > OK, another run, this time back to vfs.zfs.arc_max=512M in > > /boot/loader.conf, and a panic: > > > > panic: kmem malloc(131072): kmem map too small: 1294258176 total > > allocated > > > > I admit I do not fully understand what metrics are important to > > proper analysis of this issue. ??In this case, I was watching the > > following within 1 second of the panic: > > > > sysctl kstat.zfs.misc.arcstats.size: 41739944 > > sysctl vfs.numvnodes: 678 > > sysctl vfs.zfs.arc_max: 536870912 > > sysctl vfs.zfs.arc_meta_limit: 134217728 > > sysctl vfs.zfs.arc_meta_used: 7228584 > > sysctl vfs.zfs.arc_min: 67108864 > > sysctl vfs.zfs.cache_flush_disable: 0 > > sysctl vfs.zfs.debug: 0 > > sysctl vfs.zfs.mdcomp_disable: 0 > > sysctl vfs.zfs.prefetch_disable: 1 > > sysctl vfs.zfs.recover: 0 > > sysctl vfs.zfs.scrub_limit: 10 > > sysctl vfs.zfs.super_owner: 0 > > sysctl vfs.zfs.txg.synctime: 5 > > sysctl vfs.zfs.txg.timeout: 30 > > sysctl vfs.zfs.vdev.aggregation_limit: 131072 > > sysctl vfs.zfs.vdev.cache.bshift: 16 > > sysctl vfs.zfs.vdev.cache.max: 16384 > > sysctl vfs.zfs.vdev.cache.size: 10485760 > > sysctl vfs.zfs.vdev.max_pending: 35 > > sysctl vfs.zfs.vdev.min_pending: 4 > > sysctl vfs.zfs.vdev.ramp_rate: 2 > > sysctl vfs.zfs.vdev.time_shift: 6 > > sysctl vfs.zfs.version.acl: 1 > > sysctl vfs.zfs.version.dmu_backup_header: 2 > > sysctl vfs.zfs.version.dmu_backup_stream: 1 > > sysctl vfs.zfs.version.spa: 13 > > sysctl vfs.zfs.version.vdev_boot: 1 > > sysctl vfs.zfs.version.zpl: 3 > > sysctl vfs.zfs.zfetch.array_rd_sz: 1048576 > > sysctl vfs.zfs.zfetch.block_cap: 256 > > sysctl vfs.zfs.zfetch.max_streams: 8 > > sysctl vfs.zfs.zfetch.min_sec_reap: 2 > > sysctl vfs.zfs.zil_disable: 0 > > sysctl vm.kmem_size: 1327202304 > > sysctl vm.kmem_size_max: 329853485875 > > sysctl vm.kmem_size_min: 0 > > sysctl vm.kmem_size_scale: 3 > > > > > > vmstat -z | egrep -i 'zfs|zil|arc|zio|files' > > ITEM SIZE LIMIT USED FREE REQUESTS > > zio_cache: 720, 0, 53562, 98, 86386955, > > arc_buf_hdr_t: 208, 0, 1193, 31, 11990, > > arc_buf_t: 72, 0, 1180, 120, 11990, > > zil_lwb_cache: 200, 0, 11580, 2594, 62407, > > zfs_znode_cache: 376, 0, 605, 55, 654, > > > > vmstat -m |grep solaris|sed 's/K//'|awk '{print "vm.solaris:", $3*1024}' > > > > > > solaris: 1285068800 > > > > The value I see as the culprit is vmstat -m | grep solaris. ??This > > value fluctuates wildly during the run and is always near kmem_size > > at the time of the panic. > > > > Again, I'm not sure what to look for here, and you are patiently > > helping me along in this process. ??If you have any tips or can > > point me to docs on how to easily monitor these values, I will > > endeavor to do so. > > The only really important ones should be kstat.zfs.misc.arcstats.size > (which you very rarely print) and vm.kmem_size. The "solaris" entry > above should be near kstat.zfs.misc.arcstats.size in all cases. > > But I don't have any more ideas here. Try taking this post (also > include kstst.zfs.misc.arcstats.size) to the freebsd-fs@ mailing list. > Thank you for your help, I will take this over to the freebsd-fs list. -- Regards, Doug From owner-freebsd-fs@FreeBSD.ORG Thu Jan 14 20:03:23 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D254E106568B; Thu, 14 Jan 2010 20:03:23 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [IPv6:2001:16d8:fffb:4::abba]) by mx1.freebsd.org (Postfix) with ESMTP id 734D88FC1B; Thu, 14 Jan 2010 20:03:23 +0000 (UTC) Received: from orion.SpringDaemons.com (adsl-99-48-191-9.dsl.snfc21.sbcglobal.net [99.48.191.9]) by mx0.deglitch.com (Postfix) with ESMTPA id 572D68FC4E; Thu, 14 Jan 2010 23:03:18 +0300 (MSK) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id 1C1D139858; Thu, 14 Jan 2010 12:03:41 -0800 (PST) Date: Thu, 14 Jan 2010 12:03:41 -0800 From: Stanislav Sedov To: Mikle Krutov Message-Id: <20100114120341.7353de18.stas@FreeBSD.org> In-Reply-To: <884554e61001140748n7f7fd3b7ic085bdee5ca4695a@mail.gmail.com> References: <884554e60912090842x1bf1e8e4u842cbce4647aa63@mail.gmail.com> <4b21a7b9.5744f10a.3378.6447@mx.google.com> <884554e60912110938n79439fc1p4a3d41f81bb88991@mail.gmail.com> <5da0588e0912112203v52653fb2ic71d97970b615547@mail.gmail.com> <884554e60912120149t6cd0df5me349b1fdf485b0c@mail.gmail.com> <884554e60912120547i1ccca7acob8714fd5121a5508@mail.gmail.com> <4b2a6cad.9713f30a.573f.ffffad80@mx.google.com> <20091217180445.GA4115@a91-153-117-195.elisa-laajakaista.fi> <884554e60912180038w3442dc67of9a1ed755f399df0@mail.gmail.com> <884554e61001021740h40294fabn95c5a7c1f0bca788@mail.gmail.com> <884554e61001140748n7f7fd3b7ic085bdee5ca4695a@mail.gmail.com> Organization: The FreeBSD Project X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, jh@freebsd.org Subject: Re: [8.0-RELEASE] ext2fs mount fails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2010 20:03:23 -0000 On Thu, 14 Jan 2010 18:48:12 +0300 Mikle Krutov mentioned: > If anyone is still interested, i've figured out why my fs's could not > be mounted (same thing went on one more of my machines). Disk inside > of it was gpt, just like both on first pc, and after recreating > partition table with mbr - everything worked flawlessly. Are you sure you were using the correct devide node with GPT? I can't see how GPT could influence the ext2fs operation. -- Stanislav Sedov ST4096-RIPE From owner-freebsd-fs@FreeBSD.ORG Fri Jan 15 13:46:32 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFF291065692 for ; Fri, 15 Jan 2010 13:46:32 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (email.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id 9B96F8FC0A for ; Fri, 15 Jan 2010 13:46:32 +0000 (UTC) Received: by email.octopus.com.au (Postfix, from userid 1002) id A66945CB8ED; Sat, 16 Jan 2010 00:22:36 +1100 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: * X-Spam-Status: No, score=1.9 required=10.0 tests=ALL_TRUSTED, FH_DATE_PAST_20XX autolearn=no version=3.2.3 Received: from [10.20.30.104] (60.218.233.220.static.exetel.com.au [220.233.218.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id 8B80A5CB8BB; Sat, 16 Jan 2010 00:22:32 +1100 (EST) Message-ID: <4B5071A1.6050805@modulus.org> Date: Sat, 16 Jan 2010 00:46:09 +1100 From: Andrew Snow User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: Ivan Voras , freebsd-fs@freebsd.org References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <200603241518.01027.mi+mx@aldan.algebra.com> <20060325103927.GE703@turion.vk2pj.dyndns.org> <200603250920.14208@aldan> <20060325190333.GD7001@funkthat.com> <4B4F7CF5.4040307@aldan.algebra.com> <4B4F9781.2050508@modulus.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: An old gripe: Reading via mmap stinks X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2010 13:46:33 -0000 Ivan Voras wrote: >> Hi Mikhail, I assume these tests were done on UFS. Have you tried >> ZFS? I'm curious to see the results. > I suspect it would be noticably worse :) AFAIK ZFS integration with mmap > does at least one extra in-memory data copy. True but ZFS also has its own quite aggressive read-ahead algorithm - Andrew From owner-freebsd-fs@FreeBSD.ORG Fri Jan 15 16:57:15 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC5FA106566C; Fri, 15 Jan 2010 16:57:15 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gx0-f218.google.com (mail-gx0-f218.google.com [209.85.217.218]) by mx1.freebsd.org (Postfix) with ESMTP id 575978FC12; Fri, 15 Jan 2010 16:57:15 +0000 (UTC) Received: by gxk10 with SMTP id 10so607295gxk.3 for ; Fri, 15 Jan 2010 08:57:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=006Hy+SzcLW5RyLs66GlTLo0Ty7S4T3T3FNQ3kWQARs=; b=KMhZ2H02rMjQ6mq4nSwiwkfA1VIyRW7VNAuD3puPYyORayUWBZD4/1e5sgjrvbtaCG qv+YFvzsqFxyj5LwfD8qWf9MJt+2U7zlejEQ+NclZoajKS3LXfan+3402/NwUuYnnw69 ombxBBjQP0WHBWRZy1WCAJT0urN+1ZDlx9Kgw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=pv6gkrGn3LavsPCWB83E/Fy/cc/Z298CEFzwYJO1vhyBwY3ilLHcPbQWxs2TQAESfa oKOF94ZICFUS7NnonEJSJ7+eYJHVFgSPAPVc1BHz2/ANInKW1VMMvxUO7jhxJtUN2kV+ R+DrAi2vcZzC4R/V9bsGz3EsBc54+MXvukadc= MIME-Version: 1.0 Received: by 10.101.214.5 with SMTP id r5mr5024557anq.3.1263574623123; Fri, 15 Jan 2010 08:57:03 -0800 (PST) Date: Fri, 15 Jan 2010 18:57:03 +0200 Message-ID: From: Dan Naumov To: Rick Macklem , freebsd-questions@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: (SOLVED) Re: installing FreeBSD 8 on SSDs and UFS2 - partition alignment, block sizes, what does one need to know? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2010 16:57:15 -0000 On Fri, Jan 15, 2010 at 6:38 PM, Rick Macklem wrote: > > > On Tue, 12 Jan 2010, Dan Naumov wrote: > >> For my upcoming storage system, the OS install is going to be on a >> 80gb Intel SSD disk and for various reasons, I am now pretty convinced >> to stick with UFS2 for the root partition (the actual data pool will >> be ZFS using traditional SATA disks). I am probably going to use GPT >> partitioning and have the SSD host the swap, boot, root and a few >> other partitions. What do I need to know in regards to partition >> alignment and filesystem block sizes to get the best performance out >> of the Intel SSDs? >> > I can't help with your question, but I thought I'd mention that there > was a recent post (on freebsd-current, I think?) w.r.t. using an SSD > for the ZFS log file. It suggested that that helped with ZFS perf., so > you might want to look for the message. > > rick I have managed to figure out the essential things to know by know, I just wish there was a single, easy to grasp webpage or HOWTO describing and whys and hows so I wouldn't have had had to spend the entire day googling things to get a proper grasp on the issue :) To (perhaps a bit too much) simplify things, if you are using an SSD with FreeeBSD, you: 1) Should use GPT 2) Should create the freebsd-boot partition as normal (to ensure compatibility with some funky BIOSes) 3) All additional partitions should be aligned, meaning that their boundaries should be dividable by 1024kb (that's 2048 "logical blocks" in gpart). Ie, having created your freeebsd-boot, your next partition should start at block 2048 and the partition size should be dividable by 2048 blocks. This applies to ALL further partitions added to the disk, so you WILL end up having some empty space between them, but a few MBs worth of space will be lost at most. P.S: My oversimplification was in that MOST SSDs will be just fine with a 512 kb / 1024 block alignment. However, _ALL_ SSDs will be fine with 1024 kb / 2048 block alignment. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Sat Jan 16 10:01:49 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69F491065693; Sat, 16 Jan 2010 10:01:49 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 40B858FC13; Sat, 16 Jan 2010 10:01:49 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0GA1ng9011231; Sat, 16 Jan 2010 10:01:49 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0GA1nrK011227; Sat, 16 Jan 2010 10:01:49 GMT (envelope-from linimon) Date: Sat, 16 Jan 2010 10:01:49 GMT Message-Id: <201001161001.o0GA1nrK011227@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/142872: [zfs] ZFS ZVOL Lockmgr Deadlock X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2010 10:01:49 -0000 Old Synopsis: ZFS ZVOL Lockmgr Deadlock New Synopsis: [zfs] ZFS ZVOL Lockmgr Deadlock Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jan 16 10:01:32 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=142872 From owner-freebsd-fs@FreeBSD.ORG Sat Jan 16 10:36:17 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBC571065672; Sat, 16 Jan 2010 10:36:17 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6CCCF8FC0C; Sat, 16 Jan 2010 10:36:17 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0GAaHOA037755; Sat, 16 Jan 2010 10:36:17 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0GAaHZu037751; Sat, 16 Jan 2010 10:36:17 GMT (envelope-from jh) Date: Sat, 16 Jan 2010 10:36:17 GMT Message-Id: <201001161036.o0GAaHZu037751@freefall.freebsd.org> To: bruce@cran.org.uk, jh@FreeBSD.org, freebsd-fs@FreeBSD.org, jh@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/138367: [tmpfs] [panic] 'panic: Assertion pages > 0 failed' when running regression/tmpfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2010 10:36:17 -0000 Synopsis: [tmpfs] [panic] 'panic: Assertion pages > 0 failed' when running regression/tmpfs State-Changed-From-To: open->feedback State-Changed-By: jh State-Changed-When: Sat Jan 16 10:31:57 UTC 2010 State-Changed-Why: The reported panic should not occur after r201773. Can you confirm? Responsible-Changed-From-To: freebsd-fs->jh Responsible-Changed-By: jh Responsible-Changed-When: Sat Jan 16 10:31:57 UTC 2010 Responsible-Changed-Why: I'll try to look at the other issues raised by Gleb Kurtsou. http://www.freebsd.org/cgi/query-pr.cgi?pr=138367 From owner-freebsd-fs@FreeBSD.ORG Sat Jan 16 12:20:08 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FF9A1065672 for ; Sat, 16 Jan 2010 12:20:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 362E38FC0A for ; Sat, 16 Jan 2010 12:20:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0GCK85U036863 for ; Sat, 16 Jan 2010 12:20:08 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0GCK8qK036862; Sat, 16 Jan 2010 12:20:08 GMT (envelope-from gnats) Date: Sat, 16 Jan 2010 12:20:08 GMT Message-Id: <201001161220.o0GCK8qK036862@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Justin Head Cc: Subject: Re: kern/142872: [zfs] ZFS ZVOL Lockmgr Deadlock X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Justin Head List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2010 12:20:08 -0000 The following reply was made to PR kern/142872; it has been noted by GNATS. From: Justin Head To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/142872: [zfs] ZFS ZVOL Lockmgr Deadlock Date: Sat, 16 Jan 2010 06:17:19 -0600 Debug output posted at below url. https://docs.google.com/Doc?docid=0ASw5sDNUQTGfZHpnanZ2bl8zZnE2cmNtaGc&hl=en From owner-freebsd-fs@FreeBSD.ORG Sat Jan 16 12:36:59 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C8F9106566C; Sat, 16 Jan 2010 12:36:59 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 147058FC13; Sat, 16 Jan 2010 12:36:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0GCawDu055937; Sat, 16 Jan 2010 12:36:58 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0GCawHj055933; Sat, 16 Jan 2010 12:36:58 GMT (envelope-from linimon) Date: Sat, 16 Jan 2010 12:36:58 GMT Message-Id: <201001161236.o0GCawHj055933@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/142878: [zfs] [vfs] lock order reversal X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2010 12:36:59 -0000 Synopsis: [zfs] [vfs] lock order reversal Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jan 16 12:36:43 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=142878 From owner-freebsd-fs@FreeBSD.ORG Sat Jan 16 16:58:15 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 347B2106566C for ; Sat, 16 Jan 2010 16:58:15 +0000 (UTC) (envelope-from lehmann@ans-netz.de) Received: from avocado.salatschuessel.net (avocado.salatschuessel.net [78.111.72.186]) by mx1.freebsd.org (Postfix) with SMTP id AAD018FC17 for ; Sat, 16 Jan 2010 16:58:14 +0000 (UTC) Received: (qmail 51461 invoked by uid 89); 16 Jan 2010 16:31:32 -0000 Received: from unknown (HELO kartoffel.salatschuessel.net) (78.111.72.187) by avocado.salatschuessel.net with SMTP; 16 Jan 2010 16:31:32 -0000 Date: Sat, 16 Jan 2010 17:31:31 +0100 From: Oliver Lehmann To: fs@freebsd.org Message-Id: <20100116173131.07629963.lehmann@ans-netz.de> X-Mailer: Sylpheed 2.7.1 (GTK+ 2.18.5; amd64-portbld-freebsd8.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Subject: UF filesystem performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2010 16:58:15 -0000 Hi, I had to recreate my /usr filesystem so I played around with the -f and -b switches of newfs. ======> newfs -U -o time -f 8192 -b 65536 /dev/ada1s1f Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nudel.salatschue 6G 96 99 50529 26 24165 17 251 99 57469 16 174.7 10 Latency 88795us 380ms 682ms 63203us 39098us 2737ms Version 1.96 ------Sequential Create------ --------Random Create-------- nudel.salatschuesse -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 962 9 +++++ +++ 2310 15 931 8 +++++ +++ 2268 15 Latency 248ms 66us 12908us 26791us 57us 32782us Then I created a RAID-1 with gmirror - the 2nd disk is of the same type and started bonnie++ again: Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nudel.salatschue 6G 89 99 59174 31 27738 18 250 99 63858 17 308.5 17 Latency 172ms 200ms 1763ms 44152us 123ms 308ms Version 1.96 ------Sequential Create------ --------Random Create-------- nudel.salatschuesse -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 12276 90 +++++ +++ 12796 96 14750 94 +++++ +++ 10542 97 Latency 18836us 72us 258us 18891us 82us 312us Then I've read tuning(7) and I thought that maybe newfs with standard -f and -b would be better because of FreeBSD performs best when using 8K or 16K file system block sizes. The default file system block size is 16K, which provides best performance for most applications, [...] ...] Using a block size larger than 16K can cause fragmentation of the buffer cache and lead to lower performance. Then I recreated the filesystem (still mirrored): ======> newfs -U -o time /dev/mirror/gm0s1f Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nudel.salatschue 6G 96 99 42068 23 21395 18 251 99 43891 12 308.8 13 Latency 86066us 132ms 361ms 54776us 120ms 295ms Version 1.96 ------Sequential Create------ --------Random Create-------- nudel.salatschuesse -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7817 72 +++++ +++ 30164 99 6111 41 +++++ +++ 29030 96 Latency 562ms 56us 72us 284ms 78us 98us So... why is it now massivly slower? -- Oliver Lehmann http://www.pofo.de/ http://wishlist.ans-netz.de/ From owner-freebsd-fs@FreeBSD.ORG Sat Jan 16 20:20:03 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2099A1065670 for ; Sat, 16 Jan 2010 20:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E971A8FC18 for ; Sat, 16 Jan 2010 20:20:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0GKK27K054301 for ; Sat, 16 Jan 2010 20:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0GKK2Mu054300; Sat, 16 Jan 2010 20:20:02 GMT (envelope-from gnats) Date: Sat, 16 Jan 2010 20:20:02 GMT Message-Id: <201001162020.o0GKK2Mu054300@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Justin Head Cc: Subject: Re: kern/142872: [zfs] ZFS ZVOL Lockmgr Deadlock X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Justin Head List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2010 20:20:03 -0000 The following reply was made to PR kern/142872; it has been noted by GNATS. From: Justin Head To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/142872: [zfs] ZFS ZVOL Lockmgr Deadlock Date: Sat, 16 Jan 2010 14:12:25 -0600 This also occurs on 9.0-CURRENT though it takes a few runs through the loop and waiting a bit between the runs.