From owner-freebsd-fs@FreeBSD.ORG Sun Oct 12 20:40:41 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6228441F; Sun, 12 Oct 2014 20:40:41 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 021FF643; Sun, 12 Oct 2014 20:40:41 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jGFK31HN8zTC; Sun, 12 Oct 2014 22:40:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413146433; x=1415738434; bh=sjHenPGix6lixGKB+tLxm1xd3 8Hpi3a8Lwzk0a9KC4k=; b=oz+Tox6sfOfLQczdobwcpNkBOi1xenRbO8xfOoJZp lVsCcPMt6DYIpCYHNJYk4uJsh6krSOQ6ZdpXoYhT9ZRgfaHdrzi16MQUMb/qgdkZ QOPIo8ixeCriDlK3f1VkK3YfLWeJrCCFLInPD0HUr/yQtRxhY7dqmIX9mWkwHOaa 6k= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id du5XWCIkoZTI; Sun, 12 Oct 2014 22:40:33 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Sun, 12 Oct 2014 22:40:33 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jGFJx09zbz1P5; Sun, 12 Oct 2014 22:40:32 +0200 (CEST) Message-ID: <543AE740.7000808@ijs.si> Date: Sun, 12 Oct 2014 22:40:32 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-fs@freebsd.org Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> In-Reply-To: <543731F3.8090701@ijs.si> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 20:40:41 -0000 I made available an image copy (dd) of my 4 GiB partition (compressed down to a 384 MiB file), holding one of my partitions (a small bsd root) that can no longer be imported into a 10.1-RC1 or 10.1-BETA3, as described in my first posting (below): http://www.ijs.si/usr/mark/bsd/ I would appreciate if it can be confirmed that such pool (one of several I have with this symptom) causes 'zpool import' to hang on 10.1 or 10-STABLE: - download, xz -d sys1boot.img.xz # mdconfig -f sys1boot.img # zpool import sys1boot ... and advise on a solution. Considering that 'zdb -e -cc' is happy and there were no other prior trouble with these pools, except for an upgrade to 10.1-BETA3/-RC1 from 10-STABLE as of cca. late September, it is my belief that these pools are still healthy, just non-importable. I'm reluctant to upgrade any other system from 10.0 to 10.1 without finding out what went wrong here. Mark On 10/10/2014 03:02, Steven Hartland wrote: > Sorry to be a pain but could you attach the full output or link it > somewhere as mail has messed up the formatting :( Now at http://www.ijs.si/usr/mark/bsd/procstat-kka-2074.txt On 2014-10-10 Mark Martinec wrote: > In short, after upgrading to 10.1-BETA3 or -RC1 I ended up with several > zfs pools that can no longer be imported. The zpool import command > (with no arguments) does show all available pools, but trying to > import one just hangs and the command cannot be aborted, although > the rest of the system is still alive and fine: > > # zpool import -f > > Typing ^T just shows an idle process, waiting on [tx->tx_sync_done_cv]: > > load: 0.20 cmd: zpool 939 [tx->tx_sync_done_cv] 5723.65r 0.01u 0.02s 0% 8220k > load: 0.16 cmd: zpool 939 [tx->tx_sync_done_cv] 5735.73r 0.01u 0.02s 0% 8220k > load: 0.14 cmd: zpool 939 [tx->tx_sync_done_cv] 5741.83r 0.01u 0.02s 0% 8220k > load: 0.13 cmd: zpool 939 [tx->tx_sync_done_cv] 5749.16r 0.01u 0.02s 0% 8220k > > ps shows (on a system re-booted to a LiveCD running FreeBSD-10.1-RC1): > > PID TID COMM TDNAME CPU PRI STATE WCHAN > 939 100632 zpool - 5 122 sleep tx->tx_s > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 939 801 0 22 0 107732 8236 tx->tx_s D+ v0 0:00.04 > zpool import -f -o cachefile=/tmp/zpool.cache -R /tmp/sys0boot sys0boot > > NWCHAN > fffff8007b0f2a20 > > # procstat -kk 939 > > PID TID COMM TDNAME KSTACK > 939 100632 zpool - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d txg_wait_synced+0x85 spa_load+0x1cd1 spa_load_best+0x6f spa_import+0x1ff zfs_ioc_pool_import+0x137 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb > > > Background story: the system where this happened was being kept > to a fairly recent 10-STABLE. The last upgrade was very close to > a BETA3 release. There are a couple of zfs pools there, one on a > mirrored pair of SSDs mostly holding the OS, one with a mirrored > pair of large spindles, and three more small ones (4 GiB each), > mostly for boot redundancy or testing - these small ones are on > old smallish disks. These disks are different, and attached to > different SATA controllers (LSI and onboard Intel). Pools were > mostly kept up-to-date to the most recent zpool features set > through their lifetime (some starting their life with 9.0, some > with 10.0). > > About two weeks ago after a reboot to a 10-STABLE of the day > the small pools became unavailable, but the regular two large > pools were still normal. At first I wasn't giving much attention > to that, as these pools were on oldish disks and nonessential > for normal operation, blaming a potentially crappy hw. > > Today I needed to do a reboot (for unrelated reason), and the > machine was no longer able to mount the boot pool. > The first instinct was - disks are malfunctioning - but ... > > Booting it to a FreeBSD-10.1-RC1 LiveCD was successful. > smartmon disk test shows no problems. dd is able to read whole > partititions of each problematic pool. And most importantly, > running a 'zdb -e -cc' on each (non-imported) pool was churning > normally and steadily, producing a stats report at the end > and reported no errors. > > As a final proof that disks are fine I sacrificed one of the broken > 4 GiB GPT partitions with one of the problematic pools, and > did a fresh 10.1-RC1 install on it from a distribution ISO DVD. > The installation went fine and the system does boot and run > fine from the newly installed OS. Trying to import one of the > remaining old pools hangs the import command as before. > > As a final proof, I copied (with dd) one of the broken 4 GiB > partitions to a file on another system (running 10.1-BETA3, > which did not suffer from this problem), made a memory disk > out of this file, then run zfs import on this pool - and it hangs > there too! So hardware was not a problem - either these partitions > are truly broken (even though zdb -cc says they are fine), > or the new OS is somehow no longer able to import them. > > Please advise. > > I have a copy of the 4 GiB partition on a 400 MB compressed > file available, if somebody would be willing to play with it. > > Also have a ktrace of the 'zpool import' command. It's last > actions before it hangs are: > > 939 zpool RET madvise 0 > 939 zpool CALL madvise(0x80604e000,0x1000,MADV_FREE) > 939 zpool RET madvise 0 > 939 zpool CALL close(0x6) > 939 zpool RET close 0 > 939 zpool CALL ioctl(0x3,0xc0185a05,0x7fffffffbf00) > 939 zpool RET ioctl -1 errno 2 No such file or directory > 939 zpool CALL madvise(0x802c71000,0x10000,MADV_FREE) > 939 zpool RET madvise 0 > 939 zpool CALL madvise(0x802ca5000,0x1000,MADV_FREE) > 939 zpool RET madvise 0 > 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) > 939 zpool RET ioctl 0 > 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) > 939 zpool RET ioctl 0 > 939 zpool CALL stat(0x802c380e0,0x7fffffffbc58) > 939 zpool NAMI "/tmp" > 939 zpool STRU struct stat {dev=273, ino=2, mode=041777, nlink=8, uid=0, gid=0, rdev=96, atime=1412866648, stime=1412871393, ctime=1412871393, birthtime=1412866648, size=512, blksize=32768, blocks=8, flags=0x0 } > 939 zpool RET stat 0 > 939 zpool CALL ioctl(0x3,0xc0185a02,0x7fffffffbc60) > > > Mark From owner-freebsd-fs@FreeBSD.ORG Sun Oct 12 21:00:10 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C82B78D9 for ; Sun, 12 Oct 2014 21:00:10 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9EDE18A6 for ; Sun, 12 Oct 2014 21:00:10 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s9CL0AOn021194 for ; Sun, 12 Oct 2014 21:00:10 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201410122100.s9CL0AOn021194@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 12 Oct 2014 21:00:10 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 21:00:10 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ----------------+-----------+------------------------------------------------- Needs MFC | 136470 | [nfs] Cannot mount / in read-only, over NFS Needs MFC | 139651 | [nfs] mount(8): read-only remount of NFS volume Needs MFC | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non 3 problems total for which you should take action. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 00:54:21 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE61A9DB; Mon, 13 Oct 2014 00:54:21 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 2CB3368; Mon, 13 Oct 2014 00:54:20 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 8D0BC20E708AC; Mon, 13 Oct 2014 00:54:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id A360420E708AA; Mon, 13 Oct 2014 00:54:09 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 01:54:05 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 00:54:22 -0000 I have reproduced the issue with your image. It seems like the import is waiting a transation group to complete. It looks like its waiting on the zio from dsl_pool_sync_mos which for some reason is never returning. The pool does seem to be happy importing read only with: zpool import -N -f -o readonly=on sys1boot print dp->dp_tx $3 = { tx_cpu = 0xfffffe0004a05000, tx_sync_lock = { lock_object = { lo_name = 0xffffffff815f2941 "tx->tx_sync_lock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0 }, sx_lock = 1 }, tx_open_txg = 11733519, tx_quiesced_txg = 0, tx_syncing_txg = 11733518, tx_synced_txg = 0, tx_open_time = 120562502889, tx_sync_txg_waiting = 11733518, tx_quiesce_txg_waiting = 11733519, tx_sync_more_cv = { cv_description = 0xffffffff815f2953 "tx->tx_sync_more_cv", cv_waiters = 0 }, tx_sync_done_cv = { cv_description = 0xffffffff815f2968 "tx->tx_sync_done_cv", cv_waiters = 1 }, tx_quiesce_more_cv = { cv_description = 0xffffffff815f297d "tx->tx_quiesce_more_cv", cv_waiters = 1 }, tx_quiesce_done_cv = { cv_description = 0xffffffff815f2995 "tx->tx_quiesce_done_cv", cv_waiters = 0 }, tx_timeout_cv = { cv_description = 0x0, cv_waiters = 0 }, tx_exit_cv = { cv_description = 0xffffffff815f29ad "tx->tx_exit_cv", cv_waiters = 0 }, tx_threads = 2 '\002', tx_exiting = 0 '\0', tx_sync_thread = 0xfffff80041505490, tx_quiesce_thread = 0xfffff80041509920, tx_commit_cb_taskq = 0x0 } Relavent threads: #0 sched_switch (td=0xfffff800446bc000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80021f4b220, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80021f4b220, lock=0xfffff80021f4b1b8) at /usr/src/sys/kern/kern_condvar.c:139 #5 0xffffffff8151ec4b in txg_wait_synced (dp=0xfffff80021f4b000, txg=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:631 #6 0xffffffff81510937 in spa_load (spa=0xfffffe0003fe1000, state=SPA_LOAD_IMPORT, type=, mosconfig=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2777 #7 0xffffffff8150e5df in spa_load_best (spa=0xfffffe0003fe1000, state=SPA_LOAD_IMPORT, mosconfig=1, max_request=, rewind_flags=1) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2873 #8 0xffffffff8150e023 in spa_import (pool=0xfffffe0002e2a000 "sys1boot", config=0xfffff80044f24ba0, props=0x0, flags=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:4207 #9 0xffffffff81566d78 in zfs_ioc_pool_import (zc=0xfffffe0002e2a000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1594 #10 0xffffffff81563c4f in zfsdev_ioctl (dev=, zcmd=, arg=, flag=, td=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6158 #11 0xffffffff805299eb in devfs_ioctl_f (fp=0xfffff8004f020aa0, com=3222821378, data=0xfffff80038123e60, cred=, td=0xfffff800446bc000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #12 0xffffffff8066201b in kern_ioctl (td=, fd=, com=) at file.h:320 #13 0xffffffff80661d9c in sys_ioctl (td=0xfffff800446bc000, uap=0xfffffe011e49fa40) at /usr/src/sys/kern/sys_generic.c:702 #14 0xffffffff8088defa in amd64_syscall (td=0xfffff800446bc000, traced=0) at subr_syscall.c:134 #15 0xffffffff80870ecb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 Thread 509 (Thread 100533): #0 sched_switch (td=0xfffff800446bf000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064e323 in sleepq_timedwait (wchan=0xfffff80044e47210, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:652 #4 0xffffffff805b84f0 in _cv_timedwait_sbt (cvp=0xfffff80044e47210, lock=0xfffff80044e471b8, sbt=, pr=, flags=) at /usr/src/sys/kern/kern_condvar.c:325 #5 0xffffffff8151f72e in txg_thread_wait (tx=, cpr=0xfffffe011dc76a00, cv=0xfffff80044e47210, time=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:249 #6 0xffffffff8151e930 in txg_sync_thread (arg=0xfffff80044e47000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:483 #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 , arg=0xfffff80044e47000, frame=0xfffffe011dc76ac0) at /usr/src/sys/kern/kern_fork.c:996 #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #9 0x0000000000000000 in ?? () Thread 508 (Thread 100532): #0 sched_switch (td=0xfffff80048204920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044e47230, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044e47230, lock=0xfffff80044e471b8) at /usr/src/sys/kern/kern_condvar.c:139 #5 0xffffffff8151f73b in txg_thread_wait (tx=, cpr=0xfffffe011dc6da00, cv=, time=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:251 #6 0xffffffff8151e450 in txg_quiesce_thread (arg=0xfffff80044e47000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:556 #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e030 , arg=0xfffff80044e47000, frame=0xfffffe011dc6dac0) at /usr/src/sys/kern/kern_fork.c:996 #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #9 0x0000000000000000 in ?? () Thread 512 (Thread 100338): #0 sched_switch (td=0xfffff80041505490, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044acf320, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044acf320, lock=0xfffff80044acf300) at /usr/src/sys/kern/kern_condvar.c:139 #5 0xffffffff81547c9b in zio_wait (zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1442 #6 0xffffffff814f056c in dsl_pool_sync (dp=0xfffff80021f4b000, txg=11733518) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:531 #7 0xffffffff81514030 in spa_sync (spa=0xfffffe0003fe1000, txg=11733518) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6604 #8 0xffffffff8151e6cd in txg_sync_thread (arg=0xfffff80021f4b000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:518 #9 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 , arg=0xfffff80021f4b000, frame=0xfffffe011e6fdac0) at /usr/src/sys/kern/kern_fork.c:996 #10 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #11 0x0000000000000000 in ?? () ----- Original Message ----- From: "Mark Martinec" To: ; Sent: Sunday, October 12, 2014 9:40 PM Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] >I made available an image copy (dd) of my 4 GiB partition > (compressed down to a 384 MiB file), holding one of my > partitions (a small bsd root) that can no longer be imported > into a 10.1-RC1 or 10.1-BETA3, as described in my first > posting (below): > > http://www.ijs.si/usr/mark/bsd/ > > I would appreciate if it can be confirmed that such pool > (one of several I have with this symptom) causes > 'zpool import' to hang on 10.1 or 10-STABLE: > > - download, xz -d sys1boot.img.xz > # mdconfig -f sys1boot.img > # zpool import sys1boot > > ... and advise on a solution. > > Considering that 'zdb -e -cc' is happy and there were no > other prior trouble with these pools, except for an upgrade > to 10.1-BETA3/-RC1 from 10-STABLE as of cca. late September, > it is my belief that these pools are still healthy, just > non-importable. I'm reluctant to upgrade any other system > from 10.0 to 10.1 without finding out what went wrong here. > > Mark > > > On 10/10/2014 03:02, Steven Hartland wrote: > > Sorry to be a pain but could you attach the full output or link it > > somewhere as mail has messed up the formatting :( > > Now at > http://www.ijs.si/usr/mark/bsd/procstat-kka-2074.txt > > On 2014-10-10 Mark Martinec wrote: >> In short, after upgrading to 10.1-BETA3 or -RC1 I ended up with several >> zfs pools that can no longer be imported. The zpool import command >> (with no arguments) does show all available pools, but trying to >> import one just hangs and the command cannot be aborted, although >> the rest of the system is still alive and fine: >> >> # zpool import -f >> >> Typing ^T just shows an idle process, waiting on [tx->tx_sync_done_cv]: >> >> load: 0.20 cmd: zpool 939 [tx->tx_sync_done_cv] 5723.65r 0.01u 0.02s 0% 8220k >> load: 0.16 cmd: zpool 939 [tx->tx_sync_done_cv] 5735.73r 0.01u 0.02s 0% 8220k >> load: 0.14 cmd: zpool 939 [tx->tx_sync_done_cv] 5741.83r 0.01u 0.02s 0% 8220k >> load: 0.13 cmd: zpool 939 [tx->tx_sync_done_cv] 5749.16r 0.01u 0.02s 0% 8220k >> >> ps shows (on a system re-booted to a LiveCD running FreeBSD-10.1-RC1): >> >> PID TID COMM TDNAME CPU PRI STATE WCHAN >> 939 100632 zpool - 5 122 sleep tx->tx_s >> UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND >> 0 939 801 0 22 0 107732 8236 tx->tx_s D+ v0 0:00.04 >> zpool import -f -o cachefile=/tmp/zpool.cache -R /tmp/sys0boot sys0boot >> >> NWCHAN >> fffff8007b0f2a20 >> >> # procstat -kk 939 >> >> PID TID COMM TDNAME KSTACK >> 939 100632 zpool - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d txg_wait_synced+0x85 >> spa_load+0x1cd1 spa_load_best+0x6f spa_import+0x1ff zfs_ioc_pool_import+0x137 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 >> kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb >> >> >> Background story: the system where this happened was being kept >> to a fairly recent 10-STABLE. The last upgrade was very close to >> a BETA3 release. There are a couple of zfs pools there, one on a >> mirrored pair of SSDs mostly holding the OS, one with a mirrored >> pair of large spindles, and three more small ones (4 GiB each), >> mostly for boot redundancy or testing - these small ones are on >> old smallish disks. These disks are different, and attached to >> different SATA controllers (LSI and onboard Intel). Pools were >> mostly kept up-to-date to the most recent zpool features set >> through their lifetime (some starting their life with 9.0, some >> with 10.0). >> >> About two weeks ago after a reboot to a 10-STABLE of the day >> the small pools became unavailable, but the regular two large >> pools were still normal. At first I wasn't giving much attention >> to that, as these pools were on oldish disks and nonessential >> for normal operation, blaming a potentially crappy hw. >> >> Today I needed to do a reboot (for unrelated reason), and the >> machine was no longer able to mount the boot pool. >> The first instinct was - disks are malfunctioning - but ... >> >> Booting it to a FreeBSD-10.1-RC1 LiveCD was successful. >> smartmon disk test shows no problems. dd is able to read whole >> partititions of each problematic pool. And most importantly, >> running a 'zdb -e -cc' on each (non-imported) pool was churning >> normally and steadily, producing a stats report at the end >> and reported no errors. >> >> As a final proof that disks are fine I sacrificed one of the broken >> 4 GiB GPT partitions with one of the problematic pools, and >> did a fresh 10.1-RC1 install on it from a distribution ISO DVD. >> The installation went fine and the system does boot and run >> fine from the newly installed OS. Trying to import one of the >> remaining old pools hangs the import command as before. >> >> As a final proof, I copied (with dd) one of the broken 4 GiB >> partitions to a file on another system (running 10.1-BETA3, >> which did not suffer from this problem), made a memory disk >> out of this file, then run zfs import on this pool - and it hangs >> there too! So hardware was not a problem - either these partitions >> are truly broken (even though zdb -cc says they are fine), >> or the new OS is somehow no longer able to import them. >> >> Please advise. >> >> I have a copy of the 4 GiB partition on a 400 MB compressed >> file available, if somebody would be willing to play with it. >> >> Also have a ktrace of the 'zpool import' command. It's last >> actions before it hangs are: >> >> 939 zpool RET madvise 0 >> 939 zpool CALL madvise(0x80604e000,0x1000,MADV_FREE) >> 939 zpool RET madvise 0 >> 939 zpool CALL close(0x6) >> 939 zpool RET close 0 >> 939 zpool CALL ioctl(0x3,0xc0185a05,0x7fffffffbf00) >> 939 zpool RET ioctl -1 errno 2 No such file or directory >> 939 zpool CALL madvise(0x802c71000,0x10000,MADV_FREE) >> 939 zpool RET madvise 0 >> 939 zpool CALL madvise(0x802ca5000,0x1000,MADV_FREE) >> 939 zpool RET madvise 0 >> 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) >> 939 zpool RET ioctl 0 >> 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) >> 939 zpool RET ioctl 0 >> 939 zpool CALL stat(0x802c380e0,0x7fffffffbc58) >> 939 zpool NAMI "/tmp" >> 939 zpool STRU struct stat {dev=273, ino=2, mode=041777, nlink=8, uid=0, gid=0, rdev=96, atime=1412866648, >> stime=1412871393, ctime=1412871393, birthtime=1412866648, size=512, blksize=32768, blocks=8, flags=0x0 } >> 939 zpool RET stat 0 >> 939 zpool CALL ioctl(0x3,0xc0185a02,0x7fffffffbc60) >> >> >> Mark > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 03:54:33 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 847859EE; Mon, 13 Oct 2014 03:54:33 +0000 (UTC) Received: from mail-yh0-x232.google.com (mail-yh0-x232.google.com [IPv6:2607:f8b0:4002:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3351E3B5; Mon, 13 Oct 2014 03:54:33 +0000 (UTC) Received: by mail-yh0-f50.google.com with SMTP id a41so3265925yho.23 for ; Sun, 12 Oct 2014 20:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Y8Pu4pQN5ssvKh1dxtAFY4vwN4CziU13Dc6OZwmPSGQ=; b=Fh4FNi2ovq6/C87wRWOwZDVnYLzFhhL83XYs5HBdMbzOcLV3YYTN0suhbIhtA2goOq Mrrq+RrZgfnWtLsGzu08fGYllJvFR7W7to+9DqVmXyGyYSzT0Ms3L8Mux5CIJkzZYUbE duc/rYDlqaGZ1TqRFoHZh6j8EILsyyF2uNjWeKjq9QoN5kvXZK1EKH3cjzpStcFozZyE fg1NbsQGx0OFdY5wpd29gaRCW+0p9gDi/a8co82um/WD0e2aiV2M69215YlxCBlJJb9t q8Nl0q0oPWnvFp5pKiwIh49lUQGGRtulh7p0+6gxiqjC/09Otp4l/8DwtaWlNEqHaVYU 1EGg== MIME-Version: 1.0 X-Received: by 10.236.32.4 with SMTP id n4mr33142798yha.16.1413172472105; Sun, 12 Oct 2014 20:54:32 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Sun, 12 Oct 2014 20:54:32 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> Date: Sun, 12 Oct 2014 20:54:32 -0700 X-Google-Sender-Auth: Ed-LGs5HF9Eo1bT7G0bE8hehL2Q Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 03:54:33 -0000 Relavent threads: > > #0 sched_switch (td=0xfffff800446bc000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80021f4b220, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:617 > #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80021f4b220, > lock=0xfffff80021f4b1b8) at /usr/src/sys/kern/kern_condvar.c:139 > #5 0xffffffff8151ec4b in txg_wait_synced (dp=0xfffff80021f4b000, > txg=) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:631 > #6 0xffffffff81510937 in spa_load (spa=0xfffffe0003fe1000, > state=SPA_LOAD_IMPORT, type=, mosconfig= optimized out>) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2777 > #7 0xffffffff8150e5df in spa_load_best (spa=0xfffffe0003fe1000, > state=SPA_LOAD_IMPORT, mosconfig=1, max_request=, > rewind_flags=1) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2873 > #8 0xffffffff8150e023 in spa_import (pool=0xfffffe0002e2a000 "sys1boot", > config=0xfffff80044f24ba0, props=0x0, flags=) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/spa.c:4207 > #9 0xffffffff81566d78 in zfs_ioc_pool_import (zc=0xfffffe0002e2a000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/zfs_ioctl.c:1594 > #10 0xffffffff81563c4f in zfsdev_ioctl (dev=, > zcmd=, arg=, flag= optimized out>, td=) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6158 > #11 0xffffffff805299eb in devfs_ioctl_f (fp=0xfffff8004f020aa0, > com=3222821378, data=0xfffff80038123e60, cred=, > td=0xfffff800446bc000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 > #12 0xffffffff8066201b in kern_ioctl (td=, fd= optimized out>, com=) at file.h:320 > #13 0xffffffff80661d9c in sys_ioctl (td=0xfffff800446bc000, > uap=0xfffffe011e49fa40) at /usr/src/sys/kern/sys_generic.c:702 > #14 0xffffffff8088defa in amd64_syscall (td=0xfffff800446bc000, traced=0) > at subr_syscall.c:134 > #15 0xffffffff80870ecb in Xfast_syscall () at /usr/src/sys/amd64/amd64/ > exception.S:391 > > > Thread 509 (Thread 100533): > #0 sched_switch (td=0xfffff800446bf000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064e323 in sleepq_timedwait (wchan=0xfffff80044e47210, > pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:652 > #4 0xffffffff805b84f0 in _cv_timedwait_sbt (cvp=0xfffff80044e47210, > lock=0xfffff80044e471b8, sbt=, pr= out>, flags=) at /usr/src/sys/kern/kern_condvar.c:325 > #5 0xffffffff8151f72e in txg_thread_wait (tx=, > cpr=0xfffffe011dc76a00, cv=0xfffff80044e47210, time=) > at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:249 > #6 0xffffffff8151e930 in txg_sync_thread (arg=0xfffff80044e47000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:483 > #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 > , arg=0xfffff80044e47000, frame=0xfffffe011dc76ac0) at > /usr/src/sys/kern/kern_fork.c:996 > #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/ > exception.S:606 > #9 0x0000000000000000 in ?? () > > Thread 508 (Thread 100532): > #0 sched_switch (td=0xfffff80048204920, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044e47230, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:617 > #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044e47230, > lock=0xfffff80044e471b8) at /usr/src/sys/kern/kern_condvar.c:139 > #5 0xffffffff8151f73b in txg_thread_wait (tx=, > cpr=0xfffffe011dc6da00, cv=, time= out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:251 > #6 0xffffffff8151e450 in txg_quiesce_thread (arg=0xfffff80044e47000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:556 > #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e030 > , arg=0xfffff80044e47000, frame=0xfffffe011dc6dac0) at > /usr/src/sys/kern/kern_fork.c:996 > #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/ > exception.S:606 > #9 0x0000000000000000 in ?? () > > > Thread 512 (Thread 100338): > #0 sched_switch (td=0xfffff80041505490, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044acf320, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:617 > #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044acf320, > lock=0xfffff80044acf300) at /usr/src/sys/kern/kern_condvar.c:139 > #5 0xffffffff81547c9b in zio_wait (zio=) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/zio.c:1442 > #6 0xffffffff814f056c in dsl_pool_sync (dp=0xfffff80021f4b000, > txg=11733518) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:531 > #7 0xffffffff81514030 in spa_sync (spa=0xfffffe0003fe1000, txg=11733518) > at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/spa.c:6604 > #8 0xffffffff8151e6cd in txg_sync_thread (arg=0xfffff80021f4b000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:518 > #9 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 > , arg=0xfffff80021f4b000, frame=0xfffffe011e6fdac0) at > /usr/src/sys/kern/kern_fork.c:996 > #10 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/ > exception.S:606 > #11 0x0000000000000000 in ?? () > A recent quick read of the code would lead me to believe that zio_wait not returning there means that the zio never reached the zio_done stage. Parent zios seem to yield in a couple of stages in the pipeline if they have incomplete children. They determine this by calling zio_wait_for_children with zio child types and their corresponding wait type. In so doing they set the io_stall to the count of the number of waiters of the first non-zero check. This parent I/O will be resumed by the last child zio of that type and wait state in zio_notify_parent. I'm sure you know all this - but I wrote it to preface asking for the following fields of the zio being waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children is pointed to) *io_stall, io_children[*][*], io_child_list (at a first glance just the addresses). The other alternative is that it reexecuting has gotten in to a bad place in the state machine so io_reexecute. Thanks. -K From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 07:15:45 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3CEC11D4 for ; Mon, 13 Oct 2014 07:15:45 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED829A09 for ; Mon, 13 Oct 2014 07:15:43 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id 9CF42FAD3; Mon, 13 Oct 2014 07:15:35 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id Ws-I_KT4D7EU; Mon, 13 Oct 2014 07:15:33 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id 54A66FAC2; Mon, 13 Oct 2014 07:15:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413184533; bh=V+gn5V6wmP5Tni2rnV50jcwPAayKzQ+91hPdrCnyWSg=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=oiKAqGezCvZsGdKm1trzzneB0W/RvKln7cM+Qf00+KJqaG9qvuLbRcvLNBE9w37FR 9hiI8MyOeo7rQ5Vsh3q5yADiuGo13jcZxNtRcQedQC9BxD/TZhv8l01A0MO/+nbiFm McvpHzKEhos0OK77+S2WuZl+UOgdF2k2ucq41swE= Mime-Version: 1.0 Date: Mon, 13 Oct 2014 07:15:32 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable Message-ID: <8ca92a8e507970c5bc3e34c31c30561e@mail.unix-experience.fr> X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: NFSv4 nobody issue To: "Rick Macklem" In-Reply-To: <1738545148.62071361.1412941900737.JavaMail.root@uoguelph.ca> References: <1738545148.62071361.1412941900737.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 07:15:45 -0000 Hi,=0Aof course i have it. On each node:=0A=0A# cat /etc/master.passwd | = grep nobody=0Areturns:=0Anobody:*:65534:65534::0:0:Unprivileged user:/non= existent:/usr/sbin/nologin=0A=0AIt's why i do a report here :)=0A=0ARegar= ds,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security Engineer=0A= http://www.unix-experience.fr=0A=0A10 octobre 2014 13:51 "Rick Macklem" <= rmacklem@uoguelph.ca> a =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Hello= @freebsd-fs,=0A>> i'm trying to do jail hosting over NFSv4 with ezjail a= nd i'm=0A>> experimenting an issue that i can't resolve. When i extract= =0A>> base.txz (with ezjail) or i set nobody user on a file, i have this= =0A>> error:=0A>> =0A>> chown nobody:nobody /usr/jails/fulljail/mnt/=0A>>= No name and/or group mapping for uid,gid:(65534,65534)=0A>> chown: /usr/= jails/fulljail/mnt/: Operation not permitted=0A>> =0A>> No problem if i s= et:=0A>> chown mysql:nobody /usr/jails/fulljail/mnt/=0A>> =0A>> Problem a= ppears on all files.=0A> =0A> Do you have a user by the name of "nobody" = in your password database?=0A> (NFSv4 uses names and not numbers on the w= ire, so no name-->no mapping=0A> and chown can't be done.)=0A> =0A> rick= =0A> =0A>> On my ZFS+NFSv4 server i do a dataset, exported in NFS=0A>> = =0A>> /etc/exports:=0A>> V4: /=0A>> =0A>> zfs get sharenfs pool/jails:=0A= >> -network=3D10.99.99.0 -mask=3D255.255.255.0 -maproot=3Droot=0A>> =0A>>= nfsuserd and nfsv4_server_enable=3DYES on both client and server, plus= =0A>> nfsbcd on client.=0A>> =0A>> On the client here is the fstab entry= =0A>> 10.99.99.99:/pool/jails /usr/jails nfs rw,nfsv4 0 0=0A>> =0A>> What= i'm doing wrong ?=0A>> =0A>> Thanks in advance=0A>> Regards,=0A>> =0A>> = Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> htt= p://www.unix-experience.fr =0A>> _______________________________=0A>> =0A= >> freebsd-fs@freebsd.org mailing list=0A>> http://lists.freebsd.org/mail= man/listinfo/freebsd-fs=0A>> To unsubscribe, send any mail to "freebsd-fs= -unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 07:39:20 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A8CDB724 for ; Mon, 13 Oct 2014 07:39:20 +0000 (UTC) Received: from mail-vc0-x22a.google.com (mail-vc0-x22a.google.com [IPv6:2607:f8b0:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A962BFA for ; Mon, 13 Oct 2014 07:39:20 +0000 (UTC) Received: by mail-vc0-f170.google.com with SMTP id hy10so5401541vcb.29 for ; Mon, 13 Oct 2014 00:39:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=OdRfZlSwQdEM4uyEQgxmh65TS8sYd+PnZ+BmIHihYdc=; b=CGEr5Wr1UuylXpcB6oMAqjSFJP2QWLmv4w3PxNUB4kSTH0gV7Xey+0mjjiBo+lazKY Meeb8BImAKaA+9mmB/TVZiyqh15JRNgQAQeC5LTJgceEJ+nl+vvVr0sDAg821l7EnMyO u4VjHomi7iOtD/RvlGzfAZce4c/g5ieHDzdZ9+zfnmFkfb54FVRQ1zO2rQltb1d+G1iv i/S7WYJS0Np3WalC2ufra/3/5fM3bk0VfMUQZFr8x/Tpi3K01jvFdoaqWwu5bI1mAvU9 CUQiB0Hc47D3AgO23VXPLc2tAA7hlyxo1FimYB4Qk/oMoU1uwTrJiaY5VXiJstWzt+pg 8upA== MIME-Version: 1.0 X-Received: by 10.220.168.74 with SMTP id t10mr4265294vcy.35.1413185959383; Mon, 13 Oct 2014 00:39:19 -0700 (PDT) Received: by 10.220.238.14 with HTTP; Mon, 13 Oct 2014 00:39:19 -0700 (PDT) Date: Mon, 13 Oct 2014 03:39:19 -0400 Message-ID: Subject: ZFS Checksum errors without any Checksum errors? From: Zaphod Beeblebrox To: freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 07:39:20 -0000 This bit confuses me. The errors are on 'vr2' and 'raidz1-0' ... but not on any of the disks. I can let this complete and there's some 400 data errors spread all around, but the disks will all read zero errors. What do errors on the array the the vdev mean, then? How did I get here? vr2-d2c (in my notation, dxy is the x'th drive in the array, version y --- so d2c is the third time I've replaced d2) replaced d2b... which is now refusing to spin up. Stay away from 1.5T drives. Anyways... If I let the resilver finish, it will finish without finding errors on any particular drive --- but it will still report hundreds of errors on vr2 (the array) and raidz1-0 (the 1st vdev). NAME STATE READ WRITE CKSUM vr2 ONLINE 0 0 7 raidz1-0 ONLINE 0 0 14 label/vr2-d0 ONLINE 0 0 0 label/vr2-d1 ONLINE 0 0 0 gpt/vr2-d2c ONLINE 0 0 0 (resilvering) gpt/vr2-d3b ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-d4a ONLINE 0 0 0 block size: 512B configured, 4096B native ada14 ONLINE 0 0 0 label/vr2-d6 ONLINE 0 0 0 label/vr2-d7c ONLINE 0 0 0 label/vr2-d8 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 gpt/vr2-e0 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-e1 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-e2 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-e3 ONLINE 0 0 0 gpt/vr2-e4 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-e5 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-e6 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/vr2-e7 ONLINE 0 0 0 block size: 512B configured, 4096B native logs gpt/vr2log ONLINE 0 0 0 cache gpt/vr2cache ONLINE 0 0 0 errors: 427 data errors, use '-v' for a list From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 08:00:09 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8759AA5F for ; Mon, 13 Oct 2014 08:00:09 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 74FAFE12 for ; Mon, 13 Oct 2014 08:00:09 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s9D8097M037336 for ; Mon, 13 Oct 2014 08:00:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Message-Id: <201410130800.s9D8097M037336@kenobi.freebsd.org> From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [FreeBSD Bugzilla] Commit Needs MFC MIME-Version: 1.0 X-Bugzilla-Type: whine X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated Date: Mon, 13 Oct 2014 08:00:09 +0000 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:00:09 -0000 Hi, You have a bug in the "Needs MFC" state which has not been touched in 7 or more days. This email serves as a reminder that you may want to MFC this bug or marked it as completed. In the event you have a longer MFC timeout you may update this bug with a comment and I won't remind you again for 7 days. This reminder is only sent on Mondays. Please file a bug about concerns you may have. This search was scheduled by eadler@FreeBSD.org. (3 bugs) Bug 136470: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=136470 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [nfs] Cannot mount / in read-only, over NFS Bug 139651: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=139651 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [nfs] mount(8): read-only remount of NFS volume does not work Bug 144447: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=144447 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [zfs] sharenfs fsunshare() & fsshare_main() non functional From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 08:06:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 58439D6F; Mon, 13 Oct 2014 08:06:12 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 17B23F08; Mon, 13 Oct 2014 08:06:11 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 86B5220E708CA; Mon, 13 Oct 2014 08:06:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 3A10F20E708C8; Mon, 13 Oct 2014 08:06:08 +0000 (UTC) Message-ID: <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si><644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk><54372EBA.1000908@ijs.si><543731F3.8090701@ijs.si><543AE740.7000808@ijs.si> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 09:06:04 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:06:12 -0000 ----- Original Message ----- From: "K. Macy" > A recent quick read of the code would lead me to believe that zio_wait not > returning there means that the zio never reached the zio_done stage. Parent > zios seem to yield in a couple of stages in the pipeline if they have > incomplete children. They determine this by calling zio_wait_for_children > with zio child types and their corresponding wait type. In so doing they > set the io_stall to the count of the number of waiters of the first > non-zero check. This parent I/O will be resumed by the last child zio of > that type and wait state in zio_notify_parent. I'm sure you know all this - > but I wrote it to preface asking for the following fields of the zio being > waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children > is pointed to) *io_stall, io_children[*][*], io_child_list (at a first > glance just the addresses). The other alternative is that it reexecuting >has gotten in to a bad place in the state machine so io_reexecute. Yer I would have got the zio details but typically its "optimised out" by the compiler, so will need some effort to track that down unfortunately :( Regards Steve From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 08:10:08 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40B78EFF for ; Mon, 13 Oct 2014 08:10:08 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA038F40 for ; Mon, 13 Oct 2014 08:10:06 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id AEEE1FB8D; Mon, 13 Oct 2014 08:10:04 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id hX_tzAewal1t; Mon, 13 Oct 2014 08:10:02 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id 8D025FB80; Mon, 13 Oct 2014 08:10:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413187802; bh=2w54PwUM+BBYaQicE+T55t2H+DA6I5YedGZBBN2WM/I=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=Zi8DDvKJBjk/yJB5feaedEZ2Z+KiuD/iPz19YP6LUBiNRD+Xd1HM7V0W6txF2hDd4 elWhMgnILXxAvpOE0FeDr9JTyOusFsxGxi64fd8K4CPTmoM6zz8dTJDkaDMlFqIEhJ qEnUoAVkZxNDwzKcLvwE96KMy85M7ickfLDkLG4g= Mime-Version: 1.0 Date: Mon, 13 Oct 2014 08:10:02 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable Message-ID: <1ffeae65b7b297266ee2d59dc0289d07@mail.unix-experience.fr> X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: NFSv4 nobody issue To: "Rick Macklem" In-Reply-To: <8ca92a8e507970c5bc3e34c31c30561e@mail.unix-experience.fr> References: <8ca92a8e507970c5bc3e34c31c30561e@mail.unix-experience.fr> <1738545148.62071361.1412941900737.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:10:08 -0000 Hi,=0Ai tryed some other things=0A=0AUser nobody (65534)=0A-> chown nobod= y /usr/jail/test.file =3D> problem=0A=0AGroup nogroup (65533)=0A-> chown = :nogroup /usr/jail/test.file =3D> same problem=0A=0AGroup nobody (65534)= =0A-> chown :nobody /usr/jail/test.file =3D> no problem=0A=0AChange user = nobody UID from 65534 to 65533 =3D> same problem. It's not a UID number p= roblem but a name problem.=0A=0AThen, user nobody and group nogroup (not = the integer values) are problematic. I looked at nfsuserd.c and i see:=0A= u_char *defaultuser =3D "nobody";=0Au_char *defaultgroup =3D "nogroup";= =0A=0AI think it's related.=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Sys= tems, Network and Security Engineer=0Ahttp://www.unix-experience.fr=0A=0A= 13 octobre 2014 09:15 "Lo=C3=AFc Blot" a = =C3=A9crit: =0A> Hi,=0A> of course i have it. On each node:=0A> =0A> # ca= t /etc/master.passwd | grep nobody=0A> returns:=0A> nobody:*:65534:65534:= :0:0:Unprivileged user:/nonexistent:/usr/sbin/nologin=0A> =0A> It's why i= do a report here :)=0A> =0A> Regards,=0A> =0A> Lo=C3=AFc Blot,=0A> UNIX = Systems, Network and Security Engineer=0A> http://www.unix-experience.fr= =0A> =0A> 10 octobre 2014 13:51 "Rick Macklem" a = =C3=A9crit:=0A> =0A>> Loic Blot wrote:=0A>> =0A>>> Hello @freebsd-fs,=0A>= >> i'm trying to do jail hosting over NFSv4 with ezjail and i'm=0A>>> exp= erimenting an issue that i can't resolve. When i extract=0A>>> base.txz (= with ezjail) or i set nobody user on a file, i have this=0A>>> error:=0A>= >> =0A>>> chown nobody:nobody /usr/jails/fulljail/mnt/=0A>>> No name and/= or group mapping for uid,gid:(65534,65534)=0A>>> chown: /usr/jails/fullja= il/mnt/: Operation not permitted=0A>>> =0A>>> No problem if i set:=0A>>> = chown mysql:nobody /usr/jails/fulljail/mnt/=0A>>> =0A>>> Problem appears = on all files.=0A>> =0A>> Do you have a user by the name of "nobody" in yo= ur password database?=0A>> (NFSv4 uses names and not numbers on the wire,= so no name-->no mapping=0A>> and chown can't be done.)=0A>> =0A>> rick= =0A>> =0A>>> On my ZFS+NFSv4 server i do a dataset, exported in NFS=0A>>>= =0A>>> /etc/exports:=0A>>> V4: /=0A>>> =0A>>> zfs get sharenfs pool/jail= s:=0A>>> -network=3D10.99.99.0 -mask=3D255.255.255.0 -maproot=3Droot=0A>>= > =0A>>> nfsuserd and nfsv4_server_enable=3DYES on both client and server= , plus=0A>>> nfsbcd on client.=0A>>> =0A>>> On the client here is the fst= ab entry=0A>>> 10.99.99.99:/pool/jails /usr/jails nfs rw,nfsv4 0 0=0A>>> = =0A>>> What i'm doing wrong ?=0A>>> =0A>>> Thanks in advance=0A>>> Regard= s,=0A>>> =0A>>> Lo=C3=AFc Blot,=0A>>> UNIX Systems, Network and Security = Engineer=0A>>> http://www.unix-experience.fr =0A>>> _____________________= __________=0A>>> =0A>>> freebsd-fs@freebsd.org mailing list=0A>>> http://= lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>> To unsubscribe, send = any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A> =0A> _______________= ________________=0A> =0A> freebsd-fs@freebsd.org mailing list=0A> http://= lists.freebsd.org/mailman/listinfo/freebsd-fs=0A> To unsubscribe, send an= y mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 08:11:49 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BD31F84; Mon, 13 Oct 2014 08:11:49 +0000 (UTC) Received: from mail-yh0-x234.google.com (mail-yh0-x234.google.com [IPv6:2607:f8b0:4002:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF485FDB; Mon, 13 Oct 2014 08:11:48 +0000 (UTC) Received: by mail-yh0-f52.google.com with SMTP id f10so3361152yha.39 for ; Mon, 13 Oct 2014 01:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=tA7AS5zwDI5HVrZ6Xn0uKpprexnAHw7Smzpb3vx6r8w=; b=UuV/zrbTF0VDXj8b8cNhvrUhQzZbyp1wxDYGZbiwOLNIpfR9HI0phq3iq95U7PkR1o c0J9v7P1BBJppCC7z8VwEkUoY34J3G7sfnkVlTyLtgC/YP5h+TfxUtNlFpGTn29IE0wk +TzZtazYauC7cQ3yKs2rssGhmy1P+pqTsEKapNtUBFNZOyP2Jak2NMXqKQIzaPYTPbDF ffvung5Vlb96C5JRdBYiCMD3HWMkjYmKUY6VCL4ukgZp/erICBI2ANrgy6huHn0Xeiq0 SCp6buFM/n++wmV3OmHBLvaq5c/pXRnlMoVlMSmGs4QHkB1Ivd5DeZDyMd2zyYtAdxzT 216A== MIME-Version: 1.0 X-Received: by 10.236.54.101 with SMTP id h65mr36033279yhc.47.1413187907802; Mon, 13 Oct 2014 01:11:47 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 01:11:47 -0700 (PDT) In-Reply-To: <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 01:11:47 -0700 X-Google-Sender-Auth: FzUffdsgjnUxbaw-euX2JdaqQKQ Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:11:49 -0000 On Mon, Oct 13, 2014 at 1:06 AM, Steven Hartland wrote: > ----- Original Message ----- From: "K. Macy" > > A recent quick read of the code would lead me to believe that zio_wait not >> returning there means that the zio never reached the zio_done stage. >> Parent >> zios seem to yield in a couple of stages in the pipeline if they have >> incomplete children. They determine this by calling zio_wait_for_children >> with zio child types and their corresponding wait type. In so doing they >> set the io_stall to the count of the number of waiters of the first >> non-zero check. This parent I/O will be resumed by the last child zio of >> that type and wait state in zio_notify_parent. I'm sure you know all this >> - >> but I wrote it to preface asking for the following fields of the zio being >> waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children >> is pointed to) *io_stall, io_children[*][*], io_child_list (at a first >> glance just the addresses). The other alternative is that it reexecuting >> has gotten in to a bad place in the state machine so io_reexecute. >> > > Yer I would have got the zio details but typically its "optimised out" by > the > compiler, so will need some effort to track that down unfortunately :( > Well, let me know if you can. Re-creating a new 10.x VM is taking a while as it's taking me forever to checkout the sources. Things like that need to somehow continue to be accessible. Cheers. -K From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 12:43:10 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C561A7DC for ; Mon, 13 Oct 2014 12:43:10 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 8BD40DB2 for ; Mon, 13 Oct 2014 12:43:09 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqsEACnIO1SDaFve/2dsb2JhbABbg2FYBIMCyBYKhnlUAoE1AX2EAgEBAQMBAQEBIAQnIAsFFhgCAg0HEgIpAQkmBggHBAEcBIgVCA2vJpUCAQEBAQEFAQEBAQEBARuBLI5IAQEbATMHGIJfgVQFljuEDIQyPJApg36EEyEvB4EIOYECAQEB X-IronPort-AV: E=Sophos;i="5.04,710,1406606400"; d="scan'208";a="160876252" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 13 Oct 2014 08:43:02 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 62C22E7956; Mon, 13 Oct 2014 08:43:02 -0400 (EDT) Date: Mon, 13 Oct 2014 08:43:02 -0400 (EDT) From: Rick Macklem To: =?utf-8?B?TG/Dr2M=?= Blot Message-ID: <1626547992.63435100.1413204182279.JavaMail.root@uoguelph.ca> In-Reply-To: <1ffeae65b7b297266ee2d59dc0289d07@mail.unix-experience.fr> Subject: Re: NFSv4 nobody issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 12:43:10 -0000 Loic Blot wrote: > Hi, > i tryed some other things >=20 > User nobody (65534) > -> chown nobody /usr/jail/test.file =3D> problem >=20 > Group nogroup (65533) > -> chown :nogroup /usr/jail/test.file =3D> same problem >=20 > Group nobody (65534) > -> chown :nobody /usr/jail/test.file =3D> no problem >=20 > Change user nobody UID from 65534 to 65533 =3D> same problem. It's not > a UID number problem but a name problem. >=20 Yes, for NFSv4 it is the names that go in the RPC request and not the numbers. However, since there are the numbers in the AUTH_SYS credential in the header (unless you are using Kerberized mounts), the numbers for the names need to be consistent between client and server. > Then, user nobody and group nogroup (not the integer values) are > problematic. I looked at nfsuserd.c and i see: > u_char *defaultuser =3D "nobody"; > u_char *defaultgroup =3D "nogroup"; >=20 These are used if no mapping is found in the user or group database for whatever name is in the RPC on the wire. If you want to see what is happening, I suggest that you capture packets when you do the "chown" (You can use "tcpdump -s 0 -w file.pcap hos= t XXX".) then look at them in wireshark. In wireshark, look for the Setattr RPC and then look in the setable attribu= tes. You should find Owner which looks like "nobody@ and Owner_group which looks the same (or "nogroup@" if you used nogroup). "nogroup" must be in your group database (/etc/group or what= ever you use for a group database) and the number must be consistent across clie= nt and server. Also, see what the reply to the Setattr RPC is (it is actually a Compound R= PC labelled "Setattr" for NFSv4). If there is no Setattr RPC, then the mapping is failing in the client. If the stuff looks correct on the wire, then it is most likely a server sid= e issue. rick > I think it's related. >=20 > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 13 octobre 2014 09:15 "Lo=C3=AFc Blot" a > =C3=A9crit: > > Hi, > > of course i have it. On each node: > >=20 > > # cat /etc/master.passwd | grep nobody > > returns: > > nobody:*:65534:65534::0:0:Unprivileged > > user:/nonexistent:/usr/sbin/nologin > >=20 > > It's why i do a report here :) > >=20 > > Regards, > >=20 > > Lo=C3=AFc Blot, > > UNIX Systems, Network and Security Engineer > > http://www.unix-experience.fr > >=20 > > 10 octobre 2014 13:51 "Rick Macklem" a > > =C3=A9crit: > >=20 > >> Loic Blot wrote: > >>=20 > >>> Hello @freebsd-fs, > >>> i'm trying to do jail hosting over NFSv4 with ezjail and i'm > >>> experimenting an issue that i can't resolve. When i extract > >>> base.txz (with ezjail) or i set nobody user on a file, i have > >>> this > >>> error: > >>>=20 > >>> chown nobody:nobody /usr/jails/fulljail/mnt/ > >>> No name and/or group mapping for uid,gid:(65534,65534) > >>> chown: /usr/jails/fulljail/mnt/: Operation not permitted > >>>=20 > >>> No problem if i set: > >>> chown mysql:nobody /usr/jails/fulljail/mnt/ > >>>=20 > >>> Problem appears on all files. > >>=20 > >> Do you have a user by the name of "nobody" in your password > >> database? > >> (NFSv4 uses names and not numbers on the wire, so no name-->no > >> mapping > >> and chown can't be done.) > >>=20 > >> rick > >>=20 > >>> On my ZFS+NFSv4 server i do a dataset, exported in NFS > >>>=20 > >>> /etc/exports: > >>> V4: / > >>>=20 > >>> zfs get sharenfs pool/jails: > >>> -network=3D10.99.99.0 -mask=3D255.255.255.0 -maproot=3Droot > >>>=20 > >>> nfsuserd and nfsv4_server_enable=3DYES on both client and server, > >>> plus > >>> nfsbcd on client. > >>>=20 > >>> On the client here is the fstab entry > >>> 10.99.99.99:/pool/jails /usr/jails nfs rw,nfsv4 0 0 > >>>=20 > >>> What i'm doing wrong ? > >>>=20 > >>> Thanks in advance > >>> Regards, > >>>=20 > >>> Lo=C3=AFc Blot, > >>> UNIX Systems, Network and Security Engineer > >>> http://www.unix-experience.fr > >>> _______________________________ > >>>=20 > >>> freebsd-fs@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>> To unsubscribe, send any mail to > >>> "freebsd-fs-unsubscribe@freebsd.org" > >=20 > > _______________________________ > >=20 > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to > > "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 13:14:48 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 79D191C3 for ; Mon, 13 Oct 2014 13:14:48 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 36094152 for ; Mon, 13 Oct 2014 13:14:47 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id 68642FF17; Mon, 13 Oct 2014 13:14:44 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id lx6eXhobNtTX; Mon, 13 Oct 2014 13:14:42 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id 43208FF0C; Mon, 13 Oct 2014 13:14:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413206082; bh=IIVRktR5j7JyKU85d7ygQApooa/RanOfC0uwyRheAeU=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=cGgmVbXE8oiSyrMPlONgq24QdbbXA1KnxaZpxGTBmGvMJFwquKVtbtNfBtXh0jHc4 uL5b15DUSe2wVBy5cn8a/hC8HnycIHukBRvfdwKDVDqrgXjZneqcYsl+/YHIpV98AE UaBEKRHIArUc5PFdirZhMHLpczpsRxzOPN8f0Ym0= Mime-Version: 1.0 Date: Mon, 13 Oct 2014 13:14:41 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable Message-ID: X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: NFSv4 nobody issue To: "Rick Macklem" In-Reply-To: <1626547992.63435100.1413204182279.JavaMail.root@uoguelph.ca> References: <1626547992.63435100.1413204182279.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 13:14:48 -0000 Hi Rick,=0Ano request is done.=0AIn /var/log/messages on the client i hav= e:=0A=0AOct 13 15:10:46 machine kernel: No name and/or group mapping for = uid,gid:(65534,-1)=0A=0AThe FreeBSD kernel refuses to change the owner.= =0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, Network and Security = Engineer=0Ahttp://www.unix-experience.fr=0A=0A13 octobre 2014 14:43 "Rick= Macklem" a =C3=A9crit: =0A> Loic Blot wrote:=0A> = =0A>> Hi,=0A>> i tryed some other things=0A>> =0A>> User nobody (65534)= =0A>> -> chown nobody /usr/jail/test.file =3D> problem=0A>> =0A>> Group n= ogroup (65533)=0A>> -> chown :nogroup /usr/jail/test.file =3D> same probl= em=0A>> =0A>> Group nobody (65534)=0A>> -> chown :nobody /usr/jail/test.f= ile =3D> no problem=0A>> =0A>> Change user nobody UID from 65534 to 65533= =3D> same problem. It's not=0A>> a UID number problem but a name problem= .=0A> =0A> Yes, for NFSv4 it is the names that go in the RPC request and = not the=0A> numbers. However, since there are the numbers in the AUTH_SYS= credential=0A> in the header (unless you are using Kerberized mounts), t= he numbers for=0A> the names need to be consistent between client and ser= ver.=0A> =0A>> Then, user nobody and group nogroup (not the integer value= s) are=0A>> problematic. I looked at nfsuserd.c and i see:=0A>> u_char *d= efaultuser =3D "nobody";=0A>> u_char *defaultgroup =3D "nogroup";=0A> =0A= > These are used if no mapping is found in the user or group database=0A>= for whatever name is in the RPC on the wire.=0A> =0A> If you want to see= what is happening, I suggest that you capture=0A> packets when you do th= e "chown" (You can use "tcpdump -s 0 -w file.pcap host XXX".)=0A> then lo= ok at them in wireshark.=0A> In wireshark, look for the Setattr RPC and t= hen look in the setable attributes.=0A> You should find Owner which looks= like "nobody@ and=0A> Owner_group which looks the same = (or "nogroup@" if you=0A> used nogroup). "nogroup" must = be in your group database (/etc/group or whatever=0A> you use for a group= database) and the number must be consistent across client=0A> and server= .=0A> Also, see what the reply to the Setattr RPC is (it is actually a Co= mpound RPC=0A> labelled "Setattr" for NFSv4).=0A> =0A> If there is no Set= attr RPC, then the mapping is failing in the client.=0A> =0A> If the stuf= f looks correct on the wire, then it is most likely a server side=0A> iss= ue.=0A> =0A> rick=0A> =0A>> I think it's related.=0A>> =0A>> Regards,=0A>= > =0A>> Lo=C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer= =0A>> http://www.unix-experience.fr=0A>> =0A>> 13 octobre 2014 09:15 "Lo= =C3=AFc Blot" a=0A>> =C3=A9crit:=0A>>> Hi,= =0A>>> of course i have it. On each node:=0A>>> =0A>>> # cat /etc/master.= passwd | grep nobody=0A>>> returns:=0A>>> nobody:*:65534:65534::0:0:Unpri= vileged=0A>>> user:/nonexistent:/usr/sbin/nologin=0A>>> =0A>>> It's why i= do a report here :)=0A>>> =0A>>> Regards,=0A>>> =0A>>> Lo=C3=AFc Blot,= =0A>>> UNIX Systems, Network and Security Engineer=0A>>> http://www.unix-= experience.fr=0A>>> =0A>>> 10 octobre 2014 13:51 "Rick Macklem" a=0A>>> =C3=A9crit:=0A>>> =0A>>>> Loic Blot wrote:=0A>>>> = =0A>>>>> Hello @freebsd-fs,=0A>>>>> i'm trying to do jail hosting over NF= Sv4 with ezjail and i'm=0A>>>>> experimenting an issue that i can't resol= ve. When i extract=0A>>>>> base.txz (with ezjail) or i set nobody user on= a file, i have=0A>>>>> this=0A>>>>> error:=0A>>>>> =0A>>>>> chown nobody= :nobody /usr/jails/fulljail/mnt/=0A>>>>> No name and/or group mapping for= uid,gid:(65534,65534)=0A>>>>> chown: /usr/jails/fulljail/mnt/: Operation= not permitted=0A>>>>> =0A>>>>> No problem if i set:=0A>>>>> chown mysql:= nobody /usr/jails/fulljail/mnt/=0A>>>>> =0A>>>>> Problem appears on all f= iles.=0A>>>> =0A>>>> Do you have a user by the name of "nobody" in your p= assword=0A>>>> database?=0A>>>> (NFSv4 uses names and not numbers on the = wire, so no name-->no=0A>>>> mapping=0A>>>> and chown can't be done.)=0A>= >>> =0A>>>> rick=0A>>>> =0A>>>>> On my ZFS+NFSv4 server i do a dataset, e= xported in NFS=0A>>>>> =0A>>>>> /etc/exports:=0A>>>>> V4: /=0A>>>>> =0A>>= >>> zfs get sharenfs pool/jails:=0A>>>>> -network=3D10.99.99.0 -mask=3D25= 5.255.255.0 -maproot=3Droot=0A>>>>> =0A>>>>> nfsuserd and nfsv4_server_en= able=3DYES on both client and server,=0A>>>>> plus=0A>>>>> nfsbcd on clie= nt.=0A>>>>> =0A>>>>> On the client here is the fstab entry=0A>>>>> 10.99.= 99.99:/pool/jails /usr/jails nfs rw,nfsv4 0 0=0A>>>>> =0A>>>>> What i'm d= oing wrong ?=0A>>>>> =0A>>>>> Thanks in advance=0A>>>>> Regards,=0A>>>>> = =0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network and Security Engin= eer=0A>>>>> http://www.unix-experience.fr=0A>>>>> =0A>> _________________= ______________=0A>> =0A>>>>> =0A>>>>> freebsd-fs@freebsd.org mailing list= =0A>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A>>>>> To = unsubscribe, send any mail to=0A>>>>> "freebsd-fs-unsubscribe@freebsd.org= "=0A>>> =0A>>> =0A>> _______________________________=0A>> =0A>>> =0A>>> f= reebsd-fs@freebsd.org mailing list=0A>>> http://lists.freebsd.org/mailman= /listinfo/freebsd-fs=0A>>> To unsubscribe, send any mail to=0A>>> "freebs= d-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 18:40:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BC8CF7D1; Mon, 13 Oct 2014 18:40:12 +0000 (UTC) Received: from mail-yh0-x22d.google.com (mail-yh0-x22d.google.com [IPv6:2607:f8b0:4002:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CDE0D6E; Mon, 13 Oct 2014 18:40:12 +0000 (UTC) Received: by mail-yh0-f45.google.com with SMTP id b6so3886715yha.32 for ; Mon, 13 Oct 2014 11:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=GdkDgud9ZwTDGxKB4JcbBLYVup8Tc1bc/vmglFgvaPQ=; b=v5kmjvXon+D/5MwtcYJEB9SNU1KjAZzTQYAHkNmseDvyUpcNC4FrPpEd0d2QUHbgYi LhgTBQvf30lBJVM/Un1bFKUGcr4G5yspEYIZLRH4tjVGO2WsNCpgDqjpMIG5DY/NM4nO y/J5FFk4mK6Suel9TjXHelawJyEBgE7DOF3SxIpDjbP4vsqjkD343ZMvcnCc2nB8A+pu Y87QUFc45YrOZIdM0vMFcHAM0Bn3bwkA+3P5FjBgW7QpHN5MkuXKc4uDMjl2zxfLoArX JBFgJ6cfqa2SG5WxXauBzNOi0e2lk2LNjxqlDOtWWHq8hEE+lvwCVHj+J5uKiXGBiN1e GZTQ== MIME-Version: 1.0 X-Received: by 10.236.96.197 with SMTP id r45mr536807yhf.52.1413225611417; Mon, 13 Oct 2014 11:40:11 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 11:40:11 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 11:40:11 -0700 X-Google-Sender-Auth: Fs7HlWehuGXSlb7kaqKeWaNiw2E Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 18:40:12 -0000 >> Yer I would have got the zio details but typically its "optimised out" b= y >> the >> compiler, so will need some effort to track that down unfortunately :( > > > Well, let me know if you can. Re-creating a new 10.x VM is taking a while= as > it's taking me forever to checkout the sources. > > Things like that need to somehow continue to be accessible. So there is an outstanding child, but it isn't clear what state it's in because the child pointer in the zio_link isn't valid. It's as if the memory got re-used for something else. I'm dumping my findings below on the off-chance that it helps you push this along. (kgdb) thread 459 [Switching to thread 459 (Thread 101524)]#0 sched_switch (td=3D0xfffff80063111000, newtd=3D, flags=3D) at /usr/home/kmacy/devel/svn/10/sys/kern/sched_ule.c:1945 1945 cpuid =3D PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=3D0xfffff80063111000, newtd=3D, flags=3D) at /usr/home/kmacy/devel/svn/10/sys/kern/sched_ule.c:1945 #1 0xffffffff807aa199 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/home/kmacy/devel/svn/10/sys/kern/kern_synch.c:494 #2 0xffffffff807e6e82 in sleepq_switch (wchan=3D, pri=3D) at /usr/home/kmacy/devel/svn/10/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff807e6ce3 in sleepq_wait (wchan=3D0xfffff8004ddf4a50, pri=3D0) at /usr/home/kmacy/devel/svn/10/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff80750d7a in _cv_wait (cvp=3D0xfffff8004ddf4a50, lock=3D0xfffff8004ddf4a30) at /usr/home/kmacy/devel/svn/10/sys/kern/kern_condvar.c:139 #5 0xffffffff817d145b in zio_wait (zio=3D) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/zio.c:1442 #6 0xffffffff81779d3c in dsl_pool_sync (dp=3D0xfffff8004d364800, txg=3D117= 33518) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/dsl_pool.c:531 #7 0xffffffff8179d800 in spa_sync (spa=3D0xfffffe000372f000, txg=3D1173351= 8) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/spa.c:6604 #8 0xffffffff817a7e9d in txg_sync_thread (arg=3D0xfffff8004d364800) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/txg.c:518 #9 0xffffffff8076ed34 in fork_exit (callout=3D0xffffffff817a7c50 , arg=3D0xfffff8004d364800, frame=3D0xfffffe012043fac0) at /usr/home/kmacy/devel/svn/10/sys/kern/kern_fork.c:996 #10 0xffffffff80b96b3e in fork_trampoline () at /usr/home/kmacy/devel/svn/10/sys/amd64/amd64/exception.S:606 #11 0x0000000000000000 in ?? () (kgdb) f 5 #5 0xffffffff817d145b in zio_wait (zio=3D) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/zio.c:1442 1442 cv_wait(&zio->io_cv, &zio->io_lock); (kgdb) disassemble zio_wait Dump of assembler code for function zio_wait: 0xffffffff817d13c0 : push %rbp 0xffffffff817d13c1 : mov %rsp,%rbp 0xffffffff817d13c4 : push %r15 0xffffffff817d13c6 : push %r14 0xffffffff817d13c8 : push %r12 0xffffffff817d13ca : push %rbx 0xffffffff817d13cb : mov %rdi,%r14 0xffffffff817d13ce : cmpl $0x1,0x254(%r14) 0xffffffff817d13d6 : je 0xffffffff817d13f0 0xffffffff817d13d8 : mov $0xffffffff81883de9,%rdi 0xffffffff817d13df : mov $0xffffffff81883b20,%rsi 0xffffffff817d13e6 : mov $0x599,%edx 0xffffffff817d13eb : callq 0xffffffff81a19200 0xffffffff817d13f0 : cmpq $0x0,0x2f0(%r14) 0xffffffff817d13f8 : je 0xffffffff817d1412 0xffffffff817d13fa : mov $0xffffffff8188410c,%rdi 0xffffffff817d1401 : mov $0xffffffff81883b20,%rsi 0xffffffff817d1408 : mov $0x59a,%edx 0xffffffff817d140d : callq 0xffffffff81a19200 0xffffffff817d1412 : mov %gs:0x0,%rax 0xffffffff817d141b : mov %rax,0x2f8(%r14) 0xffffffff817d1422 : mov %r14,%rdi 0xffffffff817d1425 : callq 0xffffffff817d24c0 0xffffffff817d142a : lea 0x300(%r14),%r15 0xffffffff817d1431 : xor %esi,%esi 0xffffffff817d1433 : mov $0xffffffff81883b20,%rdx 0xffffffff817d143a : mov $0x5a0,%ecx 0xffffffff817d143f : mov %r15,%rdi 0xffffffff817d1442 : callq 0xffffffff807a8270 <_sx_xloc= k> 0xffffffff817d1447 : lea 0x320(%r14),%rbx 0xffffffff817d144e : jmp 0xffffffff817d145b 0xffffffff817d1450 : mov %rbx,%rdi 0xffffffff817d1453 : mov %r15,%rsi 0xffffffff817d1456 : callq 0xffffffff80750ba0 <_cv_wait= > 0xffffffff817d145b : cmpq $0x0,0x2f0(%r14) 0xffffffff817d1463 : jne 0xffffffff817d1450 0xffffffff817d1465 : mov $0xffffffff81883b20,%rsi 0xffffffff817d146c : mov $0x5a3,%edx 0xffffffff817d1471 : mov %r15,%rdi 0xffffffff817d1474 : callq 0xffffffff807a8630 <_sx_xunl= ock> 0xffffffff817d1479 : mov 0x268(%r14),%r12d 0xffffffff817d1480 : lea 0xf0(%r14),%rdi 0xffffffff817d1487 : callq 0xffffffff8172d480 0xffffffff817d148c : lea 0x110(%r14),%rdi 0xffffffff817d1493 : callq 0xffffffff8172d480 0xffffffff817d1498 : mov %r15,%rdi 0xffffffff817d149b : callq 0xffffffff807a8010 0xffffffff817d14a0 : mov %rbx,%rdi 0xffffffff817d14a3 : callq 0xffffffff80750b50 0xffffffff817d14a8 : mov 0xffffffff818af6e0,%rdi 0xffffffff817d14b0 : mov %r14,%rsi 0xffffffff817d14b3 : callq 0xffffffff81a19400 0xffffffff817d14b8 : mov %r12d,%eax 0xffffffff817d14bb : pop %rbx 0xffffffff817d14bc : pop %r12 0xffffffff817d14be : pop %r14 0xffffffff817d14c0 : pop %r15 0xffffffff817d14c2 : pop %rbp 0xffffffff817d14c3 : retq End of assembler dump. 0xffffffff817d1422 : mov %r14,%rdi 0xffffffff817d1425 : callq 0xffffffff817d24c0 0xffffffff817d142a : lea 0x300(%r14),%r15 0xffffffff817d1431 : xor %esi,%esi 0xffffffff817d1433 : mov $0xffffffff81883b20,%rdx 0xffffffff817d143a : mov $0x5a0,%ecx 0xffffffff817d143f : mov %r15,%rdi 0xffffffff817d1442 : callq 0xffffffff807a8270 <_sx_xloc= k> 0xffffffff817d1447 : lea 0x320(%r14),%rbx 0xffffffff817d144e : jmp 0xffffffff817d145b 0xffffffff817d1450 : mov %rbx,%rdi 0xffffffff817d1453 : mov %r15,%rsi 0xffffffff817d1456 : callq 0xffffffff80750ba0 <_cv_wait= > (kgdb) p *(zio_t *)$r14 $1 =3D { io_bookmark =3D { zb_objset =3D 0, zb_object =3D 0, zb_level =3D 0, zb_blkid =3D 0 }, io_prop =3D { zp_checksum =3D ZIO_CHECKSUM_INHERIT, zp_compress =3D ZIO_COMPRESS_INHERIT, zp_type =3D DMU_OT_NONE, zp_level =3D 0 '\0', zp_copies =3D 0 '\0', zp_dedup =3D 0, zp_dedup_verify =3D 0, zp_nopwrite =3D 0 }, io_type =3D ZIO_TYPE_NULL, io_child_type =3D ZIO_CHILD_LOGICAL, io_cmd =3D 0, io_priority =3D ZIO_PRIORITY_NOW, io_reexecute =3D 2 '\002', io_state =3D "\001\001", io_txg =3D 0, io_spa =3D 0xfffffe000372f000, io_bp =3D 0x0, io_bp_override =3D 0x0, io_bp_copy =3D { blk_dva =3D {{ dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }}, blk_prop =3D 0, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D {0, 0, 0, 0} } }, io_parent_list =3D { list_size =3D 48, list_offset =3D 16, list_head =3D { list_next =3D 0xfffff800b435c850, list_prev =3D 0xfffff800b435c850 } }, io_child_list =3D { list_size =3D 48, list_offset =3D 32, list_head =3D { list_next =3D 0xfffff80003585770, list_prev =3D 0xfffff80003585770 } }, io_walk_link =3D 0x0, io_logical =3D 0x0, io_transform_stack =3D 0x0, io_ready =3D 0, io_physdone =3D 0, io_done =3D 0, io_private =3D 0x0, io_prev_space_delta =3D 0, io_bp_orig =3D { blk_dva =3D {{ dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }}, blk_prop =3D 0, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D {0, 0, 0, 0} } }, io_data =3D 0x0, io_orig_data =3D 0x0, io_size =3D 0, io_orig_size =3D 0, io_vd =3D 0x0, io_vsd =3D 0x0, io_vsd_ops =3D 0x0, io_offset =3D 0, io_timestamp =3D 0, io_queue_node =3D { avl_child =3D {0x0, 0x0}, avl_pcb =3D 0 }, io_flags =3D 0, io_stage =3D ZIO_STAGE_DONE, io_pipeline =3D 2162688, io_orig_flags =3D 0, io_orig_stage =3D ZIO_STAGE_OPEN, io_orig_pipeline =3D 2162688, io_error =3D 0, io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 0}, {0, 0}, {0, 0}, {0, 0}}, io_child_count =3D 1, io_phys_children =3D 0, io_parent_count =3D 1, io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0, io_executor =3D 0xfffff8006314a000, io_waiter =3D 0xfffff80063111000, io_lock =3D { lock_object =3D { lo_name =3D 0xffffffff8188655c "zio->io_lock", lo_flags =3D 41091072, lo_data =3D 0, lo_witness =3D 0xfffffe00006eca80 }, sx_lock =3D 1 }, io_cv =3D { cv_description =3D 0xffffffff8188656a "zio->io_cv", cv_waiters =3D 1 }, io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D { tqent_task =3D { ta_link =3D { stqe_next =3D 0x0 }, ta_pending =3D 0, ta_priority =3D 0, ta_func =3D 0, ta_context =3D 0x0 }, tqent_func =3D 0, tqent_arg =3D 0x0 }, io_trim_node =3D { avl_child =3D {0x0, 0x0}, avl_pcb =3D 0 }, io_trim_link =3D { list_next =3D 0x0, list_prev =3D 0x0 } } All of its children have called zio_notify_parent on this zio and are error free: io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 0}, {0, 0}, {0, 0}, {0, 0}}, (kgdb) p ((zio_t *)$r14)->io_stall $5 =3D (uint64_t *) 0x0 One child has itself not reached the end of zio_done: io_child_count =3D 1, (kgdb) p ((zio_t *)$r14)->io_child_list $2 =3D { list_size =3D 48, list_offset =3D 32, list_head =3D { list_next =3D 0xfffff80003585770, list_prev =3D 0xfffff80003585770 } (kgdb) p *(zio_link_t *)0xfffff80003585770 $15 =3D { zl_parent =3D 0xfffff8004ddf4850, zl_child =3D 0xfffff8004ddf4850, zl_parent_node =3D { list_next =3D 0xfffff8004dd62398, list_prev =3D 0xfffff8004d4d2730 }, zl_child_node =3D { list_next =3D 0x0, list_prev =3D 0x0 } } I don't understand why the parent and child are the same, of course that could be part of the problem. That address isn't a valid zio and there and the child node entry doesn't point to anything. It has a thread executing it (which itself is probably in cv_wait): io_executor =3D 0xfffff8006314a000, (kgdb) p *((struct thread *)0xfffff8006314a000) $11 =3D { td_lock =3D 0xffffffff81224b30, td_proc =3D 0xffffffff814438d8, td_plist =3D { tqe_next =3D 0xfffff8006312b920, tqe_prev =3D 0xfffff8006312a4a0 }, td_runq =3D { tqe_next =3D 0x0, tqe_prev =3D 0xffffffff811ef378 }, td_slpq =3D { tqe_next =3D 0x0, tqe_prev =3D 0xfffff80020e204c0 }, td_lockq =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0 }, td_hash =3D { le_next =3D 0x0, le_prev =3D 0xfffffe0000879fc8 }, td_cpuset =3D 0xfffff800631d2318, td_sel =3D 0x0, td_sleepqueue =3D 0x0, td_turnstile =3D 0xfffff80063128480, td_rlqe =3D 0x0, td_umtxq =3D 0xfffff8004d7a6380, td_tid =3D 100345, td_sigqueue =3D { sq_signals =3D { __bits =3D {0, 0, 0, 0} }, sq_kill =3D { __bits =3D {0, 0, 0, 0} }, sq_list =3D { tqh_first =3D 0x0, tqh_last =3D 0xfffff8006314a0b8 }, sq_proc =3D 0xffffffff814438d8, sq_flags =3D 1 }, td_lend_user_pri =3D 255 '=C3=BF', td_flags =3D 4, td_inhibitors =3D 2, td_pflags =3D 2097152, td_dupfd =3D 0, td_sqqueue =3D 0, td_wchan =3D 0xfffff800b48a2900, td_wmesg =3D 0xffffffff80d64d7e "-", td_lastcpu =3D 0 '\0', td_oncpu =3D 255 '=C3=BF', td_owepreempt =3D 0 '\0', td_tsqueue =3D 0 '\0', td_locks =3D 0, td_rw_rlocks =3D 0, td_lk_slocks =3D 0, td_stopsched =3D 0, td_blocked =3D 0x0, td_lockname =3D 0x0, td_contested =3D { lh_first =3D 0x0 }, td_sleeplocks =3D 0xffffffff81369470, ---Type to continue, or q to quit--- td_intr_nesting_level =3D 0, td_pinned =3D 0, td_ucred =3D 0xfffff800027f1e00, td_estcpu =3D 0, td_slptick =3D 226829, td_blktick =3D 0, td_swvoltick =3D 226829, td_cow =3D 0, td_ru =3D { ru_utime =3D { tv_sec =3D 0, tv_usec =3D 0 }, ru_stime =3D { tv_sec =3D 0, tv_usec =3D 0 }, ru_maxrss =3D 0, ru_ixrss =3D 0, ru_idrss =3D 0, ru_isrss =3D 0, ru_minflt =3D 0, ru_majflt =3D 0, ru_nswap =3D 0, ru_inblock =3D 0, ru_oublock =3D 0, ru_msgsnd =3D 0, ru_msgrcv =3D 0, ru_nsignals =3D 0, ru_nvcsw =3D 6, ru_nivcsw =3D 0 }, td_rux =3D { rux_runtime =3D 0, rux_uticks =3D 0, rux_sticks =3D 0, rux_iticks =3D 0, rux_uu =3D 0, rux_su =3D 0, rux_tu =3D 0 }, td_incruntime =3D 410296, td_runtime =3D 410296, td_pticks =3D 0, td_sticks =3D 0, td_iticks =3D 0, td_uticks =3D 0, td_intrval =3D 0, td_oldsigmask =3D { __bits =3D {0, 0, 0, 0} }, td_generation =3D 6, td_sigstk =3D { ss_sp =3D 0x0, ss_size =3D 0, ss_flags =3D 0 }, td_xsig =3D 0, td_profil_addr =3D 0, td_profil_ticks =3D 0, td_name =3D "zio_write_intr_7\000\000\000", td_fpop =3D 0x0, td_dbgflags =3D 0, td_dbgksi =3D { ksi_link =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0 }, ksi_info =3D { si_signo =3D 0, si_errno =3D 0, si_code =3D 0, si_pid =3D 0, si_uid =3D 0, si_status =3D 0, si_addr =3D 0x0, si_value =3D { sival_int =3D 0, sival_ptr =3D 0x0, sigval_int =3D 0, sigval_ptr =3D 0x0 }, _reason =3D { _fault =3D { _trapno =3D 0 }, _timer =3D { _timerid =3D 0, _overrun =3D 0 }, _mesgq =3D { _mqd =3D 0 }, _poll =3D { _band =3D 0 }, __spare__ =3D { __spare1__ =3D 0, __spare2__ =3D {0, 0, 0, 0, 0, 0, 0} } } }, ksi_flags =3D 0, ksi_sigq =3D 0x0 }, td_ng_outbound =3D 0, td_osd =3D { osd_nslots =3D 0, osd_slots =3D 0x0, osd_next =3D { le_next =3D 0x0, le_prev =3D 0x0 } }, td_map_def_user =3D 0x0, td_dbg_forked =3D 0, td_vp_reserv =3D 0, td_no_sleeping =3D 0, td_dom_rr_idx =3D 0, td_sigmask =3D { __bits =3D {0, 0, 0, 0} }, td_rqindex =3D 21 '\025', td_base_pri =3D 84 'T', td_priority =3D 84 'T', td_pri_class =3D 3 '\003', td_user_pri =3D 172 '=C2=AC', td_base_user_pri =3D 172 '=C2=AC', td_pcb =3D 0xfffffe012060eb80, td_state =3D TDS_INHIBITED, td_retval =3D {0, 0}, td_slpcallout =3D { ---Type to continue, or q to quit--- c_links =3D { le =3D { le_next =3D 0x0, le_prev =3D 0x0 }, sle =3D { sle_next =3D 0x0 }, tqe =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0 } }, c_time =3D 0, c_precision =3D 0, c_arg =3D 0x0, c_func =3D 0, c_lock =3D 0x0, c_flags =3D 16, c_cpu =3D 0 }, td_frame =3D 0xfffffe012060eac0, td_kstack_obj =3D 0xfffff8006314cd00, td_kstack =3D 18446741879524470784, td_kstack_pages =3D 4, td_critnest =3D 1, td_md =3D { md_spinlock_count =3D 1, md_saved_flags =3D 582, md_spurflt_addr =3D 0 }, td_sched =3D 0xfffff8006314a468, td_ar =3D 0x0, td_lprof =3D {{ lh_first =3D 0x0 }, { lh_first =3D 0x0 }}, td_dtrace =3D 0xfffff800b437ce00, td_errno =3D 0, td_vnet =3D 0x0, td_vnet_lpush =3D 0x0, td_intr_frame =3D 0x0, td_rfppwait_p =3D 0x0, td_ma =3D 0x0, td_ma_cnt =3D 0 } Interesting bits about the executing thread: td_name =3D "zio_write_intr_7\000\000\000", td_critnest =3D 1, td_md =3D { md_spinlock_count =3D 1, md_saved_flags =3D 582, md_spurflt_addr =3D 0 }, Unfortunately executor is never cleared, so this is just the last thread that happened to execute this i/o. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 18:43:47 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 480C4987; Mon, 13 Oct 2014 18:43:47 +0000 (UTC) Received: from mail-yh0-x232.google.com (mail-yh0-x232.google.com [IPv6:2607:f8b0:4002:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE488E2F; Mon, 13 Oct 2014 18:43:46 +0000 (UTC) Received: by mail-yh0-f50.google.com with SMTP id a41so3905591yho.9 for ; Mon, 13 Oct 2014 11:43:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=3MfuhrtClGMRcJI4ReBhfBwre3FoIcn3vzxn2vjuBIg=; b=OpNi31hwVi3Pyb7OHyc+289I2vPOXI6MGIs9HkT5f8PgAml8lTLDtW17hazfa1svOC +nzhvCBdMeq+PO5QVIcuUat0UphiwS9WVOVuzcRG36zf+/D9LFmoDlIYUcfWjurUwnAL C5e+arEmCjfHWZYc99AOFu34QiZNcjiYby+ilU7nHxWy68D5kKXF9J7F6KP4PXaroAJl VvzAp2P2hrUxBEfJAKXLJ6IlBl4eTrXvQzV26dAMxMMqw5ppgLBgK8EFARg9Z2Q+OyVc 6XiF0+LzTeELdYXKwmnYe518Gspl/uOkYyJTH5iaJDFQ/FE1k25gRuulCSwJOHW6UsJx ACZQ== MIME-Version: 1.0 X-Received: by 10.236.103.170 with SMTP id f30mr659352yhg.4.1413225825965; Mon, 13 Oct 2014 11:43:45 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 11:43:45 -0700 (PDT) In-Reply-To: <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 11:43:45 -0700 X-Google-Sender-Auth: Sv6PFUGBZE66F3NhqfSBLjHi2ys Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 18:43:47 -0000 >> A recent quick read of the code would lead me to believe that > > Yer I would have got the zio details but typically its "optimised out" by > the > compiler, so will need some effort to track that down unfortunately :( FYI it isn't actually optimized out. The debug info accounting just isn't very good about tracking the values in registers. You'll see in my last mail that just looking at the assembler makes it pretty obvious that zio was in %r14. HTH. Cheers. -K From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 19:02:21 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 522B5405; Mon, 13 Oct 2014 19:02:21 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id B198EB6; Mon, 13 Oct 2014 19:02:20 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 3D0C020E708EB; Mon, 13 Oct 2014 19:02:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 157FE20E708E8; Mon, 13 Oct 2014 19:02:15 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 20:02:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:02:21 -0000 ----- Original Message ----- From: "K. Macy" > On Mon, Oct 13, 2014 at 1:06 AM, Steven Hartland > wrote: > >> ----- Original Message ----- From: "K. Macy" >> >> A recent quick read of the code would lead me to believe that zio_wait not >>> returning there means that the zio never reached the zio_done stage. >>> Parent >>> zios seem to yield in a couple of stages in the pipeline if they have >>> incomplete children. They determine this by calling zio_wait_for_children >>> with zio child types and their corresponding wait type. In so doing they >>> set the io_stall to the count of the number of waiters of the first >>> non-zero check. This parent I/O will be resumed by the last child zio of >>> that type and wait state in zio_notify_parent. I'm sure you know all this >>> - >>> but I wrote it to preface asking for the following fields of the zio being >>> waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children >>> is pointed to) *io_stall, io_children[*][*], io_child_list (at a first >>> glance just the addresses). The other alternative is that it reexecuting >>> has gotten in to a bad place in the state machine so io_reexecute. >>> >> >> Yer I would have got the zio details but typically its "optimised out" by >> the >> compiler, so will need some effort to track that down unfortunately :( >> > > Well, let me know if you can. Re-creating a new 10.x VM is taking a while > as it's taking me forever to checkout the sources. > > Things like that need to somehow continue to be accessible. I believe there's some pool corruption here somewhere as every once in a while I trip and ASSERT panic: panic: solaris assert: size >= SPA_MINBLOCKSIZE || range_tree_space(msp->ms_tree) == 0, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c, line: 1636 #3 0xffffffff80607ed3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:688 #4 0xffffffff8179321d in assfail (a=, f=, l=) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 #5 0xffffffff81501ecf in metaslab_passivate (msp=0xfffff800091e5800, size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1636 #6 0xffffffff81501ca7 in metaslab_group_alloc (mg=0xfffff80044ef7400, psize=512, asize=512, txg=11733518, min_distance=536281088, dva=, d=-1) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:2206 #7 0xffffffff81500c9d in metaslab_alloc_dva (spa=0xfffffe00022bb000, mc=0xfffff800045d0c00, psize=512, dva=0xfffffe000e409640, d=0, hintdva=, flags=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:2381 #8 0xffffffff81500709 in metaslab_alloc (spa=0xfffffe00022bb000, mc=0xfffff800045d0c00, psize=512, bp=0xfffffe000e409640, ndvas=3, txg=11733518, hintbp=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:2586 #9 0xffffffff8154ca8a in zio_dva_allocate (zio=0xfffff80044a57398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2444 #10 0xffffffff81548d54 in zio_execute (zio=0xfffff80044a57398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #11 0xffffffff8154d25f in zio_ready (zio=0xfffff800487d2730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3031 #12 0xffffffff81548d54 in zio_execute (zio=0xfffff800487d2730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #13 0xffffffff80651410 in taskqueue_run_locked (queue=0xfffff80054291000) at /usr/src/sys/kern/subr_taskqueue.c:342 #14 0xffffffff80651dcb in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:563 (kgdb) frame 5 #5 0xffffffff81501ecf in metaslab_passivate (msp=0xfffff800091e5800, size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1636 1636 ASSERT(size >= SPA_MINBLOCKSIZE || range_tree_space(msp->ms_tree) == 0); (kgdb) print size $5 = 0 (kgdb) print msp->ms_tree $6 = (range_tree_t *) 0xfffff800091e5400 (kgdb) print *msp->ms_tree $7 = { rt_root = { avl_root = 0xfffff80048fab840, avl_compar = 0xffffffff81502850 , avl_offset = 0, avl_numnodes = 1, avl_size = 64 }, rt_space = 5632, rt_ops = 0xffffffff8160b110, rt_arg = 0xfffff800091e5800, rt_histogram = {0 , 1, 0 }, rt_lock = 0xfffff800091e5800 } (kgdb) print *msp $8 = { ms_lock = { lock_object = { lo_name = 0xffffffff815eb310 "msp->ms_lock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0 }, sx_lock = 18446735279027798016 }, ms_load_cv = { cv_description = 0xffffffff815eb31e "msp->ms_load_cv", cv_waiters = 0 }, ms_sm = 0xfffff8000924bb80, ms_ops = 0xffffffff8160b100, ms_id = 119, ms_start = 3992977408, ms_size = 33554432, ms_fragmentation = 90, ms_alloctree = {0xfffff800091e5000, 0xfffff800091ec000, 0xfffff800091e3c00, 0xfffff800091e3400}, ms_freetree = {0xfffff800091e4c00, 0xfffff800091e4000, 0xfffff800091e4400, 0xfffff800091e3000}, ms_defertree = {0xfffff800091e3800, 0xfffff800091df400}, ms_tree = 0xfffff800091e5400, ms_condensing = 0, ms_condense_wanted = 0, ms_loaded = 1, ms_loading = 0, ms_deferspace = 0, ms_weight = 13835058055282163712, ms_access_txg = 11733526, ms_size_tree = { avl_root = 0xfffff80048fab858, avl_compar = 0xffffffff81502360 , avl_offset = 24, avl_numnodes = 1, avl_size = 64 }, ms_lbas = {0, 0, 0, 0, 0, 0, 0, 0, 0, 4026524672, 4026530816, 0 }, ms_group = 0xfffff80044ef7400, ms_group_node = { avl_child = {0x0, 0x0}, avl_pcb = 18446735278773074705 }, ms_txg_node = { tn_next = {0x0, 0xfffff800091d7b28, 0xfffff800091e2328, 0x0}, tn_member = "\000\001\001" } } Also when it stalls I've seen this in the zfs debug:- 13 39363 :zfs-dprintf dnode.c - dnode_free_range:1655: ds=mos obj=31 blkid=0 nblks=1125899906842624 txg=3078494 13 39363 :zfs-dprintf dbuf.c - dbuf_free_range:816: ds=mos obj=31 start=0 end=0 13 39363 :zfs-dprintf dbuf.c - dbuf_dirty:1133: ds=mos obj=40 lvl=0 blkid=-1 size=140 13 39363 :zfs-dprintf dnode.c - dnode_setdirty:1286: ds=mos obj=40 txg=3078494 nblks looks suspisiously large and I'm not sure blkid should be -1? For reference to see this I'm using: sysctl debug.zfs_flags=64 dtrace -n 'zfs-dprintf {printf("%s - %s:%d: %s", stringof(arg0), stringof(arg1), arg2, stringof(arg3))}' With all that said the following looks like it might indicate the issue: zdb -e -m sys1boot Metaslabs: vdev 0 metaslabs 127 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 417 free 0 metaslab 1 offset 2000000 spacemap 24 free 0 metaslab 2 offset 4000000 spacemap 64 free 0 metaslab 3 offset 6000000 spacemap 1094 free 0 metaslab 4 offset 8000000 spacemap 1091 free 0 metaslab 5 offset a000000 spacemap 1093 free 0 metaslab 6 offset c000000 spacemap 1095 free 0 metaslab 7 offset e000000 spacemap 1096 free 0 metaslab 8 offset 10000000 spacemap 1098 free 0 metaslab 9 offset 12000000 spacemap 1097 free 0 metaslab 10 offset 14000000 spacemap 1099 free 0 metaslab 11 offset 16000000 spacemap 1102 free 0 metaslab 12 offset 18000000 spacemap 113 free 0 metaslab 13 offset 1a000000 spacemap 110 free 0 metaslab 14 offset 1c000000 spacemap 115 free 0 metaslab 15 offset 1e000000 spacemap 1103 free 0 metaslab 16 offset 20000000 spacemap 416 free 0 metaslab 17 offset 22000000 spacemap 81 free 0 metaslab 18 offset 24000000 spacemap 57 free 0 metaslab 19 offset 26000000 spacemap 102 free 0 metaslab 20 offset 28000000 spacemap 100 free 0 metaslab 21 offset 2a000000 spacemap 104 free 0 metaslab 22 offset 2c000000 spacemap 125 free 0 metaslab 23 offset 2e000000 spacemap 105 free 0 metaslab 24 offset 30000000 spacemap 108 free 0 metaslab 25 offset 32000000 spacemap 126 free 0 metaslab 26 offset 34000000 spacemap 248 free 0 metaslab 27 offset 36000000 spacemap 55 free 0 metaslab 28 offset 38000000 spacemap 82 free 0 metaslab 29 offset 3a000000 spacemap 149 free 0 metaslab 30 offset 3c000000 spacemap 152 free 0 metaslab 31 offset 3e000000 spacemap 155 free 0 metaslab 32 offset 40000000 spacemap 156 free 0 metaslab 33 offset 42000000 spacemap 72 free 0 metaslab 34 offset 44000000 spacemap 96 free 0 metaslab 35 offset 46000000 spacemap 159 free 0 metaslab 36 offset 48000000 spacemap 158 free 0 metaslab 37 offset 4a000000 spacemap 160 free 0 metaslab 38 offset 4c000000 spacemap 1271 free 0 metaslab 39 offset 4e000000 spacemap 161 free 0 metaslab 40 offset 50000000 spacemap 1273 free 0 metaslab 41 offset 52000000 spacemap 1277 free 0 metaslab 42 offset 54000000 spacemap 35 free 0 metaslab 43 offset 56000000 spacemap 25 free 0 metaslab 44 offset 58000000 spacemap 32 free 0 metaslab 45 offset 5a000000 spacemap 150 free 0 metaslab 46 offset 5c000000 spacemap 151 free 0 metaslab 47 offset 5e000000 spacemap 163 free 0 metaslab 48 offset 60000000 spacemap 415 free 0 metaslab 49 offset 62000000 spacemap 98 free 0 metaslab 50 offset 64000000 spacemap 109 free 0 metaslab 51 offset 66000000 spacemap 171 free 0 metaslab 52 offset 68000000 spacemap 80 free 0 metaslab 53 offset 6a000000 spacemap 134 free 0 metaslab 54 offset 6c000000 spacemap 135 free 0 metaslab 55 offset 6e000000 spacemap 154 free 0 metaslab 56 offset 70000000 spacemap 140 free 0 metaslab 57 offset 72000000 spacemap 141 free 0 metaslab 58 offset 74000000 spacemap 1272 free 0 metaslab 59 offset 76000000 spacemap 138 free 0 metaslab 60 offset 78000000 spacemap 139 free 0 metaslab 61 offset 7a000000 spacemap 18 free 0 metaslab 62 offset 7c000000 spacemap 148 free 0 metaslab 63 offset 7e000000 spacemap 1270 free 0 metaslab 64 offset 80000000 spacemap 114 free 0 metaslab 65 offset 82000000 spacemap 112 free 0 metaslab 66 offset 84000000 spacemap 116 free 0 metaslab 67 offset 86000000 spacemap 164 free 0 metaslab 68 offset 88000000 spacemap 243 free 0 metaslab 69 offset 8a000000 spacemap 128 free 0 metaslab 70 offset 8c000000 spacemap 1101 free 0 metaslab 71 offset 8e000000 spacemap 153 free 0 metaslab 72 offset 90000000 spacemap 120 free 0 metaslab 73 offset 92000000 spacemap 62 free 0 metaslab 74 offset 94000000 spacemap 122 free 0 metaslab 75 offset 96000000 spacemap 131 free 0 metaslab 76 offset 98000000 spacemap 129 free 0 metaslab 77 offset 9a000000 spacemap 157 free 0 metaslab 78 offset 9c000000 spacemap 133 free 0 metaslab 79 offset 9e000000 spacemap 137 free 0 metaslab 80 offset a0000000 spacemap 165 free 0 metaslab 81 offset a2000000 spacemap 168 free 0 metaslab 82 offset a4000000 spacemap 170 free 0 metaslab 83 offset a6000000 spacemap 127 free 0 metaslab 84 offset a8000000 spacemap 180 free 0 metaslab 85 offset aa000000 spacemap 162 free 0 metaslab 86 offset ac000000 spacemap 1100 free 0 metaslab 87 offset ae000000 spacemap 107 free 0 metaslab 88 offset b0000000 spacemap 119 free 0 metaslab 89 offset b2000000 spacemap 61 free 0 metaslab 90 offset b4000000 spacemap 60 free 0 metaslab 91 offset b6000000 spacemap 123 free 0 metaslab 92 offset b8000000 spacemap 130 free 0 metaslab 93 offset ba000000 spacemap 59 free 0 metaslab 94 offset bc000000 spacemap 167 free 0 metaslab 95 offset be000000 spacemap 136 free 0 metaslab 96 offset c0000000 spacemap 144 free 0 metaslab 97 offset c2000000 spacemap 166 free 0 metaslab 98 offset c4000000 spacemap 169 free 0 metaslab 99 offset c6000000 spacemap 58 free 0 metaslab 100 offset c8000000 spacemap 56 free 0 metaslab 101 offset ca000000 spacemap 54 free 0 metaslab 102 offset cc000000 spacemap 53 free 0 metaslab 103 offset ce000000 spacemap 52 free 0 metaslab 104 offset d0000000 spacemap 106 free 0 metaslab 105 offset d2000000 spacemap 51 free 0 metaslab 106 offset d4000000 spacemap 50 free 0 metaslab 107 offset d6000000 spacemap 121 free 0 metaslab 108 offset d8000000 spacemap 124 free 0 metaslab 109 offset da000000 spacemap 49 free 0 metaslab 110 offset dc000000 spacemap 142 free 0 metaslab 111 offset de000000 spacemap 132 free 0 metaslab 112 offset e0000000 spacemap 48 free 0 metaslab 113 offset e2000000 spacemap 47 free 0 metaslab 114 offset e4000000 spacemap 46 free 0 metaslab 115 offset e6000000 spacemap 45 free 0 metaslab 116 offset e8000000 spacemap 143 free 0 metaslab 117 offset ea000000 spacemap 44 free 0 metaslab 118 offset ec000000 spacemap 43 free 177K metaslab 119 offset ee000000 spacemap 42 free 13.0K metaslab 120 offset f0000000 spacemap 41 free 0 metaslab 121 offset f2000000 spacemap 179 free 0 metaslab 122 offset f4000000 spacemap 40 free 0 metaslab 123 offset f6000000 spacemap 39 free 0 metaslab 124 offset f8000000 spacemap 38 free 0 metaslab 125 offset fa000000 spacemap 37 free 0 metaslab 126 offset fc000000 spacemap 36 free 0 Only two metaslabs with any free? zfs list -r sys1boot NAME USED AVAIL REFER MOUNTPOINT sys1boot 1.76G 2.08G 11K /sys1boot sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - So basically the zpool says its out of space even though zfs says its under half full? Given this I'm guessing that on import when it goes to write an update its hitting a brick wall and can't, with the zio being flagged as can't fail it seems to get stuck. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 19:14:00 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55534B95; Mon, 13 Oct 2014 19:14:00 +0000 (UTC) Received: from mail-yh0-x22f.google.com (mail-yh0-x22f.google.com [IPv6:2607:f8b0:4002:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 067D11F1; Mon, 13 Oct 2014 19:13:59 +0000 (UTC) Received: by mail-yh0-f47.google.com with SMTP id c41so3897462yho.34 for ; Mon, 13 Oct 2014 12:13:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Ea7Yw4RhAHySmYJCtQSRSqE3Bd1Y69auwHcMR4b5ME8=; b=O9l5alfrZBEuTq/aT0d3XKiYz7mpM00ajBByGZyluc7Lo+W0z3IjnxVaGozJmNl2hO WEjLYGNByL9CNzaTMVlFBfeF8VTRC+rfhBhw9VtMKuTbzTiJ2ue9aBu8enBheWN15fVY G4Y6+8/LQjSh/y8fJjiecAbGM58H81g0IEOWxkLxdHvAOhMHE1tOB47k6Fe6uONqC75G 2CZpFkykHIGGO+31/4uWZzMGCvY4ikCiau7GP/wvCdmXmLyJzw1RgqQa42K+7ADRJmEi Kk3h+wVdZ7wA95kqppWLEJMhWAIYT9XzzNg65Oej5FTLRIxjZyvWKOdcKzHTawlmBmu9 wIGQ== MIME-Version: 1.0 X-Received: by 10.236.25.1 with SMTP id y1mr814329yhy.62.1413227639056; Mon, 13 Oct 2014 12:13:59 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 12:13:58 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 12:13:58 -0700 X-Google-Sender-Auth: F6WId1xapj9yD1oj_DAfO2MWkvQ Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:14:00 -0000 >>> Yer I would have got the zio details but typically its "optimised out" by >>> the >>> compiler, so will need some effort to track that down unfortunately :( >>> >> >> Well, let me know if you can. Re-creating a new 10.x VM is taking a while >> as it's taking me forever to checkout the sources. >> >> Things like that need to somehow continue to be accessible. > > > I believe there's some pool corruption here somewhere as every once in a > while > I trip and ASSERT panic: > panic: solaris assert: size >= SPA_MINBLOCKSIZE || > range_tree_space(msp->ms_tree) == 0, file: > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c, > line: 1636 > <... snip> You are correct. (kgdb) p ((zio_t *)$r14)->io_reexecute $32 = 2 '\002' (kgdb) p ((zio_t *)$r14)->io_flags $33 = 0 (kgdb) p ((zio_t *)$r14)->io_spa->spa_suspended $34 = 1 '\001' This means zio_suspend has been called from zio_done: else if (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND) { /* * We'd fail again if we reexecuted now, so suspend * until conditions improve (e.g. device comes online). */ zio_suspend(spa, zio); } If failure mode were panic we would have panicked when attempting the import: void zio_suspend(spa_t *spa, zio_t *zio) { if (spa_get_failmode(spa) == ZIO_FAILURE_MODE_PANIC) fm_panic("Pool '%s' has encountered an uncorrectable I/O " "failure and the failure mode property for this pool " "is set to panic.", spa_name(spa)); From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 19:14:36 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4E09CF2; Mon, 13 Oct 2014 19:14:36 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 92DBD208; Mon, 13 Oct 2014 19:14:36 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id CD8AB20E708EE; Mon, 13 Oct 2014 19:14:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 264B920E708EB; Mon, 13 Oct 2014 19:14:34 +0000 (UTC) Message-ID: <15EB5C3A83504A949CBDA96420361ADB@multiplay.co.uk> From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 20:14:30 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:14:36 -0000 ----- Original Message ----- From: "K. Macy" To: "Steven Hartland" Cc: "Mark Martinec" ; "freebsd-fs@FreeBSD.org" ; "FreeBSD Stable" Sent: Monday, October 13, 2014 7:43 PM Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] >>> A recent quick read of the code would lead me to believe that >> >> Yer I would have got the zio details but typically its "optimised out" by >> the >> compiler, so will need some effort to track that down unfortunately :( > > FYI it isn't actually optimized out. The debug info accounting just > isn't very good about tracking the values in registers. You'll see in > my last mail that just looking at the assembler makes it pretty > obvious that zio was in %r14. HTH. > > Cheers. > > > -K > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 19:24:43 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C7C63B4 for ; Mon, 13 Oct 2014 19:24:43 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 9F28338B for ; Mon, 13 Oct 2014 19:24:42 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqsEAPIlPFSDaFve/2dsb2JhbABbg2FYBIMCyCIKhnlUAoE0AX2EAgEBAQMBAQEBIAQnIAsFFhgCAg0HEgIpAQkVEQYIBwQBHASIFQgNsAyUZgEBAQEBBQEBAQEBAQEbgSyOSAEBGwEzBxiCHkESgUIFljuEDIQyPJApg36EEyEvB4EIOYECAQEB X-IronPort-AV: E=Sophos;i="5.04,712,1406606400"; d="scan'208";a="160939301" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 13 Oct 2014 15:24:40 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 6737DB403A; Mon, 13 Oct 2014 15:24:40 -0400 (EDT) Date: Mon, 13 Oct 2014 15:24:40 -0400 (EDT) From: Rick Macklem To: =?utf-8?B?TG/Dr2M=?= Blot Message-ID: <1003039765.63581639.1413228280410.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: NFSv4 nobody issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:24:43 -0000 Loic Blot wrote: > Hi Rick, > no request is done. > In /var/log/messages on the client i have: >=20 > Oct 13 15:10:46 machine kernel: No name and/or group mapping for > uid,gid:(65534,-1) >=20 > The FreeBSD kernel refuses to change the owner. >=20 Ok, I took a look and it is a restriction enforced by the server. If you want it to work, you need to comment out these lines in sys/fs/nfsserver/nfs_nfsdsubs.c: if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D nfsrv_defaultuid) 1547 =09|| (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D nfsrv_defaultgid))= { 1548 =09error =3D NFSERR_BADOWNER; 1549 =09goto out; 1550 =09} (Line#s 1546->1550 in head.) It is done because some clients try to set the owner when there is no valid mapping by sending "nobody@" to the server. Unfortunately for you "nobody" is the traditional name for "no mapping". For example, if "chown rick " was done on a client where "rick" is not in the client's passwd database, some clients will send "nobody@" and the above code makes sure that doesn't work. So, if you want this to work, comment out the above lines in your NFSv4 ser= ver's kernel. rick > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr >=20 > 13 octobre 2014 14:43 "Rick Macklem" a =C3=A9crit: > > Loic Blot wrote: > >=20 > >> Hi, > >> i tryed some other things > >>=20 > >> User nobody (65534) > >> -> chown nobody /usr/jail/test.file =3D> problem > >>=20 > >> Group nogroup (65533) > >> -> chown :nogroup /usr/jail/test.file =3D> same problem > >>=20 > >> Group nobody (65534) > >> -> chown :nobody /usr/jail/test.file =3D> no problem > >>=20 > >> Change user nobody UID from 65534 to 65533 =3D> same problem. It's > >> not > >> a UID number problem but a name problem. > >=20 > > Yes, for NFSv4 it is the names that go in the RPC request and not > > the > > numbers. However, since there are the numbers in the AUTH_SYS > > credential > > in the header (unless you are using Kerberized mounts), the numbers > > for > > the names need to be consistent between client and server. > >=20 > >> Then, user nobody and group nogroup (not the integer values) are > >> problematic. I looked at nfsuserd.c and i see: > >> u_char *defaultuser =3D "nobody"; > >> u_char *defaultgroup =3D "nogroup"; > >=20 > > These are used if no mapping is found in the user or group database > > for whatever name is in the RPC on the wire. > >=20 > > If you want to see what is happening, I suggest that you capture > > packets when you do the "chown" (You can use "tcpdump -s 0 -w > > file.pcap host XXX".) > > then look at them in wireshark. > > In wireshark, look for the Setattr RPC and then look in the setable > > attributes. > > You should find Owner which looks like "nobody@ > > and > > Owner_group which looks the same (or "nogroup@" if > > you > > used nogroup). "nogroup" must be in your group database (/etc/group > > or whatever > > you use for a group database) and the number must be consistent > > across client > > and server. > > Also, see what the reply to the Setattr RPC is (it is actually a > > Compound RPC > > labelled "Setattr" for NFSv4). > >=20 > > If there is no Setattr RPC, then the mapping is failing in the > > client. > >=20 > > If the stuff looks correct on the wire, then it is most likely a > > server side > > issue. > >=20 > > rick > >=20 > >> I think it's related. > >>=20 > >> Regards, > >>=20 > >> Lo=C3=AFc Blot, > >> UNIX Systems, Network and Security Engineer > >> http://www.unix-experience.fr > >>=20 > >> 13 octobre 2014 09:15 "Lo=C3=AFc Blot" = a > >> =C3=A9crit: > >>> Hi, > >>> of course i have it. On each node: > >>>=20 > >>> # cat /etc/master.passwd | grep nobody > >>> returns: > >>> nobody:*:65534:65534::0:0:Unprivileged > >>> user:/nonexistent:/usr/sbin/nologin > >>>=20 > >>> It's why i do a report here :) > >>>=20 > >>> Regards, > >>>=20 > >>> Lo=C3=AFc Blot, > >>> UNIX Systems, Network and Security Engineer > >>> http://www.unix-experience.fr > >>>=20 > >>> 10 octobre 2014 13:51 "Rick Macklem" a > >>> =C3=A9crit: > >>>=20 > >>>> Loic Blot wrote: > >>>>=20 > >>>>> Hello @freebsd-fs, > >>>>> i'm trying to do jail hosting over NFSv4 with ezjail and i'm > >>>>> experimenting an issue that i can't resolve. When i extract > >>>>> base.txz (with ezjail) or i set nobody user on a file, i have > >>>>> this > >>>>> error: > >>>>>=20 > >>>>> chown nobody:nobody /usr/jails/fulljail/mnt/ > >>>>> No name and/or group mapping for uid,gid:(65534,65534) > >>>>> chown: /usr/jails/fulljail/mnt/: Operation not permitted > >>>>>=20 > >>>>> No problem if i set: > >>>>> chown mysql:nobody /usr/jails/fulljail/mnt/ > >>>>>=20 > >>>>> Problem appears on all files. > >>>>=20 > >>>> Do you have a user by the name of "nobody" in your password > >>>> database? > >>>> (NFSv4 uses names and not numbers on the wire, so no name-->no > >>>> mapping > >>>> and chown can't be done.) > >>>>=20 > >>>> rick > >>>>=20 > >>>>> On my ZFS+NFSv4 server i do a dataset, exported in NFS > >>>>>=20 > >>>>> /etc/exports: > >>>>> V4: / > >>>>>=20 > >>>>> zfs get sharenfs pool/jails: > >>>>> -network=3D10.99.99.0 -mask=3D255.255.255.0 -maproot=3Droot > >>>>>=20 > >>>>> nfsuserd and nfsv4_server_enable=3DYES on both client and server, > >>>>> plus > >>>>> nfsbcd on client. > >>>>>=20 > >>>>> On the client here is the fstab entry > >>>>> 10.99.99.99:/pool/jails /usr/jails nfs rw,nfsv4 0 0 > >>>>>=20 > >>>>> What i'm doing wrong ? > >>>>>=20 > >>>>> Thanks in advance > >>>>> Regards, > >>>>>=20 > >>>>> Lo=C3=AFc Blot, > >>>>> UNIX Systems, Network and Security Engineer > >>>>> http://www.unix-experience.fr > >>>>>=20 > >> _______________________________ > >>=20 > >>>>>=20 > >>>>> freebsd-fs@freebsd.org mailing list > >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>>>> To unsubscribe, send any mail to > >>>>> "freebsd-fs-unsubscribe@freebsd.org" > >>>=20 > >>>=20 > >> _______________________________ > >>=20 > >>>=20 > >>> freebsd-fs@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>> To unsubscribe, send any mail to > >>> "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 20:10:19 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70875F3B; Mon, 13 Oct 2014 20:10:19 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id D984598C; Mon, 13 Oct 2014 20:10:18 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 95A7720E708F1; Mon, 13 Oct 2014 20:10:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: *** X-Spam-Status: No, score=3.1 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE, STOX_REPLY_TYPE_WITHOUT_QUOTES autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 0E33D20E708EF; Mon, 13 Oct 2014 20:10:14 +0000 (UTC) Message-ID: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Subject: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 21:10:11 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 20:10:19 -0000 ----- Original Message ----- From: "K. Macy" > You are correct. > > (kgdb) p ((zio_t *)$r14)->io_reexecute > $32 = 2 '\002' > (kgdb) p ((zio_t *)$r14)->io_flags > $33 = 0 > (kgdb) p ((zio_t *)$r14)->io_spa->spa_suspended > $34 = 1 '\001' > > This means zio_suspend has been called from zio_done: > else if (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND) { > /* > * We'd fail again if we reexecuted now, so suspend > * until conditions improve (e.g. device comes online). > */ > zio_suspend(spa, zio); > } > > If failure mode were panic we would have panicked when attempting the import: > void > zio_suspend(spa_t *spa, zio_t *zio) > { > if (spa_get_failmode(spa) == ZIO_FAILURE_MODE_PANIC) > fm_panic("Pool '%s' has encountered an uncorrectable I/O " > "failure and the failure mode property for this pool " > "is set to panic.", spa_name(spa)); Yep and forcing that panic I got the following stack: #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff80607977 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff80607e85 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80607ed3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:688 #4 0xffffffff81548dfa in zio_suspend (spa=, zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1527 #5 0xffffffff8154ec66 in zio_done (zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3264 #6 0xffffffff81548d54 in zio_execute (zio=0xfffff80044a0dac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #7 0xffffffff8154ebfc in zio_done (zio=0xfffff8004884b398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #8 0xffffffff81548d54 in zio_execute (zio=0xfffff8004884b398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #9 0xffffffff8154ebfc in zio_done (zio=0xfffff80044c0a000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #10 0xffffffff81548d54 in zio_execute (zio=0xfffff80044c0a000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #11 0xffffffff8154ebfc in zio_done (zio=0xfffff80044a2fac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #12 0xffffffff81548d54 in zio_execute (zio=0xfffff80044a2fac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #13 0xffffffff8154ebfc in zio_done (zio=0xfffff80044853398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #14 0xffffffff81548d54 in zio_execute (zio=0xfffff80044853398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #15 0xffffffff8154ea2a in zio_done (zio=0xfffff8004877e398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3313 #16 0xffffffff81548d54 in zio_execute (zio=0xfffff8004877e398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #17 0xffffffff8154ea2a in zio_done (zio=0xfffff80044cb0730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3313 #18 0xffffffff81548d54 in zio_execute (zio=0xfffff80044cb0730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #19 0xffffffff80651410 in taskqueue_run_locked (queue=0xfffff800488cf400) at /usr/src/sys/kern/subr_taskqueue.c:342 #20 0xffffffff80651dcb in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:563 Along with: (kgdb) print (*(zio_t *)0xfffff80044853398)->io_error $20 = 28 (kgdb) print (*(zio_t *)0xfffff80044a2fac8)->io_error $21 = 28 grep 28 /usr/include/sys/errno.h #define ENOSPC 28 /* No space left on device */ So the issue is simply the pool is out of space to perform the import as that process, when not readonly, requires space to write to the pool. The problem with that is that during this process it has the pool lock so any subsequent zpool actions are dead in the water as they will block waiting on that lock. Something to discuss with the openzfs guys, but I would say the import should fail with a no space error. So Mark the mystery is solved, when you upgraded you ran the pool so low on space that it now can't be imported RW as that requires a write. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 20:55:43 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03DAB7A1; Mon, 13 Oct 2014 20:55:43 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 80D49DE6; Mon, 13 Oct 2014 20:55:42 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jGsbw4pMNzNr; Mon, 13 Oct 2014 22:55:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:from:from :date:date:message-id:received:received:received; s=jakla4; t= 1413233736; x=1415825737; bh=6vVXfSie0JrMZ/XRKkdzqrFhYP73Kvzoe9z 5MW9q7cg=; b=FcmCvr5bkoqpBHw1IS279ppSl6ayjJqUiaDl0rgvZ8J+QuEkfZ2 rU94jkMQko9tvYD853yVtoBLmt8CR2d6mk25JiOH/DHNLMvFvV4NlwDkzVKxse3H eG1/ck+EEqLs6cNuMSgTvzMP7QDcF2KKP+gkhh5AMUagPC0LEDxhp3bo= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id nPgztBolmqQc; Mon, 13 Oct 2014 22:55:36 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Mon, 13 Oct 2014 22:55:36 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jGsbq6N3lzSJ; Mon, 13 Oct 2014 22:55:35 +0200 (CEST) Message-ID: <543C3C47.4010208@ijs.si> Date: Mon, 13 Oct 2014 22:55:35 +0200 From: mark User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> In-Reply-To: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 20:55:43 -0000 On 10/13/2014 22:10, Steven Hartland wrote: > So the issue is simply the pool is out of space to perform the import > as that process, when not readonly, requires space to write to the pool. > > The problem with that is that during this process it has the pool lock so > any subsequent zpool actions are dead in the water as they will block > waiting on that lock. > > Something to discuss with the openzfs guys, but I would say the import > should fail with a no space error. > > So Mark the mystery is solved, when you upgraded you ran the pool so low > on space that it now can't be imported RW as that requires a write. > > Regards > Steve Thank you both for analysis and effort! I can't rule out the possibility that my main system pool on a SSD was low on space at some point in time, but the three 4 GiB cloned pools (sys1boot and its brothers) were all created as a zfs send / receive copies of the main / (root) file system and I haven't noticed anything unusual during syncing. This syncing was done manually (using zxfer) and independently from the upgrade on the system - on a steady/quiet system, when the source file system definitely had sufficient free space. The source file system now shows 1.2 GiB of usage shown by df: shiny/ROOT 61758388 1271620 60486768 2% / Seems unlikely that the 1.2 GiB has grown to 4 GiB space on a cloned filesystem. Will try to import the main two pools after re-creating a sane boot pool... Mark From owner-freebsd-fs@FreeBSD.ORG Mon Oct 13 21:56:56 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07E5680A; Mon, 13 Oct 2014 21:56:56 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id B67AB67A; Mon, 13 Oct 2014 21:56:55 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 4284820E708FA; Mon, 13 Oct 2014 21:56:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id D2AB920E708F8; Mon, 13 Oct 2014 21:56:51 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "mark" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 22:56:48 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 21:56:56 -0000 ----- Original Message ----- From: "mark" To: ; Cc: "FreeBSD Stable" Sent: Monday, October 13, 2014 9:55 PM Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] > On 10/13/2014 22:10, Steven Hartland wrote: >> So the issue is simply the pool is out of space to perform the import >> as that process, when not readonly, requires space to write to the pool. >> >> The problem with that is that during this process it has the pool lock so >> any subsequent zpool actions are dead in the water as they will block >> waiting on that lock. >> >> Something to discuss with the openzfs guys, but I would say the import >> should fail with a no space error. >> >> So Mark the mystery is solved, when you upgraded you ran the pool so low >> on space that it now can't be imported RW as that requires a write. >> >> Regards >> Steve > > Thank you both for analysis and effort! > > I can't rule out the possibility that my main system pool > on a SSD was low on space at some point in time, but the > three 4 GiB cloned pools (sys1boot and its brothers) were all > created as a zfs send / receive copies of the main / (root) > file system and I haven't noticed anything unusual during > syncing. This syncing was done manually (using zxfer) and > independently from the upgrade on the system - on a steady/quiet > system, when the source file system definitely had sufficient > free space. > > The source file system now shows 1.2 GiB of usage shown > by df: > shiny/ROOT 61758388 1271620 60486768 2% / > Seems unlikely that the 1.2 GiB has grown to 4 GiB space > on a cloned filesystem. > > Will try to import the main two pools after re-creating > a sane boot pool... Yer zfs list only shows around 2-3GB used too but zpool list shows the pool is out of space. Cant rule out an accounting issue though. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 01:15:59 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C5426B0; Tue, 14 Oct 2014 01:15:59 +0000 (UTC) Received: from mail-yk0-x235.google.com (mail-yk0-x235.google.com [IPv6:2607:f8b0:4002:c07::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A26BD6C; Tue, 14 Oct 2014 01:15:59 +0000 (UTC) Received: by mail-yk0-f181.google.com with SMTP id q200so3925603ykb.40 for ; Mon, 13 Oct 2014 18:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=oqg72uyd7A6ysA/Jre/9gGgkOX8P5fjPtNjpQ0dRirI=; b=nccjuB8TWlrxdg7X2JIQ3UMQ0IQ1yaD2gZpw0YO/VNcaBV2paBJ5qUYR0FCatyk3z9 J6wrMVKPcG//A8JNXZRlIhwLMv2aXx/7oHVPypXgyBMiPqmaXE8totx0Ya7XKyseFaTp 1LF7GELvD/H3rhge39KBRgvw1iHDa3/D9r/amC/DoBgx104NhjCqcPEMRqnbUap3JMyc 8Mq66qjBZORZNbFqumLtr9BAzy7M+kBb6/j8Sdqr4BytlHd6Zp4z3Z7ekX0JadvNBm3d EgqXN7K8ngy12zgyJ+4BVFFsvPXmJp/Y9nmvvwLBIOxtFz3yw5DgOdjM4t7Cp97Z2LYn EyDQ== MIME-Version: 1.0 X-Received: by 10.236.199.46 with SMTP id w34mr3092953yhn.20.1413249358423; Mon, 13 Oct 2014 18:15:58 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 18:15:58 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Date: Mon, 13 Oct 2014 18:15:58 -0700 X-Google-Sender-Auth: MmfIWfMdJ_ON90WeEuTiHDaEu-g Message-ID: Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable , mark X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 01:15:59 -0000 >> Thank you both for analysis and effort! >> >> I can't rule out the possibility that my main system pool >> on a SSD was low on space at some point in time, but the >> three 4 GiB cloned pools (sys1boot and its brothers) were all >> created as a zfs send / receive copies of the main / (root) >> file system and I haven't noticed anything unusual during >> syncing. This syncing was done manually (using zxfer) and >> independently from the upgrade on the system - on a steady/quiet >> system, when the source file system definitely had sufficient >> free space. >> >> The source file system now shows 1.2 GiB of usage shown >> by df: >> shiny/ROOT 61758388 1271620 60486768 2% / >> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >> on a cloned filesystem. >> >> Will try to import the main two pools after re-creating >> a sane boot pool... > > > Yer zfs list only shows around 2-3GB used too but zpool list > shows the pool is out of space. Cant rule out an accounting > issue though. > What is using the extra space in the pool? Is there an unmounted dataset or snapshot? Do you know how to easily tell? Unlike txg and zio processing I don't have the luxury of having just read that part of the codebase. Thanks. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 01:24:26 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BD7F8C0; Tue, 14 Oct 2014 01:24:26 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id AAA4FE51; Tue, 14 Oct 2014 01:24:25 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jGzZ00r7gz1Pj; Tue, 14 Oct 2014 03:24:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413249860; x=1415841861; bh=E00J03d8mnC6AmiSMuV81G4q/ 4Li+rNK8OluLpXLO8w=; b=NYM7VzCHHu12F0S1O94X/tU+PB5tZoNSVlbkupIdH FfNZBxWgqt0NqzpoMsH1Ps7UXCznZTrFfVDzkO+stidS9kKnHEdeImdjbVOqCpFE CLU2PXdXGtMXyGyqMOVORL9OYeLBwwjC5Kz6Pt/uW25zis+SOx08J2ivWEB+DW9c Po= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id 95NqP2gMnm1A; Tue, 14 Oct 2014 03:24:20 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Tue, 14 Oct 2014 03:24:20 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jGzYv6YGTzqr; Tue, 14 Oct 2014 03:24:19 +0200 (CEST) Message-ID: <543C7B43.5020301@ijs.si> Date: Tue, 14 Oct 2014 03:24:19 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-fs@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 01:24:26 -0000 On 10/14/2014 03:15, K. Macy wrote: > What is using the extra space in the pool? Is there an unmounted > dataset or snapshot? Do you know how to easily tell? Unlike txg and > zio processing I don't have the luxury of having just read that part > of the codebase. Most likely the snapshots (regular periodic snapshots). Changes after upgrading an OS can maybe take an additional 50% of space (just guessing). Btw, ashift=12. Still can't see how that would amount to 4 GiB, but it's possible. Mark From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 01:32:29 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15AA3AD8; Tue, 14 Oct 2014 01:32:29 +0000 (UTC) Received: from mail-yk0-x234.google.com (mail-yk0-x234.google.com [IPv6:2607:f8b0:4002:c07::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6BDFF1D; Tue, 14 Oct 2014 01:32:28 +0000 (UTC) Received: by mail-yk0-f180.google.com with SMTP id 142so3981126ykq.11 for ; Mon, 13 Oct 2014 18:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=MM2GVLyT6Pehvu1oKKeJ4UpP2MiIgQ6VqLc9yOfdDts=; b=z1bNNuOfrK0vIXEmo/JeWKq/CxTQAv/lk+NIVz41KHBOdBn9ylniupC+lvBNhd/JxD fXe/XZQEfqHUtG3BT3RsDDtROc1AZ3R36qfRBrAskb2x/Qb3d2ZGdLvPxUqrwrOZCzx4 gIElWQxuHvH/zD1zFgYSGmKXW0MCEWDH5afeAr0l0UB84lFW+3Kda2HW534klvJ5Yb2k S63BeW+MkkWEDgIKV+inkJcus+L6XyeXcjxZW+WR/eOrOGO9feCNgn+Ud8RviNl5/b3M zDUmoaJ70WFYnkc33IShp5xaBu+hTnxbiY5dzY84zdEK0CbUFXFPbf2jisPLSJk7z63M rc2A== MIME-Version: 1.0 X-Received: by 10.236.231.161 with SMTP id l31mr2441674yhq.104.1413250347868; Mon, 13 Oct 2014 18:32:27 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 18:32:27 -0700 (PDT) In-Reply-To: <543C7B43.5020301@ijs.si> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <543C7B43.5020301@ijs.si> Date: Mon, 13 Oct 2014 18:32:27 -0700 X-Google-Sender-Auth: VfHbR0ZPCoKAbKWaADfhCwJ9OWQ Message-ID: Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Mark Martinec Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 01:32:29 -0000 On Mon, Oct 13, 2014 at 6:24 PM, Mark Martinec wrote: > On 10/14/2014 03:15, K. Macy wrote: >> >> What is using the extra space in the pool? Is there an unmounted >> dataset or snapshot? Do you know how to easily tell? Unlike txg and >> zio processing I don't have the luxury of having just read that part >> of the codebase. > > > Most likely the snapshots (regular periodic snapshots). > Changes after upgrading an OS can maybe take an additional 50% > of space (just guessing). Btw, ashift=12. > Still can't see how that would amount to 4 GiB, but it's possible. > Disconcerting. Is this something that others are likely to hit? Should accounting for writes fail with ENOSPC a bit earlier so that we never reach a state like this? I.e. non-metadata writes will fail at a lower threshold than data or if that is already the case, reduce the threshold further. -K From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 08:15:07 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 982E7BF4; Tue, 14 Oct 2014 08:15:07 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id E4C7B8F9; Tue, 14 Oct 2014 08:15:06 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 1C80A20E70935; Tue, 14 Oct 2014 08:14:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 42F1120E70934; Tue, 14 Oct 2014 08:14:57 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 09:14:54 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , mark , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:15:07 -0000 ----- Original Message ----- From: "K. Macy" >>> Thank you both for analysis and effort! >>> >>> I can't rule out the possibility that my main system pool >>> on a SSD was low on space at some point in time, but the >>> three 4 GiB cloned pools (sys1boot and its brothers) were all >>> created as a zfs send / receive copies of the main / (root) >>> file system and I haven't noticed anything unusual during >>> syncing. This syncing was done manually (using zxfer) and >>> independently from the upgrade on the system - on a steady/quiet >>> system, when the source file system definitely had sufficient >>> free space. >>> >>> The source file system now shows 1.2 GiB of usage shown >>> by df: >>> shiny/ROOT 61758388 1271620 60486768 2% / >>> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >>> on a cloned filesystem. >>> >>> Will try to import the main two pools after re-creating >>> a sane boot pool... >> >> >> Yer zfs list only shows around 2-3GB used too but zpool list >> shows the pool is out of space. Cant rule out an accounting >> issue though. >> > > What is using the extra space in the pool? Is there an unmounted > dataset or snapshot? Do you know how to easily tell? Unlike txg and > zio processing I don't have the luxury of having just read that part > of the codebase. Its not clear but I believe it could just be fragmention even though its ashift=9. I sent the last snapshot to another pool of the same size and it resulted in: NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - I believe FRAG is 0% as the feature wasn't enabled for the lifetime of the pool hence its simply not showing a valid value. zfs list -t all -r sys1boot NAME USED AVAIL REFER MOUNTPOINT sys1boot 1.76G 2.08G 11K /sys1boot sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-19_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-20_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-21_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-22_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-23_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-24_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-26_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-27_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-28_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-29_04.00 128K - 1.19G - sys1boot/ROOT@auto-2014-08-31_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-01_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-02_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-03_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-04_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-05_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-07_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-08_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-09_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-10_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_16.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-11_20.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-12_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_20.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-16_00.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-16_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_16.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-17_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_23.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_01.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_02.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_03.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_05.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_06.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_07.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_09.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_10.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_11.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_13.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_14.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_15.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_16.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-19_17.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-19_18.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_18.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_18.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.30 0 - 1.20G - Regards Steve From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 08:25:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D9B1EEF4; Tue, 14 Oct 2014 08:25:12 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 923FC9EE; Tue, 14 Oct 2014 08:25:12 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id DA42B20E70936; Tue, 14 Oct 2014 08:25:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 834ED20E70934; Tue, 14 Oct 2014 08:25:10 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" , "Mark Martinec" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <543C7B43.5020301@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 09:25:07 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:25:13 -0000 ----- Original Message ----- From: "K. Macy" > On Mon, Oct 13, 2014 at 6:24 PM, Mark Martinec > wrote: >> On 10/14/2014 03:15, K. Macy wrote: >>> >>> What is using the extra space in the pool? Is there an unmounted >>> dataset or snapshot? Do you know how to easily tell? Unlike txg and >>> zio processing I don't have the luxury of having just read that part >>> of the codebase. >> >> >> Most likely the snapshots (regular periodic snapshots). >> Changes after upgrading an OS can maybe take an additional 50% >> of space (just guessing). Btw, ashift=12. >> Still can't see how that would amount to 4 GiB, but it's possible. >> > > Disconcerting. Is this something that others are likely to hit? Should > accounting for writes fail with ENOSPC a bit earlier so that we never > reach a state like this? I.e. non-metadata writes will fail at a lower > threshold than data or if that is already the case, reduce the > threshold further. I thought I remembered seeing some recent changes in this area, but I can't find them ATM. Something to raise on the openzfs list. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 08:51:07 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A1AB1368 for ; Tue, 14 Oct 2014 08:51:07 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 514D7C44 for ; Tue, 14 Oct 2014 08:51:05 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id E6594114CB; Tue, 14 Oct 2014 08:50:56 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id 8d1ZqN_Ky9rh; Tue, 14 Oct 2014 08:50:54 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id 18DC5114BF; Tue, 14 Oct 2014 08:50:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413276654; bh=+pBeH0TWA5uHK2AJKRQDi14+ZMnpiwoYx4gmvmCzrow=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=IgtNT9Z/wKJ5MFnQsDYqFVZ77HVSyISCYoVTftUPPrLUe4bbivtIF33MeuiuFBhdw 3i906izvUJmGQerMGFK/W6+ZQM6wji3gdL+Tjxq8s+UtoXiym8CjSjhf/lNTws73Rj 3sQCdbf42jkVYVEo0VWSIXlh5qXp4mJy2iLpzVr4= Mime-Version: 1.0 Date: Tue, 14 Oct 2014 08:50:53 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable Message-ID: <726222de616461ce67f35e77dfaac5fe@mail.unix-experience.fr> X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: NFSv4 nobody issue To: "Rick Macklem" In-Reply-To: <1003039765.63581639.1413228280410.JavaMail.root@uoguelph.ca> References: <1003039765.63581639.1413228280410.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:51:07 -0000 Hi Rick,=0Athanks for your tip. It works perfect.=0AI think creating a sy= sctl variable must be fine to handle this precise case, no ?=0A=0AI'll lo= ok at a patch today.=0A=0ARegards,=0A=0ALo=C3=AFc Blot,=0AUNIX Systems, N= etwork and Security Engineer=0Ahttp://www.unix-experience.fr=0A=0A13 octo= bre 2014 21:24 "Rick Macklem" a =C3=A9crit: =0A> L= oic Blot wrote:=0A> =0A>> Hi Rick,=0A>> no request is done.=0A>> In /var/= log/messages on the client i have:=0A>> =0A>> Oct 13 15:10:46 machine ker= nel: No name and/or group mapping for=0A>> uid,gid:(65534,-1)=0A>> =0A>> = The FreeBSD kernel refuses to change the owner.=0A> =0A> Ok, I took a loo= k and it is a restriction enforced by the server.=0A> If you want it to w= ork, you need to comment out these lines in=0A> sys/fs/nfsserver/nfs_nfsd= subs.c:=0A> if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D nfsrv_defau= ltuid)=0A> 1547 || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D nfsrv_de= faultgid)) {=0A> 1548 error =3D NFSERR_BADOWNER;=0A> 1549 goto out;=0A> 1= 550 }=0A> (Line#s 1546->1550 in head.)=0A> =0A> It is done because some c= lients try to set the owner when there is no=0A> valid mapping by sending= "nobody@" to the server.=0A> Unfortunately for you "nob= ody" is the traditional name for "no mapping".=0A> For example, if "chown= rick " was done on a client where "rick"=0A> is not in the client'= s passwd database, some clients will send "nobody@"=0A> = and the above code makes sure that doesn't work.=0A> =0A> So, if you want= this to work, comment out the above lines in your NFSv4 server's=0A> ker= nel.=0A> =0A> rick=0A> =0A>> Regards,=0A>> =0A>> Lo=C3=AFc Blot,=0A>> UNI= X Systems, Network and Security Engineer=0A>> http://www.unix-experience.= fr=0A>> =0A>> 13 octobre 2014 14:43 "Rick Macklem" = a =C3=A9crit:=0A>>> Loic Blot wrote:=0A>>> =0A>>>> Hi,=0A>>>> i tryed so= me other things=0A>>>> =0A>>>> User nobody (65534)=0A>>>> -> chown nobody= /usr/jail/test.file =3D> problem=0A>>>> =0A>>>> Group nogroup (65533)=0A= >>>> -> chown :nogroup /usr/jail/test.file =3D> same problem=0A>>>> =0A>>= >> Group nobody (65534)=0A>>>> -> chown :nobody /usr/jail/test.file =3D> = no problem=0A>>>> =0A>>>> Change user nobody UID from 65534 to 65533 =3D>= same problem. It's=0A>>>> not=0A>>>> a UID number problem but a name pro= blem.=0A>>> =0A>>> Yes, for NFSv4 it is the names that go in the RPC requ= est and not=0A>>> the=0A>>> numbers. However, since there are the numbers= in the AUTH_SYS=0A>>> credential=0A>>> in the header (unless you are usi= ng Kerberized mounts), the numbers=0A>>> for=0A>>> the names need to be c= onsistent between client and server.=0A>>> =0A>>>> Then, user nobody and = group nogroup (not the integer values) are=0A>>>> problematic. I looked a= t nfsuserd.c and i see:=0A>>>> u_char *defaultuser =3D "nobody";=0A>>>> u= _char *defaultgroup =3D "nogroup";=0A>>> =0A>>> These are used if no mapp= ing is found in the user or group database=0A>>> for whatever name is in = the RPC on the wire.=0A>>> =0A>>> If you want to see what is happening, I= suggest that you capture=0A>>> packets when you do the "chown" (You can = use "tcpdump -s 0 -w=0A>>> file.pcap host XXX".)=0A>>> then look at them = in wireshark.=0A>>> In wireshark, look for the Setattr RPC and then look = in the setable=0A>>> attributes.=0A>>> You should find Owner which looks = like "nobody@=0A>>> and=0A>>> Owner_group which looks th= e same (or "nogroup@" if=0A>>> you=0A>>> used nogroup). = "nogroup" must be in your group database (/etc/group=0A>>> or whatever=0A= >>> you use for a group database) and the number must be consistent=0A>>>= across client=0A>>> and server.=0A>>> Also, see what the reply to the Se= tattr RPC is (it is actually a=0A>>> Compound RPC=0A>>> labelled "Setattr= " for NFSv4).=0A>>> =0A>>> If there is no Setattr RPC, then the mapping i= s failing in the=0A>>> client.=0A>>> =0A>>> If the stuff looks correct on= the wire, then it is most likely a=0A>>> server side=0A>>> issue.=0A>>> = =0A>>> rick=0A>>> =0A>>>> I think it's related.=0A>>>> =0A>>>> Regards,= =0A>>>> =0A>>>> Lo=C3=AFc Blot,=0A>>>> UNIX Systems, Network and Security= Engineer=0A>>>> http://www.unix-experience.fr=0A>>>> =0A>>>> 13 octobre = 2014 09:15 "Lo=C3=AFc Blot" a=0A>>>> =C3= =A9crit:=0A>>>>> Hi,=0A>>>>> of course i have it. On each node:=0A>>>>> = =0A>>>>> # cat /etc/master.passwd | grep nobody=0A>>>>> returns:=0A>>>>> = nobody:*:65534:65534::0:0:Unprivileged=0A>>>>> user:/nonexistent:/usr/sbi= n/nologin=0A>>>>> =0A>>>>> It's why i do a report here :)=0A>>>>> =0A>>>>= > Regards,=0A>>>>> =0A>>>>> Lo=C3=AFc Blot,=0A>>>>> UNIX Systems, Network= and Security Engineer=0A>>>>> http://www.unix-experience.fr=0A>>>>> =0A>= >>>> 10 octobre 2014 13:51 "Rick Macklem" a=0A>>>>= > =C3=A9crit:=0A>>>>> =0A>>>>>> Loic Blot wrote:=0A>>>>>> =0A>>>>>>> Hell= o @freebsd-fs,=0A>>>>>>> i'm trying to do jail hosting over NFSv4 with ez= jail and i'm=0A>>>>>>> experimenting an issue that i can't resolve. When = i extract=0A>>>>>>> base.txz (with ezjail) or i set nobody user on a file= , i have=0A>>>>>>> this=0A>>>>>>> error:=0A>>>>>>> =0A>>>>>>> chown nobod= y:nobody /usr/jails/fulljail/mnt/=0A>>>>>>> No name and/or group mapping = for uid,gid:(65534,65534)=0A>>>>>>> chown: /usr/jails/fulljail/mnt/: Oper= ation not permitted=0A>>>>>>> =0A>>>>>>> No problem if i set:=0A>>>>>>> c= hown mysql:nobody /usr/jails/fulljail/mnt/=0A>>>>>>> =0A>>>>>>> Problem a= ppears on all files.=0A>>>>>> =0A>>>>>> Do you have a user by the name of= "nobody" in your password=0A>>>>>> database?=0A>>>>>> (NFSv4 uses names = and not numbers on the wire, so no name-->no=0A>>>>>> mapping=0A>>>>>> an= d chown can't be done.)=0A>>>>>> =0A>>>>>> rick=0A>>>>>> =0A>>>>>>> On my= ZFS+NFSv4 server i do a dataset, exported in NFS=0A>>>>>>> =0A>>>>>>> /e= tc/exports:=0A>>>>>>> V4: /=0A>>>>>>> =0A>>>>>>> zfs get sharenfs pool/ja= ils:=0A>>>>>>> -network=3D10.99.99.0 -mask=3D255.255.255.0 -maproot=3Droo= t=0A>>>>>>> =0A>>>>>>> nfsuserd and nfsv4_server_enable=3DYES on both cli= ent and server,=0A>>>>>>> plus=0A>>>>>>> nfsbcd on client.=0A>>>>>>> =0A>= >>>>>> On the client here is the fstab entry=0A>>>>>>> 10.99.99.99:/pool/= jails /usr/jails nfs rw,nfsv4 0 0=0A>>>>>>> =0A>>>>>>> What i'm doing wro= ng ?=0A>>>>>>> =0A>>>>>>> Thanks in advance=0A>>>>>>> Regards,=0A>>>>>>> = =0A>>>>>>> Lo=C3=AFc Blot,=0A>>>>>>> UNIX Systems, Network and Security E= ngineer=0A>>>>>>> http://www.unix-experience.fr=0A>>>>>>> =0A>>>> =0A>> _= ______________________________=0A>> =0A>>>> =0A>>>>>>> =0A>>>>>>> freebsd= -fs@freebsd.org mailing list=0A>>>>>>> http://lists.freebsd.org/mailman/l= istinfo/freebsd-fs=0A>>>>>>> To unsubscribe, send any mail to=0A>>>>>>> "= freebsd-fs-unsubscribe@freebsd.org"=0A>>>>> =0A>>>>> =0A>>>> =0A>> ______= _________________________=0A>> =0A>>>> =0A>>>>> =0A>>>>> freebsd-fs@freeb= sd.org mailing list=0A>>>>> http://lists.freebsd.org/mailman/listinfo/fre= ebsd-fs=0A>>>>> To unsubscribe, send any mail to=0A>>>>> "freebsd-fs-unsu= bscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 09:39:32 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D30A6FE4 for ; Tue, 14 Oct 2014 09:39:32 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D3B62B for ; Tue, 14 Oct 2014 09:39:32 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id D5EF93151D7; Tue, 14 Oct 2014 11:39:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:content-type:content-transfer-encoding; s=20140709; bh=rRL5MGnnn40A+JCPkgZC4S6Tl5E=; b=YUfMWqdj/m8pzPizl+f44RAYYVeVnlpdMem5jjp9hhMpkvTYidTPDBLnKihyzFR2t4zYYtJui7hzcovBmC9ylUYA8RIo2YNAA0K0Vxozk8lEA4dmT13Rr6gm4z1uZm8gei8Vx9dY6ccY30NZ+dDmQITjGfjlHTlFNQQO3Xd7pYA= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id 9DD263151CA for ; Tue, 14 Oct 2014 11:39:23 +0200 (CEST) Received: (qmail 91151 invoked by uid 89); 14 Oct 2014 11:39:23 +0200 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with CAMELLIA256-SHA encrypted SMTP; 14 Oct 2014 11:39:23 +0200 Message-ID: <543CEF4A.1010908@bytecamp.net> Date: Tue, 14 Oct 2014 11:39:22 +0200 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130330 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: using newnfs client w/ oldnfs server Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 09:39:32 -0000 Hi, I just wonder if there are any issues on using the current (new-)NFS implementation from 10.X to connect to an oldnfs server. with kind regards, Robert Schulze From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 10:09:39 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 562A0971 for ; Tue, 14 Oct 2014 10:09:39 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) by mx1.freebsd.org (Postfix) with ESMTP id 120E13F1 for ; Tue, 14 Oct 2014 10:09:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id 156144C4CDCA; Tue, 14 Oct 2014 12:01:48 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gVrl4tgKuwbW; Tue, 14 Oct 2014 12:01:45 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id A0BE64C4C20B; Tue, 14 Oct 2014 12:01:45 +0200 (CEST) Message-ID: <543CF47B.4000101@internetx.com> Date: Tue, 14 Oct 2014 12:01:31 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Robert Schulze , freebsd-fs@freebsd.org Subject: Re: using newnfs client w/ oldnfs server References: <543CEF4A.1010908@bytecamp.net> In-Reply-To: <543CEF4A.1010908@bytecamp.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 10:09:39 -0000 Am 14.10.2014 um 11:39 schrieb Robert Schulze: > Hi, > > I just wonder if there are any issues on using the current (new-)NFS > implementation from 10.X to connect to an oldnfs server. works fine for me > > with kind regards, > Robert Schulze > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 10:34:41 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00021E79 for ; Tue, 14 Oct 2014 10:34:40 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B02A0890 for ; Tue, 14 Oct 2014 10:34:39 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id E03D91161E for ; Tue, 14 Oct 2014 10:34:36 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id oonoMJpRiDFF for ; Tue, 14 Oct 2014 10:34:35 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id BF57E1160A for ; Tue, 14 Oct 2014 10:34:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413282874; bh=MWu5laHJU3Kd0lzGdSm1ExFKPEwQtU2Zpn/62uCC38k=; h=Date:From:Subject:To; b=MprcwFg4aEqjdDHks+J9FeTXkslwvyj4VmIbhvjy6rVm7lZqj3JAEtqAB0tSaMmcM /pq0+61gXELhcX5B49NnzYpYDzYAthZlOPFcT9+yq7Q2nY+NcBVT1Rs7kQ+rCz5K9+ CT9gY01Upq6yzQ9Fc3JudQkpcCBVMSRiznCZSP94= Mime-Version: 1.0 Date: Tue, 14 Oct 2014 10:34:34 +0000 Message-ID: X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: [PATCH] disable nfsd (NFSv4) nobody/nogroup check To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 10:34:41 -0000 Hi,=0A since a recent problem (see thread NFSv4 nobody issue), i think we= need a sysctl variable to disable nobody and nogroup check into the kern= el (default enabled)=0A This variable is useful in some situations, like = TFTP over NFS, jails over NFS (some files like /var/db/locate.database ne= ed nobody user).=0A=0A I added vfs.nfsd.disable_nobodycheck and vfs.nfsd.= disable_nogroupcheck to modify NFSv4 nobody/nogroup check.=0A=0A Thanks t= o Rick to tell me where the problem was.=0A=0A Can you review the patch, = and add it to kernel to avoid previous mentionned issue.=0A=0A Here is my= patch:=0A=0A --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig=C2=A0=C2=A0 =C2=A0= 2014-10-14 12:03:50.163311506 +0200=0A +++ sys/fs/nfsserver/nfs_nfsdsubs.= c=C2=A0=C2=A0 =C2=A02014-10-14 12:06:29.793304755 +0200=0A @@ -62,9 +62,1= 8 @@=0A =C2=A0SYSCTL_DECL(_vfs_nfsd);=0A =C2=A0=0A =C2=A0static int=C2=A0= =C2=A0 =C2=A0disable_checkutf8 =3D 0;=0A +static int=C2=A0=C2=A0 =C2=A0di= sable_nobodycheck =3D 0;=0A +static int=C2=A0=C2=A0 =C2=A0disable_nogroup= check =3D 0;=0A =C2=A0SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, = CTLFLAG_RW,=0A =C2=A0=C2=A0=C2=A0=C2=A0 &disable_checkutf8, 0,=0A =C2=A0= =C2=A0=C2=A0=C2=A0 "Disable the NFSv4 check for a UTF8 compliant name");= =0A +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW,=0A = +=C2=A0=C2=A0=C2=A0 &disable_nobodycheck, 0,=0A +=C2=A0=C2=A0=C2=A0 "Disa= ble the NFSv4 check when setting user nobody as owner");=0A +SYSCTL_INT(_= vfs_nfsd, OID_AUTO, disable_nogroupcheck, CTLFLAG_RW,=0A +=C2=A0=C2=A0=C2= =A0 &disable_nogroupcheck, 0,=0A +=C2=A0=C2=A0=C2=A0 "Disable the NFSv4 c= heck when setting group nogroup as owner");=0A +=0A =C2=A0=0A =C2=A0stati= c char nfsrv_hexdigit(char, int *);=0A =C2=A0=0A @@ -1543,8 +1552,8 @@=0A= =C2=A0=C2=A0=C2=A0 =C2=A0 */=0A =C2=A0=C2=A0=C2=A0 =C2=A0if (NFSVNO_NOTS= ETUID(nvap) && NFSVNO_NOTSETGID(nvap))=0A =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0goto out;=0A -=C2=A0=C2=A0 =C2=A0if ((NFSVNO_ISSETUID(nvap) = && nvap->na_uid =3D=3D nfsrv_defaultuid)=0A -=C2=A0=C2=A0 =C2=A0=C2=A0=C2= =A0=C2=A0 || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D nfsrv_defaultg= id)) {=0A +=C2=A0=C2=A0 =C2=A0if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid = =3D=3D nfsrv_defaultuid && disable_nobodycheck =3D=3D 0)=0A +=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D= =3D nfsrv_defaultgid && disable_nogroupcheck =3D=3D 0)) {=0A =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0error =3D NFSERR_BADOWNER;=0A =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0goto out;=0A =C2=A0=C2=A0=C2=A0 =C2=A0= }=0A Regards,=0A=0A Lo=C3=AFc Blot,=0A UNIX Systems, Network and Security= Engineer=0A http://www.unix-experience.fr From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 10:46:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5A7F1E8 for ; Tue, 14 Oct 2014 10:46:27 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 735D099F for ; Tue, 14 Oct 2014 10:46:27 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id hi2so11304361wib.5 for ; Tue, 14 Oct 2014 03:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=DcCIu03I0j9R4RIPg5Hf50Sp6XHUZW4JleEUlde+08o=; b=pUiV+mXjdd+jOl40pA3vZZvrSgwARzOGZ1R/A2btiOzcEWsxe7oHolqKoO95cr+7Rs 33vkDVtlnVycDBWbIUPVIvBq8QIB74uKe/m/DH5/m53j4COz3sFU7JoA0AWiSfruynxE V818lefRtMja5kqjl+eF39yqwLr4ORNuTWAeMP0NOtizyIF5YaYGsvZqk5KsFl1KAghI kuZpA6lf5m0gHcFU7VRxZSzaA32RtUXfe8U6PQBupvafpAlo0xiRGvP8c5dO30Si14i9 TOeCAodK9a35Z4GdoUCoGOXIDIYZ3PdTJq2yeAsLbixwby61SpY4I8zwN1VaShRVkwfw vFqg== MIME-Version: 1.0 X-Received: by 10.180.88.162 with SMTP id bh2mr4550997wib.77.1413283585514; Tue, 14 Oct 2014 03:46:25 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Tue, 14 Oct 2014 03:46:25 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: References: Date: Tue, 14 Oct 2014 18:46:25 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: =?UTF-8?Q?Lo=C3=AFc_Blot?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 10:46:28 -0000 Hello Blot, The patch looks reasonable. As per the email thread, seems a good approach to overcome this issue, at least for now. If Rick has no objection and no free time, I can commit the patch during this week. Best Regards, 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot : > Hi, > since a recent problem (see thread NFSv4 nobody issue), i think we need = a > sysctl variable to disable nobody and nogroup check into the kernel > (default enabled) > This variable is useful in some situations, like TFTP over NFS, jails > over NFS (some files like /var/db/locate.database need nobody user). > > I added vfs.nfsd.disable_nobodycheck and vfs.nfsd.disable_nogroupcheck t= o > modify NFSv4 nobody/nogroup check. > > Thanks to Rick to tell me where the problem was. > > Can you review the patch, and add it to kernel to avoid previous > mentionned issue. > > Here is my patch: > > --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 12:03:50.16331150= 6 > +0200 > +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 12:06:29.793304755 +02= 00 > @@ -62,9 +62,18 @@ > SYSCTL_DECL(_vfs_nfsd); > > static int disable_checkutf8 =3D 0; > +static int disable_nobodycheck =3D 0; > +static int disable_nogroupcheck =3D 0; > SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, > &disable_checkutf8, 0, > "Disable the NFSv4 check for a UTF8 compliant name"); > +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, > + &disable_nobodycheck, 0, > + "Disable the NFSv4 check when setting user nobody as owner"); > +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, CTLFLAG_RW, > + &disable_nogroupcheck, 0, > + "Disable the NFSv4 check when setting group nogroup as owner"); > + > > static char nfsrv_hexdigit(char, int *); > > @@ -1543,8 +1552,8 @@ > */ > if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) > goto out; > - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D nfsrv_defaultuid) > - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D nfsrv_defaultg= id)) { > + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D nfsrv_defaultuid = && > disable_nobodycheck =3D=3D 0) > + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D nfsrv_defaultg= id && > disable_nogroupcheck =3D=3D 0)) { > error =3D NFSERR_BADOWNER; > goto out; > } > Regards, > > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 11:20:06 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0EEEACF5; Tue, 14 Oct 2014 11:20:06 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 775F7C84; Tue, 14 Oct 2014 11:20:04 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 99E9F20E70942; Tue, 14 Oct 2014 11:20:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id E634D20E70940; Tue, 14 Oct 2014 11:20:00 +0000 (UTC) Message-ID: <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 12:19:58 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable , mark X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:20:06 -0000 ----- Original Message ----- From: "Steven Hartland" To: "K. Macy" Cc: "freebsd-fs@FreeBSD.org" ; "mark" ; "FreeBSD Stable" Sent: Tuesday, October 14, 2014 9:14 AM Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] > ----- Original Message ----- > From: "K. Macy" > > >>>> Thank you both for analysis and effort! >>>> >>>> I can't rule out the possibility that my main system pool >>>> on a SSD was low on space at some point in time, but the >>>> three 4 GiB cloned pools (sys1boot and its brothers) were all >>>> created as a zfs send / receive copies of the main / (root) >>>> file system and I haven't noticed anything unusual during >>>> syncing. This syncing was done manually (using zxfer) and >>>> independently from the upgrade on the system - on a steady/quiet >>>> system, when the source file system definitely had sufficient >>>> free space. >>>> >>>> The source file system now shows 1.2 GiB of usage shown >>>> by df: >>>> shiny/ROOT 61758388 1271620 60486768 2% / >>>> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >>>> on a cloned filesystem. >>>> >>>> Will try to import the main two pools after re-creating >>>> a sane boot pool... >>> >>> >>> Yer zfs list only shows around 2-3GB used too but zpool list >>> shows the pool is out of space. Cant rule out an accounting >>> issue though. >>> >> >> What is using the extra space in the pool? Is there an unmounted >> dataset or snapshot? Do you know how to easily tell? Unlike txg and >> zio processing I don't have the luxury of having just read that part >> of the codebase. > > Its not clear but I believe it could just be fragmention even though > its ashift=9. > > I sent the last snapshot to another pool of the same size and it > resulted in: > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT > sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - > sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - > > I believe FRAG is 0% as the feature wasn't enabled for the lifetime of > the pool hence its simply not showing a valid value. > > zfs list -t all -r sys1boot > NAME USED AVAIL REFER MOUNTPOINT > sys1boot 1.76G 2.08G 11K /sys1boot > sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT > sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - > sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - .. Well interesting issue I left this pool alone this morning literally doing nothing, and its now out of space. zpool list NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - There's something very wrong here as nothing has been accessing the pool. pool: zfs state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://illumos.org/msg/ZFS-8000-HC scan: none requested config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 2 0 md1 ONLINE 0 0 0 I tried destroying the pool and ever that failed, presumably because the pool has suspended IO. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 11:27:00 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F1B4235; Tue, 14 Oct 2014 11:27:00 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id ED1FCD89; Tue, 14 Oct 2014 11:26:59 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jHDx56fGxz1Dh; Tue, 14 Oct 2014 13:26:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413286005; x=1415878006; bh=JAltATHwlSD2F6q4NoNW9s7EL aNvB/11Qu7cvsHFvzE=; b=eReAxFzeG9+BpS7T/Ll5GYTK2ejDQwafFC51KiYlX CFmlYC8EyIPIeWvA+hqDzmau5S2q2J1lCQy8mZvwIsnMw3MCy6roV/UIMLGFjED4 YW/ErPxn7ZhszP4FCAXNTF91W8QlQVW0jKER/1uuhtBQYeEYUQV9U9S9LI9vTj2l yc= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id UUa2MEOZHAAm; Tue, 14 Oct 2014 13:26:45 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Tue, 14 Oct 2014 13:26:45 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jHDx128FMzBt; Tue, 14 Oct 2014 13:26:45 +0200 (CEST) Message-ID: <543D0874.9080809@ijs.si> Date: Tue, 14 Oct 2014 13:26:44 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:27:00 -0000 On 10/14/2014 10:14, Steven Hartland wrote: >>> Yer zfs list only shows around 2-3GB used too but zpool list >>> shows the pool is out of space. Cant rule out an accounting >>> issue though. >> >> What is using the extra space in the pool? Is there an unmounted >> dataset or snapshot? Do you know how to easily tell? Unlike txg and >> zio processing I don't have the luxury of having just read that part >> of the codebase. > > Its not clear but I believe it could just be fragmention even though > its ashift=9. > > I sent the last snapshot to another pool of the same size and it > resulted in: > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH > ALTROOT > sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - > sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - Yes, that's it! Fragmentation. > I believe FRAG is 0% as the feature wasn't enabled for the lifetime of > the pool hence its simply not showing a valid value. Indeed. The pool has a long lifetime and the feature was only recently made available. > zfs list -t all -r sys1boot > NAME USED AVAIL REFER MOUNTPOINT > sys1boot 1.76G 2.08G 11K /sys1boot > sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT > sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - > sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - > sys1boot/ROOT@auto-2014-08-19_04.00 1K - 1.19G - > [...] > sys1boot/ROOT@auto-2014-09-19_22.20 1K - 1.20G - > sys1boot/ROOT@auto-2014-09-19_22.30 0 - 1.20G - So snapshots were not consuming much, it was fragmentation. Thanks! Mark From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 11:30:33 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A3F196B2; Tue, 14 Oct 2014 11:30:33 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 2FF25DDF; Tue, 14 Oct 2014 11:30:33 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jHF1N4138z1Gc; Tue, 14 Oct 2014 13:30:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413286227; x=1415878228; bh=zLHNxpSja50NlONXqgXlIWA5a gJI6zlDpAKAum9KRSE=; b=HAYylhABxzW6VWCfGn9b6LoSwYcUJNfTmPn3i0nX4 ePT8QdCCpmpBbtHW5HkdogcMjcwkrWljoxn7+9/KfzAvMKsaB6iWurKO1n00vaDJ 0AhgKSz59CUml2BZhnVAMaM7hAeXBtmO2JkHa38y6lUhc7KmzUiKauTLoEUYEuZM vk= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id YIifCGDKsbao; Tue, 14 Oct 2014 13:30:27 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Tue, 14 Oct 2014 13:30:27 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jHF1H48xzzDy; Tue, 14 Oct 2014 13:30:27 +0200 (CEST) Message-ID: <543D0953.1070604@ijs.si> Date: Tue, 14 Oct 2014 13:30:27 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> In-Reply-To: <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:30:33 -0000 On 10/14/2014 13:19, Steven Hartland wrote: > Well interesting issue I left this pool alone this morning literally doing > nothing, and its now out of space. > zpool list > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH > ALTROOT > sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - > sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - > > There's something very wrong here as nothing has been accessing the pool. > > pool: zfs > state: ONLINE > status: One or more devices are faulted in response to IO failures. > action: Make sure the affected devices are connected, then run 'zpool > clear'. > see: http://illumos.org/msg/ZFS-8000-HC > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > zfs ONLINE 0 2 0 > md1 ONLINE 0 0 0 > > I tried destroying the pool and ever that failed, presumably because > the pool has suspended IO. That's exactly how trouble started here. Got the "One or more devices are faulted in response to IO failures" on all three small cloned boot pools one day, out of the blue. There was no activity there, except for periodic snapshoting every 10 minutes. Mark From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 11:40:51 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07ED7A4D; Tue, 14 Oct 2014 11:40:51 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id B5FF3F4F; Tue, 14 Oct 2014 11:40:50 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 980F420E70953; Tue, 14 Oct 2014 11:40:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 0729020E70942; Tue, 14 Oct 2014 11:40:47 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 12:40:45 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:40:51 -0000 ----- Original Message ----- From: "Mark Martinec" > On 10/14/2014 13:19, Steven Hartland wrote: >> Well interesting issue I left this pool alone this morning literally doing >> nothing, and its now out of space. >> zpool list >> NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH >> ALTROOT >> sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - >> sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - >> >> There's something very wrong here as nothing has been accessing the pool. >> >> pool: zfs >> state: ONLINE >> status: One or more devices are faulted in response to IO failures. >> action: Make sure the affected devices are connected, then run 'zpool >> clear'. >> see: http://illumos.org/msg/ZFS-8000-HC >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> zfs ONLINE 0 2 0 >> md1 ONLINE 0 0 0 >> >> I tried destroying the pool and ever that failed, presumably because >> the pool has suspended IO. > > That's exactly how trouble started here. Got the > "One or more devices are faulted in response to IO failures" > on all three small cloned boot pools one day, out of the blue. > There was no activity there, except for periodic snapshoting > every 10 minutes. Yer this isn't fragmentation, this is something else. I've started a thread on the openzfs list to discuss this as theres something quite odd going on. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 12:01:53 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 880AD15D; Tue, 14 Oct 2014 12:01:53 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1E904206; Tue, 14 Oct 2014 12:01:52 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqoEANAPPVSDaFve/2dsb2JhbABbg2FYBIMCySIKhnlUAoErAX2EAgEBAQMBAQEBIAQnIAsbGAICDRkCKQEJJg4HBAEIFASIFQgNsH2VEgEBAQEGAQEBAQEdgSyONxACAQEaNAeCNkESgUIFljuEDIRulCeEEyEvB4FBgQIBAQE X-IronPort-AV: E=Sophos;i="5.04,716,1406606400"; d="scan'208";a="161074542" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 14 Oct 2014 08:01:50 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 47944B403F; Tue, 14 Oct 2014 08:01:50 -0400 (EDT) Date: Tue, 14 Oct 2014 08:01:50 -0400 (EDT) From: Rick Macklem To: araujo@FreeBSD.org Message-ID: <986887451.63845723.1413288110282.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 12:01:53 -0000 Marcelo Araujo wrote: > Hello Blot, >=20 > The patch looks reasonable. > As per the email thread, seems a good approach to overcome this > issue, at > least for now. >=20 > If Rick has no objection and no free time, I can commit the patch > during > this week. >=20 > Best Regards, >=20 > 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot : >=20 > > Hi, > > since a recent problem (see thread NFSv4 nobody issue), i think we > > need a > > sysctl variable to disable nobody and nogroup check into the kernel > > (default enabled) > > This variable is useful in some situations, like TFTP over NFS, > > jails > > over NFS (some files like /var/db/locate.database need nobody > > user). > > > > I added vfs.nfsd.disable_nobodycheck and > > vfs.nfsd.disable_nogroupcheck to > > modify NFSv4 nobody/nogroup check. > > > > Thanks to Rick to tell me where the problem was. > > > > Can you review the patch, and add it to kernel to avoid previous > > mentionned issue. > > > > Here is my patch: > > > > --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 > > 12:03:50.163311506 > > +0200 > > +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 > > 12:06:29.793304755 +0200 > > @@ -62,9 +62,18 @@ > > SYSCTL_DECL(_vfs_nfsd); > > > > static int disable_checkutf8 =3D 0; > > +static int disable_nobodycheck =3D 0; > > +static int disable_nogroupcheck =3D 0; > > SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, > > &disable_checkutf8, 0, > > "Disable the NFSv4 check for a UTF8 compliant name"); > > +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, > > + &disable_nobodycheck, 0, > > + "Disable the NFSv4 check when setting user nobody as owner"); > > +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, CTLFLAG_RW, > > + &disable_nogroupcheck, 0, > > + "Disable the NFSv4 check when setting group nogroup as > > owner"); > > + > > Patch looks fine to me. Marcelo, you can commit this if you'd like. Otherwise I'll do it. Sorry it took a while for me to remember this was disabled. (My only excuse is I wrote it about 10years ago;-) rick > > static char nfsrv_hexdigit(char, int *); > > > > @@ -1543,8 +1552,8 @@ > > */ > > if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) > > goto out; > > - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > nfsrv_defaultuid) > > - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > nfsrv_defaultgid)) { > > + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > nfsrv_defaultuid && > > disable_nobodycheck =3D=3D 0) > > + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > nfsrv_defaultgid && > > disable_nogroupcheck =3D=3D 0)) { > > error =3D NFSERR_BADOWNER; > > goto out; > > } > > Regards, > > > > Lo=C3=AFc Blot, > > UNIX Systems, Network and Security Engineer > > http://www.unix-experience.fr > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to > > "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 >=20 > -- >=20 > -- > Marcelo Araujo (__)araujo@FreeBSD.org > \\\'',)http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 12:04:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B9F8920B; Tue, 14 Oct 2014 12:04:12 +0000 (UTC) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7900022B; Tue, 14 Oct 2014 12:04:12 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1Xe0pg-0003Pq-1M; Tue, 14 Oct 2014 14:04:09 +0200 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: =?iso-8859-15?Q?Lo=EFc_Blot?= , "Marcelo Araujo" , araujo@freebsd.org Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check References: Date: Tue, 14 Oct 2014 14:04:02 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.17 (Win32) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: -- X-Spam-Score: -2.9 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED, BAYES_00 autolearn=disabled version=3.3.1 X-Scan-Signature: 2d0a7f6a049cc125cd28f2ceffdc0173 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 12:04:12 -0000 I thought it is advised to make settings positively defined. So not use = = 'disable =3D 1', but 'enable =3D 0'. Ronald. On Tue, 14 Oct 2014 12:46:25 +0200, Marcelo Araujo = wrote: > Hello Blot, > > The patch looks reasonable. > As per the email thread, seems a good approach to overcome this issue,= at > least for now. > > If Rick has no objection and no free time, I can commit the patch duri= ng > this week. > > Best Regards, > > 2014-10-14 18:34 GMT+08:00 Lo=EFc Blot := > >> Hi, >> since a recent problem (see thread NFSv4 nobody issue), i think we = >> need a >> sysctl variable to disable nobody and nogroup check into the kernel >> (default enabled) >> This variable is useful in some situations, like TFTP over NFS, jail= s >> over NFS (some files like /var/db/locate.database need nobody user). >> >> I added vfs.nfsd.disable_nobodycheck and vfs.nfsd.disable_nogroupche= ck = >> to >> modify NFSv4 nobody/nogroup check. >> >> Thanks to Rick to tell me where the problem was. >> >> Can you review the patch, and add it to kernel to avoid previous >> mentionned issue. >> >> Here is my patch: >> >> --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 = >> 12:03:50.163311506 >> +0200 >> +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 12:06:29.793304755= = >> +0200 >> @@ -62,9 +62,18 @@ >> SYSCTL_DECL(_vfs_nfsd); >> >> static int disable_checkutf8 =3D 0; >> +static int disable_nobodycheck =3D 0; >> +static int disable_nogroupcheck =3D 0; >> SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, >> &disable_checkutf8, 0, >> "Disable the NFSv4 check for a UTF8 compliant name"); >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, >> + &disable_nobodycheck, 0, >> + "Disable the NFSv4 check when setting user nobody as owner"); >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, CTLFLAG_RW, >> + &disable_nogroupcheck, 0, >> + "Disable the NFSv4 check when setting group nogroup as owner");= >> + >> >> static char nfsrv_hexdigit(char, int *); >> >> @@ -1543,8 +1552,8 @@ >> */ >> if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) >> goto out; >> - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D nfsrv_default= uid) >> - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D = >> nfsrv_defaultgid)) { >> + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D nfsrv_default= uid && >> disable_nobodycheck =3D=3D 0) >> + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D nfsrv_defa= ultgid = >> && >> disable_nogroupcheck =3D=3D 0)) { >> error =3D NFSERR_BADOWNER; >> goto out; >> } >> Regards, >> >> Lo=EFc Blot, >> UNIX Systems, Network and Security Engineer >> http://www.unix-experience.fr >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= > > > From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 12:09:41 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D43F3B5; Tue, 14 Oct 2014 12:09:41 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id C4174269; Tue, 14 Oct 2014 12:09:40 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqsEADkRPVSDaFve/2dsb2JhbABbg2FYBIMCySIKhnlUAoErAX2EAgEBAQMBAQEBIAQnIAsFFhgCAg0ZAiMGAQkmBggHBAEcBIgJAwkIDbB7jlcNhi4BAQEBAQUBAQEBAQEcgSyMZ4FQEAIBGzQHgjZBEoFCBZY7hAxzg3uNU4ZUhBMhLweBQYECAQEB X-IronPort-AV: E=Sophos;i="5.04,716,1406606400"; d="scan'208";a="159872225" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 14 Oct 2014 08:09:34 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 00F0DB4035; Tue, 14 Oct 2014 08:09:34 -0400 (EDT) Date: Tue, 14 Oct 2014 08:09:33 -0400 (EDT) From: Rick Macklem To: Ronald Klop Message-ID: <2111556765.63849821.1413288573994.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org, Marcelo Araujo X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 12:09:41 -0000 Ronald Klop wrote: > I thought it is advised to make settings positively defined. So not > use > 'disable =3D 1', but 'enable =3D 0'. >=20 For the case of disable_utf8, I made it negative, since disabling the check violates RFC-3530. For these checks, there isn't anything in the RFC requiring the check AFAIK, so I personally don't care which way they are done. (If the default is disabling the check that could be a minor POLA violation.) So, you guys choose whichever you prefer to commit, rick > Ronald. >=20 >=20 > On Tue, 14 Oct 2014 12:46:25 +0200, Marcelo Araujo > wrote: >=20 > > Hello Blot, > > > > The patch looks reasonable. > > As per the email thread, seems a good approach to overcome this > > issue, at > > least for now. > > > > If Rick has no objection and no free time, I can commit the patch > > during > > this week. > > > > Best Regards, > > > > 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot > > : > > > >> Hi, > >> since a recent problem (see thread NFSv4 nobody issue), i think > >> we > >> need a > >> sysctl variable to disable nobody and nogroup check into the > >> kernel > >> (default enabled) > >> This variable is useful in some situations, like TFTP over NFS, > >> jails > >> over NFS (some files like /var/db/locate.database need nobody > >> user). > >> > >> I added vfs.nfsd.disable_nobodycheck and > >> vfs.nfsd.disable_nogroupcheck > >> to > >> modify NFSv4 nobody/nogroup check. > >> > >> Thanks to Rick to tell me where the problem was. > >> > >> Can you review the patch, and add it to kernel to avoid previous > >> mentionned issue. > >> > >> Here is my patch: > >> > >> --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 > >> 12:03:50.163311506 > >> +0200 > >> +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 > >> 12:06:29.793304755 > >> +0200 > >> @@ -62,9 +62,18 @@ > >> SYSCTL_DECL(_vfs_nfsd); > >> > >> static int disable_checkutf8 =3D 0; > >> +static int disable_nobodycheck =3D 0; > >> +static int disable_nogroupcheck =3D 0; > >> SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, > >> &disable_checkutf8, 0, > >> "Disable the NFSv4 check for a UTF8 compliant name"); > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, > >> + &disable_nobodycheck, 0, > >> + "Disable the NFSv4 check when setting user nobody as > >> owner"); > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, > >> CTLFLAG_RW, > >> + &disable_nogroupcheck, 0, > >> + "Disable the NFSv4 check when setting group nogroup as > >> owner"); > >> + > >> > >> static char nfsrv_hexdigit(char, int *); > >> > >> @@ -1543,8 +1552,8 @@ > >> */ > >> if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) > >> goto out; > >> - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > >> nfsrv_defaultuid) > >> - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > >> nfsrv_defaultgid)) { > >> + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > >> nfsrv_defaultuid && > >> disable_nobodycheck =3D=3D 0) > >> + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > >> nfsrv_defaultgid > >> && > >> disable_nogroupcheck =3D=3D 0)) { > >> error =3D NFSERR_BADOWNER; > >> goto out; > >> } > >> Regards, > >> > >> Lo=C3=AFc Blot, > >> UNIX Systems, Network and Security Engineer > >> http://www.unix-experience.fr > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" > > > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 12:11:11 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 189E7460 for ; Tue, 14 Oct 2014 12:11:11 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A308280 for ; Tue, 14 Oct 2014 12:11:10 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id fb4so9937988wid.0 for ; Tue, 14 Oct 2014 05:11:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=wk2xqd33cgQ+Ko7W6qKIK0VXWArYaq/bnEFyDs4GO1s=; b=rfFCDexz7zpYCmhC3EwiowndpxvgV5WKyoQBPCHuKfWH1/srKBPKbramne8SK9wZSr m21PL25fa4NkN3i+GtxIKxABA4wknveykqjlmbV6YAsCy0Bqj3owd15HlNcZ7J78Hdfe oPQwfRY/3itdBrCMQ+EF3xZmLzfNmffJ19qPFjLbpRnXvQYSjveDbLDPWQ2yqBH6RE37 d/KADP9ZooMO6as/CFJcKU91DPgFSSPDz+Lra1615Pl7lHOgcktt9zzzDa3UtlNJ5LTb hSNT92EuWz54m09VrEoNbc7g2nsQY2JGvRlAGZvKRNQvzIQN03AJXirtq+7RQw1d2c3f 6zWw== MIME-Version: 1.0 X-Received: by 10.194.3.78 with SMTP id a14mr2172545wja.107.1413288668780; Tue, 14 Oct 2014 05:11:08 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Tue, 14 Oct 2014 05:11:08 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: <986887451.63845723.1413288110282.JavaMail.root@uoguelph.ca> References: <986887451.63845723.1413288110282.JavaMail.root@uoguelph.ca> Date: Tue, 14 Oct 2014 20:11:08 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: Rick Macklem Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 12:11:11 -0000 Thanks Rick, I will do it tomorrow (Taiwan Time). Best Regards, 2014-10-14 20:01 GMT+08:00 Rick Macklem : > Marcelo Araujo wrote: > > Hello Blot, > > > > The patch looks reasonable. > > As per the email thread, seems a good approach to overcome this > > issue, at > > least for now. > > > > If Rick has no objection and no free time, I can commit the patch > > during > > this week. > > > > Best Regards, > > > > 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot : > > > > > Hi, > > > since a recent problem (see thread NFSv4 nobody issue), i think we > > > need a > > > sysctl variable to disable nobody and nogroup check into the kernel > > > (default enabled) > > > This variable is useful in some situations, like TFTP over NFS, > > > jails > > > over NFS (some files like /var/db/locate.database need nobody > > > user). > > > > > > I added vfs.nfsd.disable_nobodycheck and > > > vfs.nfsd.disable_nogroupcheck to > > > modify NFSv4 nobody/nogroup check. > > > > > > Thanks to Rick to tell me where the problem was. > > > > > > Can you review the patch, and add it to kernel to avoid previous > > > mentionned issue. > > > > > > Here is my patch: > > > > > > --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 > > > 12:03:50.163311506 > > > +0200 > > > +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 > > > 12:06:29.793304755 +0200 > > > @@ -62,9 +62,18 @@ > > > SYSCTL_DECL(_vfs_nfsd); > > > > > > static int disable_checkutf8 =3D 0; > > > +static int disable_nobodycheck =3D 0; > > > +static int disable_nogroupcheck =3D 0; > > > SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, > > > &disable_checkutf8, 0, > > > "Disable the NFSv4 check for a UTF8 compliant name"); > > > +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, > > > + &disable_nobodycheck, 0, > > > + "Disable the NFSv4 check when setting user nobody as owner"); > > > +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, CTLFLAG_RW, > > > + &disable_nogroupcheck, 0, > > > + "Disable the NFSv4 check when setting group nogroup as > > > owner"); > > > + > > > > Patch looks fine to me. > > Marcelo, you can commit this if you'd like. Otherwise I'll do it. > > Sorry it took a while for me to remember this was disabled. (My only > excuse is I wrote it about 10years ago;-) > > rick > > > > static char nfsrv_hexdigit(char, int *); > > > > > > @@ -1543,8 +1552,8 @@ > > > */ > > > if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) > > > goto out; > > > - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > > nfsrv_defaultuid) > > > - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > > nfsrv_defaultgid)) { > > > + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > > nfsrv_defaultuid && > > > disable_nobodycheck =3D=3D 0) > > > + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > > nfsrv_defaultgid && > > > disable_nogroupcheck =3D=3D 0)) { > > > error =3D NFSERR_BADOWNER; > > > goto out; > > > } > > > Regards, > > > > > > Lo=C3=AFc Blot, > > > UNIX Systems, Network and Security Engineer > > > http://www.unix-experience.fr > > > _______________________________________________ > > > freebsd-fs@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to > > > "freebsd-fs-unsubscribe@freebsd.org" > > > > > > > > > > -- > > > > -- > > Marcelo Araujo (__)araujo@FreeBSD.org > > \\\'',)http://www.FreeBSD.org \/ \ ^ > > Power To Server. .\. /_) > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 12:12:13 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73BEC4D4 for ; Tue, 14 Oct 2014 12:12:13 +0000 (UTC) Received: from mail-wg0-x22b.google.com (mail-wg0-x22b.google.com [IPv6:2a00:1450:400c:c00::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F1F7B33B for ; Tue, 14 Oct 2014 12:12:12 +0000 (UTC) Received: by mail-wg0-f43.google.com with SMTP id m15so10567045wgh.2 for ; Tue, 14 Oct 2014 05:12:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=1A2DPgNpna61FibdyIC1rPVdIbcTZJCxv+lT194sUO0=; b=BMpenL9qedEmj/Gq8RTzlYFfzDBBPMA/urk7e3ZvLY0YV597sFFZU08KNesd0L6YaW ykC0ePA2bysoNRw1vMRdOgOkY49SGVMgjNHDLr/XFWMvGHBFpnUim1tNJY5GBwYhrjKe gepXkXEaalWFH6DS3QqVVZY9dbC3BdNrePuvZjZp9pn1V1givcoE09XVwPoZ18B1fuUf TyEc/dxOmuJiWF0LruqEZPV3pGs3fFvJwI59gSdCennpAjcQEhAjqPuTedMYCVUQjby8 Tec3T82ykoqH6OiUXh4G73uhZAzDIxJi1hK/zR8cQNiBdDQXOzbsDybDzze/fJTzmd/F eB4w== MIME-Version: 1.0 X-Received: by 10.180.101.200 with SMTP id fi8mr4901662wib.77.1413288731133; Tue, 14 Oct 2014 05:12:11 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Tue, 14 Oct 2014 05:12:11 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: <2111556765.63849821.1413288573994.JavaMail.root@uoguelph.ca> References: <2111556765.63849821.1413288573994.JavaMail.root@uoguelph.ca> Date: Tue, 14 Oct 2014 20:12:11 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: Rick Macklem Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 12:12:13 -0000 Hello All, Before I commit it, I will double check what is the best way. Thanks Ronald to point it out. Best Regards, 2014-10-14 20:09 GMT+08:00 Rick Macklem : > Ronald Klop wrote: > > I thought it is advised to make settings positively defined. So not > > use > > 'disable =3D 1', but 'enable =3D 0'. > > > For the case of disable_utf8, I made it negative, since disabling the > check violates RFC-3530. For these checks, there isn't anything in the > RFC requiring the check AFAIK, so I personally don't care which way they > are done. (If the default is disabling the check that could be a minor PO= LA > violation.) > > So, you guys choose whichever you prefer to commit, rick > > > Ronald. > > > > > > On Tue, 14 Oct 2014 12:46:25 +0200, Marcelo Araujo > > wrote: > > > > > Hello Blot, > > > > > > The patch looks reasonable. > > > As per the email thread, seems a good approach to overcome this > > > issue, at > > > least for now. > > > > > > If Rick has no objection and no free time, I can commit the patch > > > during > > > this week. > > > > > > Best Regards, > > > > > > 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot > > > : > > > > > >> Hi, > > >> since a recent problem (see thread NFSv4 nobody issue), i think > > >> we > > >> need a > > >> sysctl variable to disable nobody and nogroup check into the > > >> kernel > > >> (default enabled) > > >> This variable is useful in some situations, like TFTP over NFS, > > >> jails > > >> over NFS (some files like /var/db/locate.database need nobody > > >> user). > > >> > > >> I added vfs.nfsd.disable_nobodycheck and > > >> vfs.nfsd.disable_nogroupcheck > > >> to > > >> modify NFSv4 nobody/nogroup check. > > >> > > >> Thanks to Rick to tell me where the problem was. > > >> > > >> Can you review the patch, and add it to kernel to avoid previous > > >> mentionned issue. > > >> > > >> Here is my patch: > > >> > > >> --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 > > >> 12:03:50.163311506 > > >> +0200 > > >> +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 > > >> 12:06:29.793304755 > > >> +0200 > > >> @@ -62,9 +62,18 @@ > > >> SYSCTL_DECL(_vfs_nfsd); > > >> > > >> static int disable_checkutf8 =3D 0; > > >> +static int disable_nobodycheck =3D 0; > > >> +static int disable_nogroupcheck =3D 0; > > >> SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, > > >> &disable_checkutf8, 0, > > >> "Disable the NFSv4 check for a UTF8 compliant name"); > > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, > > >> + &disable_nobodycheck, 0, > > >> + "Disable the NFSv4 check when setting user nobody as > > >> owner"); > > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, > > >> CTLFLAG_RW, > > >> + &disable_nogroupcheck, 0, > > >> + "Disable the NFSv4 check when setting group nogroup as > > >> owner"); > > >> + > > >> > > >> static char nfsrv_hexdigit(char, int *); > > >> > > >> @@ -1543,8 +1552,8 @@ > > >> */ > > >> if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) > > >> goto out; > > >> - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > >> nfsrv_defaultuid) > > >> - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > >> nfsrv_defaultgid)) { > > >> + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > >> nfsrv_defaultuid && > > >> disable_nobodycheck =3D=3D 0) > > >> + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > >> nfsrv_defaultgid > > >> && > > >> disable_nogroupcheck =3D=3D 0)) { > > >> error =3D NFSERR_BADOWNER; > > >> goto out; > > >> } > > >> Regards, > > >> > > >> Lo=C3=AFc Blot, > > >> UNIX Systems, Network and Security Engineer > > >> http://www.unix-experience.fr > > >> _______________________________________________ > > >> freebsd-fs@freebsd.org mailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> To unsubscribe, send any mail to > > >> "freebsd-fs-unsubscribe@freebsd.org" > > > > > > > > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Tue Oct 14 18:46:05 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 570A7E7E for ; Tue, 14 Oct 2014 18:46:05 +0000 (UTC) Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 220E0641 for ; Tue, 14 Oct 2014 18:46:04 +0000 (UTC) Received: by mail-ig0-f169.google.com with SMTP id uq10so18399248igb.2 for ; Tue, 14 Oct 2014 11:45:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:reply-to:organization :user-agent:mime-version:to:subject:content-type :content-transfer-encoding; bh=36GF50kwESW0yW2NOKxhxMGyKoPqI6UiyJ4214PwAjQ=; b=lk7+1tOMoPqX2iLPMBUEmIi6D9hY1cvzKvtnedLNbI2wsuMnJvRD/tGw4ZGA28FiKL auAsFLz82kQ34VkR6c+d80GMNrZGhz0NXUZNGcCbWorMf4mrtbS4tLGsMbg7dS/XJy61 iODXq+95UQMcBmdVWb3d2BIoBg8Sj53nH1A2tbnkb3r4/HFQk/8tEZh6lOYlywwsshsh RMldbsBtNiCrIL4V3x8tTdvbMLeiV8Dti4/DHyr6jLW7eb4ZludIJ7GtrW7SCIHx1pSv AhCK0K3q+IM6mdb9K58fQQWSSDoajIfLnNIxnWcpqYdlGymbVk8P+AEmrYQ5zctkLC/H Pvog== X-Gm-Message-State: ALoCoQmhLyIU4kSOcpr6LUPxSMjPzTCX9kuwtOkpQQDgmpgCnM/w75vAHX1oKEqsIxFxUCr2D6R4 X-Received: by 10.107.155.76 with SMTP id d73mr4325817ioe.12.1413312358708; Tue, 14 Oct 2014 11:45:58 -0700 (PDT) Received: from linda-macbook.local ([63.231.252.189]) by mx.google.com with ESMTPSA id o8sm11898059igh.12.2014.10.14.11.45.57 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 Oct 2014 11:45:58 -0700 (PDT) Message-ID: <543D6F65.4090205@kateley.com> Date: Tue, 14 Oct 2014 13:45:57 -0500 From: Linda Kateley Reply-To: linda@kateley.com Organization: Kateley Company User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, zfs-discuss@zfsonlinux.org, zfs@lists.illumos.org, iloveosxzfs@gmail.com, zfs-discuss@solaris-zfs.java.net Subject: Open-ZFS Bootcamp Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 18:46:05 -0000 I just posted an Open-ZFS bootcamp. Thought people might be interested, if you don't know zfs, give it a try. http://kateleyco.com/?page_id=783 Linda K From owner-freebsd-fs@FreeBSD.ORG Wed Oct 15 02:24:48 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C1802FF5 for ; Wed, 15 Oct 2014 02:24:48 +0000 (UTC) Received: from mail-wg0-x22b.google.com (mail-wg0-x22b.google.com [IPv6:2a00:1450:400c:c00::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 45B2DA46 for ; Wed, 15 Oct 2014 02:24:48 +0000 (UTC) Received: by mail-wg0-f43.google.com with SMTP id m15so280512wgh.26 for ; Tue, 14 Oct 2014 19:24:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=n2JrRl+k2/5WpfNQ/on/IuAoQw3z6SE5Dz7TRSDCUo8=; b=zBcSAZvHdyo59+R3dJXvOCfr33ekmcB8WFeZTtJgWy9gYrEg3xgClMZvwbHMEB+jRF nRV+IifccAg04Mbi+DlS/ejqijFvLKxSJ48dDonci4aRyGIqMkkjXvEEjglHGLPQOcqw vG41H6ovt16mnxBrtR+jPS9IUS94a30RDsvsYE+axyAQIFNS8t3ooaAe3Su4XTu5ER+F WWZXn4MDfSyMSZtoS+anSOfljtZCOdel6Jvi7F0FZWg0lFdcDqROXyLF6Nxt+Ol7obtA /nWnmTPHfwl/2Qxe/pjW7knZ66y7cjJsroZZnp1K5iWO1wScsy3WBSdlLGMemSmzqK3c nq7A== MIME-Version: 1.0 X-Received: by 10.194.108.104 with SMTP id hj8mr9197677wjb.28.1413339886542; Tue, 14 Oct 2014 19:24:46 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Tue, 14 Oct 2014 19:24:46 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: References: <2111556765.63849821.1413288573994.JavaMail.root@uoguelph.ca> Date: Wed, 15 Oct 2014 10:24:46 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: Rick Macklem Content-Type: multipart/mixed; boundary=047d7b6d96c4e2889505056cd5a1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 02:24:48 -0000 --047d7b6d96c4e2889505056cd5a1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello Ronald and Blot, Here is the patch with a small rework. I consider Ronaldo's comments as well as I just change a bit the code style. If you guys agree with the patch, I will commit it today. Note: About the disable_utf8 that Rick has mention, I will rework that part later to make it as enable_utf8 instead of disable_utf8. Best Regards, 2014-10-14 20:12 GMT+08:00 Marcelo Araujo : > Hello All, > > Before I commit it, I will double check what is the best way. > Thanks Ronald to point it out. > > Best Regards, > > 2014-10-14 20:09 GMT+08:00 Rick Macklem : > >> Ronald Klop wrote: >> > I thought it is advised to make settings positively defined. So not >> > use >> > 'disable =3D 1', but 'enable =3D 0'. >> > >> For the case of disable_utf8, I made it negative, since disabling the >> check violates RFC-3530. For these checks, there isn't anything in the >> RFC requiring the check AFAIK, so I personally don't care which way they >> are done. (If the default is disabling the check that could be a minor >> POLA >> violation.) >> >> So, you guys choose whichever you prefer to commit, rick >> >> > Ronald. >> > >> > >> > On Tue, 14 Oct 2014 12:46:25 +0200, Marcelo Araujo >> > wrote: >> > >> > > Hello Blot, >> > > >> > > The patch looks reasonable. >> > > As per the email thread, seems a good approach to overcome this >> > > issue, at >> > > least for now. >> > > >> > > If Rick has no objection and no free time, I can commit the patch >> > > during >> > > this week. >> > > >> > > Best Regards, >> > > >> > > 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot >> > > : >> > > >> > >> Hi, >> > >> since a recent problem (see thread NFSv4 nobody issue), i think >> > >> we >> > >> need a >> > >> sysctl variable to disable nobody and nogroup check into the >> > >> kernel >> > >> (default enabled) >> > >> This variable is useful in some situations, like TFTP over NFS, >> > >> jails >> > >> over NFS (some files like /var/db/locate.database need nobody >> > >> user). >> > >> >> > >> I added vfs.nfsd.disable_nobodycheck and >> > >> vfs.nfsd.disable_nogroupcheck >> > >> to >> > >> modify NFSv4 nobody/nogroup check. >> > >> >> > >> Thanks to Rick to tell me where the problem was. >> > >> >> > >> Can you review the patch, and add it to kernel to avoid previous >> > >> mentionned issue. >> > >> >> > >> Here is my patch: >> > >> >> > >> --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 >> > >> 12:03:50.163311506 >> > >> +0200 >> > >> +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 >> > >> 12:06:29.793304755 >> > >> +0200 >> > >> @@ -62,9 +62,18 @@ >> > >> SYSCTL_DECL(_vfs_nfsd); >> > >> >> > >> static int disable_checkutf8 =3D 0; >> > >> +static int disable_nobodycheck =3D 0; >> > >> +static int disable_nogroupcheck =3D 0; >> > >> SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, >> > >> &disable_checkutf8, 0, >> > >> "Disable the NFSv4 check for a UTF8 compliant name"); >> > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW, >> > >> + &disable_nobodycheck, 0, >> > >> + "Disable the NFSv4 check when setting user nobody as >> > >> owner"); >> > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, >> > >> CTLFLAG_RW, >> > >> + &disable_nogroupcheck, 0, >> > >> + "Disable the NFSv4 check when setting group nogroup as >> > >> owner"); >> > >> + >> > >> >> > >> static char nfsrv_hexdigit(char, int *); >> > >> >> > >> @@ -1543,8 +1552,8 @@ >> > >> */ >> > >> if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) >> > >> goto out; >> > >> - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D >> > >> nfsrv_defaultuid) >> > >> - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D >> > >> nfsrv_defaultgid)) { >> > >> + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D >> > >> nfsrv_defaultuid && >> > >> disable_nobodycheck =3D=3D 0) >> > >> + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D >> > >> nfsrv_defaultgid >> > >> && >> > >> disable_nogroupcheck =3D=3D 0)) { >> > >> error =3D NFSERR_BADOWNER; >> > >> goto out; >> > >> } >> > >> Regards, >> > >> >> > >> Lo=C3=AFc Blot, >> > >> UNIX Systems, Network and Security Engineer >> > >> http://www.unix-experience.fr >> > >> _______________________________________________ >> > >> freebsd-fs@freebsd.org mailing list >> > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > >> To unsubscribe, send any mail to >> > >> "freebsd-fs-unsubscribe@freebsd.org" >> > > >> > > >> > > >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > >> > > > > -- > > -- > Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.Fr= eeBSD.org \/ \ ^ > Power To Server. .\. /_) > > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) --047d7b6d96c4e2889505056cd5a1 Content-Type: application/octet-stream; name="nfs-nogroup-user.patch" Content-Disposition: attachment; filename="nfs-nogroup-user.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_i1a2199h0 SW5kZXg6IHN5cy9mcy9uZnNzZXJ2ZXIvbmZzX25mc2RzdWJzLmMKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lz L2ZzL25mc3NlcnZlci9uZnNfbmZzZHN1YnMuYwkocmV2aXNpb24gMjczMTEyKQorKysgc3lzL2Zz L25mc3NlcnZlci9uZnNfbmZzZHN1YnMuYwkod29ya2luZyBjb3B5KQpAQCAtNjYsNiArNjYsMTYg QEAKICAgICAmZGlzYWJsZV9jaGVja3V0ZjgsIDAsCiAgICAgIkRpc2FibGUgdGhlIE5GU3Y0IGNo ZWNrIGZvciBhIFVURjggY29tcGxpYW50IG5hbWUiKTsKIAorc3RhdGljIGludCAgICBlbmFibGVf bm9ib2R5Y2hlY2sgPSAxOworU1lTQ1RMX0lOVChfdmZzX25mc2QsIE9JRF9BVVRPLCBlbmFibGVf bm9ib2R5Y2hlY2ssIENUTEZMQUdfUlcsCisgICAgJmVuYWJsZV9ub2JvZHljaGVjaywgMCwKKyAg ICAiRW5hYmxlIHRoZSBORlN2NCBjaGVjayB3aGVuIHNldHRpbmcgdXNlciBub2JvZHkgYXMgb3du ZXIiKTsKKworc3RhdGljIGludCAgICBlbmFibGVfbm9ncm91cGNoZWNrID0gMTsKK1NZU0NUTF9J TlQoX3Zmc19uZnNkLCBPSURfQVVUTywgZW5hYmxlX25vZ3JvdXBjaGVjaywgQ1RMRkxBR19SVywK KyAgICAmZW5hYmxlX25vZ3JvdXBjaGVjaywgMCwKKyAgICAiRW5hYmxlIHRoZSBORlN2NCBjaGVj ayB3aGVuIHNldHRpbmcgZ3JvdXAgbm9ncm91cCBhcyBvd25lciIpOworCiBzdGF0aWMgY2hhciBu ZnNydl9oZXhkaWdpdChjaGFyLCBpbnQgKik7CiAKIC8qCkBAIC0xNTQzLDggKzE1NTMsMTAgQEAK IAkgKi8KIAlpZiAoTkZTVk5PX05PVFNFVFVJRChudmFwKSAmJiBORlNWTk9fTk9UU0VUR0lEKG52 YXApKQogCQlnb3RvIG91dDsKLQlpZiAoKE5GU1ZOT19JU1NFVFVJRChudmFwKSAmJiBudmFwLT5u YV91aWQgPT0gbmZzcnZfZGVmYXVsdHVpZCkKLQkgICAgfHwgKE5GU1ZOT19JU1NFVEdJRChudmFw KSAmJiBudmFwLT5uYV9naWQgPT0gbmZzcnZfZGVmYXVsdGdpZCkpIHsKKwlpZiAoKE5GU1ZOT19J U1NFVFVJRChudmFwKSAmJiBudmFwLT5uYV91aWQgPT0gbmZzcnZfZGVmYXVsdHVpZCAmJgorICAg ICAgICAgICBlbmFibGVfbm9ib2R5Y2hlY2sgPT0gMSkKKwkgICAgfHwgKE5GU1ZOT19JU1NFVEdJ RChudmFwKSAmJiBudmFwLT5uYV9naWQgPT0gbmZzcnZfZGVmYXVsdGdpZCAmJgorICAgICAgICAg ICBlbmFibGVfbm9ncm91cGNoZWNrID09IDEpKSB7CiAJCWVycm9yID0gTkZTRVJSX0JBRE9XTkVS OwogCQlnb3RvIG91dDsKIAl9Cg== --047d7b6d96c4e2889505056cd5a1-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 15 04:52:15 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 16B65F4F; Wed, 15 Oct 2014 04:52:15 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id A21B6AEB; Wed, 15 Oct 2014 04:52:14 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 4FD4B20E7088B; Wed, 15 Oct 2014 04:52:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 985EE20E70885; Wed, 15 Oct 2014 04:52:10 +0000 (UTC) Message-ID: <8F4036C658724468B34B20CCBA658E43@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Wed, 15 Oct 2014 05:52:08 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 04:52:15 -0000 ----- Original Message ----- From: "Steven Hartland" To: "Mark Martinec" ; ; Sent: Tuesday, October 14, 2014 12:40 PM Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] > ----- Original Message ----- > From: "Mark Martinec" > > >> On 10/14/2014 13:19, Steven Hartland wrote: >>> Well interesting issue I left this pool alone this morning literally doing >>> nothing, and its now out of space. >>> zpool list >>> NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH >>> ALTROOT >>> sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - >>> sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - >>> >>> There's something very wrong here as nothing has been accessing the pool. >>> >>> pool: zfs >>> state: ONLINE >>> status: One or more devices are faulted in response to IO failures. >>> action: Make sure the affected devices are connected, then run 'zpool >>> clear'. >>> see: http://illumos.org/msg/ZFS-8000-HC >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> zfs ONLINE 0 2 0 >>> md1 ONLINE 0 0 0 >>> >>> I tried destroying the pool and ever that failed, presumably because >>> the pool has suspended IO. >> >> That's exactly how trouble started here. Got the >> "One or more devices are faulted in response to IO failures" >> on all three small cloned boot pools one day, out of the blue. >> There was no activity there, except for periodic snapshoting >> every 10 minutes. > > Yer this isn't fragmentation, this is something else. I've started a > thread on the openzfs list to discuss this as theres something quite > odd going on. After bisecting the kernel versions in stable/10 the problem commit appears to be: https://svnweb.freebsd.org/base?view=revision&revision=268650 Removing it or using a pool without async_destory enabled prevents the leak. More debugging tomorrow. Regards steve From owner-freebsd-fs@FreeBSD.ORG Wed Oct 15 07:21:30 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C3F2BD4; Wed, 15 Oct 2014 07:21:30 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AFA05A47; Wed, 15 Oct 2014 07:21:28 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id ACDACFCBF; Wed, 15 Oct 2014 07:21:25 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id z210S1zSLmAp; Wed, 15 Oct 2014 07:21:20 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id EAE21FCA1; Wed, 15 Oct 2014 07:21:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413357680; bh=GuzdxyEK0Fulyhc0Gtf13CojsmWYLaRzBlKt4X00AaM=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=lmXf8VjGck+o2QkqjmR6kgNvdWVk8JbTq7v2KPoaxh9y3ArLvIS3kQBuLMCYqf8d4 iZnrx+OvpE9xM1Tk3b7UlQ81g9DtABbv8R+JQ7qnxMGTCMPcSN+igqfq+ziIsqgH2B fly/nwTzzXcoAmBDSi7g86AMdCjoGpYIIkfbf9f0= Mime-Version: 1.0 Date: Wed, 15 Oct 2014 07:21:19 +0000 Message-ID: <345e74ad56f643496a0fa158dda30733@mail.unix-experience.fr> X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check To: araujo@freebsd.org, "Rick Macklem" In-Reply-To: References: <2111556765.63849821.1413288573994.JavaMail.root@uoguelph.ca> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 07:21:30 -0000 Hi,=0A i agree, thanks for your rework !=0A=0A Regards,=0A=0A Lo=C3=AFc B= lot,=0A UNIX Systems, Network and Security Engineer=0A http://www.unix-ex= perience.fr=0A 15 octobre 2014 04:24 "Marcelo Araujo" a =C3=A9crit: =0A= =0A =C2=A0 =0A Hello Ronald and Blot, =0A=C2=A0 =0AHere is the patch with= a small rework. I consider Ronaldo's comments as well as I just change a= bit the code style. =0A=C2=A0 =0AIf you guys agree with the patch, I wil= l commit it today. =0A=C2=A0 =0ANote: About the=C2=A0disable_utf8 that R= ick has mention, I will rework that part later to make it as enable_utf8 = instead of disable_utf8. =0A=C2=A0 =0ABest Regards, =0A=C2=A0 =0A2014-10= -14 20:12 GMT+08:00 Marcelo Araujo :=0A=0A =C2=A0 Hello All, =0A=C2=A0 = =0ABefore I commit it, I will double check what is the best way. =0AThank= s Ronald to point it out. =0A=C2=A0 =0ABest Regards, =0A=C2=A0 =0A2014-1= 0-14 20:09 GMT+08:00 Rick Macklem : Ronald Klop wrote:=0A > I thought it = is advised to make settings positively defined. So not=0A > use=0A > 'dis= able =3D 1', but 'enable =3D 0'.=0A >=0A For the case of disable_utf8, I = made it negative, since disabling the=0A check violates RFC-3530. For the= se checks, there isn't anything in the=0A RFC requiring the check AFAIK, = so I personally don't care which way they=0A are done. (If the default is= disabling the check that could be a minor POLA=0A violation.)=0A=0A So, = you guys choose whichever you prefer to commit, rick =0A > Ronald.=0A >= =0A >=0A > On Tue, 14 Oct 2014 12:46:25 +0200, Marcelo Araujo=0A > wrote= :=0A >=0A > > Hello Blot,=0A > >=0A > > The patch looks reasonable.=0A > = > As per the email thread, seems a good approach to overcome this=0A > > = issue, at=0A > > least for now.=0A > >=0A > > If Rick has no objection an= d no free time, I can commit the patch=0A > > during=0A > > this week.=0A= > >=0A > > Best Regards,=0A > >=0A > > 2014-10-14 18:34 GMT+08:00 Lo=C3= =AFc Blot=0A > > :=0A > >=0A > >> Hi,=0A > >>=C2=A0 since a recent proble= m (see thread NFSv4 nobody issue), i think=0A > >>=C2=A0 we=0A > >> need = a=0A > >> sysctl variable to disable nobody and nogroup check into the=0A= > >> kernel=0A > >> (default enabled)=0A > >>=C2=A0 This variable is use= ful in some situations, like TFTP over NFS,=0A > >>=C2=A0 jails=0A > >> o= ver NFS (some files like /var/db/locate.database need nobody=0A > >> user= ).=0A > >>=0A > >>=C2=A0 I added vfs.nfsd.disable_nobodycheck and=0A > >>= =C2=A0 vfs.nfsd.disable_nogroupcheck=0A > >> to=0A > >> modify NFSv4 nobo= dy/nogroup check.=0A > >>=0A > >>=C2=A0 Thanks to Rick to tell me where t= he problem was.=0A > >>=0A > >>=C2=A0 Can you review the patch, and add i= t to kernel to avoid previous=0A > >> mentionned issue.=0A > >>=0A > >>= =C2=A0 Here is my patch:=0A > >>=0A > >>=C2=A0 --- sys/fs/nfsserver/nfs_n= fsdsubs.c.orig=C2=A0 =C2=A0 2014-10-14=0A > >> 12:03:50.163311506=0A > >>= +0200=0A > >>=C2=A0 +++ sys/fs/nfsserver/nfs_nfsdsubs.c=C2=A0 =C2=A0 201= 4-10-14=0A > >>=C2=A0 12:06:29.793304755=0A > >> +0200=0A > >>=C2=A0 @@ -= 62,9 +62,18 @@=0A > >>=C2=A0 =C2=A0SYSCTL_DECL(_vfs_nfsd);=0A > >>=0A > >= >=C2=A0 =C2=A0static int=C2=A0 =C2=A0 disable_checkutf8 =3D 0;=0A > >>=C2= =A0 +static int=C2=A0 =C2=A0 disable_nobodycheck =3D 0;=0A > >>=C2=A0 +st= atic int=C2=A0 =C2=A0 disable_nogroupcheck =3D 0;=0A > >>=C2=A0 =C2=A0SYS= CTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW,=0A > >>=C2=A0= =C2=A0 =C2=A0 =C2=A0&disable_checkutf8, 0,=0A > >>=C2=A0 =C2=A0 =C2=A0 = =C2=A0"Disable the NFSv4 check for a UTF8 compliant name");=0A > >>=C2=A0= +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, CTLFLAG_RW,=0A > >= >=C2=A0 +=C2=A0 =C2=A0 &disable_nobodycheck, 0,=0A > >>=C2=A0 +=C2=A0 =C2= =A0 "Disable the NFSv4 check when setting user nobody as=0A > >>=C2=A0 ow= ner");=0A > >>=C2=A0 +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupchec= k,=0A > >>=C2=A0 CTLFLAG_RW,=0A > >>=C2=A0 +=C2=A0 =C2=A0 &disable_nogrou= pcheck, 0,=0A > >>=C2=A0 +=C2=A0 =C2=A0 "Disable the NFSv4 check when set= ting group nogroup as=0A > >>=C2=A0 owner");=0A > >>=C2=A0 +=0A > >>=0A >= >>=C2=A0 =C2=A0static char nfsrv_hexdigit(char, int *);=0A > >>=0A > >>= =C2=A0 @@ -1543,8 +1552,8 @@=0A > >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 */=0A > >= >=C2=A0 =C2=A0 =C2=A0 =C2=A0if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGI= D(nvap))=0A > >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;=0A > >= >=C2=A0 -=C2=A0 =C2=A0 if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D= =0A > >>=C2=A0 nfsrv_defaultuid)=0A > >>=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2= =A0 || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D=0A > >> nfsrv_defaul= tgid)) {=0A > >>=C2=A0 +=C2=A0 =C2=A0 if ((NFSVNO_ISSETUID(nvap) && nvap-= >na_uid =3D=3D=0A > >>=C2=A0 nfsrv_defaultuid &&=0A > >> disable_nobodych= eck =3D=3D 0)=0A > >>=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 || (NFSVNO_ISSET= GID(nvap) && nvap->na_gid =3D=3D=0A > >>=C2=A0 nfsrv_defaultgid=0A > >> &= &=0A > >> disable_nogroupcheck =3D=3D 0)) {=0A > >>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0error =3D NFSERR_BADOWNER;=0A > >>=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0goto out;=0A > >>=C2=A0 =C2=A0 =C2=A0 =C2=A0}=0A = > >>=C2=A0 Regards,=0A > >>=0A > >>=C2=A0 Lo=C3=AFc Blot,=0A > >>=C2=A0 U= NIX Systems, Network and Security Engineer=0A > >>=C2=A0 http://www.unix-= experience.fr (http://www.unix-experience.fr)=0A > >> ___________________= ____________________________=0A > >> freebsd-fs@freebsd.org (mailto:freeb= sd-fs@freebsd.org) mailing list=0A > >> http://lists.freebsd.org/mailman/= listinfo/freebsd-fs (http://lists.freebsd.org/mailman/listinfo/freebsd-fs= )=0A > >> To unsubscribe, send any mail to=0A > >> "freebsd-fs-unsubscrib= e@freebsd.org (mailto:freebsd-fs-unsubscribe@freebsd.org)"=0A > >=0A > >= =0A > >=0A > _______________________________________________=0A > freebsd= -fs@freebsd.org (mailto:freebsd-fs@freebsd.org) mailing list=0A > http://= lists.freebsd.org/mailman/listinfo/freebsd-fs (http://lists.freebsd.org/m= ailman/listinfo/freebsd-fs)=0A > To unsubscribe, send any mail to "freebs= d-fs-unsubscribe@freebsd.org (mailto:freebsd-fs-unsubscribe@freebsd.org)"= =0A > =C2=A0 =0A=C2=A0 -- =0A=C2=A0 =0A -- Marcelo Araujo (__) ar= aujo@FreeBSD.org (mailto:araujo@FreeBSD.org) \'',) http://www.FreeBSD.org= (http://www.freebsd.org/) / ^ Power To Server. .. /_) =C2=A0 =0A= =C2=A0 -- =0A=C2=A0 =0A -- Marcelo Araujo (__) araujo@FreeBSD.org (mail= to:araujo@FreeBSD.org) \'',) http://www.FreeBSD.org (http://www.freebsd.o= rg/) / ^ Power To Server. .. /_) =0A=0A =C2=A0 From owner-freebsd-fs@FreeBSD.ORG Wed Oct 15 23:46:14 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 63A7E2AF; Wed, 15 Oct 2014 23:46:14 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id E4E29347; Wed, 15 Oct 2014 23:46:13 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqwEAJMGP1SDaFve/2dsb2JhbABbg2FYBIMCyHoKhnlUAoEmAX2EAgEBAQMBAQEBIAQnIAsbGAICDRkCIwYBCSYOBwQBHASICQMJCA2zYY5GDYY9AQEBBwEBAQEBHYEsjG6BUhACAQEaNAeCNkESgUIFlkOEDnODfY1ZhlaEEyEvB4FBgQIBAQE X-IronPort-AV: E=Sophos;i="5.04,728,1406606400"; d="scan'208";a="161618890" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 15 Oct 2014 19:46:05 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id C9A523CE1D; Wed, 15 Oct 2014 19:46:05 -0400 (EDT) Date: Wed, 15 Oct 2014 19:46:05 -0400 (EDT) From: Rick Macklem To: araujo@FreeBSD.org Message-ID: <1865571459.65576954.1413416765814.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 23:46:14 -0000 Marcelo Araujo wrote: >=20 > Hello Ronald and Blot, >=20 >=20 >=20 > Here is the patch with a small rework. I consider Ronaldo's comments > as well as I just change a bit the code style. >=20 >=20 > If you guys agree with the patch, I will commit it today. >=20 Looks fine to me. >=20 > Note: About the disable_utf8 that Rick has mention, I will rework > that part later to make it as enable_utf8 instead of disable_utf8. >=20 If you do change this one, try to include something in the description string w.r.t. RFC-3530 requires it to be enabled. Thanks, rick >=20 > Best Regards, >=20 >=20 > 2014-10-14 20:12 GMT+08:00 Marcelo Araujo < araujobsdport@gmail.com > > : >=20 >=20 >=20 > Hello All, >=20 >=20 > Before I commit it, I will double check what is the best way. > Thanks Ronald to point it out. >=20 >=20 > Best Regards, >=20 >=20 >=20 >=20 > 2014-10-14 20:09 GMT+08:00 Rick Macklem < rmacklem@uoguelph.ca > : >=20 >=20 > Ronald Klop wrote: > > I thought it is advised to make settings positively defined. So not > > use > > 'disable =3D 1', but 'enable =3D 0'. > >=20 > For the case of disable_utf8, I made it negative, since disabling the > check violates RFC-3530. For these checks, there isn't anything in > the > RFC requiring the check AFAIK, so I personally don't care which way > they > are done. (If the default is disabling the check that could be a > minor POLA > violation.) >=20 > So, you guys choose whichever you prefer to commit, rick >=20 >=20 >=20 > > Ronald. > >=20 > >=20 > > On Tue, 14 Oct 2014 12:46:25 +0200, Marcelo Araujo > > < araujobsdport@gmail.com > wrote: > >=20 > > > Hello Blot, > > >=20 > > > The patch looks reasonable. > > > As per the email thread, seems a good approach to overcome this > > > issue, at > > > least for now. > > >=20 > > > If Rick has no objection and no free time, I can commit the patch > > > during > > > this week. > > >=20 > > > Best Regards, > > >=20 > > > 2014-10-14 18:34 GMT+08:00 Lo=C3=AFc Blot > > > < loic.blot@unix-experience.fr >: > > >=20 > > >> Hi, > > >> since a recent problem (see thread NFSv4 nobody issue), i think > > >> we > > >> need a > > >> sysctl variable to disable nobody and nogroup check into the > > >> kernel > > >> (default enabled) > > >> This variable is useful in some situations, like TFTP over NFS, > > >> jails > > >> over NFS (some files like /var/db/locate.database need nobody > > >> user). > > >>=20 > > >> I added vfs.nfsd.disable_nobodycheck and > > >> vfs.nfsd.disable_nogroupcheck > > >> to > > >> modify NFSv4 nobody/nogroup check. > > >>=20 > > >> Thanks to Rick to tell me where the problem was. > > >>=20 > > >> Can you review the patch, and add it to kernel to avoid previous > > >> mentionned issue. > > >>=20 > > >> Here is my patch: > > >>=20 > > >> --- sys/fs/nfsserver/nfs_nfsdsubs.c.orig 2014-10-14 > > >> 12:03:50.163311506 > > >> +0200 > > >> +++ sys/fs/nfsserver/nfs_nfsdsubs.c 2014-10-14 > > >> 12:06:29.793304755 > > >> +0200 > > >> @@ -62,9 +62,18 @@ > > >> SYSCTL_DECL(_vfs_nfsd); > > >>=20 > > >> static int disable_checkutf8 =3D 0; > > >> +static int disable_nobodycheck =3D 0; > > >> +static int disable_nogroupcheck =3D 0; > > >> SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_checkutf8, CTLFLAG_RW, > > >> &disable_checkutf8, 0, > > >> "Disable the NFSv4 check for a UTF8 compliant name"); > > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nobodycheck, > > >> CTLFLAG_RW, > > >> + &disable_nobodycheck, 0, > > >> + "Disable the NFSv4 check when setting user nobody as > > >> owner"); > > >> +SYSCTL_INT(_vfs_nfsd, OID_AUTO, disable_nogroupcheck, > > >> CTLFLAG_RW, > > >> + &disable_nogroupcheck, 0, > > >> + "Disable the NFSv4 check when setting group nogroup as > > >> owner"); > > >> + > > >>=20 > > >> static char nfsrv_hexdigit(char, int *); > > >>=20 > > >> @@ -1543,8 +1552,8 @@ > > >> */ > > >> if (NFSVNO_NOTSETUID(nvap) && NFSVNO_NOTSETGID(nvap)) > > >> goto out; > > >> - if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > >> nfsrv_defaultuid) > > >> - || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > >> nfsrv_defaultgid)) { > > >> + if ((NFSVNO_ISSETUID(nvap) && nvap->na_uid =3D=3D > > >> nfsrv_defaultuid && > > >> disable_nobodycheck =3D=3D 0) > > >> + || (NFSVNO_ISSETGID(nvap) && nvap->na_gid =3D=3D > > >> nfsrv_defaultgid > > >> && > > >> disable_nogroupcheck =3D=3D 0)) { > > >> error =3D NFSERR_BADOWNER; > > >> goto out; > > >> } > > >> Regards, > > >>=20 > > >> Lo=C3=AFc Blot, > > >> UNIX Systems, Network and Security Engineer > > >> http://www.unix-experience.fr > > >> _______________________________________________ > > >> freebsd-fs@freebsd.org mailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> To unsubscribe, send any mail to > > >> " freebsd-fs-unsubscribe@freebsd.org " > > >=20 > > >=20 > > >=20 > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to " > > freebsd-fs-unsubscribe@freebsd.org " > >=20 >=20 >=20 >=20 >=20 > -- >=20 >=20 >=20 >=20 > -- > Marcelo Araujo (__) araujo@FreeBSD.org \\\'',) > http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) >=20 >=20 >=20 > -- >=20 >=20 >=20 >=20 > -- > Marcelo Araujo (__) araujo@FreeBSD.org \\\'',) > http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 02:42:20 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C880249 for ; Thu, 16 Oct 2014 02:42:20 +0000 (UTC) Received: from mail-wg0-x231.google.com (mail-wg0-x231.google.com [IPv6:2a00:1450:400c:c00::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B7551835 for ; Thu, 16 Oct 2014 02:42:19 +0000 (UTC) Received: by mail-wg0-f49.google.com with SMTP id x12so2689543wgg.8 for ; Wed, 15 Oct 2014 19:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=beeAyhR86wNs/oOTzsK0BEpGlfvkyMC9GOw551SiX5Q=; b=NjdWSwCH0YMyiLZb5sr9fmoTJIYVn7MV5oVsny+RBvW4pMBJCTepXroKYKVdrUoL8s wjUdvrzY6hYspVGTXRAFQyxBmEUKt5Ku/DZ4FCjOsU4Vk7RxUZLwi5QZWP6urozOmFZJ muTgcLj96hZ4WABUN9ByOsnxsTHgi3ziAvaQyv26OmvO4vgQrMlidx3Ow7mR8hDw9Mcr FLgVfQ3JxXn9EoqFv8RwOShWpxTtnHqrLS6r9jMjiKtNhI2U2jtj8UMAfU3eNZzHvf5L GO4i+UIgCxC40fAGdtjSegNqbUH1Q4ZOeYApL4hprtnMsXefR94S3HEpTJ82kBv2C7e7 EC/Q== MIME-Version: 1.0 X-Received: by 10.194.3.78 with SMTP id a14mr8037858wja.107.1413427338062; Wed, 15 Oct 2014 19:42:18 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Wed, 15 Oct 2014 19:42:17 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: <1865571459.65576954.1413416765814.JavaMail.root@uoguelph.ca> References: <1865571459.65576954.1413416765814.JavaMail.root@uoguelph.ca> Date: Thu, 16 Oct 2014 10:42:17 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: Rick Macklem Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 02:42:20 -0000 2014-10-16 7:46 GMT+08:00 Rick Macklem : > Marcelo Araujo wrote: > > > > Hello Ronald and Blot, > > > > > > > > Here is the patch with a small rework. I consider Ronaldo's comments > > as well as I just change a bit the code style. > > > > > > If you guys agree with the patch, I will commit it today. > > > Looks fine to me. > Thanks Rick! Committed; I will do the MFC after two weeks if you have no objections. https://svnweb.freebsd.org/base?view=revision&revision=273159 > > > > Note: About the disable_utf8 that Rick has mention, I will rework > > that part later to make it as enable_utf8 instead of disable_utf8. > > > If you do change this one, try to include something in the description > string w.r.t. RFC-3530 requires it to be enabled. > > Thanks, rick > > Rick, here is a patch that renames the disable_utf8 to enable_utf8 and as per your request, I have changed the description of the sysctl(8) as well. Let me know if the change looks good for you as well as the description. Best Regards, -- Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 02:56:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 012AB40F; Thu, 16 Oct 2014 02:56:26 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id AC31090B; Thu, 16 Oct 2014 02:56:26 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id CF03620E7088B; Thu, 16 Oct 2014 02:56:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 52D6A20E70885; Thu, 16 Oct 2014 02:56:24 +0000 (UTC) Message-ID: <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> From: "Steven Hartland" To: "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> <8F4036C658724468B34B20CCBA658E43@mu ltiplay.co.uk> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Thu, 16 Oct 2014 03:56:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 02:56:27 -0000 ----- Original Message ----- From: "Steven Hartland" > After bisecting the kernel versions in stable/10 the problem commit > appears to be: > https://svnweb.freebsd.org/base?view=revision&revision=268650 > > Removing it or using a pool without async_destory enabled prevents > the leak. > > More debugging tomorrow. Fix for this has now been committed: https://svnweb.freebsd.org/changeset/base/273158 I'm already talking with re@ to get this in to the 10.1 release. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 02:58:02 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CCC71578 for ; Thu, 16 Oct 2014 02:58:02 +0000 (UTC) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 584FD92B for ; Thu, 16 Oct 2014 02:58:02 +0000 (UTC) Received: by mail-wi0-f176.google.com with SMTP id hi2so282453wib.3 for ; Wed, 15 Oct 2014 19:58:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=tnwEAb2ZW1jLz61MwN4rb/AzRrb04LqlVtAni0Fsy+4=; b=zosbscQMnQpfoJR6+mtgj4wbIaGSwDNI+QPA81Lb1Flb2HbAPKbn3zjQw0ZkyxOxZR Qum8oa6IbqocSGb363c9OkBCqNMn/mR34ORXMGMVuqUI8nvaayT+Ikfa4nQvwOZak5rJ XZtv/bA3uSMY3pmSWzAz008y5xgZK8vZQxz3SbkQ+2h2q9cRdsqoOq267tG14sxi8EUI 3fGPwuCQdY8NFuBv4TKpTZ7BNlfVYULe2mS5gm8snTOW4tYuJBjvEMPU2EsyCi66lcx2 nHzU6z9sYfluumlGpN6Hg3dLCm7KRgbfbnMII6QuNzwiuhoDaZbSDtS51iINrcGaDSdh dNBA== MIME-Version: 1.0 X-Received: by 10.180.88.162 with SMTP id bh2mr1877646wib.77.1413428280498; Wed, 15 Oct 2014 19:58:00 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Wed, 15 Oct 2014 19:58:00 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: References: <1865571459.65576954.1413416765814.JavaMail.root@uoguelph.ca> Date: Thu, 16 Oct 2014 10:58:00 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: Rick Macklem Content-Type: multipart/mixed; boundary=f46d04426c58951a010505816ad2 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 02:58:02 -0000 --f46d04426c58951a010505816ad2 Content-Type: text/plain; charset=UTF-8 2014-10-16 10:42 GMT+08:00 Marcelo Araujo : > > > 2014-10-16 7:46 GMT+08:00 Rick Macklem : > >> Marcelo Araujo wrote: >> > >> > Hello Ronald and Blot, >> > >> > >> > >> > Here is the patch with a small rework. I consider Ronaldo's comments >> > as well as I just change a bit the code style. >> > >> > >> > If you guys agree with the patch, I will commit it today. >> > >> Looks fine to me. >> > > Thanks Rick! Committed; I will do the MFC after two weeks if you have no > objections. > > https://svnweb.freebsd.org/base?view=revision&revision=273159 > > >> > >> > Note: About the disable_utf8 that Rick has mention, I will rework >> > that part later to make it as enable_utf8 instead of disable_utf8. >> > >> If you do change this one, try to include something in the description >> string w.r.t. RFC-3530 requires it to be enabled. >> >> Thanks, rick >> >> > Rick, here is a patch that renames the disable_utf8 to enable_utf8 and as > per your request, I have changed the description of the sysctl(8) as well. > Let me know if the change looks good for you as well as the description. > > > > Ouch, I forgot to attach the patch, spotted by kevlo@ via Skype :_) Best Regards, -- -- Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) --f46d04426c58951a010505816ad2 Content-Type: text/plain; charset=US-ASCII; name="nfsd_utf8.diff" Content-Disposition: attachment; filename="nfsd_utf8.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_i1biosbw0 SW5kZXg6IHN5cy9mcy9uZnNzZXJ2ZXIvbmZzX25mc2RzdWJzLmMKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lz L2ZzL25mc3NlcnZlci9uZnNfbmZzZHN1YnMuYwkocmV2aXNpb24gMjczMTU5KQorKysgc3lzL2Zz L25mc3NlcnZlci9uZnNfbmZzZHN1YnMuYwkod29ya2luZyBjb3B5KQpAQCAtNjEsMTAgKzYxLDEw IEBACiAKIFNZU0NUTF9ERUNMKF92ZnNfbmZzZCk7CiAKLXN0YXRpYyBpbnQJZGlzYWJsZV9jaGVj a3V0ZjggPSAwOwotU1lTQ1RMX0lOVChfdmZzX25mc2QsIE9JRF9BVVRPLCBkaXNhYmxlX2NoZWNr dXRmOCwgQ1RMRkxBR19SVywKLSAgICAmZGlzYWJsZV9jaGVja3V0ZjgsIDAsCi0gICAgIkRpc2Fi bGUgdGhlIE5GU3Y0IGNoZWNrIGZvciBhIFVURjggY29tcGxpYW50IG5hbWUiKTsKK3N0YXRpYyBp bnQJZW5hYmxlX2NoZWNrdXRmOCA9IDE7CitTWVNDVExfSU5UKF92ZnNfbmZzZCwgT0lEX0FVVE8s IGVuYWJsZV9jaGVja3V0ZjgsIENUTEZMQUdfUlcsCisgICAgJmVuYWJsZV9jaGVja3V0ZjgsIDAs CisgICAgIkVuYWJsZSB0aGUgTkZTdjQgY2hlY2sgZm9yIHRoZSBVVEY4IGNvbXBsaWFudCBuYW1l IHJlcXVpcmVkIGJ5IHJmYzM1MzAiKTsKIAogc3RhdGljIGludCAgICBlbmFibGVfbm9ib2R5Y2hl Y2sgPSAxOwogU1lTQ1RMX0lOVChfdmZzX25mc2QsIE9JRF9BVVRPLCBlbmFibGVfbm9ib2R5Y2hl Y2ssIENUTEZMQUdfUlcsCkBAIC0yMDA1LDcgKzIwMDUsNyBAQAogCQkgICAgZXJyb3IgPSAwOwog CQkgICAgZ290byBuZnNtb3V0OwogCQl9Ci0JCWlmIChkaXNhYmxlX2NoZWNrdXRmOCA9PSAwICYm CisJCWlmIChlbmFibGVfY2hlY2t1dGY4ID09IDEgJiYKIAkJICAgIG5mc3J2X2NoZWNrdXRmOCgo dV9pbnQ4X3QgKilidWZwLCBvdXRsZW4pKSB7CiAJCSAgICBuZC0+bmRfcmVwc3RhdCA9IE5GU0VS Ul9JTlZBTDsKIAkJICAgIGVycm9yID0gMDsK --f46d04426c58951a010505816ad2-- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 07:55:25 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CCA2E469; Thu, 16 Oct 2014 07:55:25 +0000 (UTC) Received: from odin.blazingdot.com (odin.blazingdot.com [204.109.60.170]) by mx1.freebsd.org (Postfix) with ESMTP id A9FC88B0; Thu, 16 Oct 2014 07:55:25 +0000 (UTC) Received: by odin.blazingdot.com (Postfix, from userid 1001) id 124AB130C8F; Thu, 16 Oct 2014 03:55:19 -0400 (EDT) Date: Thu, 16 Oct 2014 03:55:19 -0400 From: Marcus Reid To: Steven Hartland Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Message-ID: <20141016075518.GA14459@blazingdot.com> References: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> <8F4036C658724468B34B20CCBA658E43@multiplay.co.uk> <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 07:55:25 -0000 On Thu, Oct 16, 2014 at 03:56:23AM +0100, Steven Hartland wrote: > Fix for this has now been committed: > https://svnweb.freebsd.org/changeset/base/273158 > > I'm already talking with re@ to get this in to the 10.1 release. Thank you for that. I looked at your thread on the illumos zfs list, and from what I gather, if you aren't wedged into a state where you have to import read-only, you don't have to worry about leaked data in your pool, correct? I always have a small number of 'deferred free' blocks that's always somewhere between 8 and 10: 9 108K 15.5K 108K 12.0K 6.97 0.00 deferred free Also, if you run 'zdb -bb ' on a live pool, you can get a bunch of: leaked space: vdev 0, offset 0x16464c2000, size 1048576 ... and then: block traversal size 14586265600 != alloc 14667571200 (leaked 81305600) which I believe is normal and unrelated. Marcus From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 08:50:55 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D610CB2; Thu, 16 Oct 2014 08:50:55 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 1BBC7F1A; Thu, 16 Oct 2014 08:50:54 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id C1F9F20E7088C; Thu, 16 Oct 2014 08:50:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 9238D20E70885; Thu, 16 Oct 2014 08:50:51 +0000 (UTC) Message-ID: <2307495EDFE14A4FB34374307B3EBC55@multiplay.co.uk> From: "Steven Hartland" To: "Marcus Reid" References: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> <8F4036C658724468B34B20CCBA658E43@multiplay.co.uk> <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> <20141016075518.GA14459@blazingdot.com> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Thu, 16 Oct 2014 09:50:48 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 08:50:55 -0000 ----- Original Message ----- From: "Marcus Reid" > On Thu, Oct 16, 2014 at 03:56:23AM +0100, Steven Hartland wrote: >> Fix for this has now been committed: >> https://svnweb.freebsd.org/changeset/base/273158 >> >> I'm already talking with re@ to get this in to the 10.1 release. > > Thank you for that. I looked at your thread on the illumos zfs list, > and from what I gather, if you aren't wedged into a state where you have > to import read-only, you don't have to worry about leaked data in your > pool, correct? > > I always have a small number of 'deferred free' blocks that's always > somewhere between 8 and 10: > > 9 108K 15.5K 108K 12.0K 6.97 0.00 deferred free > > Also, if you run 'zdb -bb ' on a live pool, you can get a bunch > of: > > leaked space: vdev 0, offset 0x16464c2000, size 1048576 > ... > > and then: > > block traversal size 14586265600 != alloc 14667571200 (leaked 81305600) > > which I believe is normal and unrelated. Yep thats normal. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 10:03:57 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D96CAC82; Thu, 16 Oct 2014 10:03:57 +0000 (UTC) Received: from smtp.unix-experience.fr (62-210-206-43.rev.poneytelecom.eu [62.210.206.43]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6F180A79; Thu, 16 Oct 2014 10:03:56 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id 446CD11968; Thu, 16 Oct 2014 10:03:48 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id JAIfy7D_KzL8; Thu, 16 Oct 2014 10:03:42 +0000 (UTC) Received: from mail.unix-experience.fr (unknown [192.168.200.1]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id 9F0721194A; Thu, 16 Oct 2014 10:03:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1413453822; bh=GcETGIGPiMVjU5kXw4Lgw05FBbe3dm3Q6y0tglMxp14=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=dqOUBGIKhE7qcbiHclbcZdnS+kebOrXPHMB/24Vpaj+KSshd6JBBrnS/rJXx2sdN5 VBhcWFha97r3Eqhcaioh6nc3dybwcvQvjoNEgtdUoMbMk2rZpjQYUxCRK6gqTq0cGq voJu7Q6MBLq6fVwM56Akvhcg2zq9aDvEDywHpAzc= Mime-Version: 1.0 Date: Thu, 16 Oct 2014 10:03:42 +0000 Message-ID: <251df1c7fb5c9e0332facb3d6e7e838b@mail.unix-experience.fr> X-Mailer: RainLoop/1.6.9.161 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check To: araujo@freebsd.org, "Rick Macklem" In-Reply-To: References: <1865571459.65576954.1413416765814.JavaMail.root@uoguelph.ca> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: Quoted-Printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 10:03:57 -0000 Thanks for the commit !=0A=0A Regards,=0A=0A Lo=C3=AFc Blot,=0A UNIX Syst= ems, Network and Security Engineer=0A http://www.unix-experience.fr=0A 16= octobre 2014 04:58 "Marcelo Araujo" a =C3=A9crit: =0A=0A =C2=A0 =0A = =C2=A0 =0A=C2=A0 =0A2014-10-16 10:42 GMT+08:00 Marcelo Araujo :=0A=0A =C2= =A0 =C2=A0 =0A=C2=A0 =0A2014-10-16 7:46 GMT+08:00 Rick Macklem : Marcelo= Araujo wrote:=0A >=0A > Hello Ronald and Blot,=0A >=0A >=0A >=0A > Here = is the patch with a small rework. I consider Ronaldo's comments=0A > as w= ell as I just change a bit the code style.=0A >=0A >=0A > If you guys agr= ee with the patch, I will commit it today.=0A >=0A Looks fine to me. =0A= =C2=A0 =0AThanks Rick! Committed; I will do the MFC after two weeks if yo= u have no objections. =0A=C2=A0 =0Ahttps://svnweb.freebsd.org/base?view= =3Drevision&revision=3D273159 (https://svnweb.freebsd.org/base?view=3Drev= ision&revision=3D273159)=C2=A0 =0A=C2=A0 =0A>=0A > Note: About the disabl= e_utf8 that Rick has mention, I will rework=0A > that part later to make = it as enable_utf8 instead of disable_utf8.=0A >=0A If you do change this = one, try to include something in the description=0A string w.r.t. RFC-353= 0 requires it to be enabled.=0A=0A Thanks, rick =0A=C2=A0 =0A=C2=A0 =0A= Rick, here is a patch that renames the disable_utf8 to enable_utf8 and as= per your request, I have changed the description of the sysctl(8) as wel= l. =0ALet me know if the change looks good for you as well as the descrip= tion.=C2=A0 =0A=C2=A0 =0A=C2=A0 =0A=C2=A0 =0A=C2=A0 =0AOuch, I forg= ot to attach the patch, spotted by kevlo@ via Skype :_) =0A=C2=A0 =0A=C2= =A0 =0ABest Regards,=C2=A0 =0A=C2=A0 -- =0A=C2=A0 =0A -- Marcelo Arauj= o (__) araujo@FreeBSD.org (mailto:araujo@FreeBSD.org) \'',) http://www.Fr= eeBSD.org (http://www.freebsd.org/) / ^ Power To Server. .. /_) =0A= =0A =C2=A0 From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 11:33:18 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3264D833; Thu, 16 Oct 2014 11:33:18 +0000 (UTC) Received: from mail.jrv.org (adsl-70-243-84-11.dsl.austtx.swbell.net [70.243.84.11]) by mx1.freebsd.org (Postfix) with ESMTP id F109D682; Thu, 16 Oct 2014 11:33:17 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id B28D41A1D02; Thu, 16 Oct 2014 06:25:44 -0500 (CDT) Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QhW2_ibdt3fa; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id D289A1A1CFA; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra64.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Mt5NHfUP045b; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) Received: from [192.168.138.128] (BMX.housenet.jrv [192.168.3.140]) by mail.jrv.org (Postfix) with ESMTPSA id AB7B61A1CF7; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) Message-ID: <543FAB3C.4090503@jrv.org> Date: Thu, 16 Oct 2014 06:25:48 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> In-Reply-To: <54250AE9.6070609@jrv.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 11:33:18 -0000 The zfs recv / kmem arena hang happens with -CURRENT as well as 10-STABLE, on two different systems, with 16GB or 32GB of RAM, from memstick or normal multi-user environments, Hangs usually seem to hapeen 1TB to 3TB in, but last night one run hung after only 4.35MB. On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: > FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 #2 r272070M: > Wed Sep 24 17:36:56 CDT 2014 > james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 > > With current STABLE10 I am unable to replicate a ZFS pool using zfs > send/recv without zfs hanging in state "kmem arena", within the first > 4TB or so (of a 23TB Pool). > > The most recent attempt used this command line > > SUPERTEX:/root# zfs send -R BIGTEX/UNIX@syssnap | ssh BLACKIE zfs recv > -duvF BIGTOX > > though local replications fail in kmem arena too. > > The two machines I've been attempting this on have 16BG and 32GB of RAM > each and are otherwise idle. > > Any suggestions on how to get around, or investigate, "kmem arena"? > > # top > last pid: 3272; load averages: 0.22, 0.22, 0.23 up > 0+08:25:02 01:32:07 > 34 processes: 1 running, 33 sleeping > CPU: 0.0% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.9% idle > Mem: 21M Active, 82M Inact, 15G Wired, 28M Cache, 450M Free > ARC: 12G Total, 24M MFU, 12G MRU, 23M Anon, 216M Header, 47M Other > Swap: 16G Total, 16G Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 1173 root 1 52 0 86476K 7780K select 0 124:33 0.00% sshd > 1176 root 1 46 0 87276K 47732K kmem a 3 48:36 0.00% zfs > 968 root 32 20 0 12344K 1888K rpcsvc 0 0:13 0.00% nfsd > 1009 root 1 20 0 25452K 2864K select 3 0:01 0.00% ntpd > ... From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 15:25:11 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 79082868 for ; Thu, 16 Oct 2014 15:25:11 +0000 (UTC) Received: from mail-yh0-x23f.google.com (mail-yh0-x23f.google.com [IPv6:2607:f8b0:4002:c01::23f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B15B3EE for ; Thu, 16 Oct 2014 15:25:11 +0000 (UTC) Received: by mail-yh0-f63.google.com with SMTP id b6so486981yha.18 for ; Thu, 16 Oct 2014 08:25:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version:content-type; bh=7jvGIqjUS/JqJ2m7r+vBiu3UjS+SbwFkb3EI5woAsTo=; b=VGvPcgfDnVVCVVV0yuEXiceolfCXg01XZLffCTfP4wyP3DfyebwxcSQ/H+xIT1kpoI huNPwp0n1cTZHYAly7QZ5B1X3mBDLQ6baKv1V+GYkWgw7WXwjIXlv1eUdat9JfccatzN y9DPCAL6sWhfbxxjktiLWoyriHuwMGsoCwBnW9APczPR98MbkpwnlZJyDht5zUUznwBY 2HMqemsbXAI6nuDf0LyilxuxYS8JpfHHWtQNAey0it8D+rHQUAdqusjwNlemtWdOvNRh gf9EaaAABeo1e/bxgk9ouzYECOtQP4du7nBdkkRMewmKuFm/Yhc22IZzdrbZwsIFkqd8 PQjQ== X-Received: by 10.140.22.239 with SMTP id 102mr30512qgn.1.1413473110345; Thu, 16 Oct 2014 08:25:10 -0700 (PDT) X-Google-Doc-Id: a19511c891300660 X-Google-Web-Client: true Date: Thu, 16 Oct 2014 08:25:09 -0700 (PDT) From: Richard To: zfs-discuss@zfsonlinux.org Message-Id: In-Reply-To: <543D6F65.4090205@kateley.com> References: <543D6F65.4090205@kateley.com> Subject: Re: Open-ZFS Bootcamp MIME-Version: 1.0 X-Google-Token: ENXG_6EF0Xtjzs-EGRI0 X-Google-IP: 84.24.160.143 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org, zfs@lists.illumos.org, zfs-discuss@solaris-zfs.java.net, iloveosxzfs@gmail.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 15:25:11 -0000 Hi, Thanks for this beginners course on ZFS. I ended up watching the whole session and learned some new things. Regards, Richard On Tuesday, October 14, 2014 8:46:01 PM UTC+2, Linda Kateley wrote: > > I just posted an Open-ZFS bootcamp. Thought people might be interested, if > you don't know zfs, give it a try. > > http://kateleyco.com/?page_id=783 > > Linda K > > From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 16:12:35 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E39B2B71; Thu, 16 Oct 2014 16:12:34 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C20ABB92; Thu, 16 Oct 2014 16:12:34 +0000 (UTC) Received: from delphij-macbook.home.us.delphij.net (unknown [IPv6:2001:470:83bf:0:5da2:ad7b:ac1d:bf78]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id B7168238C9; Thu, 16 Oct 2014 09:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1413475954; x=1413490354; bh=wxT7FrlsPuEfeqYoRYbXS3edRIvGEGcpjiHDqajHzWQ=; h=Date:From:Reply-To:To:CC:Subject:References:In-Reply-To; b=bSoLNYyLiClWytm3+wGgBMKRWqHtrTgLmihkZepw0OvmyOtdi7qOB7rPzIcO5Z17m MHWJHtD6Exguf+syOCRuyTJbFF+jAiy3Z66ekZ4QOb2SccRQ3y4MhRDIlXUESiAjIm IghvsDp5uIjdtDXTJuyP+eCjwIM+ieyzq7ggmHfE= Message-ID: <543FEE6F.5050007@delphij.net> Date: Thu, 16 Oct 2014 09:12:31 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: "James R. Van Artsdalen" Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> <543FAB3C.4090503@jrv.org> In-Reply-To: <543FAB3C.4090503@jrv.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 16:12:35 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 10/16/14 4:25 AM, James R. Van Artsdalen wrote: > The zfs recv / kmem arena hang happens with -CURRENT as well as > 10-STABLE, on two different systems, with 16GB or 32GB of RAM, > from memstick or normal multi-user environments, > > Hangs usually seem to hapeen 1TB to 3TB in, but last night one run > hung after only 4.35MB. > > On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: >> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 #2 >> r272070M: Wed Sep 24 17:36:56 CDT 2014 >> james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 >> >> With current STABLE10 I am unable to replicate a ZFS pool using >> zfs send/recv without zfs hanging in state "kmem arena", within >> the first 4TB or so (of a 23TB Pool). >> >> The most recent attempt used this command line >> >> SUPERTEX:/root# zfs send -R BIGTEX/UNIX@syssnap | ssh BLACKIE zfs >> recv -duvF BIGTOX >> >> though local replications fail in kmem arena too. >> >> The two machines I've been attempting this on have 16BG and 32GB >> of RAM each and are otherwise idle. >> >> Any suggestions on how to get around, or investigate, "kmem >> arena"? >> >> # top last pid: 3272; load averages: 0.22, 0.22, 0.23 >> up 0+08:25:02 01:32:07 34 processes: 1 running, 33 sleeping >> CPU: 0.0% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.9% >> idle Mem: 21M Active, 82M Inact, 15G Wired, 28M Cache, 450M Free >> ARC: 12G Total, 24M MFU, 12G MRU, 23M Anon, 216M Header, 47M >> Other Swap: 16G Total, 16G Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME >> WCPU COMMAND 1173 root 1 52 0 86476K 7780K select >> 0 124:33 0.00% sshd 1176 root 1 46 0 87276K 47732K >> kmem a 3 48:36 0.00% zfs 968 root 32 20 0 12344K >> 1888K rpcsvc 0 0:13 0.00% nfsd 1009 root 1 20 0 >> 25452K 2864K select 3 0:01 0.00% ntpd ... What does procstat -kk 1176 (or the PID of your 'zfs' process that stuck in that state) say? Cheers, -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJUP+5vAAoJEJW2GBstM+ns0v4P/31s7geR2j22etrRnfReUxbb lbex0VkmLGm23TbTj2vpVce+ogBeA4zo6h4WzF/yYt2372MpWOfnEoVX2yOuuGku AFapewXS3UMXLzaRWrdTWng1KQlOyQykAHI2rvQLlYlQNTLA5AbUm6uzNXaKpD8s PbckREQ6wHnpZOiRcMN695QstjBNCal+XJHgvrwTfyp9vdFrPVD4UHnsN7MU6QSO XobxOqbuw4Tq95mgYJqrjk+xEYMgzUy2zkVp2QTCBXZn3T3yroI2RcgUZQWaw5SO xRegPa5jfJqcQJAdSxl8oVs9Sz8+5YDeksAnjCOxIQzLZBbNho+SOAzi+kjnT6W7 ijTc20z5eioQVPekdJ4MBweBsAeS1aGi8VWppuP+ZDLoirmxB0LaZyRv/W/HRQDD j4CoZswkndh+J+9Crsa9SUkfNGNvVVNjhJUGyIfTGFUsMbWTAWwa4SMj7Ad04aqW yhg+Ab4H3Yc14TahtX0jrhD3sTBer6ZoMFKE3tl8aStGXHVMyPkj0PHg5xjZEWL2 XGF86eoIgx03A9sIdbdHEZpyTMksfNatDXZk5XpPGF/sVd6txUoYP4Ch2wD8YRFM O5Ny2r6ash2rZYmlyjf19n4gvKebdGo8d8NbzOJ3oYue6OI/88cu0rv6xLV9hHSF fwgIbPo5uK4hIpEm0Dk4 =qY45 -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 16:56:09 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2A9828E for ; Thu, 16 Oct 2014 16:56:08 +0000 (UTC) Received: from mail-pd0-x233.google.com (mail-pd0-x233.google.com [IPv6:2607:f8b0:400e:c02::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C7264F77 for ; Thu, 16 Oct 2014 16:56:08 +0000 (UTC) Received: by mail-pd0-f179.google.com with SMTP id r10so3590775pdi.10 for ; Thu, 16 Oct 2014 09:56:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=O7kiK5G5Ymr5AGrB00wjEtB9I8T3jpRNlONlrOyCvgU=; b=gHh2pnaARtKy/6rFsmQld4qD9rbXo/wV8pNYOZtOy1ye4Jl91fWx0ZA3tPCHfI3q0S nkGBC7IBnP0fV/nhjbncVOZ/r7HEgBazqnCvLVWSoQAacNRSwfY+yu0YtNMu9x5qIO70 X6JtjRHNhfdL4aU29OX9dSoN+DWZePMhcGwQc6LDtVJDKXiiROYPz7gziaCCPRkNyiiJ XAKqObqr5/JM+i2Z9rWg4C7F8yZNYPUoOJTiDbMhNJIj17/UW2L0D8FfTzd73S9Xmm7W oHJKTFDVefJJNNfDU6gm8BTizVLi4ziIwvKPff5J2C6qmBvFchNPTq0MMtkTBfxrs7Kp cWOQ== MIME-Version: 1.0 X-Received: by 10.70.133.200 with SMTP id pe8mr2442410pdb.123.1413478568370; Thu, 16 Oct 2014 09:56:08 -0700 (PDT) Received: by 10.70.76.2 with HTTP; Thu, 16 Oct 2014 09:56:08 -0700 (PDT) Received: by 10.70.76.2 with HTTP; Thu, 16 Oct 2014 09:56:08 -0700 (PDT) In-Reply-To: References: <543D6F65.4090205@kateley.com> Date: Thu, 16 Oct 2014 19:56:08 +0300 Message-ID: Subject: Re: Open-ZFS Bootcamp From: Sami Halabi To: Richard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org, zfs-discuss@zfsonlinux.org, zfs@lists.illumos.org, zfs-discuss@solaris-zfs.java.net, iloveosxzfs@gmail.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 16:56:09 -0000 +1 on this. Much appreciated! Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 16 =D7=91=D7=90=D7=95=D7=A7 2014 18:25= , "Richard" =D7=9B=D7=AA=D7=91: > Hi, > > Thanks for this beginners course on ZFS. I ended up watching the whole > session and learned some new things. > > Regards, Richard > > On Tuesday, October 14, 2014 8:46:01 PM UTC+2, Linda Kateley wrote: > > > > I just posted an Open-ZFS bootcamp. Thought people might be interested, > if > > you don't know zfs, give it a try. > > > > http://kateleyco.com/?page_id=3D783 > > > > Linda K > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 19:52:34 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D6043C87 for ; Thu, 16 Oct 2014 19:52:34 +0000 (UTC) Received: from mx.bsdtec.net (mx.bsdtec.net [174.34.171.65]) by mx1.freebsd.org (Postfix) with ESMTP id A8AB86EB for ; Thu, 16 Oct 2014 19:52:34 +0000 (UTC) Received: from localhost (mx.bsdtec.net [172.16.32.2]) by mx.bsdtec.net (Postfix) with ESMTP id 5762A48989C for ; Thu, 16 Oct 2014 19:52:28 +0000 (UTC) Received: from mx.bsdtec.net ([172.16.32.2]) by localhost (mx.bsdtec.net [172.16.32.2]) (amavisd-new, port 10032) with ESMTP id Sot56r3yb7fS for ; Thu, 16 Oct 2014 19:52:23 +0000 (UTC) Received: from localhost (mx.bsdtec.net [172.16.32.2]) by mx.bsdtec.net (Postfix) with ESMTP id B30BB4898C0 for ; Thu, 16 Oct 2014 19:52:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at bsdtec.net Received: from mx.bsdtec.net ([172.16.32.2]) by localhost (mx.bsdtec.net [172.16.32.2]) (amavisd-new, port 10026) with ESMTP id 1n6y-LsideLa for ; Thu, 16 Oct 2014 19:52:23 +0000 (UTC) Received: from [192.168.1.3] (bsdtec.plus.com [84.92.41.141]) by mx.bsdtec.net (Postfix) with ESMTPSA id 20EF748989C for ; Thu, 16 Oct 2014 19:52:23 +0000 (UTC) Subject: re: FreeBSD support being added to GlusterFS From: Craig Butler To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="iso8859-1" Date: Thu, 16 Oct 2014 20:54:19 +0100 Message-ID: <1413489259.1652.27.camel@zbox.lerwick.hopto.org> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 19:52:34 -0000 Howdy Folks I have put my head on the chopping board and am looking to become port maintainer for the glusterfs port. Justin Clift from gluster.org sent out a tweet looking for a maintainer.. I answered :) I have lifted FreeNAS glusterfs and updated it to the latest beta release. I have also contacted bapt@ to ask about taking over the maintainership there as well to save double touching the same codebase. FreeBSD PR submitted (194409) to get the port into our tree. I am hoping to use redports to test building for amd64 and x86, I can also test sparc64, amd64, x86, arm locally as well. Kind Regards Craig Butler From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 21:39:50 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8820C899; Thu, 16 Oct 2014 21:39:50 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 399B635E; Thu, 16 Oct 2014 21:39:49 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEAEg6QFSDaFve/2dsb2JhbABbg2FcgwLJHodNAoErAX2EAwEBBCNWGxgCAg0ZAiM2GYgqAxENtnSOQQ2GQgEBAQEBBQEBAQEBARyBLIxtgWYaNAcWgmGBVAWWRYUBg0GOFoJYg36EEyEvgUiBAgEBAQ X-IronPort-AV: E=Sophos;i="5.04,734,1406606400"; d="scan'208";a="161812584" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 16 Oct 2014 17:39:48 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 7CE36B3EE4; Thu, 16 Oct 2014 17:39:48 -0400 (EDT) Date: Thu, 16 Oct 2014 17:39:48 -0400 (EDT) From: Rick Macklem To: araujo@FreeBSD.org Message-ID: <997140144.720996.1413495588502.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 21:39:50 -0000 Marcelo Araujo wrote: > > > > > > 2014-10-16 10:42 GMT+08:00 Marcelo Araujo < araujobsdport@gmail.com > > : > > > > > > > > 2014-10-16 7:46 GMT+08:00 Rick Macklem < rmacklem@uoguelph.ca > : > > > Marcelo Araujo wrote: > > > > Hello Ronald and Blot, > > > > > > > > Here is the patch with a small rework. I consider Ronaldo's > > comments > > as well as I just change a bit the code style. > > > > > > If you guys agree with the patch, I will commit it today. > > > Looks fine to me. > > > > Thanks Rick! Committed; I will do the MFC after two weeks if you have > no objections. > > > https://svnweb.freebsd.org/base?view=revision&revision=273159 > > > > > > > > Note: About the disable_utf8 that Rick has mention, I will rework > > that part later to make it as enable_utf8 instead of disable_utf8. > > > If you do change this one, try to include something in the > description > string w.r.t. RFC-3530 requires it to be enabled. > > Thanks, rick > > > > > > > Rick, here is a patch that renames the disable_utf8 to enable_utf8 > and as per your request, I have changed the description of the > sysctl(8) as well. > Let me know if the change looks good for you as well as the > description. > Looks fine to me, rick ps: I would suggest not MFC'ng this one, since I think it would be a POLA violation to have the sysctl name change. > > > > > > > > > > Ouch, I forgot to attach the patch, spotted by kevlo@ via Skype :_) > > > > > Best Regards, > > -- > > > > > -- > Marcelo Araujo (__) araujo@FreeBSD.org \\\'',) > http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 22:57:11 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6FCE7659 for ; Thu, 16 Oct 2014 22:57:11 +0000 (UTC) Received: from caida.org (rommie.caida.org [192.172.226.78]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 41E92D04 for ; Thu, 16 Oct 2014 22:57:10 +0000 (UTC) Message-ID: <544047B4.8000402@caida.org> Date: Thu, 16 Oct 2014 15:33:24 -0700 From: Daniel Andersen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Process enters unkillable state and somewhat wedges zfs References: <53F25402.1020907@caida.org> <201408271639.09352.jhb@freebsd.org> <53FE4C9F.7030406@caida.org> <5842681.mjgMD2kESs@ralph.baldwin.cx> <5425DF0E.4040406@caida.org> In-Reply-To: <5425DF0E.4040406@caida.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 22:57:11 -0000 A further update: we almost went 3 weeks without a hang after moving /home back off zfs. Then someone used sudo in a nullfs <-> zfs directory and we got the same thing. Interesting that it's almost always sudo. Only other hangs have been when a program coredumps. ( by nullfs <-> zfs directory, I mean we have a zfs directory /tank/foo and a nullfs mount /data/foo that people use to access ) Dan On 09/26/2014 02:47 PM, Daniel Andersen wrote: > Okay, we were finally able to collect a trace on this. I took two, just in > case. basically did a tr , continue, go back into ddb and did another. > > Tracing pid 89483 tid 101168 td 0xfffff8048e301920 > cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 > ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 > trap() at trap+0x42/frame 0xfffffe2f395e1f20 > nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 > --- trap 0x13, rip = 0xffffffff808aada0, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bf60 --- > __lockmgr_args() at __lockmgr_args+0x30/frame 0xfffffe2ffca2bf60 > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe2ffca2bf80 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9d/frame 0xfffffe2ffca2bfb0 > _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 > zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 > null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 > lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 > namei() at namei+0x504/frame 0xfffffe2ffca2c480 > vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 > vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 > null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 > VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 > vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 > vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 > kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 > --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- > > > > Tracing pid 89483 tid 101168 td 0xfffff8048e301920 > cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 > ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 > trap() at trap+0x42/frame 0xfffffe2f395e1f20 > nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 > --- trap 0x13, rip = 0xffffffff80e6110e, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bfb0 --- > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xae/frame 0xfffffe2ffca2bfb0 > _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 > zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 > null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 > lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 > namei() at namei+0x504/frame 0xfffffe2ffca2c480 > vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 > vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 > null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 > VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 > vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 > vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 > kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 > --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- > > > > > On 08/29/2014 11:24 AM, John Baldwin wrote: >> On Wednesday, August 27, 2014 02:24:47 PM Daniel Andersen wrote: >>> On 08/27/2014 01:39 PM, John Baldwin wrote: >>>> These are all blocked in "zfs" then. (For future reference, the 'mwchan' >>>> field that you see as 'STATE' in top or via 'ps O mwchan' is more detailed >>>> than the 'D' state.) >>>> >>>> To diagnose this further, you would need to see which thread holds the >>>> ZFS vnode lock these threads need. I have some gdb scripts you can use to >>>> do that at www.freebsd.org/~jhb/gdb/. You would want to download 'gdb6*' >>>> files from there and then do this as root: >>>> >>>> # cd /path/to/gdb/files >>>> # kgdb >>>> (kgdb) source gdb6 >>>> (kgdb) sleepchain 42335 >>>> >>>> Where '42335' is the pid of some process stuck in "zfs". >>> >>> I will keep this in mind the next time the machine wedges. Another data >>> point: the second procstat output I sent was the most recent. All the >>> processes listed there were after the fact. The process that started the >>> entire problem ( this time ) was sudo, and it only has this one entry in >>> procstat: >>> >>> 38003 102797 sudo - >>> >>> Of note, this does not appear to be blocked on zfs in anyway. 'ps' showed >>> it in 'R' state instead of 'D' ( I will be sure to use mwchan in the >>> future. ) It appeared to be pegging an entire CPU core at 100% usage, as >>> well, and was only single threaded. >> >> Well, if it is spinning in some sort of loop in the kernel while holding a >> ZFS vnode lock that could be blocking all the other threads. In that case, >> you don't need to do what I asked for above. Instead, we need to find out >> what that thread is doing. There are two ways of doing this. One is to >> force a panic via 'sysctl debug.kdb.panic=1' and then use kgdb on the >> crashdump to determine what the running thread is doing. Another option >> is to break into the DDB debugger on the console (note that you will need >> to build a custom kernel with DDB if you are on stable) and request a >> stack trace of the running process via 'tr '. Ideally you can do this >> over a serial console so you can just cut and paste the output of the trace >> into a mail. Over a video console you can either transcribe it by hand or >> take photos. >> > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 02:14:49 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A4691EE9 for ; Fri, 17 Oct 2014 02:14:49 +0000 (UTC) Received: from mail-wg0-x231.google.com (mail-wg0-x231.google.com [IPv6:2a00:1450:400c:c00::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C3807B for ; Fri, 17 Oct 2014 02:14:49 +0000 (UTC) Received: by mail-wg0-f49.google.com with SMTP id x12so5076759wgg.32 for ; Thu, 16 Oct 2014 19:14:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=mWcBY0FpGIYrNPUfaDm7bOh8AMgcbKIoSCzPBtaWZ3I=; b=OLaUIY4wd0gHtuGUQ40d+TTKxeTws+bfb+osqZBP92s9Enye/BfBCIVDiIvEDRVi8i yELTDUBN8B3+6odAlplbCoQT0gUKgwMdhi6XTA369GMACVTPfxmVljYqRB/ipnmghVlJ drRX1+vy2ON9opN46LYUegBTKziXYb/9HnDN68fnJvOpZyE+5BU/vodRETAg7g+t3G4O WdR6mjUdlyqZZJqqYyVEYMdd4H0FRjMEZM4bUjiPgdmJRF8HoKAlk7INkm85ucwjFWx9 8fQdsUKZtn9o9Jbvrcl0YIDs2UYshRrkG1n5tnbclWsfy/0f8olBhqtx8b4tTn3usbCO asQQ== MIME-Version: 1.0 X-Received: by 10.180.39.106 with SMTP id o10mr10410491wik.54.1413512087564; Thu, 16 Oct 2014 19:14:47 -0700 (PDT) Received: by 10.216.159.193 with HTTP; Thu, 16 Oct 2014 19:14:47 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: <997140144.720996.1413495588502.JavaMail.root@uoguelph.ca> References: <997140144.720996.1413495588502.JavaMail.root@uoguelph.ca> Date: Fri, 17 Oct 2014 10:14:47 +0800 Message-ID: Subject: Re: [PATCH] disable nfsd (NFSv4) nobody/nogroup check From: Marcelo Araujo To: Rick Macklem Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 02:14:49 -0000 2014-10-17 5:39 GMT+08:00 Rick Macklem : > > > > > Rick, here is a patch that renames the disable_utf8 to enable_utf8 > > and as per your request, I have changed the description of the > > sysctl(8) as well. > > Let me know if the change looks good for you as well as the > > description. > > > Looks fine to me, rick > ps: I would suggest not MFC'ng this one, since I think it would be > a POLA violation to have the sysctl name change. > > Rick, Committed, thanks! https://svnweb.freebsd.org/base?view=revision&revision=273202 I agree with you about the MFC. Best Regards, -- Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 03:43:10 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62555CD5; Fri, 17 Oct 2014 03:43:10 +0000 (UTC) Received: from mail.jrv.org (adsl-70-243-84-11.dsl.austtx.swbell.net [70.243.84.11]) by mx1.freebsd.org (Postfix) with ESMTP id 1A65CB08; Fri, 17 Oct 2014 03:43:09 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id 47A551A336A; Thu, 16 Oct 2014 22:43:08 -0500 (CDT) Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QL_N-UzTu8Zn; Thu, 16 Oct 2014 22:42:58 -0500 (CDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id 5A51B1A3365; Thu, 16 Oct 2014 22:42:58 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra64.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ssWaWlpx5cuS; Thu, 16 Oct 2014 22:42:58 -0500 (CDT) Received: from [192.168.138.128] (BMX.housenet.jrv [192.168.3.140]) by mail.jrv.org (Postfix) with ESMTPSA id 2EA611A3362; Thu, 16 Oct 2014 22:42:58 -0500 (CDT) Message-ID: <54409050.4070401@jrv.org> Date: Thu, 16 Oct 2014 22:43:12 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: d@delphij.net Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> <543FAB3C.4090503@jrv.org> <543FEE6F.5050007@delphij.net> In-Reply-To: <543FEE6F.5050007@delphij.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, "James R. Van Artsdalen" , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 03:43:10 -0000 On 10/16/2014 11:12 AM, Xin Li wrote: > > On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: > >> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 #2 > >> r272070M: Wed Sep 24 17:36:56 CDT 2014 > >> james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 > >> > >> With current STABLE10 I am unable to replicate a ZFS pool using > >> zfs send/recv without zfs hanging in state "kmem arena", within > >> the first 4TB or so (of a 23TB Pool). > What does procstat -kk 1176 (or the PID of your 'zfs' process that > stuck in that state) say? > > Cheers, > SUPERTEX:/root# ps -lp 866 UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 866 863 0 52 0 66800 29716 kmem are D+ 1 57:40.82 zfs recv -duvF BIGTOX SUPERTEX:/root# procstat -kk 866 PID TID COMM TDNAME KSTACK 866 101573 zfs - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d kmem_malloc+0x33 keg_alloc_slab+0xcd keg_fetch_slab+0x151 zone_fetch_slab+0x7e zone_import+0x40 uma_zalloc_arg+0x34e arc_get_data_buf+0x31a arc_buf_alloc+0xaa dmu_buf_will_fill+0x169 dmu_write+0xfc dmu_recv_stream+0xd40 zfs_ioc_recv+0x94e zfsdev_ioctl+0x5ca SUPERTEX:/root# From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 04:10:31 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D05B423D; Fri, 17 Oct 2014 04:10:31 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B4AD6CCB; Fri, 17 Oct 2014 04:10:31 +0000 (UTC) Received: from delphij-macbook.home.us.delphij.net (unknown [IPv6:2001:470:83bf:0:14f4:f167:6c7a:fce4]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 091FE3A1F; Thu, 16 Oct 2014 21:10:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1413519031; x=1413533431; bh=SfnrGzueCF9MRC2ek5cax3DnRNglwirXPkkHvyqKmFY=; h=Date:From:Reply-To:To:CC:Subject:References:In-Reply-To; b=NszhTEg2dcNN7Kps3S4ZLJqh6ipWEUh/pc2eMYI7rgE2B6V9buTHyhm35BBZ+mSo2 smdugX0FsdO0WuNJGK+M2FJObCI1yJgaUfdgF6JDSX1LzSxqI6+MqATsVcD9X/3H3g GzyWu/Vw7EmCINBjYAPZt8yPXGvv6V0gud3FrWKI= Message-ID: <544096B3.20306@delphij.net> Date: Thu, 16 Oct 2014 21:10:27 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: "James R. Van Artsdalen" , d@delphij.net Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> <543FAB3C.4090503@jrv.org> <543FEE6F.5050007@delphij.net> <54409050.4070401@jrv.org> In-Reply-To: <54409050.4070401@jrv.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, "James R. Van Artsdalen" , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 04:10:31 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 10/16/14 8:43 PM, James R. Van Artsdalen wrote: > On 10/16/2014 11:12 AM, Xin Li wrote: >>> On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: >>>> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 >>>> #2 r272070M: Wed Sep 24 17:36:56 CDT 2014 >>>> james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC >>>> amd64 >>>> >>>> With current STABLE10 I am unable to replicate a ZFS pool >>>> using zfs send/recv without zfs hanging in state "kmem >>>> arena", within the first 4TB or so (of a 23TB Pool). > >> What does procstat -kk 1176 (or the PID of your 'zfs' process >> that stuck in that state) say? >> >> Cheers, >> > SUPERTEX:/root# ps -lp 866 UID PID PPID CPU PRI NI VSZ RSS > MWCHAN STAT TT TIME COMMAND 0 866 863 0 52 0 66800 > 29716 kmem are D+ 1 57:40.82 zfs recv -duvF BIGTOX > SUPERTEX:/root# procstat -kk 866 PID TID COMM TDNAME > KSTACK 866 101573 zfs - mi_switch+0xe1 > sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d > kmem_malloc+0x33 keg_alloc_slab+0xcd keg_fetch_slab+0x151 > zone_fetch_slab+0x7e zone_import+0x40 uma_zalloc_arg+0x34e > arc_get_data_buf+0x31a arc_buf_alloc+0xaa dmu_buf_will_fill+0x169 > dmu_write+0xfc dmu_recv_stream+0xd40 zfs_ioc_recv+0x94e > zfsdev_ioctl+0x5ca Do you have any special tuning in your /boot/loader.conf? Cheers, -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJUQJazAAoJEJW2GBstM+ns6dQQAK4NM6X40d7tS7pqoTQvZbrD U0u5kid703tWgAlSFzvORxeOEB94BxcHu/z1a68nGhUlL2kip8SirWV9A1rqBpes i4T6asHYTcFj4OvaPfSoA7lSVsZIaLK+RscraN1b7hehSG9UExeYF8D7cRIguhoa 1Gnlv5iZZkjJZGjR0R6DmxC8C1CyNxAZBXnj1L+ofpgUzqH0Rw2TCW1XVKqMcxvI 5lpt+V0uu7MPNgjzgVy/1z5ZD2SUBPco0eHuN/Npj0c6HkmHkoWqd53vxrBhlyCP eDbzLw7QTO7PaV5hAuC9y9/X1JGlmTVa0GP2irKuE5t1bAbVwUPQqpn+iiFs1Le8 34fL/jkCeSBY6voYYj100CBU1/1mZOh93wuY6FdMVWPJp/bsjbDUtKZUmosGU86j ZMikfVNl5Jc5dmH30JGFCDOWzfaFq+V34toSfYIihaBQPyFov0Mr7De5MvQ7VJ7D qiXDcfAXE99CXzAboYpruwrbxyxTqhUmXlWp2uCPqvmo0WhVUsROmhhXhWXkG3tS S7L0n4X8kgklveirZWq/oDsg4JXNTP2ernNdAYyhD7TbG/N4INdFaVuqZkDVDgny ibwY0HEzg2zskJOJBqr9a21fZx6c2dvJ1n+j5BaAq6ve2Hw2NyvUVWfMTknp4I8j t/JJtsDNs9xokH/veS3J =aBKI -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 04:37:18 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35AF5778; Fri, 17 Oct 2014 04:37:18 +0000 (UTC) Received: from mail.jrv.org (adsl-70-243-84-11.dsl.austtx.swbell.net [70.243.84.11]) by mx1.freebsd.org (Postfix) with ESMTP id E1827F00; Fri, 17 Oct 2014 04:37:17 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id 299E51A3624; Thu, 16 Oct 2014 23:37:17 -0500 (CDT) Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 0b9MMkB1wxKV; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id 2D7251A361F; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra64.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id jhh0VrzWEfSt; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) Received: from [192.168.138.128] (BMX.housenet.jrv [192.168.3.140]) by mail.jrv.org (Postfix) with ESMTPSA id 085DE1A361C; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) Message-ID: <54409CFE.8070905@jrv.org> Date: Thu, 16 Oct 2014 23:37:18 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: d@delphij.net Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> <543FAB3C.4090503@jrv.org> <543FEE6F.5050007@delphij.net> <54409050.4070401@jrv.org> <544096B3.20306@delphij.net> In-Reply-To: <544096B3.20306@delphij.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, "James R. Van Artsdalen" , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 04:37:18 -0000 On 10/16/2014 11:10 PM, Xin Li wrote: > On 10/16/14 8:43 PM, James R. Van Artsdalen wrote: > > On 10/16/2014 11:12 AM, Xin Li wrote: > >>> On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: > >>>> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 > >>>> #2 r272070M: Wed Sep 24 17:36:56 CDT 2014 > >>>> james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC > >>>> amd64 > >>>> > >>>> With current STABLE10 I am unable to replicate a ZFS pool > >>>> using zfs send/recv without zfs hanging in state "kmem > >>>> arena", within the first 4TB or so (of a 23TB Pool). > > >> What does procstat -kk 1176 (or the PID of your 'zfs' process > >> that stuck in that state) say? > >> > >> Cheers, > >> > > SUPERTEX:/root# ps -lp 866 UID PID PPID CPU PRI NI VSZ RSS > > MWCHAN STAT TT TIME COMMAND 0 866 863 0 52 0 66800 > > 29716 kmem are D+ 1 57:40.82 zfs recv -duvF BIGTOX > > SUPERTEX:/root# procstat -kk 866 PID TID COMM TDNAME > > KSTACK 866 101573 zfs - mi_switch+0xe1 > > sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d > > kmem_malloc+0x33 keg_alloc_slab+0xcd keg_fetch_slab+0x151 > > zone_fetch_slab+0x7e zone_import+0x40 uma_zalloc_arg+0x34e > > arc_get_data_buf+0x31a arc_buf_alloc+0xaa dmu_buf_will_fill+0x169 > > dmu_write+0xfc dmu_recv_stream+0xd40 zfs_ioc_recv+0x94e > > zfsdev_ioctl+0x5ca > > Do you have any special tuning in your /boot/loader.conf? > > Cheers, > Below. I had forgotten some of this was there. After sending the previous message I ran kgdb to see if I could get a backtrace with function args. I didn't see how to do it for this proc, but during all this the process un-blocked and started running again. The process blocked again in kmem arena after a few minutes. SUPERTEX:/root# cat /boot/loader.conf zfs_load="YES" # ZFS vfs.root.mountfrom="zfs:SUPERTEX/UNIX" # Specify root partition in a way the # kernel understands kern.maxfiles="32K" # Set the sys. wide open files limit kern.ktrace.request_pool="512" #vfs.zfs.debug=1 vfs.zfs.check_hostid=0 loader_logo="beastie" # Desired logo: fbsdbw, beastiebw, beastie, none boot_verbose="YES" # -v: Causes extra debugging information to be printed geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8)) geom_label_load="YES" # File system labels (see glabel(8)) ahci_load="YES" siis_load="YES" mvs_load="YES" coretemp_load="YES" # Intel Core CPU temperature monitor #console="comconsole" kern.msgbufsize="131072" # Set size of kernel message buffer kern.geom.label.gpt.enable=0 kern.geom.label.gptid.enable=0 kern.geom.label.disk_ident.enable=0 SUPERTEX:/root# From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 13:54:21 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C23051EB for ; Fri, 17 Oct 2014 13:54:21 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63E93A94 for ; Fri, 17 Oct 2014 13:54:21 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s9HDsC5b012240 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 17 Oct 2014 16:54:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s9HDsC5b012240 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s9HDsBnI012239; Fri, 17 Oct 2014 16:54:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 17 Oct 2014 16:54:11 +0300 From: Konstantin Belousov To: Daniel Andersen Subject: Re: Process enters unkillable state and somewhat wedges zfs Message-ID: <20141017135411.GE2153@kib.kiev.ua> References: <53F25402.1020907@caida.org> <201408271639.09352.jhb@freebsd.org> <53FE4C9F.7030406@caida.org> <5842681.mjgMD2kESs@ralph.baldwin.cx> <5425DF0E.4040406@caida.org> <544047B4.8000402@caida.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <544047B4.8000402@caida.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 13:54:21 -0000 On Thu, Oct 16, 2014 at 03:33:24PM -0700, Daniel Andersen wrote: > A further update: we almost went 3 weeks without a hang after moving /home back off zfs. > Then someone used sudo in a nullfs <-> zfs directory and we got the same thing. Interesting > that it's almost always sudo. Only other hangs have been when a program coredumps. > > ( by nullfs <-> zfs directory, I mean we have a zfs directory /tank/foo and a nullfs mount /data/foo that people use to > access ) Read https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html for instructions how to obtain useful debugging information. I did not found any mention of the version of the FreeBSD you use. > > Dan > > On 09/26/2014 02:47 PM, Daniel Andersen wrote: > > Okay, we were finally able to collect a trace on this. I took two, just in > > case. basically did a tr , continue, go back into ddb and did another. > > > > Tracing pid 89483 tid 101168 td 0xfffff8048e301920 > > cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 > > ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 > > trap() at trap+0x42/frame 0xfffffe2f395e1f20 > > nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 > > --- trap 0x13, rip = 0xffffffff808aada0, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bf60 --- > > __lockmgr_args() at __lockmgr_args+0x30/frame 0xfffffe2ffca2bf60 > > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe2ffca2bf80 > > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9d/frame 0xfffffe2ffca2bfb0 > > _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 > > zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 > > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 > > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 > > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 > > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 > > null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 > > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 > > lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 > > namei() at namei+0x504/frame 0xfffffe2ffca2c480 > > vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 > > vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 > > null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 > > VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 > > vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 > > vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 > > kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 > > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 > > --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- > > > > > > > > Tracing pid 89483 tid 101168 td 0xfffff8048e301920 > > cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 > > ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 > > trap() at trap+0x42/frame 0xfffffe2f395e1f20 > > nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 > > --- trap 0x13, rip = 0xffffffff80e6110e, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bfb0 --- > > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xae/frame 0xfffffe2ffca2bfb0 > > _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 > > zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 > > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 > > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 > > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 > > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 > > null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 > > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 > > lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 > > namei() at namei+0x504/frame 0xfffffe2ffca2c480 > > vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 > > vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 > > null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 > > VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 > > vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 > > vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 > > kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 > > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 > > --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- > > > > > > > > > > On 08/29/2014 11:24 AM, John Baldwin wrote: > >> On Wednesday, August 27, 2014 02:24:47 PM Daniel Andersen wrote: > >>> On 08/27/2014 01:39 PM, John Baldwin wrote: > >>>> These are all blocked in "zfs" then. (For future reference, the 'mwchan' > >>>> field that you see as 'STATE' in top or via 'ps O mwchan' is more detailed > >>>> than the 'D' state.) > >>>> > >>>> To diagnose this further, you would need to see which thread holds the > >>>> ZFS vnode lock these threads need. I have some gdb scripts you can use to > >>>> do that at www.freebsd.org/~jhb/gdb/. You would want to download 'gdb6*' > >>>> files from there and then do this as root: > >>>> > >>>> # cd /path/to/gdb/files > >>>> # kgdb > >>>> (kgdb) source gdb6 > >>>> (kgdb) sleepchain 42335 > >>>> > >>>> Where '42335' is the pid of some process stuck in "zfs". > >>> > >>> I will keep this in mind the next time the machine wedges. Another data > >>> point: the second procstat output I sent was the most recent. All the > >>> processes listed there were after the fact. The process that started the > >>> entire problem ( this time ) was sudo, and it only has this one entry in > >>> procstat: > >>> > >>> 38003 102797 sudo - > >>> > >>> Of note, this does not appear to be blocked on zfs in anyway. 'ps' showed > >>> it in 'R' state instead of 'D' ( I will be sure to use mwchan in the > >>> future. ) It appeared to be pegging an entire CPU core at 100% usage, as > >>> well, and was only single threaded. > >> > >> Well, if it is spinning in some sort of loop in the kernel while holding a > >> ZFS vnode lock that could be blocking all the other threads. In that case, > >> you don't need to do what I asked for above. Instead, we need to find out > >> what that thread is doing. There are two ways of doing this. One is to > >> force a panic via 'sysctl debug.kdb.panic=1' and then use kgdb on the > >> crashdump to determine what the running thread is doing. Another option > >> is to break into the DDB debugger on the console (note that you will need > >> to build a custom kernel with DDB if you are on stable) and request a > >> stack trace of the running process via 'tr '. Ideally you can do this > >> over a serial console so you can just cut and paste the output of the trace > >> into a mail. Over a video console you can either transcribe it by hand or > >> take photos. > >> > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 21:26:10 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 53645298; Fri, 17 Oct 2014 21:26:10 +0000 (UTC) Received: from caida.org (rommie.caida.org [192.172.226.78]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3BA2AF77; Fri, 17 Oct 2014 21:26:09 +0000 (UTC) Message-ID: <54418970.8080104@caida.org> Date: Fri, 17 Oct 2014 14:26:08 -0700 From: Daniel Andersen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: John Baldwin Subject: Re: Process enters unkillable state and somewhat wedges zfs References: <53F25402.1020907@caida.org> <5842681.mjgMD2kESs@ralph.baldwin.cx> <5425DF0E.4040406@caida.org> <6662003.L0FKeJoQHN@ralph.baldwin.cx> In-Reply-To: <6662003.L0FKeJoQHN@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, avg@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 21:26:10 -0000 My apologies on delays in responding to this. I guess all the traces triggered my spam filters. :( While we rebooted the below mentioned machine just yesterday, we just had an identical event on a completely different machine. Yes, we are making heavy use of nullfs and nfs as a means of presenting identical directory structures across multiple machines. I just ran 'ktrace -p ' on the second machine, with matching the process that is in permanent 100% R state ( again, it is sudo ) ktrace.out came out as an empty file. I did run on a few random processes to make sure ktrace actually works on the system, and it did. For the record, if I didn't say this before: This is on a FreeBSD 10.0/amd64 machine. Reading the email thread you linked.. It could be the same thing. I'm not entirely sure what counts as a true deadlock, and the thread doesn't mention if one process was stuck in R state but still unkillable. Now that it is confirmed that this thing hits us on two different machines, I'm considering rearranging things without as much nullfs in an attempt to reduce our downtime. Dan On 09/27/2014 05:46 AM, John Baldwin wrote: > On Friday, September 26, 2014 02:47:58 PM Daniel Andersen wrote: >> Okay, we were finally able to collect a trace on this. I took two, just in >> case. basically did a tr , continue, go back into ddb and did another. >> >> Tracing pid 89483 tid 101168 td 0xfffff8048e301920 >> cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 >> ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 >> trap() at trap+0x42/frame 0xfffffe2f395e1f20 >> nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 >> --- trap 0x13, rip = 0xffffffff808aada0, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bf60 --- >> __lockmgr_args() at __lockmgr_args+0x30/frame 0xfffffe2ffca2bf60 >> vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe2ffca2bf80 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9d/frame 0xfffffe2ffca2bfb0 >> _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 >> zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 >> zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 >> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 >> vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 >> null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 >> lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 >> namei() at namei+0x504/frame 0xfffffe2ffca2c480 >> vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 >> vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 >> null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 >> VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 >> vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 >> vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 >> kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 >> amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 >> --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- >> >> >> >> Tracing pid 89483 tid 101168 td 0xfffff8048e301920 >> cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 >> ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 >> trap() at trap+0x42/frame 0xfffffe2f395e1f20 >> nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 >> --- trap 0x13, rip = 0xffffffff80e6110e, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bfb0 --- >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0xae/frame 0xfffffe2ffca2bfb0 >> _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 >> zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 >> zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 >> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 >> vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 >> null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 >> lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 >> namei() at namei+0x504/frame 0xfffffe2ffca2c480 >> vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 >> vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 >> null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 >> VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 >> vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 >> vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 >> kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 >> amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 >> --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- > > Hmm, if you are still in this state, can you do a ktrace of this running > process and see if it is logging any syscall events (just want to make sure it > is stuck in the kernel rather than constantly calling system calls due to a > loop in userland)? > > Also, do you have a nullfs mount of a zfs path? (It looks that way from your > stack trace). I did find this thread: > > https://www.mail-archive.com/freebsd-current@freebsd.org/msg147678.html > > Andriy (cc'd) probably has better insight into this than I. > >> On 08/29/2014 11:24 AM, John Baldwin wrote: >>> On Wednesday, August 27, 2014 02:24:47 PM Daniel Andersen wrote: >>>> On 08/27/2014 01:39 PM, John Baldwin wrote: >>>>> These are all blocked in "zfs" then. (For future reference, the 'mwchan' >>>>> field that you see as 'STATE' in top or via 'ps O mwchan' is more detailed >>>>> than the 'D' state.) >>>>> >>>>> To diagnose this further, you would need to see which thread holds the >>>>> ZFS vnode lock these threads need. I have some gdb scripts you can use to >>>>> do that at www.freebsd.org/~jhb/gdb/. You would want to download 'gdb6*' >>>>> files from there and then do this as root: >>>>> >>>>> # cd /path/to/gdb/files >>>>> # kgdb >>>>> (kgdb) source gdb6 >>>>> (kgdb) sleepchain 42335 >>>>> >>>>> Where '42335' is the pid of some process stuck in "zfs". >>>> >>>> I will keep this in mind the next time the machine wedges. Another data >>>> point: the second procstat output I sent was the most recent. All the >>>> processes listed there were after the fact. The process that started the >>>> entire problem ( this time ) was sudo, and it only has this one entry in >>>> procstat: >>>> >>>> 38003 102797 sudo - >>>> >>>> Of note, this does not appear to be blocked on zfs in anyway. 'ps' showed >>>> it in 'R' state instead of 'D' ( I will be sure to use mwchan in the >>>> future. ) It appeared to be pegging an entire CPU core at 100% usage, as >>>> well, and was only single threaded. >>> >>> Well, if it is spinning in some sort of loop in the kernel while holding a >>> ZFS vnode lock that could be blocking all the other threads. In that case, >>> you don't need to do what I asked for above. Instead, we need to find out >>> what that thread is doing. There are two ways of doing this. One is to >>> force a panic via 'sysctl debug.kdb.panic=1' and then use kgdb on the >>> crashdump to determine what the running thread is doing. Another option >>> is to break into the DDB debugger on the console (note that you will need >>> to build a custom kernel with DDB if you are on stable) and request a >>> stack trace of the running process via 'tr '. Ideally you can do this >>> over a serial console so you can just cut and paste the output of the trace >>> into a mail. Over a video console you can either transcribe it by hand or >>> take photos. >