From owner-freebsd-fs@freebsd.org Sun Mar 27 04:34:10 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2532EADF889 for ; Sun, 27 Mar 2016 04:34:10 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 154731D40 for ; Sun, 27 Mar 2016 04:34:10 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 11198ADF888; Sun, 27 Mar 2016 04:34:10 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E7C7ADF887 for ; Sun, 27 Mar 2016 04:34:10 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id 878151D3F for ; Sun, 27 Mar 2016 04:34:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 705AAD60207 for ; Sun, 27 Mar 2016 15:34:00 +1100 (AEDT) Date: Sun, 27 Mar 2016 15:34:00 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: fs@freebsd.org Subject: nfs pessimized by vnode pages changes Message-ID: <20160327144755.Y4269@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=EfU1O6SC c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=djkEBVEjuCfxqb0UTpoA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Mar 2016 04:34:10 -0000 I debugged another pessimization of nfs. ncl_getpages() is now almost always called with a count of 1 page, due to the change changing the count from faultcount to 1 in r292373 in vm_fault(). The only exception seems to be for the initial pagein for exec -- this is still normally the nfs rsize. ncl_getpages() doesn't do any readahead stuff like vnode_pager_generic_getpages() does, so it normally does read RPCs of size 1 page instead of the the nfs rsize. This gives the following increases in read RPCs for makeworld of an old world: - with rsize = 16K, from 24k to 39k (the worst case would be 4 times as many) - with rsize = 8K, from 39k to 44k (the worst case would be 2 times as many). Also, nfs_getpages() has buggy logic which works accidentally if the count is 1: X diff -c2 ./fs/nfsclient/nfs_clbio.c~ ./fs/nfsclient/nfs_clbio.c X *** ./fs/nfsclient/nfs_clbio.c~ Sun Mar 27 01:31:38 2016 X --- ./fs/nfsclient/nfs_clbio.c Sun Mar 27 02:35:32 2016 X *************** X *** 135,140 **** X */ X VM_OBJECT_WLOCK(object); X ! if (pages[npages - 1]->valid != 0 && --npages == 0) X goto out; X VM_OBJECT_WUNLOCK(object); X X --- 135,155 ---- X */ X VM_OBJECT_WLOCK(object); X ! #if 0 X ! /* This matches the comment. but doesn't work (has little effect). */ X ! if (pages[0]->valid != 0) X goto out; The comment still says that the code checks the requested page, but that is no longer passed to the function in a_reqpage. The first page is a better geuss of the requested page than the last one, but when npages is 1 these pages are the same. X + #else X + if (pages[0]->valid != 0) X + printf("ncl_getpages: page 0 valid; npages %d\n", npages); X + for (i = 0; i < npages; i++) X + if (pages[i]->valid != 0) X + printf("ncl_getpages: page %d valid; npages %d\n", X + i, npages); X + for (i = 0; i < npages; i++) X + if (pages[i]->valid != 0) X + npages = i; X + if (npages == 0) X + goto out; X + #endif Debugging and more forceful guessing code. This makes little difference except of course to spam the console. X VM_OBJECT_WUNLOCK(object); X X *************** X *** 199,202 **** X --- 214,220 ---- X KASSERT(m->dirty == 0, X ("nfs_getpages: page %p is dirty", m)); X + printf("ncl_getpages: partial page %d of %d %s\n", X + i, npages, X + pages[i]->valid != 0 ? "valid" : "invalid"); X } else { X /* X *************** X *** 210,215 **** X --- 228,239 ---- X */ X ; X + printf("ncl_getpages: short page %d of %d %s\n", X + i, npages, X + pages[i]->valid != 0 ? "valid" : "invalid"); X } X } X + for (i = 0; i < npages; i++) X + printf("ncl_getpages: page %d of %d %s\n", X + i, npages, pages[i]->valid != 0 ? "valid" : "invalid"); X out: X VM_OBJECT_WUNLOCK(object); Further debugging code. Similar debugging code in the old working version shows that normal operation for paging in a 15K file with an rsize of 16K is: - call here with npages = 4 - page in 3 full pages and 1 partial page using 1 RPC - call here again with npages = 1 for the partial page - use the optimization of returning early for this page -- don't do another RPC The buggy version does: - call here with npages = 1; page in 1 full page using 1 RPC - call here with npages = 1; page in 1 full page using 1 RPC - call here with npages = 1; page in 1 full page using 1 RPC - call here with npages = 1; page in 1 partial page using 1 RPC - call here again with npages = 1 for the partial page; the optimization works as before. The partial page isn't handled very well, but at least there is no extra physical i/o for it, at least if it is at EOF. vfs clustering handles partial pages even worse than this. Bruce From owner-freebsd-fs@freebsd.org Sun Mar 27 14:37:15 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B102ADF6C0 for ; Sun, 27 Mar 2016 14:37:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2BBB310F4 for ; Sun, 27 Mar 2016 14:37:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2REbDOr063075 for ; Sun, 27 Mar 2016 14:37:15 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 206521] Can't decrypt disks on ZFS+Geli installation after order of devices changed Date: Sun, 27 Mar 2016 14:37:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: fk@fabiankeil.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Mar 2016 14:37:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206521 Fabian Keil changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |allanjude@FreeBSD.org, | |fk@fabiankeil.de --- Comment #2 from Fabian Keil --- If I don't misinterpret the code, bsdinstall puts the plain disk name into loader.conf when specifying the keyfile. Updating the keyfile entry/entries should solve the problem. To prevent the issue in the first place, bsdinstall could use GPT labels. The ElectroBSD installer cloudiatr does this and reordering disks doesn't seem to cause any problems. Example: geli_gpt_rpool-ada0_keyfile0_load=3D"YES" geli_gpt_rpool-ada0_keyfile0_type=3D"gpt/rpool-ada0:geli_keyfile0" geli_gpt_rpool-ada0_keyfile0_name=3D"/boot/rpool.key" [...] geli_gpt_rpool-ada3_keyfile0_load=3D"YES" geli_gpt_rpool-ada3_keyfile0_type=3D"gpt/rpool-ada3:geli_keyfile0" geli_gpt_rpool-ada3_keyfile0_name=3D"/boot/rpool.key" Putting the original disk name into the label is a bit confusing when the disk name changes, though, so I'm considering using generic names like disk1, disk2 etc. in the future. CC'in Alan who worked on the relevant bsdinstall code in the past. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Mar 27 15:15:58 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCA10ADF126 for ; Sun, 27 Mar 2016 15:15:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD31F138A for ; Sun, 27 Mar 2016 15:15:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2RFFwmY041439 for ; Sun, 27 Mar 2016 15:15:58 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 206521] Can't decrypt disks on ZFS+Geli installation after order of devices changed Date: Sun, 27 Mar 2016 15:15:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Mar 2016 15:15:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206521 --- Comment #3 from Allan Jude --- (In reply to Fabian Keil from comment #2) The bit of this related to the GELIBoot commit, was meant to quickly be followed by a patch to the installer to create a single encrypted pool, wit= hout using key files (because they are not supported by GELIBoot yet) This change was delayed due to problems encountered, and my personal lack of time to fix them. I hope to have a patch for the installer soon that will n= ot use key files if the configuration allows a GELIBoot style install (GPT, ZF= S, single pool) The reason GPT labels were not used was: A) support for MBR B) GPT labels disappear if the disk_ident label is used first (this caused issues when GPT labels were used for swap in previous versions) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Mar 27 21:00:08 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5D99ADF9F0 for ; Sun, 27 Mar 2016 21:00:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4C0C1764 for ; Sun, 27 Mar 2016 21:00:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2RL01Ao072583 for ; Sun, 27 Mar 2016 21:00:08 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201603272100.u2RL01Ao072583@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 27 Mar 2016 21:00:08 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Mar 2016 21:00:09 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 4 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Sun Mar 27 23:40:20 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19721AE007A for ; Sun, 27 Mar 2016 23:40:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E446D1658 for ; Sun, 27 Mar 2016 23:40:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2RNeJFb050893 for ; Sun, 27 Mar 2016 23:40:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 199189] SWAP on ZFS can crash server Date: Sun, 27 Mar 2016 23:40:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: cmangin@arobas.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Mar 2016 23:40:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 Chris M changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cmangin@arobas.net --- Comment #17 from Chris M --- I'd like to point out this page which has relevant information for those wishing to tune the VM system:=20 https://wiki.freebsd.org/AvgPageoutAlgorithm The tunable vm.pageout_wakeup_thresh hasn't been mentioned here yet and see= ms to be important for the way the the VM subsystem works. So with that in mind, here's my current configuration for a 8Gb machine with swap on zvol. vm.v_free_severe=3D20460 vm.v_free_min=3D32768 vm.v_free_target=3D51200 vm.pageout_wakeup_thresh=3D36044 vfs.zfs.arc_free_target=3D36044 Note that I've picked: vm.pageout_wakeup_thresh =3D vm.v_free_min * 11/10 vfs.zfs.arc_free_target =3D vm.pageout_wakeup_thresh as these are the way they are calculated by default. As a reference, the defaults values for my system are: vm.v_free_reserved=3D2650 (untouched) vm.v_free_severe=3D7642 vm.v_free_min=3D12634 vm.v_free_target=3D42586 vm.pageout_wakeup_thresh=3D13893 vfs.zfs.arc_free_target=3D13893 I submitted my system to memory pressure, notably by runs of poudriere generating up to 3Gb of swap usage. I'm happy to report that my system rema= ined rock stable and very responsive during the tests. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Mar 28 09:00:59 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99F9FAE0FCE for ; Mon, 28 Mar 2016 09:00:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8395F120D for ; Mon, 28 Mar 2016 09:00:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2S90vfZ021218 for ; Mon, 28 Mar 2016 09:00:59 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 199189] SWAP on ZFS can crash server Date: Mon, 28 Mar 2016 09:00:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 09:00:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 --- Comment #18 from Andriy Gapon --- (In reply to Chris M from comment #17) The information in that link is outdated. There have been some notable chan= ges in the page out algorithms since that article was written. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Mar 28 14:32:11 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 938C3AE0A4A for ; Mon, 28 Mar 2016 14:32:11 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5E80F1B32 for ; Mon, 28 Mar 2016 14:32:11 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-oi0-x234.google.com with SMTP id d205so172097683oia.0 for ; Mon, 28 Mar 2016 07:32:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=nFHSgz7ROAgoRyFIlHeUHffdLusgdKnQIT7OaRGdziY=; b=rg9Ndt2Q/qo7kwqw4NNURS38yvUE8596F65zFns7cJWpZCSYS/NQv2fiTznUAsyBFK 8UWp9vt6RUUuKALfOL9I5DLiWeE6vsPqOtCuyN/CHkXiECzhSIxDmklCH/zSjthUFbaR RCKYassYGqu0rPQw5h6f0ULGd/38GOeC2iuAZz7yKPruLW8D3n10jcVqMyEDFSWSb636 97bFRSfvaHbtEnrrNlhQOTFqONpQQ11LZraMoe6woNkNplCprP0rXW25KeAX7q4HPaVb Re8GVMnUytGecrLRCvlEYJIXDtWbEyBNvqYSZohSiJo2GyJh3FYdY8DFvMq5yv79ZzzN TsqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=nFHSgz7ROAgoRyFIlHeUHffdLusgdKnQIT7OaRGdziY=; b=fLY8A+PgBlsGtu5skcm+57+tRO2qkpSPzKSZpIXZCUpu1lKQ8FTebplStVmBZ0U41A +JnuYPVZTVqkTCs+XZNLvL78ULfYqc8QVwmWyxBRLlRI16ZPGJ7GYJnaj6YalAnnTqGt LcN0dt9/wTATKk4mgP627Rin8TO/JvPPFC7t7SnU1jKIOgKDKsYsSbjJEDquU2NNRm4s N9vSmrALKdrBTuM9meDPUaWlnxi0otW3RDAJ6ZvWFHzRphxYinibu1Zfa39fJ8iZ2ZU5 eWWeZsfGmll+Sr+S/qcr9LEX743JzL3GMK0Wd48ktLKgEPxyVU6gv3CsnIK2CnxttHq6 rulw== X-Gm-Message-State: AD7BkJKdcTqT8NnLByFyiS0513qOW7HwJ2GpC06mQzyXSnZUwoHIVX9Wq7Ayvi3bAOEbcaMYDznbfE/AuNEf1rZU MIME-Version: 1.0 X-Received: by 10.202.200.16 with SMTP id y16mr1874176oif.92.1459175493053; Mon, 28 Mar 2016 07:31:33 -0700 (PDT) Received: by 10.157.11.143 with HTTP; Mon, 28 Mar 2016 07:31:32 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Mar 2016 07:31:32 -0700 Message-ID: Subject: Re: Process stuck in "vnread" From: Maxim Sobolev To: stable@freebsd.org, freebsd-fs@freebsd.org Cc: Kirk McKusick , kib@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 14:32:11 -0000 OK, that happened again. Now it's 10.3-RC3, funny enough, it's the same "cp" command. Any ideas about what can be wrong here? The box is still up, so if you need me to do something specific inside kgdb please let me know. Thanks! $ uname -a FreeBSD abc01.sippysoft.com 10.3-RC3 FreeBSD 10.3-RC3 #0 bedc749(master): Wed Mar 23 14:39:07 PDT 2016 sobomax@abc01.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC01 amd64 $ ps -xaHl -O lwp UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND PID LWP TT STAT TIME COMMAND 0 39143 38949 0 52 0 12368 1064 vnread DL+ 2 0:00.00 cp /usr/local/sh 39143 101176 2 DL+ 0:00.00 cp /usr/local/share/automake-1.15/compile ./compile (kgdb) thread 473 [Switching to thread 473 (Thread 101176)]#0 sched_switch (td=0xfffff803320b84b0, newtd=, flags=) at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 1945 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xfffff803320b84b0, newtd=, flags=) at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 #1 0xffffffff809364d1 in mi_switch (flags=260, newtd=0x0) at /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:491 #2 0xffffffff809761ca in sleepq_wait (wchan=0x0, pri=0) at /usr/home/sobomax/projects/freebsd103/sys/kern/subr_sleepqueue.c:618 #3 0xffffffff80935ef7 in _sleep (ident=, lock=, priority=, wmesg=, sbt=, pr=, flags=) at /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:255 #4 0xffffffff809b7c6e in bwait (bp=0xfffffe03e1012048, pri=, wchan=0xffffffff81069471 "vnread") at /usr/home/sobomax/projects/freebsd103/sys/kern/vfs_bio.c:4431 #5 0xffffffff80b56663 in vnode_pager_generic_getpages (vp=0xfffff802b7e73588, m=0xfffffe0468c4ebe0, bytecount=, reqpage=) at /usr/home/sobomax/projects/freebsd103/sys/vm/vnode_pager.c:907 #6 0xffffffff80dd7fa7 in VOP_GETPAGES_APV (vop=, a=) at vnode_if.c:2795 #7 0xffffffff80b54a0a in vnode_pager_getpages (object=, m=0xfffffe0468c4ebe0, count=, reqpage=0) at vnode_if.h:1154 #8 0xffffffff80b354a1 in vm_fault_hold (map=0xfffff800294c98c0, vaddr=34366300160, fault_type=, fault_flags=0, m_hold=0x0) at vm_pager.h:127 #9 0xffffffff80b34c07 in vm_fault (map=0xfffff800294c98c0, vaddr=, fault_type=1 '\001', fault_flags=0) at /usr/home/sobomax/projects/freebsd103/sys/vm/vm_fault.c:273 #10 0xffffffff80cbaef5 in trap_pfault (frame=0xfffffe0468c4f040, usermode=0) at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/trap.c:747 #11 0xffffffff80cba65d in trap (frame=0xfffffe0468c4f040) at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/trap.c:447 #12 0xffffffff80ca0592 in calltrap () at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:236 #13 0xffffffff80cb8b71 in copyin () at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/support.S:311 #14 0xffffffff8097d70f in uiomove_faultflag (cp=, n=, uio=0xfffffe0468c4f960, nofault=) at /usr/home/sobomax/projects/freebsd103/sys/kern/subr_uio.c:253 ---Type to continue, or q to quit--- #15 0xffffffff819f799c in dmu_write_uio_dnode () from /boot/kernel/zfs.ko #16 0xffffffff819f78b2 in dmu_write_uio_dbuf () from /boot/kernel/zfs.ko #17 0xffffffff81a91a07 in zfs_freebsd_write () from /boot/kernel/zfs.ko #18 0xffffffff80dd6219 in VOP_WRITE_APV (vop=, a=) at vnode_if.c:999 #19 0xffffffff808730b2 in null_bypass (ap=0xfffffe0468c4f7d0) at /usr/home/sobomax/projects/freebsd103/sys/fs/nullfs/null_vnops.c:280 #20 0xffffffff80dd621f in VOP_WRITE_APV (vop=, a=) at vnode_if.c:1001 #21 0xffffffff809dea28 in vn_write (fp=0xfffff8004bc0ddc0, uio=0xfffffe0468c4f960, active_cred=0x1ca5, flags=916, td=0xfffffe00006a60d0) at vnode_if.h:413 #22 0xffffffff809dad6b in vn_io_fault (fp=0xfffff8004bc0ddc0, uio=0xfffffe0468c4f960, active_cred=0x1ca5, flags=0, td=0xfffffe00006a60d0) at /usr/home/sobomax/projects/freebsd103/sys/kern/vfs_vnops.c:1225 #23 0xffffffff80981937 in dofilewrite (td=0xfffff803320b84b0, fd=4, fp=0xfffff8004bc0ddc0, auio=0xfffffe0468c4f960, offset=, flags=0) at file.h:305 #24 0xffffffff80981668 in kern_writev (td=0xfffff803320b84b0, fd=4, auio=0xfffffe0468c4f960) at /usr/home/sobomax/projects/freebsd103/sys/kern/sys_generic.c:481 #25 0xffffffff809815f3 in sys_write (td=, uap=) at /usr/home/sobomax/projects/freebsd103/sys/kern/sys_generic.c:396 #26 0xffffffff80cbb559 in amd64_syscall (td=0xfffff803320b84b0, traced=0) at subr_syscall.c:141 #27 0xffffffff80ca087b in Xfast_syscall () at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:396 #28 0x0000000800968f5a in ?? () On Wed, Mar 2, 2016 at 1:12 AM, Maxim Sobolev wrote: > Hi, I've encountered cp(1) process stuck in the vnread state on one of my > build machines that got recently upgraded to 10.3. > > 0 79596 1 0 20 0 17092 1396 wait I 1 0:00.00 > /bin/sh /usr/local/bin/autoreconf -f -i > 0 79602 79596 0 52 0 41488 9036 wait I 1 0:00.07 > /usr/local/bin/perl -w /usr/local/bin/autoreconf-2.69 -f -i > 0 79639 79602 0 72 0 0 0 - Z 1 0:00.27 > > 0 79762 79602 0 20 0 17092 1396 wait I 1 0:00.00 > /bin/sh /usr/local/bin/automake --add-missing --copy --force-missing > 0 79768 79762 0 52 0 49736 13936 wait I 1 0:00.11 > /usr/local/bin/perl -w /usr/local/bin/automake-1.15 --add-missing --copy > --force-missing > 0 79962 79768 0 20 0 12368 1024 vnread DL 1 0:00.00 > cp /usr/local/share/automake-1.15/compile ./compile > > I am not sure if it's related to that OS version upgrade, but I have not > seen any such issues on the same machine in 2-3 years running essentially > the same build process with version 9.x, 10.0, 10.1 and 10.2. > > $ uname -a > FreeBSD van01.sippysoft.com 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #1 > 80de3e2(master)-dirty: Tue Feb 2 12:19:57 PST 2016 > sobomax@abc.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC > amd64 > > The kernel stack trace is: > > (kgdb) thread 360 > [Switching to thread 360 (Thread 100515)]#0 0xffffffff8095244e in > sched_switch () > (kgdb) bt > #0 0xffffffff8095244e in sched_switch () > #1 0xffffffff809313b1 in mi_switch () > #2 0xffffffff8097089a in sleepq_wait () > #3 0xffffffff80930dd7 in _sleep () > #4 0xffffffff809b230e in bwait () > #5 0xffffffff80b511f3 in vnode_pager_generic_getpages () > #6 0xffffffff80dd1607 in VOP_GETPAGES_APV () > #7 0xffffffff80b4f59a in vnode_pager_getpages () > #8 0xffffffff80b30031 in vm_fault_hold () > #9 0xffffffff80b2f797 in vm_fault () > #10 0xffffffff80cb5a75 in trap_pfault () > #11 0xffffffff80cb51dd in trap () > #12 0xffffffff80c9b122 in calltrap () > #13 0xffffffff80cb36f1 in copyin () > #14 0xffffffff80977ddf in uiomove_faultflag () > > The FS stack configuration is somewhat unique, so I am not sure if I am > hitting some rare race condition or lock ordering issues specific to that. > It's basically ZFS (ZRAID) on top of pair or SATA SSDs with big file on > that FS attached via md(4) and UFS2 on that md(4). The build itself runs in > chroot with that UFS2 fs as its primary root. > > Just maybe additional bit of info, attempting to list the directory with > that UFS image also got my bash process stuck in "zfs" state, backtrace > from that is: > > (kgdb) thread 353 > [Switching to thread 353 (Thread 100508)]#0 0xffffffff8095244e in > sched_switch () > (kgdb) bt > #0 0xffffffff8095244e in sched_switch () > #1 0xffffffff809313b1 in mi_switch () > #2 0xffffffff8097089a in sleepq_wait () > #3 0xffffffff809069ad in sleeplk () > #4 0xffffffff809060e0 in __lockmgr_args () > #5 0xffffffff809b8b7c in vop_stdlock () > #6 0xffffffff80dd0a3b in VOP_LOCK1_APV () > #7 0xffffffff809d6d23 in _vn_lock () > #8 0xffffffff81a8c9cd in ?? () > #9 0x0000000000000000 in ?? () > > > -- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.com Skype: SippySoft From owner-freebsd-fs@freebsd.org Mon Mar 28 15:52:04 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0B07AE0036 for ; Mon, 28 Mar 2016 15:52:04 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com [IPv6:2607:f8b0:4003:c01::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7735C1753 for ; Mon, 28 Mar 2016 15:52:04 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-ob0-x22a.google.com with SMTP id fp4so102654636obb.2 for ; Mon, 28 Mar 2016 08:52:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=ZxqblWL2zXPE5FHJiOAv5cJPrAJlBB3w03BoiPS/6Ts=; b=EDZZ5vwNzvZBNv3wrPEYvBa82UqlZcTL6thnxYPFQ54TxtYkY+2p9K41oz9j+r6M/f /MzDdPaMatvnGo9eogJyOnrBO4gmdUeGxGr6FESN0nal9vk7mN5aVxm/HCdgEqT8IPqZ xjKUtd9aPzlCrB+DXvenVo36NwddKo/4moH5UGs+1rQBvJHiZe5+jpCwathc7lxpybOb r5IqVqxL1Rnonz/SjKxK4yQbCLmu9KKgqDLhwcDCw4iAp72ikmaqYUykWgdEjph1KOVQ kNtjhE7L6ybVUIFy5+FxBck3tFvtkfJvE7VeeNFAczwbzEgD1+LkMjowd9+Np2poPMr6 jI9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=ZxqblWL2zXPE5FHJiOAv5cJPrAJlBB3w03BoiPS/6Ts=; b=jLrUiwMUEflVl+PjftCEVGWTuK7GmCoNFQ65aawtIb6VxYeW+7qI3YXMotrqPgTXPU li+N6GtrLl+g7WffXB/ooaqHnlDXpi0hkZLG1Cuthk3oG0Xg0SB8F8R+s1BMZBz45y3b nz0HZcbnY1m50MR9+X7VnvrCy9k3lEBZotTizdi/pQLg835hTCZDg9epmBUlKZIvRs6B YJyHIaGIwJFMA5UFDRm0XNBPq8BuE3GFMgd6yopMqZiEb8MVInTss3qz9Mwgx6f0iFFF UogJvyXKz7tAUia17fXbZjgSb8J3HPsm/TMeKhlOoqpioPwMygepvhVV9eiYMSNnR4NG 46EQ== X-Gm-Message-State: AD7BkJJ2v5+XJGMF5+oqu7mEl5i8luMceHvXYXoxN8T2R4TSqN5+T6lE/vm9YRf67b/1MnRWt9dC9wkp3nh1zZ6k MIME-Version: 1.0 X-Received: by 10.157.10.230 with SMTP id 93mr10945143otq.53.1459180323804; Mon, 28 Mar 2016 08:52:03 -0700 (PDT) Received: by 10.157.11.143 with HTTP; Mon, 28 Mar 2016 08:52:03 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Mar 2016 08:52:03 -0700 Message-ID: Subject: Re: Process stuck in "vnread" From: Maxim Sobolev To: stable@freebsd.org, freebsd-fs@freebsd.org, Pawel Jakub Dawidek Cc: Kirk McKusick , kib@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 15:52:04 -0000 Done some head scratching, it looks like it's got page fault in the copyin() (cp(1) AFAIK mmaps source file). There might be some interlock issue between competing write to the same ZFS, the md0 device is locked forever waiting for the write operation to complete at the very same time. I am curious as to whether we are allowed to sleep in the dmu_write_uio_dbuf(), AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done earlier to avoid possible page fault in there? (kgdb) thread 465 [Switching to thread 465 (Thread 100530)]#0 sched_switch (td=0xfffff8037b8ff960, newtd=, flags=) at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 1945 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xfffff8037b8ff960, newtd=, flags=) at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 #1 0xffffffff809364d1 in mi_switch (flags=260, newtd=0x0) at /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:491 #2 0xffffffff809761ca in sleepq_wait (wchan=0x0, pri=0) at /usr/home/sobomax/projects/freebsd103/sys/kern/subr_sleepqueue.c:618 #3 0xffffffff808d843d in _cv_wait (cvp=0xfffff8001041b220, lock=0xfffff8001041b1b8) at /usr/home/sobomax/projects/freebsd103/sys/kern/kern_condvar.c:151 #4 0xffffffff81a431e5 in txg_wait_synced () from /boot/kernel/zfs.ko #5 0xffffffff81a62fa9 in zil_commit () from /boot/kernel/zfs.ko #6 0xffffffff81a9216b in zfs_freebsd_write () from /boot/kernel/zfs.ko #7 0xffffffff80dd6219 in VOP_WRITE_APV (vop=, a=) at vnode_if.c:999 #8 0xffffffff8060b69a in mdstart_vnode (sc=0xfffff80057f78000, bp=0xfffff8036d4660f8) at vnode_if.h:413 #9 0xffffffff8060caad in md_kthread (arg=0xfffff80057f78000) at /usr/home/sobomax/projects/freebsd103/sys/dev/md/md.c:1147 #10 0xffffffff808f6e6a in fork_exit (callout=0xffffffff8060c900 , arg=0xfffff80057f78000, frame=0xfffffe04689b2ac0) at /usr/home/sobomax/projects/freebsd103/sys/kern/kern_fork.c:1027 #11 0xffffffff80ca0a5e in fork_trampoline () at /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:611 #12 0x0000000000000000 in ?? () On Mon, Mar 28, 2016 at 7:31 AM, Maxim Sobolev wrote: > OK, that happened again. Now it's 10.3-RC3, funny enough, it's the same > "cp" command. Any ideas about what can be wrong here? The box is still up, > so if you need me to do something specific inside kgdb please let me know. > Thanks! > > $ uname -a > FreeBSD abc01.sippysoft.com 10.3-RC3 FreeBSD 10.3-RC3 #0 bedc749(master): > Wed Mar 23 14:39:07 PDT 2016 sobomax@abc01.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC01 > amd64 > > > $ ps -xaHl -O lwp > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME > COMMAND PID LWP TT STAT TIME COMMAND > 0 39143 38949 0 52 0 12368 1064 vnread DL+ 2 0:00.00 > cp /usr/local/sh 39143 101176 2 DL+ 0:00.00 cp > /usr/local/share/automake-1.15/compile ./compile > > > (kgdb) thread 473 > [Switching to thread 473 (Thread 101176)]#0 sched_switch > (td=0xfffff803320b84b0, newtd=, flags= out>) > at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 > 1945 cpuid = PCPU_GET(cpuid); > (kgdb) bt > #0 sched_switch (td=0xfffff803320b84b0, newtd=, > flags=) > at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 > #1 0xffffffff809364d1 in mi_switch (flags=260, newtd=0x0) at > /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:491 > #2 0xffffffff809761ca in sleepq_wait (wchan=0x0, pri=0) at > /usr/home/sobomax/projects/freebsd103/sys/kern/subr_sleepqueue.c:618 > #3 0xffffffff80935ef7 in _sleep (ident=, lock= optimized out>, priority=, wmesg=, > sbt=, pr=, flags= optimized out>) at > /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:255 > #4 0xffffffff809b7c6e in bwait (bp=0xfffffe03e1012048, pri= optimized out>, wchan=0xffffffff81069471 "vnread") > at /usr/home/sobomax/projects/freebsd103/sys/kern/vfs_bio.c:4431 > #5 0xffffffff80b56663 in vnode_pager_generic_getpages > (vp=0xfffff802b7e73588, m=0xfffffe0468c4ebe0, bytecount= out>, > reqpage=) at > /usr/home/sobomax/projects/freebsd103/sys/vm/vnode_pager.c:907 > #6 0xffffffff80dd7fa7 in VOP_GETPAGES_APV (vop=, > a=) at vnode_if.c:2795 > #7 0xffffffff80b54a0a in vnode_pager_getpages (object= out>, m=0xfffffe0468c4ebe0, count=, reqpage=0) > at vnode_if.h:1154 > #8 0xffffffff80b354a1 in vm_fault_hold (map=0xfffff800294c98c0, > vaddr=34366300160, fault_type=, fault_flags=0, > m_hold=0x0) > at vm_pager.h:127 > #9 0xffffffff80b34c07 in vm_fault (map=0xfffff800294c98c0, vaddr= optimized out>, fault_type=1 '\001', fault_flags=0) > at /usr/home/sobomax/projects/freebsd103/sys/vm/vm_fault.c:273 > #10 0xffffffff80cbaef5 in trap_pfault (frame=0xfffffe0468c4f040, > usermode=0) at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/trap.c:747 > #11 0xffffffff80cba65d in trap (frame=0xfffffe0468c4f040) at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/trap.c:447 > #12 0xffffffff80ca0592 in calltrap () at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:236 > #13 0xffffffff80cb8b71 in copyin () at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/support.S:311 > #14 0xffffffff8097d70f in uiomove_faultflag (cp=, > n=, uio=0xfffffe0468c4f960, nofault= out>) > at /usr/home/sobomax/projects/freebsd103/sys/kern/subr_uio.c:253 > ---Type to continue, or q to quit--- > #15 0xffffffff819f799c in dmu_write_uio_dnode () from /boot/kernel/zfs.ko > #16 0xffffffff819f78b2 in dmu_write_uio_dbuf () from /boot/kernel/zfs.ko > #17 0xffffffff81a91a07 in zfs_freebsd_write () from /boot/kernel/zfs.ko > #18 0xffffffff80dd6219 in VOP_WRITE_APV (vop=, > a=) at vnode_if.c:999 > #19 0xffffffff808730b2 in null_bypass (ap=0xfffffe0468c4f7d0) at > /usr/home/sobomax/projects/freebsd103/sys/fs/nullfs/null_vnops.c:280 > #20 0xffffffff80dd621f in VOP_WRITE_APV (vop=, > a=) at vnode_if.c:1001 > #21 0xffffffff809dea28 in vn_write (fp=0xfffff8004bc0ddc0, > uio=0xfffffe0468c4f960, active_cred=0x1ca5, flags=916, > td=0xfffffe00006a60d0) > at vnode_if.h:413 > #22 0xffffffff809dad6b in vn_io_fault (fp=0xfffff8004bc0ddc0, > uio=0xfffffe0468c4f960, active_cred=0x1ca5, flags=0, td=0xfffffe00006a60d0) > at /usr/home/sobomax/projects/freebsd103/sys/kern/vfs_vnops.c:1225 > #23 0xffffffff80981937 in dofilewrite (td=0xfffff803320b84b0, fd=4, > fp=0xfffff8004bc0ddc0, auio=0xfffffe0468c4f960, offset= out>, > flags=0) at file.h:305 > #24 0xffffffff80981668 in kern_writev (td=0xfffff803320b84b0, fd=4, > auio=0xfffffe0468c4f960) > at /usr/home/sobomax/projects/freebsd103/sys/kern/sys_generic.c:481 > #25 0xffffffff809815f3 in sys_write (td=, uap= optimized out>) > at /usr/home/sobomax/projects/freebsd103/sys/kern/sys_generic.c:396 > #26 0xffffffff80cbb559 in amd64_syscall (td=0xfffff803320b84b0, traced=0) > at subr_syscall.c:141 > #27 0xffffffff80ca087b in Xfast_syscall () at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:396 > #28 0x0000000800968f5a in ?? () > > > On Wed, Mar 2, 2016 at 1:12 AM, Maxim Sobolev wrote: > >> Hi, I've encountered cp(1) process stuck in the vnread state on one of my >> build machines that got recently upgraded to 10.3. >> >> 0 79596 1 0 20 0 17092 1396 wait I 1 >> 0:00.00 /bin/sh /usr/local/bin/autoreconf -f -i >> 0 79602 79596 0 52 0 41488 9036 wait I 1 >> 0:00.07 /usr/local/bin/perl -w /usr/local/bin/autoreconf-2.69 -f -i >> 0 79639 79602 0 72 0 0 0 - Z 1 >> 0:00.27 >> 0 79762 79602 0 20 0 17092 1396 wait I 1 >> 0:00.00 /bin/sh /usr/local/bin/automake --add-missing --copy --force-missing >> 0 79768 79762 0 52 0 49736 13936 wait I 1 >> 0:00.11 /usr/local/bin/perl -w /usr/local/bin/automake-1.15 --add-missing >> --copy --force-missing >> 0 79962 79768 0 20 0 12368 1024 vnread DL 1 >> 0:00.00 cp /usr/local/share/automake-1.15/compile ./compile >> >> I am not sure if it's related to that OS version upgrade, but I have not >> seen any such issues on the same machine in 2-3 years running essentially >> the same build process with version 9.x, 10.0, 10.1 and 10.2. >> >> $ uname -a >> FreeBSD van01.sippysoft.com 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #1 >> 80de3e2(master)-dirty: Tue Feb 2 12:19:57 PST 2016 >> sobomax@abc.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC >> amd64 >> >> The kernel stack trace is: >> >> (kgdb) thread 360 >> [Switching to thread 360 (Thread 100515)]#0 0xffffffff8095244e in >> sched_switch () >> (kgdb) bt >> #0 0xffffffff8095244e in sched_switch () >> #1 0xffffffff809313b1 in mi_switch () >> #2 0xffffffff8097089a in sleepq_wait () >> #3 0xffffffff80930dd7 in _sleep () >> #4 0xffffffff809b230e in bwait () >> #5 0xffffffff80b511f3 in vnode_pager_generic_getpages () >> #6 0xffffffff80dd1607 in VOP_GETPAGES_APV () >> #7 0xffffffff80b4f59a in vnode_pager_getpages () >> #8 0xffffffff80b30031 in vm_fault_hold () >> #9 0xffffffff80b2f797 in vm_fault () >> #10 0xffffffff80cb5a75 in trap_pfault () >> #11 0xffffffff80cb51dd in trap () >> #12 0xffffffff80c9b122 in calltrap () >> #13 0xffffffff80cb36f1 in copyin () >> #14 0xffffffff80977ddf in uiomove_faultflag () >> >> The FS stack configuration is somewhat unique, so I am not sure if I am >> hitting some rare race condition or lock ordering issues specific to that. >> It's basically ZFS (ZRAID) on top of pair or SATA SSDs with big file on >> that FS attached via md(4) and UFS2 on that md(4). The build itself runs in >> chroot with that UFS2 fs as its primary root. >> >> Just maybe additional bit of info, attempting to list the directory with >> that UFS image also got my bash process stuck in "zfs" state, backtrace >> from that is: >> >> (kgdb) thread 353 >> [Switching to thread 353 (Thread 100508)]#0 0xffffffff8095244e in >> sched_switch () >> (kgdb) bt >> #0 0xffffffff8095244e in sched_switch () >> #1 0xffffffff809313b1 in mi_switch () >> #2 0xffffffff8097089a in sleepq_wait () >> #3 0xffffffff809069ad in sleeplk () >> #4 0xffffffff809060e0 in __lockmgr_args () >> #5 0xffffffff809b8b7c in vop_stdlock () >> #6 0xffffffff80dd0a3b in VOP_LOCK1_APV () >> #7 0xffffffff809d6d23 in _vn_lock () >> #8 0xffffffff81a8c9cd in ?? () >> #9 0x0000000000000000 in ?? () >> >> >> > > > -- > Maksym Sobolyev > Sippy Software, Inc. > Internet Telephony (VoIP) Experts > Tel (Canada): +1-778-783-0474 > Tel (Toll-Free): +1-855-747-7779 > Fax: +1-866-857-6942 > Web: http://www.sippysoft.com > MSN: sales@sippysoft.com > Skype: SippySoft > -- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.com Skype: SippySoft From owner-freebsd-fs@freebsd.org Mon Mar 28 15:58:31 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16278AE031C for ; Mon, 28 Mar 2016 15:58:31 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-ob0-x230.google.com (mail-ob0-x230.google.com [IPv6:2607:f8b0:4003:c01::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCF021B44 for ; Mon, 28 Mar 2016 15:58:30 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-ob0-x230.google.com with SMTP id m7so102510073obh.3 for ; Mon, 28 Mar 2016 08:58:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc; bh=lMdhA966Ck33pg7Pw25CvG+6Xr1DUjkVogO2iUcdE/4=; b=NELwqPa9ggOYVbTvWs1yi7fx7TikFBqo8ASy9+/ShrJ2JAooVhFddKhveVmC2JtK7m jC211IH7vDF1mshR+rDUISCqA0c08zCAFYL4VbPrbeIEytb8JlRGhLgSmQM1m91HGAGJ hoN20CyGZ/1NH5yGG6V9nyfsgt9jCbLBH7HupoM4gybhGFlo1DGXin4bXCUhdjAyMUmW vDuLrbMRyyaBqT7Y6rthotKCycmc7dqfIU6H+p7NACaCNSPapkWBupgweZbDikkoi94Z o835x8qW+ceWcPR//vuOfWI+jrpQMEVyfe0zAJno+f9tWaj+CJxM5xrIuJy4zVQqi6Vd HpXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc; bh=lMdhA966Ck33pg7Pw25CvG+6Xr1DUjkVogO2iUcdE/4=; b=jt4skKvmXxc7zEBo7PMFgG6aSwoOqmc6sVIFJFCxnqssSkc+YboRl6f+9R+f35umWr FhzRKI+JBdF87wnzdeJr2MEK9Wl523tG+F8ijgdIW8yiY3Kng7aG4UMWmunFA74g1+0W WrCTF6MsrrM7sXJp2Y2OheVH0xloLnkSBXeJCgDw9D6OtsmQq9DJ5Xp8Irr8vfsB5kTU qw+4iYv44k92tEp+smvyObTC2C4P0qS/IrySArXm8ckU994ZFNDwMxtYc2Gy6AsyCfND koGgGyHAIDfWIEQSiZEUXVOrA7DccIhdb6Sis7QpbItcRsIzs0VF8Jdf+RE/XK2iLi+d dwoA== X-Gm-Message-State: AD7BkJI5e9y/x2NHJP9+6sO2kE58KH4ZC/P9RUaU32Uw3LgUlkg5Ckv1TVSo4YR21P9ZLYzlBpiz4t6LB89yz7q7 MIME-Version: 1.0 X-Received: by 10.60.60.3 with SMTP id d3mr8771616oer.66.1459180709858; Mon, 28 Mar 2016 08:58:29 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.157.11.143 with HTTP; Mon, 28 Mar 2016 08:58:29 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Mar 2016 08:58:29 -0700 X-Google-Sender-Auth: Vq_9D_7iSv26-dxjhMqJjv_-5tI Message-ID: Subject: Re: Process stuck in "vnread" From: Maxim Sobolev To: stable@freebsd.org, freebsd-fs@freebsd.org, Pawel Jakub Dawidek Cc: Kirk McKusick , kib@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 15:58:31 -0000 P.S. That being said, I am not sure if that write operation on md(4) happen before or after the offending lockup in the zfs_freebsd_write(), so it might be just as well be the result of that, not cause. On Mon, Mar 28, 2016 at 8:52 AM, Maxim Sobolev wrote: > Done some head scratching, it looks like it's got page fault in the > copyin() (cp(1) AFAIK mmaps source file). There might be some interlock > issue between competing write to the same ZFS, the md0 device is locked > forever waiting for the write operation to complete at the very same time. > I am curious as to whether we are allowed to sleep in the dmu_write_uio_dbuf(), > AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done > earlier to avoid possible page fault in there? > > (kgdb) thread 465 > [Switching to thread 465 (Thread 100530)]#0 sched_switch > (td=0xfffff8037b8ff960, newtd=, flags= out>) at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 > 1945 cpuid = PCPU_GET(cpuid); > (kgdb) bt > #0 sched_switch (td=0xfffff8037b8ff960, newtd=, > flags=) at > /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 > #1 0xffffffff809364d1 in mi_switch (flags=260, newtd=0x0) at > /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:491 > #2 0xffffffff809761ca in sleepq_wait (wchan=0x0, pri=0) at > /usr/home/sobomax/projects/freebsd103/sys/kern/subr_sleepqueue.c:618 > #3 0xffffffff808d843d in _cv_wait (cvp=0xfffff8001041b220, > lock=0xfffff8001041b1b8) at > /usr/home/sobomax/projects/freebsd103/sys/kern/kern_condvar.c:151 > #4 0xffffffff81a431e5 in txg_wait_synced () from /boot/kernel/zfs.ko > #5 0xffffffff81a62fa9 in zil_commit () from /boot/kernel/zfs.ko > #6 0xffffffff81a9216b in zfs_freebsd_write () from /boot/kernel/zfs.ko > #7 0xffffffff80dd6219 in VOP_WRITE_APV (vop=, > a=) at vnode_if.c:999 > #8 0xffffffff8060b69a in mdstart_vnode (sc=0xfffff80057f78000, > bp=0xfffff8036d4660f8) at vnode_if.h:413 > #9 0xffffffff8060caad in md_kthread (arg=0xfffff80057f78000) at > /usr/home/sobomax/projects/freebsd103/sys/dev/md/md.c:1147 > #10 0xffffffff808f6e6a in fork_exit (callout=0xffffffff8060c900 > , arg=0xfffff80057f78000, frame=0xfffffe04689b2ac0) at > /usr/home/sobomax/projects/freebsd103/sys/kern/kern_fork.c:1027 > #11 0xffffffff80ca0a5e in fork_trampoline () at > /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:611 > #12 0x0000000000000000 in ?? () > > > On Mon, Mar 28, 2016 at 7:31 AM, Maxim Sobolev > wrote: > >> OK, that happened again. Now it's 10.3-RC3, funny enough, it's the same >> "cp" command. Any ideas about what can be wrong here? The box is still up, >> so if you need me to do something specific inside kgdb please let me know. >> Thanks! >> >> $ uname -a >> FreeBSD abc01.sippysoft.com 10.3-RC3 FreeBSD 10.3-RC3 #0 >> bedc749(master): Wed Mar 23 14:39:07 PDT 2016 >> sobomax@abc01.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC01 >> amd64 >> >> >> $ ps -xaHl -O lwp >> UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME >> COMMAND PID LWP TT STAT TIME COMMAND >> 0 39143 38949 0 52 0 12368 1064 vnread DL+ 2 0:00.00 >> cp /usr/local/sh 39143 101176 2 DL+ 0:00.00 cp >> /usr/local/share/automake-1.15/compile ./compile >> >> >> (kgdb) thread 473 >> [Switching to thread 473 (Thread 101176)]#0 sched_switch >> (td=0xfffff803320b84b0, newtd=, flags=> out>) >> at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 >> 1945 cpuid = PCPU_GET(cpuid); >> (kgdb) bt >> #0 sched_switch (td=0xfffff803320b84b0, newtd=, >> flags=) >> at /usr/home/sobomax/projects/freebsd103/sys/kern/sched_ule.c:1945 >> #1 0xffffffff809364d1 in mi_switch (flags=260, newtd=0x0) at >> /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:491 >> #2 0xffffffff809761ca in sleepq_wait (wchan=0x0, pri=0) at >> /usr/home/sobomax/projects/freebsd103/sys/kern/subr_sleepqueue.c:618 >> #3 0xffffffff80935ef7 in _sleep (ident=, >> lock=, priority=, wmesg=> optimized out>, >> sbt=, pr=, flags=> optimized out>) at >> /usr/home/sobomax/projects/freebsd103/sys/kern/kern_synch.c:255 >> #4 0xffffffff809b7c6e in bwait (bp=0xfffffe03e1012048, pri=> optimized out>, wchan=0xffffffff81069471 "vnread") >> at /usr/home/sobomax/projects/freebsd103/sys/kern/vfs_bio.c:4431 >> #5 0xffffffff80b56663 in vnode_pager_generic_getpages >> (vp=0xfffff802b7e73588, m=0xfffffe0468c4ebe0, bytecount=> out>, >> reqpage=) at >> /usr/home/sobomax/projects/freebsd103/sys/vm/vnode_pager.c:907 >> #6 0xffffffff80dd7fa7 in VOP_GETPAGES_APV (vop=, >> a=) at vnode_if.c:2795 >> #7 0xffffffff80b54a0a in vnode_pager_getpages (object=> out>, m=0xfffffe0468c4ebe0, count=, reqpage=0) >> at vnode_if.h:1154 >> #8 0xffffffff80b354a1 in vm_fault_hold (map=0xfffff800294c98c0, >> vaddr=34366300160, fault_type=, fault_flags=0, >> m_hold=0x0) >> at vm_pager.h:127 >> #9 0xffffffff80b34c07 in vm_fault (map=0xfffff800294c98c0, vaddr=> optimized out>, fault_type=1 '\001', fault_flags=0) >> at /usr/home/sobomax/projects/freebsd103/sys/vm/vm_fault.c:273 >> #10 0xffffffff80cbaef5 in trap_pfault (frame=0xfffffe0468c4f040, >> usermode=0) at >> /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/trap.c:747 >> #11 0xffffffff80cba65d in trap (frame=0xfffffe0468c4f040) at >> /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/trap.c:447 >> #12 0xffffffff80ca0592 in calltrap () at >> /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:236 >> #13 0xffffffff80cb8b71 in copyin () at >> /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/support.S:311 >> #14 0xffffffff8097d70f in uiomove_faultflag (cp=, >> n=, uio=0xfffffe0468c4f960, nofault=> out>) >> at /usr/home/sobomax/projects/freebsd103/sys/kern/subr_uio.c:253 >> ---Type to continue, or q to quit--- >> #15 0xffffffff819f799c in dmu_write_uio_dnode () from /boot/kernel/zfs.ko >> #16 0xffffffff819f78b2 in dmu_write_uio_dbuf () from /boot/kernel/zfs.ko >> #17 0xffffffff81a91a07 in zfs_freebsd_write () from /boot/kernel/zfs.ko >> #18 0xffffffff80dd6219 in VOP_WRITE_APV (vop=, >> a=) at vnode_if.c:999 >> #19 0xffffffff808730b2 in null_bypass (ap=0xfffffe0468c4f7d0) at >> /usr/home/sobomax/projects/freebsd103/sys/fs/nullfs/null_vnops.c:280 >> #20 0xffffffff80dd621f in VOP_WRITE_APV (vop=, >> a=) at vnode_if.c:1001 >> #21 0xffffffff809dea28 in vn_write (fp=0xfffff8004bc0ddc0, >> uio=0xfffffe0468c4f960, active_cred=0x1ca5, flags=916, >> td=0xfffffe00006a60d0) >> at vnode_if.h:413 >> #22 0xffffffff809dad6b in vn_io_fault (fp=0xfffff8004bc0ddc0, >> uio=0xfffffe0468c4f960, active_cred=0x1ca5, flags=0, td=0xfffffe00006a60d0) >> at /usr/home/sobomax/projects/freebsd103/sys/kern/vfs_vnops.c:1225 >> #23 0xffffffff80981937 in dofilewrite (td=0xfffff803320b84b0, fd=4, >> fp=0xfffff8004bc0ddc0, auio=0xfffffe0468c4f960, offset=> out>, >> flags=0) at file.h:305 >> #24 0xffffffff80981668 in kern_writev (td=0xfffff803320b84b0, fd=4, >> auio=0xfffffe0468c4f960) >> at /usr/home/sobomax/projects/freebsd103/sys/kern/sys_generic.c:481 >> #25 0xffffffff809815f3 in sys_write (td=, uap=> optimized out>) >> at /usr/home/sobomax/projects/freebsd103/sys/kern/sys_generic.c:396 >> #26 0xffffffff80cbb559 in amd64_syscall (td=0xfffff803320b84b0, traced=0) >> at subr_syscall.c:141 >> #27 0xffffffff80ca087b in Xfast_syscall () at >> /usr/home/sobomax/projects/freebsd103/sys/amd64/amd64/exception.S:396 >> #28 0x0000000800968f5a in ?? () >> >> >> On Wed, Mar 2, 2016 at 1:12 AM, Maxim Sobolev >> wrote: >> >>> Hi, I've encountered cp(1) process stuck in the vnread state on one of >>> my build machines that got recently upgraded to 10.3. >>> >>> 0 79596 1 0 20 0 17092 1396 wait I 1 >>> 0:00.00 /bin/sh /usr/local/bin/autoreconf -f -i >>> 0 79602 79596 0 52 0 41488 9036 wait I 1 >>> 0:00.07 /usr/local/bin/perl -w /usr/local/bin/autoreconf-2.69 -f -i >>> 0 79639 79602 0 72 0 0 0 - Z 1 >>> 0:00.27 >>> 0 79762 79602 0 20 0 17092 1396 wait I 1 >>> 0:00.00 /bin/sh /usr/local/bin/automake --add-missing --copy --force-missing >>> 0 79768 79762 0 52 0 49736 13936 wait I 1 >>> 0:00.11 /usr/local/bin/perl -w /usr/local/bin/automake-1.15 --add-missing >>> --copy --force-missing >>> 0 79962 79768 0 20 0 12368 1024 vnread DL 1 >>> 0:00.00 cp /usr/local/share/automake-1.15/compile ./compile >>> >>> I am not sure if it's related to that OS version upgrade, but I have not >>> seen any such issues on the same machine in 2-3 years running essentially >>> the same build process with version 9.x, 10.0, 10.1 and 10.2. >>> >>> $ uname -a >>> FreeBSD van01.sippysoft.com 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #1 >>> 80de3e2(master)-dirty: Tue Feb 2 12:19:57 PST 2016 >>> sobomax@abc.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC >>> amd64 >>> >>> The kernel stack trace is: >>> >>> (kgdb) thread 360 >>> [Switching to thread 360 (Thread 100515)]#0 0xffffffff8095244e in >>> sched_switch () >>> (kgdb) bt >>> #0 0xffffffff8095244e in sched_switch () >>> #1 0xffffffff809313b1 in mi_switch () >>> #2 0xffffffff8097089a in sleepq_wait () >>> #3 0xffffffff80930dd7 in _sleep () >>> #4 0xffffffff809b230e in bwait () >>> #5 0xffffffff80b511f3 in vnode_pager_generic_getpages () >>> #6 0xffffffff80dd1607 in VOP_GETPAGES_APV () >>> #7 0xffffffff80b4f59a in vnode_pager_getpages () >>> #8 0xffffffff80b30031 in vm_fault_hold () >>> #9 0xffffffff80b2f797 in vm_fault () >>> #10 0xffffffff80cb5a75 in trap_pfault () >>> #11 0xffffffff80cb51dd in trap () >>> #12 0xffffffff80c9b122 in calltrap () >>> #13 0xffffffff80cb36f1 in copyin () >>> #14 0xffffffff80977ddf in uiomove_faultflag () >>> >>> The FS stack configuration is somewhat unique, so I am not sure if I am >>> hitting some rare race condition or lock ordering issues specific to that. >>> It's basically ZFS (ZRAID) on top of pair or SATA SSDs with big file on >>> that FS attached via md(4) and UFS2 on that md(4). The build itself runs in >>> chroot with that UFS2 fs as its primary root. >>> >>> Just maybe additional bit of info, attempting to list the directory with >>> that UFS image also got my bash process stuck in "zfs" state, backtrace >>> from that is: >>> >>> (kgdb) thread 353 >>> [Switching to thread 353 (Thread 100508)]#0 0xffffffff8095244e in >>> sched_switch () >>> (kgdb) bt >>> #0 0xffffffff8095244e in sched_switch () >>> #1 0xffffffff809313b1 in mi_switch () >>> #2 0xffffffff8097089a in sleepq_wait () >>> #3 0xffffffff809069ad in sleeplk () >>> #4 0xffffffff809060e0 in __lockmgr_args () >>> #5 0xffffffff809b8b7c in vop_stdlock () >>> #6 0xffffffff80dd0a3b in VOP_LOCK1_APV () >>> #7 0xffffffff809d6d23 in _vn_lock () >>> #8 0xffffffff81a8c9cd in ?? () >>> #9 0x0000000000000000 in ?? () >>> >>> >>> >> >> >> -- >> Maksym Sobolyev >> Sippy Software, Inc. >> Internet Telephony (VoIP) Experts >> Tel (Canada): +1-778-783-0474 >> Tel (Toll-Free): +1-855-747-7779 >> Fax: +1-866-857-6942 >> Web: http://www.sippysoft.com >> MSN: sales@sippysoft.com >> Skype: SippySoft >> > > > > -- > Maksym Sobolyev > Sippy Software, Inc. > Internet Telephony (VoIP) Experts > Tel (Canada): +1-778-783-0474 > Tel (Toll-Free): +1-855-747-7779 > Fax: +1-866-857-6942 > Web: http://www.sippysoft.com > MSN: sales@sippysoft.com > Skype: SippySoft > From owner-freebsd-fs@freebsd.org Mon Mar 28 16:09:41 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7033BAE070F; Mon, 28 Mar 2016 16:09:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A94810ED; Mon, 28 Mar 2016 16:09:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x235.google.com with SMTP id a1so21426527igb.1; Mon, 28 Mar 2016 09:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=sShoQORPffs52DA4YndDGwHF1szc7GlwPxlMKmSvrKE=; b=R4shVnLZ3J4x+50PFcs2b2LP3uy4qsxaMCwnrIezPd9oN3xUt8r4hi6GgsFF8UyOZb lQh0yM9w0UMyE1c4tNlEIoy/fxNKDxXwp60iZPd9QTxgPXD3EgLr3MCkpjRWYEBw5Qsq v+C3+qUsnOJoJwZ5V6HbOsAk+2vOE6HL1OLAZ1RdXwojw/yzOD3Iwrd6gYh3Aht80QJS bRPKCXCtpW6KMdJnyfJcQGo6svS0eXu6xIqELH5TbFbjWeI3u035c9eQsMtRLDrF+wg/ 6nQo936rM0wbIuwNwY6seS/3iQllH+Nopzams+44QkFD9rOm4WYAeLoGXrTA7YbLAD20 X9+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=sShoQORPffs52DA4YndDGwHF1szc7GlwPxlMKmSvrKE=; b=GtMziUvO3HfBMlsiHOc73DTTrXQU7PqSX0tkV49EvoivBdFUqkwP/0XARXeJoiAWMz 3zSElveFM4nsDALsxiJB2/m3WGBaSwfVCW9CEVmrp/9sh1eUYMatjh5WV3NglpmPOQtW /k4qHMbJqPffnGJfLzceE92Ss10FxjKP33sc7+E+/ME1ue6l4c7vR7RhWfHf54jZaqww zQQOvIrFM7idx9NF9N6gVf1q3PpSKPFYz227GqYR5qaP/neRSJLrcqc9qiAo2qDcjl95 hP3c2Y4XkDgL/ORzY0zAb+VzjPCzXOZhF5Ejh6VLzuiinEU61vG6zX/+hN6GKDnD37DJ WwGw== X-Gm-Message-State: AD7BkJI97hqhYy2ScbqqodmfGeurOWAHZFjY3rVhtSHMSxO9ibsqP7XkGa1Om3FaymWXY2oHEQduqBYYnCNH8w== MIME-Version: 1.0 X-Received: by 10.50.8.102 with SMTP id q6mr10818286iga.37.1459181380578; Mon, 28 Mar 2016 09:09:40 -0700 (PDT) Received: by 10.36.14.19 with HTTP; Mon, 28 Mar 2016 09:09:40 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Mar 2016 09:09:40 -0700 Message-ID: Subject: Re: Process stuck in "vnread" From: Adrian Chadd To: Maxim Sobolev Cc: "stable@freebsd.org" , FreeBSD Filesystems , Pawel Jakub Dawidek , Kirk McKusick , Konstantin Belousov Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 16:09:41 -0000 Hi, I think a bunch of the lock order checks with witness get disabled for ZFS ? Maybe compile up a witness kernel so you can get 'show alllocks' in ddb. That should help narrow down exactly what's going on. Thanks! -a From owner-freebsd-fs@freebsd.org Mon Mar 28 16:23:24 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85DE2AE0CF2; Mon, 28 Mar 2016 16:23:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0436A1D7D; Mon, 28 Mar 2016 16:23:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2SGNAoD056823 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 28 Mar 2016 19:23:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2SGNAoD056823 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2SGNAuY056822; Mon, 28 Mar 2016 19:23:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 28 Mar 2016 19:23:10 +0300 From: Konstantin Belousov To: Maxim Sobolev Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Pawel Jakub Dawidek , Kirk McKusick , kib@freebsd.org Subject: Re: Process stuck in "vnread" Message-ID: <20160328162310.GJ1741@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 16:23:24 -0000 On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: > Done some head scratching, it looks like it's got page fault in the > copyin() (cp(1) AFAIK mmaps source file). There might be some interlock > issue between competing write to the same ZFS, the md0 device is locked > forever waiting for the write operation to complete at the very same time. > I am curious as to whether we are allowed to sleep in the dmu_write_uio_dbuf(), > AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done > earlier to avoid possible page fault in there? No idea about ZFS, but if the issue is due to copyin(9) recursing into VM and then VFS while owning file system locks, it is well-known and long-standing issue. I sometimes call it 'ups deadlock', for some reasons, see tools/test/upsdl/ for the distilled test case. It is handled for UFS and NFS, read the long comment starting with 'The vn_io_fault() is a wrapper' in sys/kern/vfs_vnops.c, which describes the deadlock in details and explains the mechanism which is used to prevent it. Filesystems must opt-in into it by specifiying MNTK_NO_IOPF flag, and then being ready to get an array of pages for io instead of the buffer KVA. From owner-freebsd-fs@freebsd.org Mon Mar 28 17:19:54 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 920E4AE0DB8; Mon, 28 Mar 2016 17:19:54 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 784C31B58; Mon, 28 Mar 2016 17:19:52 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA09541; Mon, 28 Mar 2016 20:19:50 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1akapS-0003Qy-CM; Mon, 28 Mar 2016 20:19:50 +0300 Subject: Re: Process stuck in "vnread" To: Konstantin Belousov , Maxim Sobolev References: <20160328162310.GJ1741__41334.1269981631$1459182219$gmane$org@kib.kiev.ua> Cc: freebsd-fs@FreeBSD.org, Kirk McKusick , stable@FreeBSD.org, kib@FreeBSD.org From: Andriy Gapon Message-ID: <56F96792.2010800@FreeBSD.org> Date: Mon, 28 Mar 2016 20:19:14 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160328162310.GJ1741__41334.1269981631$1459182219$gmane$org@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 17:19:54 -0000 On 28/03/2016 19:23, Konstantin Belousov wrote: > On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: >> Done some head scratching, it looks like it's got page fault in the >> copyin() (cp(1) AFAIK mmaps source file). There might be some interlock >> issue between competing write to the same ZFS, the md0 device is locked >> forever waiting for the write operation to complete at the very same time. >> I am curious as to whether we are allowed to sleep in the dmu_write_uio_dbuf(), >> AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done >> earlier to avoid possible page fault in there? Maxim, is this copy from UFS to ZFS? It looks like that because the copyin() fault goes to vnode_pager_generic_getpages() -> bwait()... > No idea about ZFS, but if the issue is due to copyin(9) recursing into > VM and then VFS while owning file system locks, it is well-known and > long-standing issue. I sometimes call it 'ups deadlock', for some > reasons, see tools/test/upsdl/ for the distilled test case. > > It is handled for UFS and NFS, read the long comment starting with 'The > vn_io_fault() is a wrapper' in sys/kern/vfs_vnops.c, which describes the > deadlock in details and explains the mechanism which is used to prevent > it. Filesystems must opt-in into it by specifiying MNTK_NO_IOPF flag, > and then being ready to get an array of pages for io instead of the buffer > KVA. I don't have any idea why the thread would be stuck in bwait() and what locks and threads are involved here. But, as Kostik said, there is a general problem and I have a patch for ZFS: https://reviews.freebsd.org/D2790 -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Mar 28 19:20:26 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A03AAE130C for ; Mon, 28 Mar 2016 19:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B24B1D69 for ; Mon, 28 Mar 2016 19:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2SJKO9A016542 for ; Mon, 28 Mar 2016 19:20:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 199189] SWAP on ZFS can crash server Date: Mon, 28 Mar 2016 19:20:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vermaden@interia.pl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 19:20:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 --- Comment #19 from vermaden@interia.pl --- @Andriy Gapon Where to find up to date information then? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Mar 28 21:30:41 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 289A7AE120B for ; Mon, 28 Mar 2016 21:30:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 125B0101C for ; Mon, 28 Mar 2016 21:30:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2SLUeRb099228 for ; Mon, 28 Mar 2016 21:30:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 199189] SWAP on ZFS can crash server Date: Mon, 28 Mar 2016 21:30:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 21:30:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 --- Comment #20 from Andriy Gapon --- (In reply to vermaden from comment #19) In the source code... --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Mar 28 21:33:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44D4AAE1333 for ; Mon, 28 Mar 2016 21:33:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 36111130A for ; Mon, 28 Mar 2016 21:33:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2SLXDor009230 for ; Mon, 28 Mar 2016 21:33:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 199189] SWAP on ZFS can crash server Date: Mon, 28 Mar 2016 21:33:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vermaden@interia.pl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 21:33:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 --- Comment #21 from vermaden@interia.pl --- @Andriy Gapon I expect such comments from the Linux community ... Pity. Regards, vermaden --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Mar 28 23:51:01 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 172F9AE04B9 for ; Mon, 28 Mar 2016 23:51:01 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CF6CD1DEC for ; Mon, 28 Mar 2016 23:51:00 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-oi0-x231.google.com with SMTP id h6so135774178oia.2 for ; Mon, 28 Mar 2016 16:51:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=xOpc2G+wyYEpY8w/KSJDXtuiEj5ZRZnTKndkWUy3gDs=; b=yU5yG4EAelsWtLWkcPcSdTjD0//WukO5bQwRS2k1CQJD5UED44orpx6zikT5YFr4XU EjKrm/T9lBKPYs+bRHm8eqKth/C3CTRKtT44YcvTHelO0ITyfO4tJCsBdw7TQYoDK35i 3WnIfFUy4nH9FgjHwUZFBYeljF+R/yXypqZ/aZU3ODMZj52fxpls6oMWWRoR+QJVEBWx /HEsPL6R/6p2iC40QUV5vGwYR0ogCcpVkQWvZM60T7/4zNCq1pgizZaCGhZpDs+4RrB4 4uCUEf/oX3Oxl3jeplsQf2kJ4mD8di+lUM/zF13JiJU34358YKiPMHFPByDpFOh8JLT0 wWcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=xOpc2G+wyYEpY8w/KSJDXtuiEj5ZRZnTKndkWUy3gDs=; b=m6ZyOgjttw5+PgGEsNWllME9H6wAjGcSVlTaeEPw42aUqXCjXn4KFhckVJPvC2ssiO 51WYGhWZK6YUbW1+QfWhKfAhdxQ8v+/zAMIqVCn1bNVe9hDzhf9+s9aqy0DknB62deQh NEFyedtuH95ioPgl4i/drG/hi6BoOuSeEaRa0weiMbCn0O5UnlZr0qeC64huCAIwuav2 iHaionvcaTLD7QUmBSa3fhwuw1zn5xUwS5kmNaqpNPzwI+lCOmigl0T1Q02KgN59D9Xi Ftn1Im4IeVu9qyBjnCJrLj80jmWcERmD5rMou1oa6xQN0FswPjIfUV4NabN8TS/becuS Bk1g== X-Gm-Message-State: AD7BkJKupOYsWHS0ztpIayg9neRXNIAzhE65jRruvUY+unlbSd0Ec2AxMj27b3zQTgHJiphyjnRa5+CzbPRjDmKR MIME-Version: 1.0 X-Received: by 10.202.200.16 with SMTP id y16mr2897996oif.92.1459209060243; Mon, 28 Mar 2016 16:51:00 -0700 (PDT) Received: by 10.157.11.143 with HTTP; Mon, 28 Mar 2016 16:50:59 -0700 (PDT) In-Reply-To: <56F96792.2010800@FreeBSD.org> References: <20160328162310.GJ1741__41334.1269981631$1459182219$gmane$org@kib.kiev.ua> <56F96792.2010800@FreeBSD.org> Date: Mon, 28 Mar 2016 16:50:59 -0700 Message-ID: Subject: Re: Process stuck in "vnread" From: Maxim Sobolev To: Andriy Gapon Cc: Konstantin Belousov , freebsd-fs@freebsd.org, Kirk McKusick , stable@freebsd.org, kib@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 23:51:01 -0000 Andriy, this is file that gets copies from md(4)-baked UFS to file that is located on ZFS zraid volume that is mounted via NULLFS. The file that backs up md(4) is located on ZFS, so in a sense we have full cycle with the backing block starting on ZFS and ending up on ZFS too. cp(1) calls write(), with a pointer to mmaped area and the page is not mapped in, so it triggers pagein and waits for the page to arrive. In any case, it looks like your patch from D5738 might be working, I've put it into my 10.3-rc3 buildbox and give it some shake to see if it improves things or not. I've also made sure that I have all debug symbols installed, so that I can poke inside zfs.ko if I need to. Thanks, guys, Andriy what keep you from pushing that patch into the tree? On Mon, Mar 28, 2016 at 10:19 AM, Andriy Gapon wrote: > On 28/03/2016 19:23, Konstantin Belousov wrote: > > On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: > >> Done some head scratching, it looks like it's got page fault in the > >> copyin() (cp(1) AFAIK mmaps source file). There might be some interlock > >> issue between competing write to the same ZFS, the md0 device is locked > >> forever waiting for the write operation to complete at the very same > time. > >> I am curious as to whether we are allowed to sleep in the > dmu_write_uio_dbuf(), > >> AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done > >> earlier to avoid possible page fault in there? > > Maxim, > > is this copy from UFS to ZFS? > It looks like that because the copyin() fault goes to > vnode_pager_generic_getpages() -> bwait()... > > > No idea about ZFS, but if the issue is due to copyin(9) recursing into > > VM and then VFS while owning file system locks, it is well-known and > > long-standing issue. I sometimes call it 'ups deadlock', for some > > reasons, see tools/test/upsdl/ for the distilled test case. > > > > It is handled for UFS and NFS, read the long comment starting with 'The > > vn_io_fault() is a wrapper' in sys/kern/vfs_vnops.c, which describes the > > deadlock in details and explains the mechanism which is used to prevent > > it. Filesystems must opt-in into it by specifiying MNTK_NO_IOPF flag, > > and then being ready to get an array of pages for io instead of the > buffer > > KVA. > > > I don't have any idea why the thread would be stuck in bwait() and what > locks > and threads are involved here. But, as Kostik said, there is a general > problem > and I have a patch for ZFS: > https://reviews.freebsd.org/D2790 > > -- > Andriy Gapon > -- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.com Skype: SippySoft From owner-freebsd-fs@freebsd.org Mon Mar 28 23:59:35 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE28BAE08BE for ; Mon, 28 Mar 2016 23:59:35 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id BCE3E12AD for ; Mon, 28 Mar 2016 23:59:35 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id BC3E2AE08BD; Mon, 28 Mar 2016 23:59:35 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBDC2AE08BC for ; Mon, 28 Mar 2016 23:59:35 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail107.syd.optusnet.com.au (mail107.syd.optusnet.com.au [211.29.132.53]) by mx1.freebsd.org (Postfix) with ESMTP id 6B4E712AC; Mon, 28 Mar 2016 23:59:34 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail107.syd.optusnet.com.au (Postfix) with ESMTPS id 067DCD45414; Tue, 29 Mar 2016 10:59:26 +1100 (AEDT) Date: Tue, 29 Mar 2016 10:59:26 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans cc: fs@freebsd.org, rmacklem@freebsd.org, glebius@freebsd.org Subject: Re: nfs pessimized by vnode pages changes In-Reply-To: <20160327144755.Y4269@besplex.bde.org> Message-ID: <20160329090209.Q1020@besplex.bde.org> References: <20160327144755.Y4269@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=c+ZWOkJl c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=M5xT4KusGiiYpgd5uT0A:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 23:59:35 -0000 On Sun, 27 Mar 2016, Bruce Evans wrote: > I debugged another pessimization of nfs. > > ncl_getpages() is now almost always called with a count of 1 page, due > to the change changing the count from faultcount to 1 in r292373 in > vm_fault(). The only exception seems to be for the initial pagein for > exec -- this is still normally the nfs rsize. ncl_getpages() doesn't > do any readahead stuff like vnode_pager_generic_getpages() does, so it > normally does read RPCs of size 1 page instead of the the nfs rsize. > This gives the following increases in read RPCs for makeworld of an > old world: > - with rsize = 16K, from 24k to 39k (the worst case would be 4 times as many) > - with rsize = 8K, from 39k to 44k (the worst case would be 2 times as many). > > Also, nfs_getpages() has buggy logic which works accidentally if the > count is 1: > > X diff -c2 ./fs/nfsclient/nfs_clbio.c~ ./fs/nfsclient/nfs_clbio.c > X *** ./fs/nfsclient/nfs_clbio.c~ Sun Mar 27 01:31:38 2016 > X --- ./fs/nfsclient/nfs_clbio.c Sun Mar 27 02:35:32 2016 > X *************** > X *** 135,140 **** > X */ > X VM_OBJECT_WLOCK(object); > X ! if (pages[npages - 1]->valid != 0 && --npages == 0) > X goto out; > X VM_OBJECT_WUNLOCK(object); > X X --- 135,155 ---- > X */ > X VM_OBJECT_WLOCK(object); > X ! #if 0 > X ! /* This matches the comment. but doesn't work (has little effect). */ > X ! if (pages[0]->valid != 0) > X goto out; > > The comment still says that the code checks the requested page, but > that is no longer passed to the function in a_reqpage. The first page > is a better geuss of the requested page than the last one, but when > npages is 1 these pages are the same. > > X + #else > X + if (pages[0]->valid != 0) > X + printf("ncl_getpages: page 0 valid; npages %d\n", npages); > X + for (i = 0; i < npages; i++) > X + if (pages[i]->valid != 0) > X + printf("ncl_getpages: page %d valid; npages %d\n", > X + i, npages); > X + for (i = 0; i < npages; i++) > X + if (pages[i]->valid != 0) > X + npages = i; > X + if (npages == 0) > X + goto out; > X + #endif > > Debugging and more forceful guessing code. This makes little difference > except of course to spam the console. > > X VM_OBJECT_WUNLOCK(object); > X X *************** > X *** 199,202 **** > X --- 214,220 ---- > X KASSERT(m->dirty == 0, > X ("nfs_getpages: page %p is dirty", m)); > X + printf("ncl_getpages: partial page %d of %d %s\n", > X + i, npages, > X + pages[i]->valid != 0 ? "valid" : "invalid"); > X } else { > X /* > X *************** > X *** 210,215 **** > X --- 228,239 ---- > X */ > X ; > X + printf("ncl_getpages: short page %d of %d %s\n", > X + i, npages, > X + pages[i]->valid != 0 ? "valid" : "invalid"); > X } > X } > X + for (i = 0; i < npages; i++) > X + printf("ncl_getpages: page %d of %d %s\n", > X + i, npages, pages[i]->valid != 0 ? "valid" : "invalid"); > X out: > X VM_OBJECT_WUNLOCK(object); > > Further debugging code. Similar debugging code in the old working version > shows that normal operation for paging in a 15K file with an rsize of 16K is: > - call here with npages = 4 > - page in 3 full pages and 1 partial page using 1 RPC > - call here again with npages = 1 for the partial page > - use the optimization of returning early for this page -- don't do another > RPC > > The buggy version does: > - call here with npages = 1; page in 1 full page using 1 RPC > - call here with npages = 1; page in 1 full page using 1 RPC > - call here with npages = 1; page in 1 full page using 1 RPC > - call here with npages = 1; page in 1 partial page using 1 RPC > - call here again with npages = 1 for the partial page; the optimization > works as before. > > The partial page isn't handled very well, but at least there is no extra > physical i/o for it, at least if it is at EOF. vfs clustering handles > partial pages even worse than this. > > Bruce This seems to be very difficult to fix. Even vnode_pager_generic_getpages() does input 1 page at a time unless the fs supports bmap, and nfs doesn't support bmap. Memory mapped input was already much slower than read(). I get the following speeds for reading an 8MB file with a cold cache on the client and a warm cache on the server (8MB is the limit below which cp(1) does "optimized" i/o using mmap): - read(): 64MB/sec (doesn't depend much on rsize; the network can't go much faster than 70MB/sec; probably does depend on low network latency (~100 sec)) - FreeBSD-10 mmap(), rsize=16K: 40MB/sec - FreeBSD-10 mmap(), rsize=8K: 32MB/sec - FreeBSD-9 mmap(), rsize=8K: 16MB/sec vnode_pager_generic_getpages() has always had an even better pessimization if the fs does support bmap but the fs block size is smaller than PAGE_SIZE. Then it calls vnode_pager_input_smlfs() once for each page to reduce the input size to the fs block size which can be as low as 512. However, if the fs doesn't support bmap, then it calls vnode_pager_input_old() once for each page. vnode_pager_input_old() uses VOP_READ(), so it is not much slower than read(). vnode_pager_input_old() has chances of being better than the specialized pager methods in all cases, since read() should do vfs clustering which does more than read-ahead. vfs clustering doesn't do read-behind AFAIR, but perhaps its better read-ahead is more important. All methods have problems guessing how much read-ahead is good. The read-ahead and read-behind should certainly be enough to give fill out an fs-block. I couldn't get full-fs-block input to work right in ncl_getpages(). In the old version, vm_fault_hold() almost always calls it with the pages for exactly 1 full block, because vm_fault_additional_pages() somehow finds this many pages. nfs doesn't use vfs clustering and wouldn't benefit from it since its block size is the same as its maximum i/o size. The block size can now be MAXBCACHEBUF which can be larger than MAXBSIZE, but I think using that is a wrong way to do things -- a default small block size plus clustering works better by not using large blocks for everything. ncl_getpages() could probably expand the read size to MAXPHYS (or whatever the pbuf size limit is) just as easily as to the rsize. Bruce From owner-freebsd-fs@freebsd.org Tue Mar 29 00:16:25 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D161BAE0D14 for ; Tue, 29 Mar 2016 00:16:25 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BE4721C23 for ; Tue, 29 Mar 2016 00:16:25 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id BA371AE0D0F; Tue, 29 Mar 2016 00:16:25 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7BA1AE0D07 for ; Tue, 29 Mar 2016 00:16:25 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 96B381C21; Tue, 29 Mar 2016 00:16:25 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id u2T0GNS6049254 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 28 Mar 2016 17:16:24 -0700 (PDT) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id u2T0GNQ8049253; Mon, 28 Mar 2016 17:16:23 -0700 (PDT) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Mon, 28 Mar 2016 17:16:23 -0700 From: Gleb Smirnoff To: Bruce Evans Cc: fs@freebsd.org, rmacklem@freebsd.org Subject: Re: nfs pessimized by vnode pages changes Message-ID: <20160329001623.GC2616@FreeBSD.org> References: <20160327144755.Y4269@besplex.bde.org> <20160329090209.Q1020@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160329090209.Q1020@besplex.bde.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 00:16:25 -0000 On Tue, Mar 29, 2016 at 10:59:26AM +1100, Bruce Evans wrote: B> > I debugged another pessimization of nfs. ... B> > ncl_getpages() is now almost always called with a count of 1 page, due B> > to the change changing the count from faultcount to 1 in r292373 in B> > vm_fault(). ... B> I couldn't get full-fs-block input to work right in ncl_getpages(). In B> the old version, vm_fault_hold() almost always calls it with the pages B> for exactly 1 full block, because vm_fault_additional_pages() somehow B> finds this many pages. The last quoted paragraph is a correct observation. According to your investigations, prior to r292373 NFS was doing multiple page pageins, despite it should have reported that it can't. My reading of the code is the same: both before r292373 and after r292373 NFS should page in a single page at request. I quickly reviewed the whole codepath and I can't see how with older code it was able to do multiple page pageins. -- Totus tuus, Glebius. From owner-freebsd-fs@freebsd.org Tue Mar 29 02:24:37 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D3D1AE12A7 for ; Tue, 29 Mar 2016 02:24:37 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 5B60D1DB2 for ; Tue, 29 Mar 2016 02:24:37 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 5ABD7AE12A6; Tue, 29 Mar 2016 02:24:37 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A6C1AE12A5 for ; Tue, 29 Mar 2016 02:24:37 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 21C391DB0; Tue, 29 Mar 2016 02:24:36 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 71EB2424897; Tue, 29 Mar 2016 13:24:28 +1100 (AEDT) Date: Tue, 29 Mar 2016 13:24:27 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Gleb Smirnoff cc: Bruce Evans , fs@freebsd.org, rmacklem@freebsd.org Subject: Re: nfs pessimized by vnode pages changes In-Reply-To: <20160329001623.GC2616@FreeBSD.org> Message-ID: <20160329122721.D1549@besplex.bde.org> References: <20160327144755.Y4269@besplex.bde.org> <20160329090209.Q1020@besplex.bde.org> <20160329001623.GC2616@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=c+ZWOkJl c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=iz7yfRI9pwfItlEF7fIA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 02:24:37 -0000 On Mon, 28 Mar 2016, Gleb Smirnoff wrote: > On Tue, Mar 29, 2016 at 10:59:26AM +1100, Bruce Evans wrote: > ... > > B> I couldn't get full-fs-block input to work right in ncl_getpages(). In > B> the old version, vm_fault_hold() almost always calls it with the pages > B> for exactly 1 full block, because vm_fault_additional_pages() somehow > B> finds this many pages. > > The last quoted paragraph is a correct observation. According to your > investigations, prior to r292373 NFS was doing multiple page pageins, > despite it should have reported that it can't. My reading of the code > is the same: both before r292373 and after r292373 NFS should page in > a single page at request. I quickly reviewed the whole codepath and I > can't see how with older code it was able to do multiple page pageins. I found it. In old versions, vm_fault_additional_pages() exists and calls vnode_pager_haspage() to locate some additional pages. VOP_BMAP() is vop_stdbmap() for nfs. This doesn't locate any additional fs blocks, but vnode_pager_haspage() locates the page within the single nfs block and returns the number of pages to read behind and ahead to fill in the block. Probably similarly in most file systems that don't really support bmap. They can always use vop_stdbmap(). Probably they mostly do, so the fallback code for when VOP_BMAP() fails is mostly unreachable. Bruce From owner-freebsd-fs@freebsd.org Tue Mar 29 03:03:48 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E0956AE1C7F for ; Tue, 29 Mar 2016 03:03:48 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id CD09F1870 for ; Tue, 29 Mar 2016 03:03:48 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id C87DAAE1C7E; Tue, 29 Mar 2016 03:03:48 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C81D0AE1C7D for ; Tue, 29 Mar 2016 03:03:48 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B35D1186E; Tue, 29 Mar 2016 03:03:48 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id u2T33lNW050322 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 28 Mar 2016 20:03:47 -0700 (PDT) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id u2T33lqD050321; Mon, 28 Mar 2016 20:03:47 -0700 (PDT) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Mon, 28 Mar 2016 20:03:46 -0700 From: Gleb Smirnoff To: Bruce Evans Cc: fs@freebsd.org, rmacklem@freebsd.org Subject: Re: nfs pessimized by vnode pages changes Message-ID: <20160329030346.GI2616@FreeBSD.org> References: <20160327144755.Y4269@besplex.bde.org> <20160329090209.Q1020@besplex.bde.org> <20160329001623.GC2616@FreeBSD.org> <20160329122721.D1549@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160329122721.D1549@besplex.bde.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 03:03:49 -0000 On Tue, Mar 29, 2016 at 01:24:27PM +1100, Bruce Evans wrote: B> > On Tue, Mar 29, 2016 at 10:59:26AM +1100, Bruce Evans wrote: B> > ... B> > B> > B> I couldn't get full-fs-block input to work right in ncl_getpages(). In B> > B> the old version, vm_fault_hold() almost always calls it with the pages B> > B> for exactly 1 full block, because vm_fault_additional_pages() somehow B> > B> finds this many pages. B> > B> > The last quoted paragraph is a correct observation. According to your B> > investigations, prior to r292373 NFS was doing multiple page pageins, B> > despite it should have reported that it can't. My reading of the code B> > is the same: both before r292373 and after r292373 NFS should page in B> > a single page at request. I quickly reviewed the whole codepath and I B> > can't see how with older code it was able to do multiple page pageins. B> B> I found it. In old versions, vm_fault_additional_pages() exists and B> calls vnode_pager_haspage() to locate some additional pages. B> VOP_BMAP() is vop_stdbmap() for nfs. This doesn't locate any B> additional fs blocks, but vnode_pager_haspage() locates the page within B> the single nfs block and returns the number of pages to read behind and B> ahead to fill in the block. Ah. I also looked at stdbmap, but missed that it doesn't actually return error value. I will come up with a patch. B> Probably similarly in most file systems that don't really support bmap. B> They can always use vop_stdbmap(). Probably they mostly do, so the B> fallback code for when VOP_BMAP() fails is mostly unreachable. -- Totus tuus, Glebius. From owner-freebsd-fs@freebsd.org Tue Mar 29 08:18:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA3B2AE1F42 for ; Tue, 29 Mar 2016 08:18:19 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-ob0-x22b.google.com (mail-ob0-x22b.google.com [IPv6:2607:f8b0:4003:c01::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70CE218F7 for ; Tue, 29 Mar 2016 08:18:19 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-ob0-x22b.google.com with SMTP id fp4so6042914obb.2 for ; Tue, 29 Mar 2016 01:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc; bh=SWN6fNnscw0XX4jPjmXHvmBPW0sLZq41LiX2I/tETAc=; b=zwoBGSQTCTtCi7mAQFmtsOoeF69Zjjn8DLI5aeGBO8UtT8gHa97Sv2DWLcQ4ysGMUY QRNQH9K8Aa+fHvn4yEZMwH1XglvMfKlI9fHWhbbHkCPn2w6m7XWE5EVQuIAwsRYhUHe9 56XHCSC9IWaHd9mJY9adR8/FyrzoBztRi96/StK01eVbwahr+tCf757gts0OYFTDFjzX 4B/wRgwxCKz/6mq7A0Fu6edOTMqj1mNvJcf41tyNvGVYLU3IXZ+EUdyX6VByVUJxwkZr 74om3rzkeDqpLzI5Szxr2kr0GSwKbjsPGmDrOuB7Ugd1kgJUzTHX5KKBZkLGQiN6MD+t ZgzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc; bh=SWN6fNnscw0XX4jPjmXHvmBPW0sLZq41LiX2I/tETAc=; b=ZCv7i2zgWuJNJ+f7NmKaibZvXAptKBQ9LvNUuXYkmSpxHELPUBvBZohGohuVWu4azS v4dx07p5/yu/4vTaERZgDVEzBm5A78iFlaOKsjJHp1GE+NDlg5nHESxK+O5LZN3QHL71 AddD/D3y7gKNDmm8WXG/VhLELPIf+zvmKjmH3Fdiur7IHmHnXhKm0nbbHB1yDdFBxF/g eFW6R5WeyFxEueUZKekcIqzsNIcYzk2NV3f3VvWVgWMTm/oI5dT38gvZkbNyVfOuWKKo keIAfA/cqpYZ5NRue/rfsYYeHfbormXRtuCrdet/iUdUFkZU8R5ZXZ4c0VyYp7EttJSs CTgA== X-Gm-Message-State: AD7BkJIEFL4qY4qNZfqVPa7UYhbZYDlqwZAPDgcTLmUzZ0XRQkfA66+byuZ5iK7bb/ESpFOpoQMTYE7Ip2ID6vh/ MIME-Version: 1.0 X-Received: by 10.60.60.3 with SMTP id d3mr345478oer.66.1459239498544; Tue, 29 Mar 2016 01:18:18 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.157.11.143 with HTTP; Tue, 29 Mar 2016 01:18:18 -0700 (PDT) In-Reply-To: References: Date: Tue, 29 Mar 2016 01:18:18 -0700 X-Google-Sender-Auth: CBR6IHtKFmM7iRis-Rba3Ci97rk Message-ID: Subject: Re: Process stuck in "vnread" From: Maxim Sobolev To: Adrian Chadd Cc: "stable@freebsd.org" , FreeBSD Filesystems , Pawel Jakub Dawidek , Kirk McKusick , Konstantin Belousov Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 08:18:19 -0000 Hey Adrian, unfortunately, this is kinda our production build/test box, it's not particularly beefy and we also run some heavily contended apps on it too just to see how it performs on this size of machine, so we would not really want to put any heavy debug into kernel unless it's some issue that can be triggered quickly. This particular deadlock has been popping up every few weeks, annoying but not severely critical to jump to guns. I think, thanks to @kib and @avg we might be on the right track. At least that sounds plausible and kinda fits the pattern. On Mon, Mar 28, 2016 at 9:09 AM, Adrian Chadd wrote: > Hi, > > I think a bunch of the lock order checks with witness get disabled for ZFS > ? > > Maybe compile up a witness kernel so you can get 'show alllocks' in > ddb. That should help narrow down exactly what's going on. > > Thanks! > > > -a > > From owner-freebsd-fs@freebsd.org Tue Mar 29 18:58:35 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D89FFAE2401 for ; Tue, 29 Mar 2016 18:58:35 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher CAMELLIA256-SHA (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 869BC1D2F for ; Tue, 29 Mar 2016 18:58:34 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 63393 invoked by uid 89); 30 Mar 2016 03:58:26 +0900 X-ASJ-Track-ID: <20160329185826.63393.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 50.207.112.201 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1459277905.973411.63386@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@50.207.112.201) Received: from gw.asusa.net (HELO rd03.asusa-internal.net) (nishida@asusa.net@50.207.112.201) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 30 Mar 2016 03:58:25 +0900 Subject: Re: Problem with FUSE + fts To: freebsd-fs@freebsd.org References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> From: Hiroshi Nishida Message-ID: <56FAD050.2080707@asusa.net> Date: Tue, 29 Mar 2016 11:58:24 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <56F6148D.2030706@asusa.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 18:58:35 -0000 Now I figured out what causes this error. Originally, fts outputs an ENOENT error when: FTSENT *p; struct stat sb; stat(p->fts_accpath, &sb); p->fts_ino != sb.st_ino i.e., the inode of p is different from sb.st_ino. They are usually same but sometimes a new inode is allocated to p while scanning the dir in the following way: 1. FUSE lowelevel's forget() is called for some reason and removes all entries from the entry tables, as well as clears all inodes, while scanning the dir. 2. Since p is already removed from FUSE's entry table, FUSE adds it again and allocates a new inode. I don't know why forget() is called for the directory which is still open and clears all inodes, but according to fuse_lowlevel.h /** * Forget about an inode * * This function is called when the kernel removes an inode * from its internal caches. * * The inode's lookup count increases by one for every call to * fuse_reply_entry and fuse_reply_create. The nlookup parameter * indicates by how much the lookup count should be decreased. * * Inodes with a non-zero lookup count may receive request from * the kernel even after calls to unlink, rmdir or (when * overwriting an existing file) rename. Filesystems must handle * such requests properly and it is recommended to defer removal * of the inode until the lookup count reaches zero. Calls to * unlink, remdir or rename will be followed closely by forget * unless the file or directory is open, in which case the * kernel issues forget only after the release or releasedir * calls. * removing the inode should be deferred until the dir is closed. I haven't checked the ref count of each node yet but there seems to be a bug in the above process. Also, there is a suggestion for the hash table but I will post later. Any feedback is appreciated on it. Thank you. On 2016/03/25 21:48, Hiroshi Nishida wrote: > Thank you for your response. > > On 3/25/16 4:53 PM, Rick Macklem wrote: >> I think I see the same thing when doing an "rm -r" on a fuse/GlusterFS volume. > > Unfortunately, it happens also with "find XXX -print", though I have experienced a similar "rm -r" + "XXX: No such file or directory" problem with UFS + SUJ. > And I also verified with truss that in > > _fstat(fd, &sb); > p->fts_ino != sb.st_ino > > stat() system call is called with the same path as p's. > > Anyway, the following patch for lib/libc/gen/fts.c prevents the error but is far from a good solution. > https://github.com/scopedog/FUSE-Test/blob/master/fts.c.patch > It assumes that the filesystem id (f_type in struct statfs) of FUSE is 0xed but I am not sure if it's applicable to all FUSE filesystems. > > I'll look into FUSE source code next week. >> To be honest, I just add a "-f" to the command to shut it up and then it deleted >> the tree. >> >> I think, in general, what readdir() returns after an entry is unlink'd is undefined >> behaviour. As such, the safe way to delete all of a directory is something like: >> - in a loop until readdir() returns EOF >> - opendir() >> - readdir() the first entry >> - unlink() that entry >> - closedir() >> --> So that all you ever do is readdir() the first entry after an opendir(). > > By the way, could you delete all the files with "-f"? > I am testing with a pretty big directory containing 81,000 files/dirs and have never used "-f", but have to "rm -r" again for undeleted entries. > However, the offset problem is very interesting as it seems to be applicable to all filesystems. > > Thank you. > -- Hiroshi Nishida nishida@asusa.net From owner-freebsd-fs@freebsd.org Tue Mar 29 20:04:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE456AE2A39 for ; Tue, 29 Mar 2016 20:04:09 +0000 (UTC) (envelope-from danilo.walterino@yahoo.fr) Received: from nm49-vm4.bullet.mail.ne1.yahoo.com (nm49-vm4.bullet.mail.ne1.yahoo.com [98.138.121.132]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7AB661E67 for ; Tue, 29 Mar 2016 20:04:09 +0000 (UTC) (envelope-from danilo.walterino@yahoo.fr) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.fr; s=s2048; t=1459281842; bh=SjeGgivIg/UBBz7Pjz+xYfSPPcBIPbxCvR62dlfzxxc=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=WAO11o3iwf66V4IEb7mHGa90Fo5+ObwEUrtiqjxdOiWQZNM49ayTrPsNgPXIpcioGBH/pap2guqvH5TjP92yz4xNTs9gNPKtynpTvGphmZjaS/kxkV2Cl/XUBRVI3jtQuUnyBfYxdZr9dJxHukPeQr92lflXajJJwOlRsZw1C4ttBUQKXDth5rmr9dmpczFVkxWUxctwhrZfXcgPN9f2rHaGFIDvLlu6dh8UTHa8kh6lYTsxNvlvAEv7NuqUOgQytCO9kF12AkogwIwFfJy4yndpnpaeRRu5rzhWkuCc68TpYE8q01/XQnBzLUJiD1PVQzl9Q6K5fjVH5tv58R5d5Q== Received: from [127.0.0.1] by nm49.bullet.mail.ne1.yahoo.com with NNFMP; 29 Mar 2016 20:04:02 -0000 Received: from [98.138.100.114] by nm49.bullet.mail.ne1.yahoo.com with NNFMP; 29 Mar 2016 20:01:09 -0000 Received: from [212.82.98.50] by tm105.bullet.mail.ne1.yahoo.com with NNFMP; 29 Mar 2016 20:01:09 -0000 Received: from [212.82.98.79] by tm3.bullet.mail.ir2.yahoo.com with NNFMP; 29 Mar 2016 20:01:09 -0000 Received: from [127.0.0.1] by omp1016.mail.ir2.yahoo.com with NNFMP; 29 Mar 2016 20:01:09 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 579201.26892.bm@omp1016.mail.ir2.yahoo.com X-YMail-OSG: iVyDtp0VM1kAcpQHi3XdOml3qIvHI2_1BtUPVNlToDqkgTHQkEuXOEKjt12hLcB ULdlvlxINdchkYgMAPNJMMJobu7DhdfheL2s_1dEFaO0Mzt50AfQauDAh2g0JygwyCquYVkegZ5K Yze6c4lXW._XfID4S1TckWYyiyb6aOy3WfP07NGSeS8m1g51yUUaRVcL0xKU49CIpQ6Wbnre7Xmt TWTbcm1UrZeBZ9u3mDn4VWeMfgDFQeV5UJBpqqHolMfYx1ixJ9TIZfC9cqBmyUYzcjHjr0lbQJSK tEdOg6FDcuQwPIi11A9UuYtsaFnhR8emei67pJAU_XsiY8U1OXup3UkATke1dmbwSv6PiwZzoT3O 6nrGjhLDUW6QFMqx_lWcu6IOTYxzORM6S6GlUfDaW1v0fE.fh27osrqG39R49CHYlhUSVci.uZu6 8sJe8CpNfngOy9_NI924nW1Hf6Ii22AreUXxJ1dqUQS_LjOTRIheIIKYKuBr94z7Qok3gra2Li_R 07IRVKIXxJtf_Wo0N0LxhxZoM8w-- Received: by 212.82.98.121; Tue, 29 Mar 2016 20:01:08 +0000 Date: Tue, 29 Mar 2016 20:01:08 +0000 (UTC) From: Reply-To: To: Message-ID: <534935401.3391095.1459281668579.JavaMail.yahoo@mail.yahoo.com> Subject: partition boot problem MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit References: <534935401.3391095.1459281668579.JavaMail.yahoo.ref@mail.yahoo.com> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 20:04:09 -0000 Hello ! Here is the problem I encounter: I recently mounted a freebsd partition in rw mode by mistake under openbsd and since I can no longer boot freebsd.I can only mount partition and list files UNDER OpenBSD and that's all.I tried fsck_ufs without any options (from freebsd livecd) without success and I can't mount the partition anymore(from freebsd). May be someone meet the same problem and has solved it.Thanks for any help ! message of fsck: "invalid disklabel, segmentation fault" when I try to mount (from freebsd livecd): "Failed to read ROOTINO directory block 13280 cannot read blk: 320 unexpected soft update inconsistency" <=== From owner-freebsd-fs@freebsd.org Tue Mar 29 21:26:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84BFAAE21BC for ; Tue, 29 Mar 2016 21:26:03 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 70E8F1DBA for ; Tue, 29 Mar 2016 21:26:03 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6CB44AE21BB; Tue, 29 Mar 2016 21:26:03 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A12EAE21BA for ; Tue, 29 Mar 2016 21:26:03 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (glebi.us [96.95.210.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D5931DB8; Tue, 29 Mar 2016 21:26:02 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id u2TLPtUV058307 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 29 Mar 2016 14:25:56 -0700 (PDT) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id u2TLPtDE058306; Tue, 29 Mar 2016 14:25:55 -0700 (PDT) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@FreeBSD.org using -f Date: Tue, 29 Mar 2016 14:25:55 -0700 From: Gleb Smirnoff To: Bruce Evans Cc: fs@freebsd.org, rmacklem@freebsd.org Subject: Re: nfs pessimized by vnode pages changes Message-ID: <20160329212555.GP2616@FreeBSD.org> References: <20160327144755.Y4269@besplex.bde.org> <20160329090209.Q1020@besplex.bde.org> <20160329001623.GC2616@FreeBSD.org> <20160329122721.D1549@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160329122721.D1549@besplex.bde.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 21:26:03 -0000 On Tue, Mar 29, 2016 at 01:24:27PM +1100, Bruce Evans wrote: B> > B> I couldn't get full-fs-block input to work right in ncl_getpages(). In B> > B> the old version, vm_fault_hold() almost always calls it with the pages B> > B> for exactly 1 full block, because vm_fault_additional_pages() somehow B> > B> finds this many pages. B> > B> > The last quoted paragraph is a correct observation. According to your B> > investigations, prior to r292373 NFS was doing multiple page pageins, B> > despite it should have reported that it can't. My reading of the code B> > is the same: both before r292373 and after r292373 NFS should page in B> > a single page at request. I quickly reviewed the whole codepath and I B> > can't see how with older code it was able to do multiple page pageins. B> B> I found it. In old versions, vm_fault_additional_pages() exists and B> calls vnode_pager_haspage() to locate some additional pages. B> VOP_BMAP() is vop_stdbmap() for nfs. This doesn't locate any B> additional fs blocks, but vnode_pager_haspage() locates the page within B> the single nfs block and returns the number of pages to read behind and B> ahead to fill in the block. Hmm, the code in vnode_pager_generic_getpages() after my change does pretty much the same: /* Recalculate blocks available after/before to pages. */ poff = (foff % bsize) / PAGE_SIZE; before *= pagesperblock; before += poff; after *= pagesperblock; after += pagesperblock - (poff + 1); So, we got zeroes in 'after' and 'before' after vop_stdbmap. But here we add 'poff' to them, which should yield in the same read sizes as before. I will debug that. -- Totus tuus, Glebius. From owner-freebsd-fs@freebsd.org Tue Mar 29 22:49:46 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E4BAAE0978 for ; Tue, 29 Mar 2016 22:49:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id CC13F1944 for ; Tue, 29 Mar 2016 22:49:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:/g6ZiBEbUXxmVEfl5cH4Zp1GYnF86YWxBRYc798ds5kLTJ75oMqwAkXT6L1XgUPTWs2DsrQf27qQ6f+rADdIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/nh6bopNaKOE1hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv+YJa6jxfrw5QLpEF3xmdjltvIy4/SXEGCuO/HwHUmRetBtTAwnJ5VmuWJbqsir2v8J0wzSBNIvwQKxiChq46KI+ch7ji28iPjU69GzSwphqiatQoxasojRixIHJbYWNNLx1d/WOLpshWWNdU5MJBGR6CYSmYt5KVrJZMA== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ChAgC/BftW/61jaINdhAYuTwa6eQENgXAXCoUiSgKBdhQBAQEBAQEBAWMngi2CFQEBBAEBASAEJyALEAIBCBgCAg0ZAgInAQkmAgQIBwQBGgIEiAYOrymQaAEBAQEBAQEBAgEBAQEBAQEBFAR8hSGBe4JJhB4BAQ6DD4JWBZJ7hHGFcoUrjT2EOo8NAh4BAUKCAhqBZSAwAQaGfQcXH34BAQE X-IronPort-AV: E=Sophos;i="5.24,413,1454994000"; d="scan'208";a="273375203" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 29 Mar 2016 18:49:39 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 344BD15F55D; Tue, 29 Mar 2016 18:49:39 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id PDgPS0o0dDN3; Tue, 29 Mar 2016 18:49:38 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 34A1C15F574; Tue, 29 Mar 2016 18:49:38 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 8BlHUTr7DD6Q; Tue, 29 Mar 2016 18:49:38 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 1616615F55D; Tue, 29 Mar 2016 18:49:38 -0400 (EDT) Date: Tue, 29 Mar 2016 18:49:37 -0400 (EDT) From: Rick Macklem To: Hiroshi Nishida Cc: freebsd-fs@freebsd.org Message-ID: <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <56FAD050.2080707@asusa.net> References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> <56FAD050.2080707@asusa.net> Subject: Re: Problem with FUSE + fts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - IE7 (Win)/8.0.9_GA_6191) Thread-Topic: Problem with FUSE + fts Thread-Index: DHSG8yAYtLkabRsGtP9humiwNqnfcQ== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 22:49:46 -0000 Hiroshi Nishida wrote: > Now I figured out what causes this error. > > Originally, fts outputs an ENOENT error when: > > FTSENT *p; > struct stat sb; > > stat(p->fts_accpath, &sb); > p->fts_ino != sb.st_ino > > i.e., the inode of p is different from sb.st_ino. > They are usually same but sometimes a new inode is allocated to p while > scanning the dir in the following way: > > 1. FUSE lowelevel's forget() is called for some reason and removes all > entries from the entry tables, as well as clears all inodes, while scanning > the dir. > 2. Since p is already removed from FUSE's entry table, FUSE adds it again and > allocates a new inode. > > I don't know why forget() is called for the directory which is still open and > clears all inodes, but according to fuse_lowlevel.h > You've never mentioned what version of FreeBSD you are using? FreeBSD10 and later have a fuse client in sys/fs/fuse and I don't recall seeing this code in it. (I will grep for it, although I know there isn't a fuse_lowlevel.h.) If you are using FreeBSD9 or earlier with the fuse client in ports/sysutils, I'd suggest you try upgrading to FreeBSD10 and see if the problem exists there. rick > /** > * Forget about an inode > * > * This function is called when the kernel removes an inode > * from its internal caches. > * > * The inode's lookup count increases by one for every call to > * fuse_reply_entry and fuse_reply_create. The nlookup parameter > * indicates by how much the lookup count should be decreased. > * > * Inodes with a non-zero lookup count may receive request from > * the kernel even after calls to unlink, rmdir or (when > * overwriting an existing file) rename. Filesystems must handle > * such requests properly and it is recommended to defer removal > * of the inode until the lookup count reaches zero. Calls to > * unlink, remdir or rename will be followed closely by forget > * unless the file or directory is open, in which case the > * kernel issues forget only after the release or releasedir > * calls. > * > > removing the inode should be deferred until the dir is closed. > > I haven't checked the ref count of each node yet but there seems to be a bug > in the above process. > > Also, there is a suggestion for the hash table but I will post later. > > Any feedback is appreciated on it. > > Thank you. > > > On 2016/03/25 21:48, Hiroshi Nishida wrote: > > Thank you for your response. > > > > On 3/25/16 4:53 PM, Rick Macklem wrote: > >> I think I see the same thing when doing an "rm -r" on a fuse/GlusterFS > >> volume. > > > > Unfortunately, it happens also with "find XXX -print", though I have > > experienced a similar "rm -r" + "XXX: No such file or directory" problem > > with UFS + SUJ. > > And I also verified with truss that in > > > > _fstat(fd, &sb); > > p->fts_ino != sb.st_ino > > > > stat() system call is called with the same path as p's. > > > > Anyway, the following patch for lib/libc/gen/fts.c prevents the error but > > is far from a good solution. > > https://github.com/scopedog/FUSE-Test/blob/master/fts.c.patch > > It assumes that the filesystem id (f_type in struct statfs) of FUSE is 0xed > > but I am not sure if it's applicable to all FUSE filesystems. > > > > I'll look into FUSE source code next week. > >> To be honest, I just add a "-f" to the command to shut it up and then it > >> deleted > >> the tree. > >> > >> I think, in general, what readdir() returns after an entry is unlink'd is > >> undefined > >> behaviour. As such, the safe way to delete all of a directory is something > >> like: > >> - in a loop until readdir() returns EOF > >> - opendir() > >> - readdir() the first entry > >> - unlink() that entry > >> - closedir() > >> --> So that all you ever do is readdir() the first entry after an > >> opendir(). > > > > By the way, could you delete all the files with "-f"? > > I am testing with a pretty big directory containing 81,000 files/dirs and > > have never used "-f", but have to "rm -r" again for undeleted entries. > > However, the offset problem is very interesting as it seems to be > > applicable to all filesystems. > > > > Thank you. > > > > -- > Hiroshi Nishida > nishida@asusa.net > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Mar 29 22:55:29 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 537FBAE0B39 for ; Tue, 29 Mar 2016 22:55:29 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher CAMELLIA256-SHA (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CFB3C1C61 for ; Tue, 29 Mar 2016 22:55:28 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 68337 invoked by uid 89); 30 Mar 2016 07:55:26 +0900 X-ASJ-Track-ID: <20160329225526.68337.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 50.207.112.201 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1459292125.904400.68330@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@50.207.112.201) Received: from gw.asusa.net (HELO rd03.asusa-internal.net) (nishida@asusa.net@50.207.112.201) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 30 Mar 2016 07:55:25 +0900 Subject: Re: Problem with FUSE + fts References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> <56FAD050.2080707@asusa.net> <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> To: freebsd-fs@freebsd.org From: Hiroshi Nishida Message-ID: <56FB07DC.4000504@asusa.net> Date: Tue, 29 Mar 2016 15:55:24 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 22:55:29 -0000 Hi, I'm using FreeBSD 10.2-RELEASE-p9. The program I checked is /usr/ports/sysutils/fusefs-libs/work/libfuse-fuse_2_9_5/lib/fuse.c. If there is anything wrong with it, let me know. Shouldn't I use port's fusefs-libs? On 2016/03/29 15:49, Rick Macklem wrote: > You've never mentioned what version of FreeBSD you are using? > FreeBSD10 and later have a fuse client in sys/fs/fuse and I don't recall seeing this > code in it. (I will grep for it, although I know there isn't a fuse_lowlevel.h.) > > If you are using FreeBSD9 or earlier with the fuse client in ports/sysutils, I'd > suggest you try upgrading to FreeBSD10 and see if the problem exists there. > > rick -- Hiroshi Nishida nishida@asusa.net From owner-freebsd-fs@freebsd.org Tue Mar 29 23:07:04 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5EED0AE0EB6 for ; Tue, 29 Mar 2016 23:07:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1EB271245 for ; Tue, 29 Mar 2016 23:07:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:r8whBR1bkCh+AV/JsmDT+DRfVm0co7zxezQtwd8ZsegRLPad9pjvdHbS+e9qxAeQG96Lu7Qe1KGP6OjJYi8p39WoiDg6aptCVhsI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6kO74TNaIBjjLw09fr2zQd6CyZ7onLnps7ToICx2xxOFKYtoKxu3qQiD/uI3uqBFbpgL9x3Sv3FTcP5Xz247bXianhL7+9vitMU7q3cYk7sb+sVBSaT3ebgjBfwdVWx+cjN92Mq+lRjZShCP5zM6U34WkxZBS1zD7Qr6X5v4miLhq/F0ni+XIZulY6ozXGGY7qxoADrhgyQDOjtxpHvSg8dziK9eiA+mqAFyx5bUJoqcYqktNpjBdM8XEDISFv1aUDZMV8blN9MC X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CvBAB4CftW/61jaINdhAZ9Brx3FwqFIkoCgXkRAQEBAQEBAQFjJ4ItghQBAQEDAQEBASArIAsFCwIBCBgCAg0ZAgInAQkmAgQIBwQBHASHfggOrzCQaQEBAQEBAQEBAQEBAQEBAQEBAQETBHyFIYF7gkmEHgEBDoMPglYFl2yFcoUrkXePDQI2LIQBIDAHhn0HFx9+AQEB X-IronPort-AV: E=Sophos;i="5.24,413,1454994000"; d="scan'208";a="275125488" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 29 Mar 2016 19:06:57 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6A17F15F55D; Tue, 29 Mar 2016 19:06:57 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id HsR3ucw0RJXa; Tue, 29 Mar 2016 19:06:56 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id DB15915F574; Tue, 29 Mar 2016 19:06:56 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id mc98662_1ygc; Tue, 29 Mar 2016 19:06:56 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id BE7E915F55D; Tue, 29 Mar 2016 19:06:56 -0400 (EDT) Date: Tue, 29 Mar 2016 19:06:56 -0400 (EDT) From: Rick Macklem To: Hiroshi Nishida Cc: freebsd-fs@freebsd.org Message-ID: <2009006928.37186618.1459292816761.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <56FB07DC.4000504@asusa.net> References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> <56FAD050.2080707@asusa.net> <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> <56FB07DC.4000504@asusa.net> Subject: Re: Problem with FUSE + fts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - IE7 (Win)/8.0.9_GA_6191) Thread-Topic: Problem with FUSE + fts Thread-Index: tcIlmcjXIjuU1WhJJFNsZSevHBp1Ng== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 23:07:04 -0000 Hiroshi Nishida wrote: > Hi, > > I'm using FreeBSD 10.2-RELEASE-p9. > The program I checked is > /usr/ports/sysutils/fusefs-libs/work/libfuse-fuse_2_9_5/lib/fuse.c. > If there is anything wrong with it, let me know. > Shouldn't I use port's fusefs-libs? > There is a fuse client in sys/fs/fuse on 10.2. As far as I know, that has replaced the one in ports. (At least it is the one I use.) There should be a fuse.ko in /boot/kernel on your system. If you overwrote that with one built from ports, you can do a kernel build in /usr/src/sys and it will be built again. You might be able to: # cd /usr/src/sys/modules/fuse # make - This should work, but I haven't tried it. rick > On 2016/03/29 15:49, Rick Macklem wrote: > > You've never mentioned what version of FreeBSD you are using? > > FreeBSD10 and later have a fuse client in sys/fs/fuse and I don't recall > > seeing this > > code in it. (I will grep for it, although I know there isn't a > > fuse_lowlevel.h.) > > > > If you are using FreeBSD9 or earlier with the fuse client in > > ports/sysutils, I'd > > suggest you try upgrading to FreeBSD10 and see if the problem exists there. > > > > rick > > -- > Hiroshi Nishida > nishida@asusa.net > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Mar 29 23:20:53 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1127BAE108D for ; Tue, 29 Mar 2016 23:20:53 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher CAMELLIA256-SHA (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A675817D1 for ; Tue, 29 Mar 2016 23:20:51 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 68769 invoked by uid 89); 30 Mar 2016 08:20:50 +0900 X-ASJ-Track-ID: <20160329232050.68769.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 50.207.112.201 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1459293649.959055.68762@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@50.207.112.201) Received: from gw.asusa.net (HELO rd03.asusa-internal.net) (nishida@asusa.net@50.207.112.201) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 30 Mar 2016 08:20:49 +0900 Subject: Re: Problem with FUSE + fts To: Rick Macklem References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> <56FAD050.2080707@asusa.net> <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> <56FB07DC.4000504@asusa.net> <2009006928.37186618.1459292816761.JavaMail.zimbra@uoguelph.ca> Cc: freebsd-fs@freebsd.org From: Hiroshi Nishida Message-ID: <56FB0DD0.4000806@asusa.net> Date: Tue, 29 Mar 2016 16:20:48 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <2009006928.37186618.1459292816761.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 23:20:53 -0000 Thank you. I use fuse.ko that was in /usr/src/sys/modules/fuse. However, to use high level functions or something, I needed to install fusefs-libs from ports. So, you mean fusefs-libs is unnecessary? As long as I remember, I couldn't compile my fuse programs without fusefs-libs.... I thought the only kernel module was merged into FreeBSD's src. I will check more carefully later. On 2016/03/29 16:06, Rick Macklem wrote: > Hiroshi Nishida wrote: >> Hi, >> >> I'm using FreeBSD 10.2-RELEASE-p9. >> The program I checked is >> /usr/ports/sysutils/fusefs-libs/work/libfuse-fuse_2_9_5/lib/fuse.c. >> If there is anything wrong with it, let me know. >> Shouldn't I use port's fusefs-libs? >> > There is a fuse client in sys/fs/fuse on 10.2. As far as I know, that has > replaced the one in ports. (At least it is the one I use.) > There should be a fuse.ko in /boot/kernel on your system. If you overwrote > that with one built from ports, you can do a kernel build in /usr/src/sys > and it will be built again. You might be able to: > # cd /usr/src/sys/modules/fuse > # make > - This should work, but I haven't tried it. > > rick > >> On 2016/03/29 15:49, Rick Macklem wrote: >>> You've never mentioned what version of FreeBSD you are using? >>> FreeBSD10 and later have a fuse client in sys/fs/fuse and I don't recall >>> seeing this >>> code in it. (I will grep for it, although I know there isn't a >>> fuse_lowlevel.h.) >>> >>> If you are using FreeBSD9 or earlier with the fuse client in >>> ports/sysutils, I'd >>> suggest you try upgrading to FreeBSD10 and see if the problem exists there. >>> >>> rick >> -- >> Hiroshi Nishida >> nishida@asusa.net >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> -- Hiroshi Nishida, PhD President, ASUSA Corporation nishida@asusa.net From owner-freebsd-fs@freebsd.org Wed Mar 30 23:44:39 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A43DAAE3D9F for ; Wed, 30 Mar 2016 23:44:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 953CC1FBB for ; Wed, 30 Mar 2016 23:44:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2UNibdS012475 for ; Wed, 30 Mar 2016 23:44:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Wed, 30 Mar 2016 23:44:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: j@nitrology.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Mar 2016 23:44:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 --- Comment #17 from Jason Wolfe --- We are seeing this issue at Limelight across a good number of boxes on stable/10 @ r285800. We have a newer 10 version in testing @ r296969, but = have not used it with enough anger to empirically know if it has been fixed. It should be getting deployed further in the next months or so. Unfortunately we don't yet have a way to reproduce, so we can introduce INVARIANTS into a kernel to help capture some info. I'll report back if we= do, but in the meantime I did want to +1 this. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Mar 30 23:47:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04CB1AE3E38 for ; Wed, 30 Mar 2016 23:47:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id B7A121086 for ; Wed, 30 Mar 2016 23:47:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:YoJGPRFE2WG0hcqLU5K4NZ1GYnF86YWxBRYc798ds5kLTJ75oMqwAkXT6L1XgUPTWs2DsrQf27qQ6fGrAzNIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/nh6boqtaJPE1hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv+YJa6jxfrw5QLpEF3xmdjltvIy4/SXEGCuO/HwHUmRetBtTAwnJ5VmuWJbqsir2v8J0wzSBNIvwQKxiChq46KI+ch7ji28iPjU69GzSwphqiatQoxasojRixIHJbYWNNLx1d/WOLpshWWNdU5MJBGR6CYSmYt5KVrJZMA== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DXCAA3ZfxW/61jaINdhAcuTwa9ABcKhSJKAoIDEQEBAQEBAQEBYyeCLYIUAQEBAwEBAQEgKyALBQsCAQgYAgINGQICJwEJJgIECAcEARwEh34IDrBRkRgBAQEBAQEBAQIBAQEBAQEBARQEfIUigXyCSYQfAQEOgw+CVgWNUIoehXKFK5F3jw4CNiyEAyAwB4crBxcffgEBAQ X-IronPort-AV: E=Sophos;i="5.24,419,1454994000"; d="scan'208";a="273621897" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 30 Mar 2016 19:47:24 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C5BF215F578; Wed, 30 Mar 2016 19:47:24 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id cN0LpKDh-pZG; Wed, 30 Mar 2016 19:47:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 1C01115F579; Wed, 30 Mar 2016 19:47:24 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id NMpdyuzFFalN; Wed, 30 Mar 2016 19:47:24 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id F3CC715F578; Wed, 30 Mar 2016 19:47:23 -0400 (EDT) Date: Wed, 30 Mar 2016 19:47:23 -0400 (EDT) From: Rick Macklem To: Hiroshi Nishida Cc: freebsd-fs@freebsd.org Message-ID: <294037501.39717127.1459381643954.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <56FB0DD0.4000806@asusa.net> References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> <56FAD050.2080707@asusa.net> <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> <56FB07DC.4000504@asusa.net> <2009006928.37186618.1459292816761.JavaMail.zimbra@uoguelph.ca> <56FB0DD0.4000806@asusa.net> Subject: Re: Problem with FUSE + fts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - IE7 (Win)/8.0.9_GA_6191) Thread-Topic: Problem with FUSE + fts Thread-Index: gxIadDJV5Qvn6zkcSw6zajQOftBp3g== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Mar 2016 23:47:42 -0000 Hiroshi Nishida wrote: > Thank you. > > I use fuse.ko that was in /usr/src/sys/modules/fuse. > However, to use high level functions or something, I needed to install > fusefs-libs from ports. > So, you mean fusefs-libs is unnecessary? For what I use it for (GlusterFS), I don't think I use it. (I'd have to look to be sure, but I think everything that GlusterFS needs to build its fuse interface is in the GlusterFS source tree.) So, for me, the answer is no. For what you are doing, I have no idea. If this "forget()" is in the userland stuff, then you are in a place I know nothing about. rick > As long as I remember, I couldn't compile my fuse programs without > fusefs-libs.... > I thought the only kernel module was merged into FreeBSD's src. > I will check more carefully later. > > On 2016/03/29 16:06, Rick Macklem wrote: > > Hiroshi Nishida wrote: > >> Hi, > >> > >> I'm using FreeBSD 10.2-RELEASE-p9. > >> The program I checked is > >> /usr/ports/sysutils/fusefs-libs/work/libfuse-fuse_2_9_5/lib/fuse.c. > >> If there is anything wrong with it, let me know. > >> Shouldn't I use port's fusefs-libs? > >> > > There is a fuse client in sys/fs/fuse on 10.2. As far as I know, that has > > replaced the one in ports. (At least it is the one I use.) > > There should be a fuse.ko in /boot/kernel on your system. If you overwrote > > that with one built from ports, you can do a kernel build in /usr/src/sys > > and it will be built again. You might be able to: > > # cd /usr/src/sys/modules/fuse > > # make > > - This should work, but I haven't tried it. > > > > rick > > > >> On 2016/03/29 15:49, Rick Macklem wrote: > >>> You've never mentioned what version of FreeBSD you are using? > >>> FreeBSD10 and later have a fuse client in sys/fs/fuse and I don't recall > >>> seeing this > >>> code in it. (I will grep for it, although I know there isn't a > >>> fuse_lowlevel.h.) > >>> > >>> If you are using FreeBSD9 or earlier with the fuse client in > >>> ports/sysutils, I'd > >>> suggest you try upgrading to FreeBSD10 and see if the problem exists > >>> there. > >>> > >>> rick > >> -- > >> Hiroshi Nishida > >> nishida@asusa.net > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >> > > -- > Hiroshi Nishida, PhD > President, ASUSA Corporation > nishida@asusa.net > From owner-freebsd-fs@freebsd.org Thu Mar 31 03:49:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D33AAAE3222 for ; Thu, 31 Mar 2016 03:49:42 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6237D1AE7 for ; Thu, 31 Mar 2016 03:49:42 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-lb0-x22c.google.com with SMTP id bc4so44238347lbc.2 for ; Wed, 30 Mar 2016 20:49:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to; bh=oKpVWvQ5bIhtXjWUJbIalhGHJCpUxwun6p4ge0Yc3I0=; b=HW8U3nhcWmexjjuVlou1vhG0R76/pwm1BoDMh50u3XZ34X+3t24erF/Py0/dARI1Mr S5lL9BbSEj3ta4H73rN1ZkTv2WbfGVIWXs1Jac2hNt+eb9igx1M2ge+34PjTGbXIdnA/ 8i9goRI7lnWbp3QvCzNNnQVOdwHzMEYvITHbCMr2R8bRgGEIPxQG/eQRay4E/ftJ4eeg utdxdMA5oM/0AS0sOMZbVZuBIUXDnlfoJ8fUI5bCBN59fmRvE/n+VSoZhWO9IZOURRWx utqWAbUJ4s//7068wgyBd24QAe68boEYEugo3sxN3HEv/lou8nARnWZ618gf7nEF9c06 kuUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=oKpVWvQ5bIhtXjWUJbIalhGHJCpUxwun6p4ge0Yc3I0=; b=EACDhbHvb3Z0r0FEVirk0nlVml2vg5k/52Zkv/5K2wiPuSZ5c1IVZaJC26avY4uC4Y rT65bXcNJ6pP6XluuM/Lg6MvnQsHKJtCZB4/8XA8L+BAwWlcdrPeZRlHvtdlq9D1B49P WgjdZY4ax0hPYLTLWbU65XW3Z1Aeh0KH1g5/qT6X/bK97ky7HQpHBMeDJVJDosUHKZYm 8ArbjSFcMuGKpp+1b3aehPdbj8ENwC9O3ZZAsQT6V6hD9Ly2BmgsP97Ivmx7FjthXabW eFE+FartBy3vxBb1jKMJftr17RyW+6rHtWffFaSd4nwJg6dPkmQzwVgCcqMYrpOpg3x7 QXig== X-Gm-Message-State: AD7BkJIm7UChr54/xDfy47jiAboUVO998NnAIRX3vtJsbV0oUCc76OUcHmh5e85NxWgFaySRNLCeNDI/7azF/A== MIME-Version: 1.0 X-Received: by 10.112.72.135 with SMTP id d7mr5683520lbv.126.1459396180595; Wed, 30 Mar 2016 20:49:40 -0700 (PDT) Received: by 10.112.44.68 with HTTP; Wed, 30 Mar 2016 20:49:40 -0700 (PDT) Received: by 10.112.44.68 with HTTP; Wed, 30 Mar 2016 20:49:40 -0700 (PDT) Date: Thu, 31 Mar 2016 06:49:40 +0300 Message-ID: Subject: zfsOnRoot From: Sami Halabi To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 03:49:42 -0000 hi everybody, I have a fbsd box indtalled that was upgraded using freebsd-update from 9.2 to 10.3-RC2. The box has a single freebsd-ufs / partition on the almost the whole disk beside a small swap partiotion. is there a known way to convert to zfs whithout installing from media? ps. i intend to upgrade to 10.3-R. Thanks in advance, Sami From owner-freebsd-fs@freebsd.org Thu Mar 31 05:43:47 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BDD5AE4C0D for ; Thu, 31 Mar 2016 05:43:47 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (vps.rulingia.com [103.243.244.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.rulingia.com", Issuer "Let's Encrypt Authority X1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F02F819E2 for ; Thu, 31 Mar 2016 05:43:46 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (c122-106-195-17.belrs5.nsw.optusnet.com.au [122.106.195.17]) by vps.rulingia.com (8.15.2/8.15.2) with ESMTPS id u2V5hY5N094932 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 31 Mar 2016 16:43:39 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id u2V5hQxk028709 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 31 Mar 2016 16:43:26 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id u2V5hPwn028708; Thu, 31 Mar 2016 16:43:25 +1100 (AEDT) (envelope-from peter) Date: Thu, 31 Mar 2016 16:43:25 +1100 From: Peter Jeremy To: Sami Halabi Cc: freebsd-fs@freebsd.org Subject: Re: zfsOnRoot Message-ID: <20160331054325.GB44174@server.rulingia.com> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="WIyZ46R2i8wDzkSu" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.24 (2015-08-30) X-Greylist: Sender succeeded STARTTLS authentication, not delayed by milter-greylist-4.4.3 (vps.rulingia.com [103.243.244.15]); Thu, 31 Mar 2016 16:43:40 +1100 (AEDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 05:43:47 -0000 --WIyZ46R2i8wDzkSu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2016-Mar-31 06:49:40 +0300, Sami Halabi wrote: >The box has a single freebsd-ufs / partition on the almost the whole disk >beside a small swap partiotion. > >is there a known way to convert to zfs whithout installing from media? You can't do an in-place conversion of UFS to ZFS - you'll need to dump your UFS somewhere, build a ZFS root and restore the backup. Note that restore(8) is completely userland so you can use dump/restore. If your used space is small enough, you might be able to use the swap partition to store the backup. --=20 Peter Jeremy --WIyZ46R2i8wDzkSu Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJW/Lj9XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0n5kP/j2JdyHvF7p3fPwAd2lyxpci XBiqPuEl8BKMlCHt3oEdlz5wHcKOPbsGWs0lEr+mOApBbsC/ph8VQelFel0FFMoH FyoiN1yp2mEew/1wyZUVpAVVWS1Y1Z/5ITtmvkBsrQSZj9qlvum6KBx++PaJDA/P TKa5xYdmcsVntwuSLCTQ7hk0aCvKl9blmWlLpe9CipVDwRYpT4ZksQt4L4XIG3Ry 5UR/9ufqLw7/jbTkmQdn+VRS+i6ukmdG8xAIBVQdL9OZKhxvrDNPxmIoUhdVv4Kt TRiEmdWAfcm2qmnc23BcisVtuKOEYMaP8rJF2ZkJS3PcNP6PK3Y+Huye/hpmiy2T tSP/HFN3FI9aJV0nTzCJJAVylVeZkYCULBKQzcK4kUVVnctnkv6SvqVVcgZaF+62 Jug9Cc7VHAv0EyCvCRKUInza3gkE5f9/xsp4ZKffRkNs4KJmCB3tnDMpIhW2yp+z VEFpHkwnLZg85MaBs6qdGy/Y7oLneP3Mly7SIN5YOgwL78LpcnvSp9ulmvn6At6+ i6EzdoehSvzblQ/ViQu02JniYSUNVm7LH6ZBdtWybmYZeMaT86PS5C+IoSQAoufg NXo4dPEI2+4tWVahJnDXXXnnZXAgypWPOS72T5g09QEyOo/Fbp73d6feoSRTYlx6 G1S6YKWsI1w12aUM3+OY =KwXg -----END PGP SIGNATURE----- --WIyZ46R2i8wDzkSu-- From owner-freebsd-fs@freebsd.org Thu Mar 31 07:10:22 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71C79AE2DA7 for ; Thu, 31 Mar 2016 07:10:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 622DD1CB7 for ; Thu, 31 Mar 2016 07:10:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u2V7AM1d096180 for ; Thu, 31 Mar 2016 07:10:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207464] Panic when destroying ZFS snapshot on boot filesystem Date: Thu, 31 Mar 2016 07:10:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: matthias.pfaller@familie-pfaller.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 07:10:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207464 --- Comment #5 from Matthias Pfaller = --- We also get these panics (but not on the boot pool): FreeBSD nibbler 10.2-RELEASE-p9 FreeBSD 10.2-RELEASE-p9 #0: Thu Jan 14 01:3= 2:46 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERI= C=20 amd64 vputx: negative ref count 0xfffff8026e11e588: tag zfs, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) VI_LOCKed lock type zfs: EXCL by thread 0xfffff80dae520000 (pid 28114, = zfs, tid 100862) panic: vputx: negative ref cnt cpuid =3D 2 KDB: stack backtrace: #0 0xffffffff80984ef0 at kdb_backtrace+0x60 #1 0xffffffff80948aa6 at vpanic+0x126 #2 0xffffffff80948973 at panic+0x43 #3 0xffffffff809eb7d5 at vputx+0x2d5 #4 0xffffffff809e4f59 at dounmount+0x689 #5 0xffffffff81a4bdd4 at zfs_unmount_snap+0x114 #6 0xffffffff81a4efc1 at zfs_ioc_destroy_snaps+0xc1 #7 0xffffffff81a4dae0 at zfsdev_ioctl+0x5f0 #8 0xffffffff80830019 at devfs_ioctl_f+0x139 #9 0xffffffff8099cde5 at kern_ioctl+0x255 #10 0xffffffff8099cae0 at sys_ioctl+0x140 #11 0xffffffff80d4b3e7 at amd64_syscall+0x357 #12 0xffffffff80d30acb at Xfast_syscall+0xfb #0 doadump (textdump=3D) at pcpu.h:219 #1 0xffffffff80948702 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff80948ae5 in vpanic (fmt=3D,=20 ap=3D) at /usr/src/sys/kern/kern_shutdown.c:758 #3 0xffffffff80948973 in panic (fmt=3D0x0) at /usr/src/sys/kern/kern_shutdown.c:687 #4 0xffffffff809eb7d5 in vputx (vp=3D,=20 func=3D) at /usr/src/sys/kern/vfs_subr.c:2269 #5 0xffffffff809e4f59 in dounmount (mp=3D0xfffff81bc9ada000,=20 flags=3D, td=3D) at /usr/src/sys/kern/vfs_mount.c:1362 #6 0xffffffff81a4bdd4 in zfs_unmount_snap (snapname=3D) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/z= fs_ioctl.c:3484 #7 0xffffffff81a4efc1 in zfs_ioc_destroy_snaps ( poolname=3D0xfffffe1025f72000 "zmnt", innvl=3D,=20 outnvl=3D0xfffff804303ca740) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/z= fs_ioctl.c:3557 #8 0xffffffff81a4dae0 in zfsdev_ioctl (dev=3D,=20 zcmd=3D, arg=3D,=20 flag=3D, td=3D) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/z= fs_ioctl.c:6198 #9 0xffffffff80830019 in devfs_ioctl_f (fp=3D0xfffff81aca04a870,=20 com=3D3222821411, data=3D0xfffffe20260458e0, cred=3D,=20 td=3D0xfffff80dae520000) at /usr/src/sys/fs/devfs/devfs_vnops.c:785 #10 0xffffffff8099cde5 in kern_ioctl (td=3D0xfffff80dae520000,=20 fd=3D, com=3D0) at file.h:320 #11 0xffffffff8099cae0 in sys_ioctl (td=3D0xfffff80dae520000,=20 uap=3D0xfffffe2026045a40) at /usr/src/sys/kern/sys_generic.c:718 #12 0xffffffff80d4b3e7 in amd64_syscall (td=3D0xfffff80dae520000, traced=3D= 0) at subr_syscall.c:134 #13 0xffffffff80d30acb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #14 0x0000000801a01fda in ?? () --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 31 15:51:49 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03D59AE3381 for ; Thu, 31 Mar 2016 15:51:49 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher CAMELLIA256-SHA (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 986171E33 for ; Thu, 31 Mar 2016 15:51:47 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 22861 invoked by uid 89); 1 Apr 2016 00:51:38 +0900 X-ASJ-Track-ID: <20160331155138.22861.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 50.207.112.201 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1459439498.543875.22853@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@50.207.112.201) Received: from gw.asusa.net (HELO rd03.asusa-internal.net) (nishida@asusa.net@50.207.112.201) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 1 Apr 2016 00:51:38 +0900 Subject: Re: Problem with FUSE + fts To: Rick Macklem References: <56F42EF4.5000505@asusa.net> <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> <56F6148D.2030706@asusa.net> <56FAD050.2080707@asusa.net> <765991039.37160180.1459291777879.JavaMail.zimbra@uoguelph.ca> <56FB07DC.4000504@asusa.net> <2009006928.37186618.1459292816761.JavaMail.zimbra@uoguelph.ca> <56FB0DD0.4000806@asusa.net> <294037501.39717127.1459381643954.JavaMail.zimbra@uoguelph.ca> Cc: freebsd-fs@freebsd.org From: Hiroshi Nishida Message-ID: <56FD4788.5030808@asusa.net> Date: Thu, 31 Mar 2016 08:51:36 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <294037501.39717127.1459381643954.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 15:51:49 -0000 Since you do stuff much closer to the real filesystem, it is understandable that you did not know the interfaces provided through libfuse. However for many people who start writing their own FUSE filesystems, using high/low level interfaces in libfuse is very common. A tutorial like http://www.ibm.com/developerworks/library/l-fuse/ or an example at https://github.com/libfuse/libfuse/blob/master/example/hello.c always shows how to use them and I guess many FUSE based filesystems use similar techniques. Therefore, /usr/ports/sysutils/fusefs-libs is necessary in many cases, not to mention fuse.ko at /usr/src/sys/fs/fuse, for using or creating a FUSE based filesystem. Since my problem is likely to happen also with Linux, I will continue discussing at fuse-devel ML. Thank you. On 2016/03/30 16:47, Rick Macklem wrote: > For what I use it for (GlusterFS), I don't think I use it. (I'd have to > look to be sure, but I think everything that GlusterFS needs to build its > fuse interface is in the GlusterFS source tree.) > > So, for me, the answer is no. For what you are doing, I have no idea. > > If this "forget()" is in the userland stuff, then you are in a place I > know nothing about. -- Hiroshi Nishida nishida@asusa.net From owner-freebsd-fs@freebsd.org Fri Apr 1 12:04:31 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 155FEAEBB31 for ; Fri, 1 Apr 2016 12:04:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0662413C2 for ; Fri, 1 Apr 2016 12:04:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u31C4U24091494 for ; Fri, 1 Apr 2016 12:04:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Fri, 01 Apr 2016 12:04:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: cejkar@fit.vutbr.cz X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Apr 2016 12:04:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 Rudolf =C4=8Cejka changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cejkar@fit.vutbr.cz --- Comment #18 from Rudolf =C4=8Cejka --- Hello, another +1 for us. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Apr 1 12:14:10 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A64CAEBE7F for ; Fri, 1 Apr 2016 12:14:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 698861A49 for ; Fri, 1 Apr 2016 12:14:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u31CEA0N042846 for ; Fri, 1 Apr 2016 12:14:10 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Fri, 01 Apr 2016 12:14:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: cejkar@fit.vutbr.cz X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Apr 2016 12:14:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 --- Comment #19 from Rudolf =C4=8Cejka --- And now the debugging info. Unfortunately, we are new for this bug, so we h= ave the kernel without INVARIANTS and without debugging.patch yet. We have locked bacula-fd, which is backup client with similar function to f= ind & stat and it occured during incremental backup operation: procstat -kk (I can send full procstat -kk -a personally, it has 429876 byt= es): 1331 101013 bacula-fd - mi_switch+0xbe sleepq_catch_signals+0xab sleepq_wait_sig+0xc _cv_wait_sig+0x17a seltdwait+= 0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x321 Xfast_syscall+0xfb=20 1331 101035 bacula-fd - mi_switch+0xbe sleepq_catch_signals+0xab sleepq_timedwait_sig+0xf _sleep+0x238 umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x321 Xfast_syscall+0xfb=20 1331 102066 bacula-fd - mi_switch+0xbe sleepq_wait+0= x3a _sleep+0x287 vnode_create_vobject+0x100 ufs_lookup_ino+0xa0 VOP_CACHEDLOOKUP_APV+0x7a vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0x7a lookup+0x571 namei+0x344 kern_statat_vnhook+0xae sys_lstat+0x30 amd64_syscall+0x321 Xfast_syscall+0xfb=20 1331 103370 bacula-fd - mi_switch+0xbe sleepq_catch_signals+0xab sleepq_timedwait_sig+0xf _cv_timedwait_sig_sbt+0x= 19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x321 Xfast_syscall+0xfb=20 where the important thread is TID 102066, with vnode_create_vobject. (kgdb) info threads ... 701 Thread 102066 (PID=3D1331: bacula-fd) sched_switch (td=3D0xfffff803ec5= 26960, newtd=3D0x0, flags=3D) at /usr/src/sys/kern/sched_ule.= c:1945 ... (kgdb) thread 701 [Switching to thread 701 (Thread 102066)]#0 sched_switch (td=3D0xfffff803ec526960, newtd=3D0x0, flags=3D) at /usr/src/sys/kern/sched_ule.c:1945 1945 cpuid =3D PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=3D0xfffff803ec526960, newtd=3D0x0, flags=3D) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff80580d4e in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/sys/kern/kern_synch.c:491 #2 0xffffffff805c155a in sleepq_wait (wchan=3D0x0, pri=3D0) at /usr/src/sys/kern/subr_sleepqueue.c:618 #3 0xffffffff80580837 in _sleep (ident=3D, lock=3D, priority=3D, wmesg=3D, sbt=3D, pr=3D, flags=3D) at /usr/src/sys/kern/kern_synch.c:255 #4 0xffffffff80766e60 in vnode_create_vobject (vp=3D0xfffff816fe4f3588, isize=3D512, td=3D0xfffff803ec526960) at /usr/src/sys/vm/vnode_pager.c:120 #5 0xffffffff8072fe80 in ufs_lookup_ino (vdp=3D0xfffff816fe4f3588, vpp=3D0xfffffe17f2e63858, cnp=3D0xfffffe17f2e63880, dd_ino=3D0x0) at /usr/src/sys/ufs/ufs/ufs_lookup.c:259 #6 0xffffffff807f897a in VOP_CACHEDLOOKUP_APV (vop=3D, a=3D) at vnode_if.c:197 #7 0xffffffff806092e6 in vfs_cache_lookup (ap=3D) at vnode_if.h:80 #8 0xffffffff807f883a in VOP_LOOKUP_APV (vop=3D, a=3D= ) at vnode_if.c:129 #9 0xffffffff806111f1 in lookup (ndp=3D0xfffffe17f2e637f8) at vnode_if.h:54 #10 0xffffffff80610a04 in namei (ndp=3D0xfffffe17f2e637f8) at /usr/src/sys/kern/vfs_lookup.c:302 #11 0xffffffff8062631e in kern_statat_vnhook (td=3D0xfffff803ec526960, flag=3D, fd=3D-100, path=3D0x43c18128 , pathseg=3DUIO_USERSPACE, sbp=3D0xfffffe17f2e63910, hook=3D0) at /usr/src/sys/kern/vfs_syscalls.c:2300 #12 0xffffffff80626420 in sys_lstat (td=3D0x0, uap=3D0xfffffe17f2e63a40) at /usr/src/sys/kern/vfs_syscalls.c:2280 #13 0xffffffff8078acd1 in amd64_syscall (td=3D0xfffff803ec526960, traced=3D= 0) at subr_syscall.c:141 #14 0xffffffff8076fffb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #15 0x00000000425a731a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) frame 4 #4 0xffffffff80766e60 in vnode_create_vobject (vp=3D0xfffff816fe4f3588, isize=3D512, td=3D0xfffff803ec526960) at /usr/src/sys/vm/vnode_pager.c:120 120 VM_OBJECT_SLEEP(object, object, PDROP | PVM, "vodea= d", 0); (kgdb) p *vp $1 =3D {v_tag =3D 0xffffffff8085c651 "ufs", v_op =3D 0xffffffff80b0f950, v_= data =3D 0xfffff81488311160, v_mount =3D 0xfffff8019a27d000, v_nmntvnodes =3D {tqe_n= ext =3D 0xfffff816894dc3b0, tqe_prev =3D 0xfffff80b0d4991f8}, v_un =3D {vu_mount = =3D 0x0,=20 vu_socket =3D 0x0, vu_cdev =3D 0x0, vu_fifoinfo =3D 0x0}, v_hashlist = =3D {le_next =3D 0xfffff80824c363b0, le_prev =3D 0xfffff81742127b48}, v_cache_src =3D {lh_fi= rst =3D 0x0}, v_cache_dst =3D {tqh_first =3D 0xfffff80e5f6a5af0,=20 tqh_last =3D 0xfffff80e5f6a5b10}, v_cache_dd =3D 0xfffff80e5f6a5af0, v_= lock =3D {lock_object =3D {lo_name =3D 0xffffffff8085c651 "ufs", lo_flags =3D 117112= 832, lo_data =3D 0, lo_witness =3D 0x0}, lk_lock =3D 1, lk_exslpfail =3D 0, lk_t= imo =3D 51,=20 lk_pri =3D 96}, v_interlock =3D {lock_object =3D {lo_name =3D 0xfffffff= f80870687 "vnode interlock", lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0x0= }, mtx_lock =3D 4}, v_vnlock =3D 0xfffff816fe4f35f0, v_actfreelist =3D { tqe_next =3D 0xfffff80b0d4991d8, tqe_prev =3D 0xfffff816894dc470}, v_bu= fobj =3D {bo_lock =3D {lock_object =3D {lo_name =3D 0xffffffff80870697 "bufobj inter= lock", lo_flags =3D 86179840, lo_data =3D 0, lo_witness =3D 0x0}, rw_lock =3D 1},= =20 bo_ops =3D 0xffffffff80afc070, bo_object =3D 0xfffff802f8423d00, bo_syn= clist =3D {le_next =3D 0x0, le_prev =3D 0x0}, bo_private =3D 0xfffff816fe4f3588, __bo= _vnode =3D 0xfffff816fe4f3588, bo_clean =3D {bv_hd =3D {tqh_first =3D 0x0,=20 tqh_last =3D 0xfffff816fe4f36a8}, bv_root =3D {pt_root =3D 0}, bv_c= nt =3D 0}, bo_dirty =3D {bv_hd =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff816fe4f36c8= }, bv_root =3D {pt_root =3D 0}, bv_cnt =3D 0}, bo_numoutput =3D 0, bo_flag =3D 0, bo_bsize= =3D 32768},=20 v_pollinfo =3D 0x0, v_label =3D 0x0, v_lockf =3D 0x0, v_rl =3D {rl_waiter= s =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff816fe4f3710}, rl_currdep =3D 0x0}, = v_cstart =3D 0, v_lasta =3D 0, v_lastw =3D 0, v_clen =3D 0, v_holdcnt =3D 8, v_usecount = =3D 7,=20 v_iflag =3D 512, v_vflag =3D 0, v_writecount =3D 0, v_hash =3D 76467851, = v_type =3D VDIR} (kgdb) p *(vp->v_bufobj.bo_object) $2 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff8088aeb6 "vm objec= t", lo_flags =3D 90374144, lo_data =3D 0, lo_witness =3D 0x0}, rw_lock =3D 1}, object_li= st =3D {tqe_next =3D 0xfffff802f8423e00, tqe_prev =3D 0xfffff802f8423c20}, shadow_= head =3D { lh_first =3D 0x0}, shadow_list =3D {le_next =3D 0x0, le_prev =3D 0xfffff801744afa30}, memq =3D {tqh_first =3D 0xfffff8183ea423d0, tqh_last = =3D 0xfffff8183ea423e0}, rtree =3D {rt_root =3D 18446735381746688977, rt_flags = =3D 0 '\0'}, size =3D 1,=20 generation =3D 1, ref_count =3D 0, shadow_count =3D 0, memattr =3D 6 '\00= 6', type =3D 2 '\002', flags =3D 16392, pg_color =3D 1608, paging_in_progress =3D 0, resident_page_count =3D 1, backing_object =3D 0x0, backing_object_offset = =3D 0,=20 pager_object_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, rvq =3D {lh_f= irst =3D 0x0}, cache =3D {rt_root =3D 0, rt_flags =3D 0 '\0'}, handle =3D 0xfffff816fe4f35= 88, un_pager =3D {vnp =3D {vnp_size =3D 512, writemappings =3D 0}, devp =3D {devp_pglist= =3D { tqh_first =3D 0x200, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, = sgp =3D {sgp_pglist =3D {tqh_first =3D 0x200, tqh_last =3D 0x0}}, swp =3D {swp_tmpf= s =3D 0x200, swp_bcount =3D 0}}, cred =3D 0x0, charge =3D 0} (kgdb) p *td $3 =3D {td_lock =3D 0xffffffff80b9d4d0, td_proc =3D 0xfffff8025dd694f0, td_= plist =3D {tqe_next =3D 0xfffff801744ac4b0, tqe_prev =3D 0xfffff805374dd970}, td_runq= =3D {tqe_next =3D 0x0, tqe_prev =3D 0xffffffff80b728b0}, td_slpq =3D { tqe_next =3D 0xfffff80389d11960, tqe_prev =3D 0xfffff80223243e80}, td_l= ockq =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe00003849b8}, td_hash =3D {le_next = =3D 0x0, le_prev =3D 0xfffffe00007d7590}, td_cpuset =3D 0xfffff801100593a8,=20 td_sel =3D 0xfffff806ce36eb00, td_sleepqueue =3D 0x0, td_turnstile =3D 0xfffff80174d07e40, td_rlqe =3D 0xfffff8017477ba28, td_umtxq =3D 0xfffff803a5a44f00, td_tid =3D 102066, td_sigqueue =3D {sq_signals =3D {__b= its =3D {0, 0, 0, 0}}, sq_kill =3D { __bits =3D {0, 0, 0, 0}}, sq_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff803ec526a18}, sq_proc =3D 0xfffff8025dd694f0, sq_flags =3D 1}, td_lend_user_pri =3D 255 '=CB=99', td_flags =3D 4, td_inhibitors =3D 2, td_= pflags =3D 0, td_dupfd =3D 0,=20 td_sqqueue =3D 0, td_wchan =3D 0xfffff802f8423d00, td_wmesg =3D 0xfffffff= f8088be0a "vodead", td_lastcpu =3D 13 '\r', td_oncpu =3D 255 '=CB=99', td_owepreempt = =3D 0 '\0', td_tsqueue =3D 0 '\0', td_locks =3D 26721, td_rw_rlocks =3D 0, td_lk_slocks= =3D 0,=20 td_stopsched =3D 0, td_blocked =3D 0x0, td_lockname =3D 0x0, td_contested= =3D {lh_first =3D 0x0}, td_sleeplocks =3D 0x0, td_intr_nesting_level =3D 0, td_= pinned =3D 0, td_ucred =3D 0xfffff8017438e800, td_estcpu =3D 0, td_slptick =3D 1266348= 642,=20 td_blktick =3D 0, td_swvoltick =3D 1266348642, td_cow =3D 0, td_ru =3D {r= u_utime =3D {tv_sec =3D 0, tv_usec =3D 0}, ru_stime =3D {tv_sec =3D 0, tv_usec =3D 0}, = ru_maxrss =3D 7952, ru_ixrss =3D 21233268, ru_idrss =3D 1299996, ru_isrss =3D 13866624,=20 ru_minflt =3D 1, ru_majflt =3D 0, ru_nswap =3D 0, ru_inblock =3D 210560= 8, ru_oublock =3D 0, ru_msgsnd =3D 1531599, ru_msgrcv =3D 100, ru_nsignals =3D= 0, ru_nvcsw =3D 1685809, ru_nivcsw =3D 9633}, td_rux =3D {rux_runtime =3D 2226241352945= ,=20 rux_uticks =3D 46227, rux_sticks =3D 62106, rux_iticks =3D 0, rux_uu = =3D 365363300, rux_su =3D 490865795, rux_tu =3D 856229096}, td_incruntime =3D 0, td_runtim= e =3D 2226241352945, td_pticks =3D 0, td_sticks =3D 0, td_iticks =3D 0, td_uticks= =3D 0,=20 td_intrval =3D 0, td_oldsigmask =3D {__bits =3D {0, 0, 0, 0}}, td_generat= ion =3D 1695442, td_sigstk =3D {ss_sp =3D 0x0, ss_size =3D 0, ss_flags =3D 0}, td_x= sig =3D 0, td_profil_addr =3D 0, td_profil_ticks =3D 0,=20 td_name =3D "bacula-fd\000\000\000\000\000\000\000\000\000\000", td_fpop = =3D 0x0, td_dbgflags =3D 0, td_dbgksi =3D {ksi_link =3D {tqe_next =3D 0x0, tqe_prev = =3D 0x0}, ksi_info =3D {si_signo =3D 0, si_errno =3D 0, si_code =3D 0, si_pid =3D 0, = si_uid =3D 0,=20 si_status =3D 0, si_addr =3D 0x0, si_value =3D {sival_int =3D 0, siva= l_ptr =3D 0x0, sigval_int =3D 0, sigval_ptr =3D 0x0}, _reason =3D {_fault =3D {_trapno =3D= 0}, _timer =3D {_timerid =3D 0, _overrun =3D 0}, _mesgq =3D {_mqd =3D 0}, _poll =3D {_band= =3D 0},=20 __spare__ =3D {__spare1__ =3D 0, __spare2__ =3D {0, 0, 0, 0, 0, 0, = 0}}}}, ksi_flags =3D 0, ksi_sigq =3D 0x0}, td_ng_outbound =3D 0, td_osd =3D {osd_n= slots =3D 0, osd_slots =3D 0x0, osd_next =3D {le_next =3D 0x0, le_prev =3D 0x0}},=20 td_map_def_user =3D 0x0, td_dbg_forked =3D 0, td_vp_reserv =3D 0, td_no_s= leeping =3D 0, td_dom_rr_idx =3D 0, td_sigmask =3D {__bits =3D {0, 0, 0, 0}}, td_rqinde= x =3D 48 '0', td_base_pri =3D 152 '\230', td_priority =3D 152 '\230',=20 td_pri_class =3D 3 '\003', td_user_pri =3D 183 '=CB=87', td_base_user_pri= =3D 183 '=CB=87', td_pcb =3D 0xfffffe17f2e63b80, td_state =3D TDS_INHIBITED, td_retval =3D {0, 1084620448}, td_slpcallout =3D {c_links =3D {le =3D {le_next =3D 0x0,=20 le_prev =3D 0xfffffe0000ad9288}, sle =3D {sle_next =3D 0x0}, tqe = =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe0000ad9288}}, c_time =3D 5430875852315327, c_= precision =3D 26843, c_arg =3D 0xfffff803ec526960,=20 c_func =3D 0xffffffff805c1370 , c_lock =3D 0x0, c_flags= =3D 2, c_iflags =3D 272, c_cpu =3D 11}, td_frame =3D 0xfffffe17f2e63ac0, td_kstack= _obj =3D 0xfffff8076ec32a00, td_kstack =3D 18446741977545703424, td_kstack_pages =3D= 4,=20 td_critnest =3D 1, td_md =3D {md_spinlock_count =3D 1, md_saved_flags =3D= 582, md_spurflt_addr =3D 1131679744}, td_sched =3D 0xfffff803ec526de0, td_ar =3D= 0x0, td_lprof =3D {{lh_first =3D 0x0}, {lh_first =3D 0x0}}, td_dtrace =3D 0x0, t= d_errno =3D 0,=20 td_vnet =3D 0x0, td_vnet_lpush =3D 0x0, td_intr_frame =3D 0x0, td_rfppwai= t_p =3D 0xfffff80616f0a000, td_ma =3D 0x0, td_ma_cnt =3D 0, td_su =3D 0x0, td_dbg_s= c_code =3D 0, td_dbg_sc_narg =3D 0, td_emuldata =3D 0x0} (kgdb) p *object $4 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff8088aeb6 "vm objec= t", lo_flags =3D 90374144, lo_data =3D 0, lo_witness =3D 0x0}, rw_lock =3D 1}, object_li= st =3D {tqe_next =3D 0xfffff802f8423e00, tqe_prev =3D 0xfffff802f8423c20}, shadow_= head =3D { lh_first =3D 0x0}, shadow_list =3D {le_next =3D 0x0, le_prev =3D 0xfffff801744afa30}, memq =3D {tqh_first =3D 0xfffff8183ea423d0, tqh_last = =3D 0xfffff8183ea423e0}, rtree =3D {rt_root =3D 18446735381746688977, rt_flags = =3D 0 '\0'}, size =3D 1,=20 generation =3D 1, ref_count =3D 0, shadow_count =3D 0, memattr =3D 6 '\00= 6', type =3D 2 '\002', flags =3D 16392, pg_color =3D 1608, paging_in_progress =3D 0, resident_page_count =3D 1, backing_object =3D 0x0, backing_object_offset = =3D 0,=20 pager_object_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, rvq =3D {lh_f= irst =3D 0x0}, cache =3D {rt_root =3D 0, rt_flags =3D 0 '\0'}, handle =3D 0xfffff816fe4f35= 88, un_pager =3D {vnp =3D {vnp_size =3D 512, writemappings =3D 0}, devp =3D {devp_pglist= =3D { tqh_first =3D 0x200, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, = sgp =3D {sgp_pglist =3D {tqh_first =3D 0x200, tqh_last =3D 0x0}}, swp =3D {swp_tmpf= s =3D 0x200, swp_bcount =3D 0}}, cred =3D 0x0, charge =3D 0} --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Apr 1 14:35:14 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8837AAEB8ED for ; Fri, 1 Apr 2016 14:35:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 78DCC1174 for ; Fri, 1 Apr 2016 14:35:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u31EZBYk066318 for ; Fri, 1 Apr 2016 14:35:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Fri, 01 Apr 2016 14:35:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Apr 2016 14:35:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 --- Comment #20 from Konstantin Belousov --- (In reply to Rudolf =C4=8Cejka from comment #19) >From your report, I am almost certain (with ~95% level) that this issue, and the issue reported in PR204426 are the same. I would follow on that PR sin= ce the reporter seems to have low time to reproduce, and more involved debuggi= ng patches are already in flight there. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Apr 2 16:46:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BDDFB004AD for ; Sat, 2 Apr 2016 16:46:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F0C931D19 for ; Sat, 2 Apr 2016 16:46:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u32GkdEv061055 for ; Sat, 2 Apr 2016 16:46:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204705] zfs receive mounts received dataset even if -u is given Date: Sat, 02 Apr 2016 16:46:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Apr 2016 16:46:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204705 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |avg@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Apr 2 16:57:23 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC128B008C0 for ; Sat, 2 Apr 2016 16:57:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BCB6812A7 for ; Sat, 2 Apr 2016 16:57:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u32GvNxQ080994 for ; Sat, 2 Apr 2016 16:57:23 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204358] zfs loader zfs_probe_args secsz is too small, causing memory corruption Date: Sat, 02 Apr 2016 16:57:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: allanjude@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Apr 2016 16:57:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204358 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |allanjude@FreeBSD.org --- Comment #3 from Andriy Gapon --- Assign to the committer. --=20 You are receiving this mail because: You are the assignee for the bug.=