From owner-freebsd-fs@freebsd.org Sun Aug 26 17:42:03 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B26C8107CC81; Sun, 26 Aug 2018 17:42:03 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 29207897AD; Sun, 26 Aug 2018 17:42:02 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w7QHgEaA015390 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 26 Aug 2018 10:42:15 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w7QHgE57015389; Sun, 26 Aug 2018 10:42:14 -0700 (PDT) (envelope-from fbsd) Date: Sun, 26 Aug 2018 10:42:13 -0700 From: bob prohaska To: Ian Lepore Cc: Kirk McKusick , FreeBSD Filesystems , freebsd-arm@freebsd.org, bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems Message-ID: <20180826174213.GA15325@www.zefox.net> References: <20180824000637.GA2157@www.zefox.net> <201808240027.w7O0Rh7f062555@chez.mckusick.com> <20180824161443.GB2157@www.zefox.net> <1535127958.1488.31.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1535127958.1488.31.camel@freebsd.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2018 17:42:03 -0000 On Fri, Aug 24, 2018 at 10:25:58AM -0600, Ian Lepore wrote: > A usb sdcard reader/writer will definitely mask the trim capability of > an sdcard. Aside from that, ALL sdcards support trim on freebsd, > regardless of what any support folks at sandisk might tell you. The > trim operation is supported in the mmcsd driver by issuing sd erase > commands (CMD32/33/38 sequence) which has been in the sd spec from the > beginning. > > That said, it's my experience that different cards implement CMD38 > different ways. Some cards treat it like a TRIM -- it's a hint that > tells the card "the data in these sectors need not be preserved during > future updates to the erase block they're embedded in", and thus they > happen very fast and improve efficiency. Other cards treat it as a more > synchronous operation, physically erasing the blocks involved while- > you-wait, which can be *horrible* for performance on ufs (although > potentially it might get better with the consolidation of BIO_DELETEs). This raises a practical question: How does throughput to microSD compare to throughput to USB, assuming the flash controllers are well-executed? Sizewise there's no need for two devices, a single microSD has plenty of space for my purposes. I thought that having a separate device for swap and maybe /usr would speed things up, but perhaps that's quaint notion leftover from st506 days. Thanks for reading! bob prohaska From owner-freebsd-fs@freebsd.org Sun Aug 26 18:03:21 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8646107D53A for ; Sun, 26 Aug 2018 18:03:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 621888A16E for ; Sun, 26 Aug 2018 18:03:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 27114107D539; Sun, 26 Aug 2018 18:03:21 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13290107D538 for ; Sun, 26 Aug 2018 18:03:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D0398A16D for ; Sun, 26 Aug 2018 18:03:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id EB2F7A7CB for ; Sun, 26 Aug 2018 18:03:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7QI3Jix025664 for ; Sun, 26 Aug 2018 18:03:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7QI3JGb025663 for fs@FreeBSD.org; Sun, 26 Aug 2018 18:03:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Sun, 26 Aug 2018 18:03:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mi@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2018 18:03:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 Mikhail Teterin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mi@FreeBSD.org --- Comment #6 from Mikhail Teterin --- Seeing this manifested pretty bad today - on a freshly-rebuilt world (r338291M). The other thing that changed is the RAM upgrade -- from 10GB to 40GB. Could it be, the kernel is trying to flush ALL filesystems, rather than just the one being remounted? Because, unlike the / on UFS, other filesystems co= uld be quite busy... While waiting, I press Ctrl-T once in a while, and get: root@narawntapu:/etc/mail (1022) mount -oro -u / load: 0.93 cmd: mount 26321 [biowr] 230.01r 0.00u 9.01s 2% 2224k load: 0.47 cmd: mount 26321 [biowr] 283.84r 0.00u 10.99s 1% 2224k load: 0.57 cmd: mount 26321 [biowr] 318.95r 0.00u 12.25s 2% 2224k The actual change to the /-filesystem is trivial. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 26 18:52:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B81A107EE1F for ; Sun, 26 Aug 2018 18:52:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E7C7E8CDE8 for ; Sun, 26 Aug 2018 18:52:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id ACD68107EE1E; Sun, 26 Aug 2018 18:52:45 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9BA7D107EE1D for ; Sun, 26 Aug 2018 18:52:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3DE748CDE5 for ; Sun, 26 Aug 2018 18:52:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 58BC7AED7 for ; Sun, 26 Aug 2018 18:52:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7QIqiIl074313 for ; Sun, 26 Aug 2018 18:52:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7QIqinO074312 for fs@FreeBSD.org; Sun, 26 Aug 2018 18:52:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Sun, 26 Aug 2018 18:52:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mi@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2018 18:52:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 --- Comment #7 from Mikhail Teterin --- (In reply to Kirk McKusick from comment #3) > include the -o noatime option No obvious difference: root@narawntapu:/etc/mail (1024) mount -orw,noatime / root@narawntapu:/etc/mail (1025) touch q root@narawntapu:/etc/mail (1026) rm q root@narawntapu:/etc/mail (1027) mount -oro -u / load: 0.71 cmd: mount 27675 [biowr] 7.92r 0.00u 0.72s 4% 2224k load: 0.52 cmd: mount 27675 [biowr] 54.83r 0.00u 2.64s 2% 2224k load: 0.47 cmd: mount 27675 [biowr] 68.05r 0.00u 3.12s 2% 2224k load: 0.58 cmd: mount 27675 [biowr] 121.83r 0.00u 5.08s 3% 2224k load: 0.36 cmd: mount 27675 [biowr] 254.59r 0.00u 9.97s 2% 2224k ... > time -l root@narawntapu:/etc/mail (1034) /usr/bin/time -l mount -oro / 330,18 real 0,00 user 13,01 sys 2212 maximum resident set size 22 average shared memory size 7 average unshared data size 121 average unshared stack size 112 page reclaims 0 page faults 0 swaps 0 block input operations 344120 block output operations 0 messages sent 0 messages received 0 signals received 344178 voluntary context switches 1014 involuntary context switches The entire FS is: Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/ada0s1a 22808 16268 4715 78% / *what* 344120 "block output operations"?! --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 26 20:03:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C99BE1080B49 for ; Sun, 26 Aug 2018 20:03:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 57BF98F431 for ; Sun, 26 Aug 2018 20:03:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1CCE11080B48; Sun, 26 Aug 2018 20:03:59 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B8D11080B47 for ; Sun, 26 Aug 2018 20:03:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A23828F42D for ; Sun, 26 Aug 2018 20:03:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id ADC7BB89A for ; Sun, 26 Aug 2018 20:03:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7QK3vSM061232 for ; Sun, 26 Aug 2018 20:03:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7QK3vER061231 for fs@FreeBSD.org; Sun, 26 Aug 2018 20:03:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Sun, 26 Aug 2018 20:03:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2018 20:04:00 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cem@freebsd.org --- Comment #8 from Conrad Meyer --- (In reply to Mikhail Teterin from comment #7) Perhaps more interestingly, > 344178 voluntary context switches Almost every single one of those bios is written synchronously, it looks li= ke. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 26 20:05:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 381451080BE7 for ; Sun, 26 Aug 2018 20:05:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C2F828F4E2 for ; Sun, 26 Aug 2018 20:05:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 848831080BE5; Sun, 26 Aug 2018 20:05:48 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 734211080BE4 for ; Sun, 26 Aug 2018 20:05:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 148B08F4DD for ; Sun, 26 Aug 2018 20:05:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 6D02BB89F for ; Sun, 26 Aug 2018 20:05:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7QK5lME062963 for ; Sun, 26 Aug 2018 20:05:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7QK5lE3062962 for fs@FreeBSD.org; Sun, 26 Aug 2018 20:05:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Sun, 26 Aug 2018 20:05:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2018 20:05:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 --- Comment #9 from Conrad Meyer --- How many inodes does the filesystem have? Does mount -oro,noatime -u (in addition to rw,noatime) help in any way? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 26 21:01:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 925C6108244E for ; Sun, 26 Aug 2018 21:01:46 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 31F907126D for ; Sun, 26 Aug 2018 21:01:46 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id EB2981082441; Sun, 26 Aug 2018 21:01:45 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C95BE1082440 for ; Sun, 26 Aug 2018 21:01:45 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5264C71267 for ; Sun, 26 Aug 2018 21:01:45 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 87674C28D for ; Sun, 26 Aug 2018 21:01:44 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7QL1imU029235 for ; Sun, 26 Aug 2018 21:01:44 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7QL1iKH029222 for fs@FreeBSD.org; Sun, 26 Aug 2018 21:01:44 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201808262101.w7QL1iKH029222@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 26 Aug 2018 21:01:44 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2018 21:01:46 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off New | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Aug 27 05:08:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 760F9108C731; Mon, 27 Aug 2018 05:08:02 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qt0-x22c.google.com (mail-qt0-x22c.google.com [IPv6:2607:f8b0:400d:c0d::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D9A77E0E0; Mon, 27 Aug 2018 05:08:01 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qt0-x22c.google.com with SMTP id o15-v6so16641276qtk.6; Sun, 26 Aug 2018 22:08:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3P3z6fO2RrnrhjXgZp+XpBBEOudswwkpJn9jxe4jW6E=; b=TOO6n5xQLNfgKGH0Ct+gaRC3hEZgnsPhXu96sx/97WA9939HNlCdPddXv5vftvdKBP HS1hzC/tW4aGR9RgnPQwyePcrplFCzX1sJclao2LGRb2rqDuBOws+XPbFsvmYeyk3j6C UwWmTbDp6mHUmahru8Cjm3awRx1f2DsvfKEQ8EZ5j0ySZ8WHyHPNivZXHZaqEqfPBzul tTJZ4ZWUGCNfgVTHVizLOrj/3GktVnT6K6VeWypsX0ihXK/nZ2Qr4Jg0Ck+35JAEcdQ/ AYesFdKDf6kdynKETuJlsbWV/jfzLSFS0IliCEH7XttKAgtPjt3tcQ9Me0gx9Rn8Q+sK WM4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3P3z6fO2RrnrhjXgZp+XpBBEOudswwkpJn9jxe4jW6E=; b=hBIbeXdqh/xps05k2zBMYN5PSs+B+HdUdQGARAPiaTlCrviK6subVpRn8Wzk1m30Nj QsHZf5Aiqn/0Q6B67eJTDJzxq2paLwqHwoSuGzK/S7qN8IYzjB62qzjTsj0AemFMq6Hi +yxnGIkIyRDXem2LZCRJkERfhp4Ob6ZSzxvIAAZd8yp+oqEQS/tTDeK2SIQJUQA02+7k 136SkJQ13KXhK5EIeP679XysuKwg+1bJrSAXjc8W1Vy4M0rVbv9NE79tTIMlICSm8GnW qEJJGV8s8A0EhsGRpSNKoORVGxhbJF8Y4NK8z+VrvuWfHUoPLvfcxPQsVgBcF3jkoUpu Og8Q== X-Gm-Message-State: APzg51AtLH/OvZnjjc1q63B8NxDTT1vVk5RRi6cppTPcEOtHxXT1k9Gu InvIL9mOCK+yoEPLVNUSJ6BnKzYS3ulAnpycqr0= X-Google-Smtp-Source: ANB0VdYKgSHIHuzKzCagTkdi6Cwa3j3T5JqZq0l3LM7m9sah6RZ7oBwclLp0n2/75yOlwGH6juCO9ui2YngvfBgCjFI= X-Received: by 2002:a0c:869b:: with SMTP id 27-v6mr11972513qvf.92.1535346481417; Sun, 26 Aug 2018 22:08:01 -0700 (PDT) MIME-Version: 1.0 References: <114c2000-17ba-3cf6-1d4a-1354f42d8875@gmail.com> In-Reply-To: <114c2000-17ba-3cf6-1d4a-1354f42d8875@gmail.com> From: Sami Halabi Date: Mon, 27 Aug 2018 08:07:45 +0300 Message-ID: Subject: Re: Bhyve guest boot error, Buffer I/O error on dev vda1, logical block 0, lost async page write To: Rajil Saraswat , freebsd-fs@freebsd.org Cc: FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 05:08:02 -0000 Hi, IIRC pool shouldn't be above 90% used otherwise bad things happen.. but at 69%???? Can someone explain thid please? Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=91=D7=B3, 27 = =D7=91=D7=90=D7=95=D7=92=D7=B3 2018, 03:24, =D7=9E=D7=90=D7=AA Rajil Sarasw= at =E2=80=8F: > On 08/26/2018 06:06 PM, Rajil Saraswat wrote: > > Hello, > > > > My bhyve guests are failing to boot. I upgraded from 11.1 to 11.2 > > recently, not sure if that is the cause. > > Any idea what could be the issue? > > > Figured out this one. Seems my pool had reached a capacity of 69%. After > deleting unnecessary zvols, and reducing the pool to 52% the guest > booted up fine. > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Aug 27 13:47:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A021108A667 for ; Mon, 27 Aug 2018 13:47:51 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D24F70CD9 for ; Mon, 27 Aug 2018 13:47:51 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [212.48.125.108] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fuHrp-000F1v-It; Mon, 27 Aug 2018 15:47:49 +0200 Date: Mon, 27 Aug 2018 15:47:27 +0200 From: Yamagi Burmeister To: freebsd-fs@freebsd.org Cc: MacCrest@gmail.com Subject: ZFS (ARC) performance regression in r321610 Message-Id: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Mon__27_Aug_2018_15_47_28_+0200_hPWCv9E4_UHjaX1S" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 13:47:51 -0000 --Signature=_Mon__27_Aug_2018_15_47_28_+0200_hPWCv9E4_UHjaX1S Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, after upgrading our database server from FreeBSD 11.1 to 11.2 we're seeing a major performance regression. On FreeBSD 11.1 everything was fine, both the master and the replication slaves were able to scope with the workload. On FreeBSD 11.2 the master is rather slow and the single threaded replication slaves are falling behind. I managed to track this down to ZFS, especially the ARC seems to be the culprit. The hardware: * 2x Intel Xeon E5-2643v3 with latest microcode. * 128GB RAM * 8x Intel DC 3700 800GB SSD connected to one LSI / Avago / Broadcom=20 SAS9300-8i HBA running the latest firmware (version 16.00.01.00) The software: * FreeBSD/amd64 11.1 and 11.2 with GENERIC kernel * ZFS configured as RAIDZ1 over 6 SSDs, one SSD as spare. * The system itself is running on UFS. * MySQL 5.1 (don't ask, the application needs such an old version) Configuration: * vfs.zfs.arc_max=3D"16G" (The memory is better used by MySQL) * vfs.zfs.compressed_arc_enabled=3D"0" * lz4 compression is enabled on the dataset holding the MySQL data. * Powermanagement (C-States, powerd) ist turned off. The workload is somewhat special. There're hundereds of tables in the database, but >95% of all workload is on only one ~18GB MyISAM table. As long as _only_ this big table get's all the load everythings okay. As soon as MySQL iterates over another big table ("select *" for example) the performance degrades and never recovers. While analyzing the problem I observed that reading over another table seems to "poison" the ARC. While the zfs-stats script insists on a >99% hit rate, the ARCs performance decreases by about 30% as soon as it was "poisoned". And even more strange: A very small change to the ARCs configuration recovers the performance. For example it's enough to lower or increase vfs.zfs.arc_max by one byte. I was able to come up with a test case. The C code can be found inline or here: https://gist.github.com/Yamagi/992733b5eed8d1b7e209b72e42d78509 To run this programm you'll need a box with a ZFS pool and two files on it. Both files must be bigger than the maximum ARC size. For example if vfs.zfs.arc_max=3D"2G" the files should be at least 2GB, better 4GB. All tests should be done after a reboot, to be sure that the ARC is empty. The program does what's decripted above, in more detail: 1. At a first step it does 1.000.000 random reads over file A. This should (and does) show normal cache behavior, the reads are getting faster as the ARC fills up. 2. As second step file B is read from start to end. That "poisons" the ARC. 3. The third step does again 1.000.000 reads over file A. This should start slow (because data from file B is in the ARC) and get faster over time. But on FreeBSD 11.2 it doesn't get any faster, the performance stays degraded. Like the ARC is never hit. 4. The fourth step lowers vfs.zfs.arc_max by one byte.=20 5. And as a final step another 1.000.000 random reads from file A. On FreeBSD 11.2 the performance is now good, just like it was in step 1. With this program I was able to bisect the source and identify commit r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culprit: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 Of course the big question is: Is this really a regression or have I just the found the one workload were a generally good and useful change is a pessimation? And why is a small change to vfs.zfs.arc_max enough to restore performance? Anyway, running this every 15 minutes through cron just to keep our database servers at good performance seems very, very wrong: #!/bin/sh CURRENT=3D$(sysctl -n vfs.zfs.arc_max) NORMAL=3D"17179869184" if [ $CURRENT -eq $NORMAL ] ; then sysctl vfs.zfs.arc_max=3D$(($NORMAL-1))>/dev/null 2>&1 else sysctl vfs.zfs.arc_max=3D$NORMAL >/dev/null 2>&1 fi Regards, Yamagi -------- These measurements were taken on a cheap desktop box with Intel Core i5 i5-4430, 24GB RAM and a Samsung 840 EVO 120GB SSD. The effect is not as pronounced as on the much faster production servers but still easily observable. r321609 (good): The performance hit in step 3 is about 12%. In step 5 the performance is fully restored. 1. Doing 1000000 random reads from /mnt/file_a: - 100000: 1960 Reads per second - 200000: 2020 Reads per second - 300000: 2040 Reads per second - 400000: 2040 Reads per second - 500000: 2049 Reads per second - 600000: 2054 Reads per second - 700000: 2058 Reads per second - 800000: 2056 Reads per second - 900000: 2059 Reads per second - 1000000: 2061 Reads per second 2. Reading /mnt/file_b from start to end. 3. Doing 1000000 random reads from /mnt/file_a: - 100000: 1724 Reads per second - 200000: 1739 Reads per second - 300000: 1754 Reads per second - 400000: 1754 Reads per second - 500000: 1760 Reads per second - 600000: 1764 Reads per second - 700000: 1767 Reads per second - 800000: 1769 Reads per second - 900000: 1768 Reads per second - 1000000: 1769 Reads per second 4. Shrinking the ARC by one byte. 5. Doing 1000000 random reads from /mnt/file_a: - 100000: 2040 Reads per second - 200000: 2061 Reads per second - 300000: 2054 Reads per second - 400000: 2061 Reads per second - 500000: 2066 Reads per second - 600000: 2061 Reads per second - 700000: 2064 Reads per second - 800000: 2061 Reads per second - 900000: 2064 Reads per second - 1000000: 2066 Reads per second =20 r321610 (bad): Reads are generally much slower than on r321609 (good). The performance hit in step 3 is about 27%. In step 5 the performance get's fully restored. 1. Doing 1000000 random reads from /mnt/file_a: - 100000: 1754 Reads per second - 200000: 1785 Reads per second - 300000: 1807 Reads per second - 400000: 1809 Reads per second - 500000: 1818 Reads per second - 600000: 1818 Reads per second - 700000: 1822 Reads per second - 800000: 1826 Reads per second - 900000: 1825 Reads per second - 1000000: 1828 Reads per second 2. Reading /mnt/file_b from start to end. 3. Doing 1000000 random reads from /mnt/file_a: - 100000: 1265 Reads per second - 200000: 1290 Reads per second - 300000: 1293 Reads per second - 400000: 1298 Reads per second - 500000: 1298 Reads per second - 600000: 1298 Reads per second - 700000: 1301 Reads per second - 800000: 1300 Reads per second - 900000: 1302 Reads per second - 1000000: 1302 Reads per second 4. Shrinking the ARC by one byte. 5. Doing 1000000 random reads from /mnt/file_a: - 100000: 1694 Reads per second - 200000: 1694 Reads per second - 300000: 1704 Reads per second - 400000: 1709 Reads per second - 500000: 1706 Reads per second - 600000: 1709 Reads per second - 700000: 1707 Reads per second - 800000: 1709 Reads per second - 900000: 1707 Reads per second - 1000000: 1709 Reads per second=20 -------- // clang -g -Wall -O2 -o zfsarc zfsarc.c #include #include #include #include #include #include #include #include #include #define ITERATIONS 1000000 void randreads(const char *file, struct stat *sb) { FILE *fd; char buffer[256 * 1024]; struct timeval start; gettimeofday(&start, NULL); if ((fd =3D fopen(file, "r")) =3D=3D NULL) { fprintf(stderr, "Couldn't open %s: %s\n", file, strerror(errno)); exit(1); } for (int i =3D 1; i <=3D ITERATIONS; i++) { uint32_t bytestoread =3D arc4random_uniform(sizeof(buffer)); uint32_t position =3D arc4random_uniform(sb->st_size - sizeof(buffer)); bytestoread =3D bytestoread ? bytestoread : 1; if (fseek(fd, position, SEEK_SET) !=3D 0) { fprintf(stderr, "Could't seek to %i in %s: %s\n", position, file, strerr= or(errno)); fclose(fd); exit(1); } if (fread(buffer, bytestoread, 1, fd) !=3D 1) { fprintf(stderr, "Couldn't read %i bytes from %s: %s\n", bytestoread, fil= e, strerror(errno)); fclose(fd); exit(1); } if (!(i % 100000)) { struct timeval now; gettimeofday(&now, NULL); printf(" - %i: %li Reads per second\n", i, i / (now.tv_sec - st= art.tv_sec));=20 } } fclose(fd); } void fullread(const char *file, struct stat *sb) { FILE *fd; char buffer[256 * 1024]; const uint32_t iterations =3D sb->st_size / sizeof(buffer); if ((fd =3D fopen(file, "r")) =3D=3D NULL) { fprintf(stderr, "Couldn't open %s: %s\n", file, strerror(errno)); exit(1); } for (int i =3D 1; i <=3D iterations; i++) { uint32_t bytestoread =3D (i =3D=3D iterations) ? sb->st_size % sizeof(buf= fer) : sizeof(buffer); if (bytestoread > 0)=20 { if (fread(buffer, bytestoread, 1, fd) !=3D 1) { fprintf(stderr, "Couldn't read %i bytes from %s: %s\n", bytestoread, fi= le, strerror(errno)); fclose(fd); exit(1); } } } fclose(fd); } int main(int argc, char *argv[]) { // No stdout buffering. setbuf(stdout, NULL); // Get arguments. if (argc !=3D 3) { fprintf(stderr, "Usage: ./test file_a file_b\n"); exit(1); } const char *file_a =3D argv[1]; const char *file_b =3D argv[2]; =09 // Check if the files are large enough. // We need at least the size of the ARC. uint64_t arcmax; size_t arglen =3D sizeof(arcmax); if (sysctlbyname("vfs.zfs.arc_max", &arcmax, &arglen, NULL, 0) !=3D 0) { fprintf(stderr, "Couldn't get vfs.zfs.arc_max: %s\n", strerror(errno)); exit(1); } =09 struct stat sb_a; if (stat(file_a, &sb_a) !=3D 0) { fprintf(stderr, "Couldn't stat %s: %s\n", file_a, strerror(errno)); exit(1); } if (!S_ISREG(sb_a.st_mode)) { fprintf(stderr, "%s is not a file.\n", file_a); exit(1); } if (sb_a.st_size < arcmax) { fprintf(stderr, "%s is to small: Must be at least %lu bytes\n", file_a, a= rcmax); exit(1); } struct stat sb_b; if (stat(file_b, &sb_b) !=3D 0) { fprintf(stderr, "Couldn't stat %s: %s\n", file_b, strerror(errno)); exit(1); } if (!S_ISREG(sb_b.st_mode)) { fprintf(stderr, "%s is not a file.\n", file_b); exit(1); } if (sb_b.st_size < arcmax) { fprintf(stderr, "%s is to small: Must be at least %lu bytes\n", file_b, a= rcmax); exit(1); } // Do ITERATIONS random reads on file A. This will fill up the ARC. // In theory this should start slow and get faster as the ARC fills // up. printf("1. Doing %i random reads from %s:\n", ITERATIONS, file_a); randreads(file_a, &sb_a); // Poison the ARC by reading once through file B. This will write // contents from file B into the ARC, replacing contents from file // A. printf("2. Reading %s from start to end.\n", file_b); fullread(file_b, &sb_b); // Do ITERATIONS random reads on file A. This should start slow and // get faster as contents of file B is replaced by contents of file // A. This can be seen in 11.1-RELEASE, but not in 11.2-RELEASE. printf("3. Doing %i random reads from %s:\n", ITERATIONS, file_a); randreads(file_a, &sb_a); // Shrink the ARC by 1 byte to alter it's configuration. printf("4. Shrinking the ARC by one byte.\n"); arcmax--; if (sysctlbyname("vfs.zfs.arc_max", NULL, NULL, &arcmax, arglen) !=3D 0) { fprintf(stderr, "Couldn't set vfs.zfs.arc_max: %s\n", strerror(errno)); exit(1); } // Do ITERATIONS random reads on file A. In 11.2-RELEASE this will show // good performance, again. printf("5. Doing %i random reads from %s:\n", ITERATIONS, file_a); randreads(file_a, &sb_a); return 0; } --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Mon__27_Aug_2018_15_47_28_+0200_hPWCv9E4_UHjaX1S Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluEAPAACgkQ6xRy5x1Q JRW3QBAAupdIEY1Vu7TKz8R95VLCH8Qewu7/UterkJidYZEWCKt45PUpHf8/y7QR NHnOYILIfohB2r7Wcu3D8Z8s9bCASaG1f+RI5ewHm/Zm+rQTBZmAxg7/VmoPB3na VnS6iuPphNZqx7iSEvqdVGutys0pXV3slSbnOOx/+O/GF8pYDx7tmSYHozHcMjIu TkZxmUuVff+zSS5sUL1MO5m1plU+jPMuGkCfX48BkaeVrYSoT+8T5KxPz1xdbG74 eL6gawp/eaQSn+3TXZ6J6AbY/+kPhyV7PzcAdxIhYVyfGDb5uRq5zr1angpB7t+9 0NzTJfVcpuytSIsCpKILDF2A0K6j/qw03ekj7uk3jUUvnNx0idYw+rR++fb8ZaAd /RocNaG3HJY6QmQ+ZhDJmIQ+gs+KpdoDP6YmXqJQMkLXgj2Tnv/Q5863fGAHw41q wPaC6tsMydkSt+DGUCgl8Wqol6VuJhJrSSN3bmortreBomsN94Os31U59LvpTIyj SxPIofZlj1iz7B7H9lQbCiFCG9ff/mD6K0P8QO/RNhrawmBdXFJaK/P3FWG4CvHi KkjRV+HoTvJgDht1a61Qfil5VKIhY9wBn58ZYnwElEPwctj5jXiZkEmu1EiQJeOi ElGAJEJMUEkvwuBGK6KF85jDbQnPC7CjA7zXebcHTX4cfqcXidM= =70AT -----END PGP SIGNATURE----- --Signature=_Mon__27_Aug_2018_15_47_28_+0200_hPWCv9E4_UHjaX1S-- From owner-freebsd-fs@freebsd.org Mon Aug 27 14:19:18 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B788A108B619 for ; Mon, 27 Aug 2018 14:19:18 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4446B72591 for ; Mon, 27 Aug 2018 14:19:18 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 53632F01; Mon, 27 Aug 2018 17:19:11 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: ZFS (ARC) performance regression in r321610 To: Yamagi Burmeister , freebsd-fs@freebsd.org Cc: MacCrest@gmail.com References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> Date: Mon, 27 Aug 2018 17:19:10 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 14:19:18 -0000 On 27.08.2018 16:47, Yamagi Burmeister wrote: > With this program I was able to bisect the source and identify commit > r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culprit: > https://svnweb.freebsd.org/base?view=revision&revision=321610 You could try to set "zfs_abd_scatter_enabled = 0" via kgdb and repeat tests to be sure, that this code is a problem. BTW, it is not clear, that FreeBSD needs this at all. -- // Lev Serebryakov From owner-freebsd-fs@freebsd.org Mon Aug 27 15:21:23 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2227B108DAAE for ; Mon, 27 Aug 2018 15:21:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B288E755D7 for ; Mon, 27 Aug 2018 15:21:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 78271108DAAB; Mon, 27 Aug 2018 15:21:22 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 671D0108DAAA for ; Mon, 27 Aug 2018 15:21:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 081C0755CE for ; Mon, 27 Aug 2018 15:21:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5722815E6B for ; Mon, 27 Aug 2018 15:21:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RFLLok019790 for ; Mon, 27 Aug 2018 15:21:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RFLLVX019789 for fs@FreeBSD.org; Mon, 27 Aug 2018 15:21:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Mon, 27 Aug 2018 15:21:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 15:21:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: mckusick Date: Mon Aug 27 15:20:42 UTC 2018 New revision: 338340 URL: https://svnweb.freebsd.org/changeset/base/338340 Log: When doing a -S "safe copy", the install command should do an fsync(2) system call after copying the installed file to ensure that it is on stable storage. PR: 230851 Reviewed by: kib Approved by: re (marius) Changes: head/usr.bin/xinstall/xinstall.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 16:22:21 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79A2C108F6E9 for ; Mon, 27 Aug 2018 16:22:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1604F77F28 for ; Mon, 27 Aug 2018 16:22:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CFCEF108F6E8; Mon, 27 Aug 2018 16:22:20 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEB3E108F6E7 for ; Mon, 27 Aug 2018 16:22:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5FDDA77F24 for ; Mon, 27 Aug 2018 16:22:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 963D5166FF for ; Mon, 27 Aug 2018 16:22:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RGMJIw064791 for ; Mon, 27 Aug 2018 16:22:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RGMJfK064790 for fs@FreeBSD.org; Mon, 27 Aug 2018 16:22:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Mon, 27 Aug 2018 16:22:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 16:22:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org CC| |rmacklem@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 16:27:44 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A7C9108F882 for ; Mon, 27 Aug 2018 16:27:44 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1FD0B781E1; Mon, 27 Aug 2018 16:27:44 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [2001:470:6845:1:52d4:a912:70e6:df0e] (helo=yuuta.localdomain) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fuKMZ-000FN1-5x; Mon, 27 Aug 2018 18:27:42 +0200 Date: Mon, 27 Aug 2018 18:27:23 +0200 From: Yamagi Burmeister To: lev@FreeBSD.org Cc: freebsd-fs@freebsd.org, MacCrest@gmail.com Subject: Re: ZFS (ARC) performance regression in r321610 Message-Id: <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> In-Reply-To: <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Mon__27_Aug_2018_18_27_23_+0200_C52TiSw_Mx8WrCMb" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 16:27:44 -0000 --Signature=_Mon__27_Aug_2018_18_27_23_+0200_C52TiSw_Mx8WrCMb Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, 27 Aug 2018 17:19:10 +0300 Lev Serebryakov wrote: > On 27.08.2018 16:47, Yamagi Burmeister wrote: >=20 > > With this program I was able to bisect the source and identify commit > > r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culprit: > > https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 > You could try to set "zfs_abd_scatter_enabled =3D 0" via kgdb and repeat > tests to be sure, that this code is a problem. With my testcase this looks very, very good. I'll give it a try in production and report back. That'll take at least 12 hours to be sure that it really works. --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Mon__27_Aug_2018_18_27_23_+0200_C52TiSw_Mx8WrCMb Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluEJmsACgkQ6xRy5x1Q JRXXzg/+K9zUlMg95NoJKxqVk0u08YH7X8Kcw7fvNnLEQNrDvX9BeLjp3I9en0Ar oR6brCV2dId6VshF/ytWOXf7dmNHLlgrWK2DWy1GOk8uQKja/Wg9srWOWXT6PzKO fzdF7y/NQxoDlnd1SJs1LVe3tNZMxJTsB5M1ZMEkTdpnRRuwjgeVDHCC0cNLnwCM En+/KljkYvXTlDGeeZm3dNRCijZUmoSpeJwTzDT4gqRlMwa9kpmqogpqriS2adCd ZhOxsYLCcc9cBfgBD4zDeCdLbJpRYFvuEsZV6+Jvyi75a+gDw69CFLX3gJiTohcA Ij7bHTN3RH4CXrkhF2nBqIN8/PM1HtGLGUmovWH03yf7FeHOVi6Yqt6ytNHxCq8O oCv8QmSwnz1QJEZ5FtQgQ0uLV0jRlBMD29l2rWNu9zh3UsKpWKbleXqJJDAamdsq qzkMrDTZZQFm9yAO4EGnEFPe5t9HrbLBDuDpGuM10WxKN3ffLeoRgaUWAmfWObQT tMiIZqP216b8jOBxH9yPDywnJdPiFZQ6QCgGQ1Vv01dkmuTIWukRcu28yVNQKeew yJg0PWuja8r9W4ffROas2OfKoOLl93Vk5/IIKOWqeMa0TX50hpnn4HP+2QN3jBAB Iwwi3b78ekmYQB0N/XC9B/qxZlflVRc370rHFbNrfb/yEDHZEjc= =jzs8 -----END PGP SIGNATURE----- --Signature=_Mon__27_Aug_2018_18_27_23_+0200_C52TiSw_Mx8WrCMb-- From owner-freebsd-fs@freebsd.org Mon Aug 27 16:50:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4CFE510901B9; Mon, 27 Aug 2018 16:50:28 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from vps-mail.nomadlogic.org (mail.nomadlogic.org [IPv6:2607:f2f8:a098::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C4E479744; Mon, 27 Aug 2018 16:50:27 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.106] (cpe-76-175-75-27.socal.res.rr.com [76.175.75.27]) by vps-mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id dced2b94 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Mon, 27 Aug 2018 09:50:25 -0700 (PDT) Subject: Re: Bhyve guest boot error, Buffer I/O error on dev vda1, logical block 0, lost async page write To: Sami Halabi , Rajil Saraswat , freebsd-fs@freebsd.org Cc: FreeBSD virtualization References: <114c2000-17ba-3cf6-1d4a-1354f42d8875@gmail.com> From: Pete Wright Message-ID: Date: Mon, 27 Aug 2018 09:50:23 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 16:50:28 -0000 On 8/26/18 10:07 PM, Sami Halabi wrote: > Hi, > IIRC pool shouldn't be above 90% used otherwise bad things happen.. but at > 69%???? > Can someone explain thid please? it's possible the filesystem was heavily fragmented?  seems suspicious - did scrubbing your pool find any errors? -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-fs@freebsd.org Mon Aug 27 20:39:50 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E42A109591C for ; Mon, 27 Aug 2018 20:39:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id AFF4F826C4 for ; Mon, 27 Aug 2018 20:39:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 70D55109591B; Mon, 27 Aug 2018 20:39:49 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F6E0109591A for ; Mon, 27 Aug 2018 20:39:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0168C826C3 for ; Mon, 27 Aug 2018 20:39:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4F6EA18A49 for ; Mon, 27 Aug 2018 20:39:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RKdmJC013490 for ; Mon, 27 Aug 2018 20:39:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RKdm8E013489 for fs@FreeBSD.org; Mon, 27 Aug 2018 20:39:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Mon, 27 Aug 2018 20:39:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 20:39:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 --- Comment #1 from Rick Macklem --- Patch looks fine to me. Unfortunately the timing is bad to commit it to head/current. It is in code freeze and I'll won't be able to do any commits for at least 2 weeks starting Wednesday (moving across Canada to BC). Maybe someone else could ask re@ for permission to commit it? If not, I will do so when I can. Thanks for submitting the patch, rick --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 20:43:50 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B6891095AF7 for ; Mon, 27 Aug 2018 20:43:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A87E082A08 for ; Mon, 27 Aug 2018 20:43:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 696BA1095AF6; Mon, 27 Aug 2018 20:43:49 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 581F61095AF5 for ; Mon, 27 Aug 2018 20:43:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8F9F82A05 for ; Mon, 27 Aug 2018 20:43:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 2792518B9E for ; Mon, 27 Aug 2018 20:43:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RKhm8u025709 for ; Mon, 27 Aug 2018 20:43:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RKhmr8025708 for fs@FreeBSD.org; Mon, 27 Aug 2018 20:43:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Mon, 27 Aug 2018 20:43:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 20:43:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |markj@FreeBSD.org --- Comment #2 from Mark Johnston --- (In reply to Rick Macklem from comment #1) I'd be happy to get it in head/ provided that you're not expecting any extra testing from me. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 21:51:03 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F6441097307 for ; Mon, 27 Aug 2018 21:51:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BF44B84B1D for ; Mon, 27 Aug 2018 21:51:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8051F1097304; Mon, 27 Aug 2018 21:51:02 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F0B41097303 for ; Mon, 27 Aug 2018 21:51:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0514784B17 for ; Mon, 27 Aug 2018 21:51:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5292C19428 for ; Mon, 27 Aug 2018 21:51:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RLp1vw027372 for ; Mon, 27 Aug 2018 21:51:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RLp1jo027371 for fs@FreeBSD.org; Mon, 27 Aug 2018 21:51:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230962] Kernel panic when writing extended attributes with soft updates enabled Date: Mon, 27 Aug 2018 21:51:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 21:51:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230962 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 22:01:08 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31FDA10976B2 for ; Mon, 27 Aug 2018 22:01:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BFE6285151 for ; Mon, 27 Aug 2018 22:01:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 80FA210976B1; Mon, 27 Aug 2018 22:01:07 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FCE710976B0 for ; Mon, 27 Aug 2018 22:01:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 118738514D for ; Mon, 27 Aug 2018 22:01:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5151E19588 for ; Mon, 27 Aug 2018 22:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RM16Ne060350 for ; Mon, 27 Aug 2018 22:01:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RM1659060345 for fs@FreeBSD.org; Mon, 27 Aug 2018 22:01:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Mon, 27 Aug 2018 22:01:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 22:01:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open --- Comment #3 from Rick Macklem --- If you'd like to commit it that sounds fine to me. I gave it a quick test a= nd it is a very straightforward patch. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 22:18:15 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F8EC1097A9E for ; Mon, 27 Aug 2018 22:18:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 32502859CF for ; Mon, 27 Aug 2018 22:18:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E22D41097A9D; Mon, 27 Aug 2018 22:18:14 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF5781097A9C for ; Mon, 27 Aug 2018 22:18:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66973859CB for ; Mon, 27 Aug 2018 22:18:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id AEBB519844 for ; Mon, 27 Aug 2018 22:18:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RMIDpI002636 for ; Mon, 27 Aug 2018 22:18:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RMIDCt002633 for fs@FreeBSD.org; Mon, 27 Aug 2018 22:18:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Mon, 27 Aug 2018 22:18:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 22:18:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 27 22:21:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F28971097B95 for ; Mon, 27 Aug 2018 22:21:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8C4AA85C1C for ; Mon, 27 Aug 2018 22:21:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 5007A1097B93; Mon, 27 Aug 2018 22:21:05 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EA881097B92 for ; Mon, 27 Aug 2018 22:21:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D55C585C15 for ; Mon, 27 Aug 2018 22:21:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 2DDEA19861 for ; Mon, 27 Aug 2018 22:21:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7RML4sY006843 for ; Mon, 27 Aug 2018 22:21:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7RML4Z7006834 for fs@FreeBSD.org; Mon, 27 Aug 2018 22:21:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230962] Kernel panic when writing extended attributes with soft updates enabled Date: Mon, 27 Aug 2018 22:21:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 22:21:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230962 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |panic --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 28 06:17:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 328D0108088A for ; Tue, 28 Aug 2018 06:17:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C3AF574901 for ; Tue, 28 Aug 2018 06:17:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 852A41080889; Tue, 28 Aug 2018 06:17:29 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73F6C1080888 for ; Tue, 28 Aug 2018 06:17:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F336F748FC for ; Tue, 28 Aug 2018 06:17:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5B3DB1DB11 for ; Tue, 28 Aug 2018 06:17:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7S6HS0s091004 for ; Tue, 28 Aug 2018 06:17:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7S6HS3G090995 for fs@FreeBSD.org; Tue, 28 Aug 2018 06:17:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Tue, 28 Aug 2018 06:17:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: aliovx@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 06:17:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 --- Comment #8 from Ali Abdallah --- (In reply to commit-hook from comment #7) Thanks a lot for your effort, this will make UFS even better.=20 Unfortunately not all softwares follow the safe update practice. For example I've lost my .zsh_history on crash yesterday. I assume that zsh don't fsync= on every command entered in the shell. I think I can do nothing in those situations unless I gjournal my home UFS partition or I can just use backups/snapshots.=20 I had never experienced crashes in the past, but all my troubles started wh= en I moved my production system from 11.1 to 11.2 (vboxdrv caused me all kind of troubles when running my Windows 10 vm machine, that unfortunately I need it for my work).=20 Thanks! --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 28 06:40:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7680310811F5 for ; Tue, 28 Aug 2018 06:40:43 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1BE6C7577A for ; Tue, 28 Aug 2018 06:40:43 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: by mailman.ysv.freebsd.org (Postfix) id D183110811F3; Tue, 28 Aug 2018 06:40:42 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF93210811F1; Tue, 28 Aug 2018 06:40:42 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.2 with cipher DHE-RSA-CAMELLIA128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 54A497576F; Tue, 28 Aug 2018 06:40:38 +0000 (UTC) (envelope-from Alexander@leidinger.net) Date: Tue, 28 Aug 2018 08:40:15 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leidinger.net; s=outgoing-alex; t=1535438431; bh=KbChgs+O+OiOdF4C2qNqGu4GIKkMlT3WMTqVq0TvJtk=; h=Date:From:To:Subject; b=PWLqJheVFQb7DLTpNwd6dNDLZ3xNdyWqfgbb41tVbvWw1uNRIeEn950FRWK9D74nN /jam4byuxlsROoAzCuyB6sQzQLWYGLI38zzOI01/V8TIqFh2TYOI7nejYF0Z+NzFmz M0JyAsl5EadXdmgnWYxwlp0l+2faxGWpLAxw6HIwviwGZtwBe/Enf2AWQZnTzeTAvJ 14n4v95KNcPfU9EG68gNkTtZjFfQXqjsiSZPUQhPTF2sHNuVbwU0QTxUFyJ/c5F2Kv KDownBOCEFw4ws3DqaYxFmIgf3p4wslc4VYmHMEnekY502WhPH9YoX2y30AAz3TWlw iHRqtVA8IZmUw== Message-ID: <20180828084015.Horde.JUeG828X_SDlJjeJdfmH7pV@webmail.leidinger.net> From: Alexander Leidinger To: fs@freebsd.org, current@freebsd.org Subject: old top and new -current: missing arcstat sysctl User-Agent: Horde Application Framework 5 Accept-Language: de,en Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 06:40:43 -0000 Hi, top reports missing sysctl kstat.zfs.misc.arcstats.other_size for 12.0-alpha3 with a top from an old-ish -current. Is/will this be handled via a compat-11 sysctl (my kernel is without compat-xx), or did this slip through? Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From owner-freebsd-fs@freebsd.org Tue Aug 28 08:51:23 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDE011084924 for ; Tue, 28 Aug 2018 08:51:22 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 878E17A1E7; Tue, 28 Aug 2018 08:51:22 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [212.48.125.108] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fuZiR-000I4Q-7C; Tue, 28 Aug 2018 10:51:17 +0200 Date: Tue, 28 Aug 2018 10:50:48 +0200 From: Yamagi Burmeister To: lev@FreeBSD.org Cc: freebsd-fs@freebsd.org, MacCrest@gmail.com Subject: Re: ZFS (ARC) performance regression in r321610 Message-Id: <20180828105048.d5a4133c8a932eff900bd758@yamagi.org> In-Reply-To: <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi." X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 08:51:23 -0000 --Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi. Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, 27 Aug 2018 18:27:23 +0200 Yamagi Burmeister wrote: > On Mon, 27 Aug 2018 17:19:10 +0300 > Lev Serebryakov wrote: >=20 > > On 27.08.2018 16:47, Yamagi Burmeister wrote: > >=20 > > > With this program I was able to bisect the source and identify commit > > > r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culpr= it: > > > https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 > > You could try to set "zfs_abd_scatter_enabled =3D 0" via kgdb and repe= at > > tests to be sure, that this code is a problem. >=20 > With my testcase this looks very, very good. I'll give it a try in > production and report back. That'll take at least 12 hours to be sure > that it really works. Yes, it works. The database is being able to keep up with the workload. The CPU load even seems to be lower than under FreeBSD 11.1, but it's too early to be sure about that. ZFS on Linux exposes zfs_abd_scatter_enabled through a module parameter. What about turning it into a sysctl? Something like this: diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c b/sys/cdd= l/contrib/opensolaris/uts/common/fs/zfs/abd.c index 2cd87c2b87b..88b76d1ffbc 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c @@ -140,6 +140,13 @@ static abd_stats_t abd_stats =3D { */ boolean_t zfs_abd_scatter_enabled =3D B_TRUE; =20 +#if defined(__FreeBSD__) && defined(_KERNEL) +SYSCTL_DECL(_vfs_zfs); + +SYSCTL_UINT(_vfs_zfs, OID_AUTO, abd_scatter_enabled, CTLFLAG_RWTUN, + &zfs_abd_scatter_enabled, 1, "Make all future ABDs linear"); +#endif + /* * The size of the chunks ABD allocates. Because the sizes allocated from = the * kmem_cache can't change, this tunable can only be modified at boot. Cha= nging @@ -151,9 +158,7 @@ boolean_t zfs_abd_scatter_enabled =3D B_TRUE; size_t zfs_abd_chunk_size =3D 4096; =20 #if defined(__FreeBSD__) && defined(_KERNEL) -SYSCTL_DECL(_vfs_zfs); - -SYSCTL_ULONG(_vfs_zfs, OID_AUTO, abd_chunk_size, CTLFLAG_RDTUN, +SYSCTL_ULONG(_vfs_zfs, OID_AUTO, abd_chunk_size, CTLFLAG_RWTUN, &zfs_abd_chunk_size, 0, "The size of the chunks ABD allocates"); #endif --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi. Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluFDOkACgkQ6xRy5x1Q JRW3/g//bpkp/B5mAWB6qlCQzgmK7BD92+8p3QDeIw1ke8dqkAZEZ3boX8vyc7+2 dgeCDDgBqabxa8clPlTjDBlK3TK4KcFKlZzAZOLxGsTAKT9seIfHMkUdDNByp5ZF xLtDlLM77JaQfAgqDoaFVMmkfTu9Rl0nOVDgCWimrZI0HbJiu+MpZRJX9T0YdtsG zv67sdzC5UG3FmT7k/xIH3Q4SPUDus6flYOzVajyVUz6f1S7JgsBHrEYASenl+SI Gle4lro/UQp/jk5iRDQ3Txp3YyVUqRTuhvpiQUEo5NZM026aSoBewzCQnv/Ygl2A ESSFIL+Xlv/gvB7sxTKDDnsUA71/LevrjpGd1UE3uVG9u++OnvlRe9iVhOzVLi06 bWYkHiN+jm8GXAF0oOJom9Gyqc1OW0s5VzgXqZv/e6Be8DwWR+V+4woOOo0nRjfp z44Lw1XFchWGBs5I8YHNO/UpyhxLtB9m3GK/kteFA1yyLQQ8BM5RI60Ix02FjW1I Rj05ucrSxU8Phh+anx8LdLdwlszDf0MrMZJSF5quR3JOrZhWhz3uN4+rcofWN1tH BWYm8bEvfgflfVDlbGjtoeBps46XmiplxIw0HB7ClGlfD0J+UP+2NHYYf3KSZRxs zgt134qWMBK/Qvq+iSeap/fOmi12+93PTnc4AkTR5mYrKiRRXYg= =dN3I -----END PGP SIGNATURE----- --Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi.-- From owner-freebsd-fs@freebsd.org Tue Aug 28 09:06:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 257BC108552B for ; Tue, 28 Aug 2018 09:06:11 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B81677A96B; Tue, 28 Aug 2018 09:06:10 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [212.48.125.108] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fuZwp-000I6C-3y; Tue, 28 Aug 2018 11:06:09 +0200 Date: Tue, 28 Aug 2018 11:05:43 +0200 From: Yamagi Burmeister To: lev@FreeBSD.org Cc: freebsd-fs@freebsd.org, MacCrest@gmail.com Subject: Re: ZFS (ARC) performance regression in r321610 Message-Id: <20180828110543.7cc7fa65a0e04e29a06a32b1@yamagi.org> In-Reply-To: <20180828105048.d5a4133c8a932eff900bd758@yamagi.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> <20180828105048.d5a4133c8a932eff900bd758@yamagi.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Tue__28_Aug_2018_11_05_43_+0200_Cvc2V3Yc1zsmbKcz" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 09:06:11 -0000 --Signature=_Tue__28_Aug_2018_11_05_43_+0200_Cvc2V3Yc1zsmbKcz Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, 28 Aug 2018 10:50:48 +0200 Yamagi Burmeister wrote: > ZFS on Linux exposes zfs_abd_scatter_enabled through a module > parameter. What about turning it into a sysctl? Something like this: Ups, there sneaked an unrelated edit into the patch. New version here: diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c b/sys/cdd= l/contrib/opensolaris/uts/common/fs/zfs/abd.c index 2cd87c2b87b..5de74ef3325 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c @@ -140,6 +140,13 @@ static abd_stats_t abd_stats =3D { */ boolean_t zfs_abd_scatter_enabled =3D B_TRUE; =20 +#if defined(__FreeBSD__) && defined(_KERNEL) +SYSCTL_DECL(_vfs_zfs); + +SYSCTL_UINT(_vfs_zfs, OID_AUTO, abd_scatter_enabled, CTLFLAG_RWTUN, + &zfs_abd_scatter_enabled, 1, "Make all future ABDs linear"); +#endif + /* * The size of the chunks ABD allocates. Because the sizes allocated from = the * kmem_cache can't change, this tunable can only be modified at boot. Cha= nging @@ -151,8 +158,6 @@ boolean_t zfs_abd_scatter_enabled =3D B_TRUE; size_t zfs_abd_chunk_size =3D 4096; =20 #if defined(__FreeBSD__) && defined(_KERNEL) -SYSCTL_DECL(_vfs_zfs); - SYSCTL_ULONG(_vfs_zfs, OID_AUTO, abd_chunk_size, CTLFLAG_RDTUN, &zfs_abd_chunk_size, 0, "The size of the chunks ABD allocates"); #endif --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Tue__28_Aug_2018_11_05_43_+0200_Cvc2V3Yc1zsmbKcz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluFEGgACgkQ6xRy5x1Q JRWBng//bVpzxhS7YfS13mxk9pqy2uyNNAsd8yGbZLNTbwqm+n8lcAqn4iUtQhyS iUjCaAaPPJT2y+cQ39o8wzxDVs45EqMGRpMOeM6oKhPlBfHdKCmwtUHyI35bKWcK H7SuQ5FWhXYGxUydD4pBwlYYSEg5FblkiIHTG5X9EtuywVtNigTFfGGaEGnsjapo gv4iRUEULa8p83hgmtUEKQJhQc1QjgIh4Uv4oihlrYA8JxsbxidA8hr+b7TM8vdw eclTokjeil8xejy1khh7Hg48l1m3ifxERKTK5xXDgnip73FI3AkGJu59U8mIh9vH qqxkp1p42OT6F1L5TdTvp9auYcw0AbjfqFS8aOek9wf0xc+nv73WO99b96le3YHw 0cce5JuscrKrwiqeniCCUuSbaXEMq2LJA5yLwGh5vkKDNq1GgZiK3EShNP2DYBly g0j1AlXmzho5SyIpohfHRDq53nXVxZtH8zK1xX9hZkiI3s00grFZbNrou9EMxfl5 HIKfWJTP/NlU5bCQ9uLf+c6QIvfL0bRmfnb2ZG6znZJ3uxj7SHpBmdVlZ6xlNBhJ dLHXLvk3op/g9JIiubJ5iZH40wokdfpAhpr65FBdysXV6rfLvMtgV6p2gvrgrdnm U8sWo0cm6C+t7hVu6Tkb2lIt3c0rA9a+OGcb/+PvzNMwSxIkFJE= =9icN -----END PGP SIGNATURE----- --Signature=_Tue__28_Aug_2018_11_05_43_+0200_Cvc2V3Yc1zsmbKcz-- From owner-freebsd-fs@freebsd.org Tue Aug 28 13:10:38 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8074B108BB73 for ; Tue, 28 Aug 2018 13:10:38 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id E82668318A for ; Tue, 28 Aug 2018 13:10:37 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 87807C4; Tue, 28 Aug 2018 16:10:36 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: ZFS (ARC) performance regression in r321610 To: Yamagi Burmeister Cc: freebsd-fs@freebsd.org, MacCrest@gmail.com References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> <20180828105048.d5a4133c8a932eff900bd758@yamagi.org> <20180828110543.7cc7fa65a0e04e29a06a32b1@yamagi.org> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Tue, 28 Aug 2018 16:10:35 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20180828110543.7cc7fa65a0e04e29a06a32b1@yamagi.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 13:10:38 -0000 On 28.08.2018 12:05, Yamagi Burmeister wrote: >> ZFS on Linux exposes zfs_abd_scatter_enabled through a module >> parameter. What about turning it into a sysctl? Something like this: > > Ups, there sneaked an unrelated edit into the patch. New version here: I think, you need to create Phabricator review ("change") and push it HARD, or patch will be lost and never go to sources :-( -- // Lev Serebryakov From owner-freebsd-fs@freebsd.org Tue Aug 28 13:20:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FDC0108BFB3 for ; Tue, 28 Aug 2018 13:20:56 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1BE1E837EC for ; Tue, 28 Aug 2018 13:20:55 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x542.google.com with SMTP id s10-v6so1339805edb.11 for ; Tue, 28 Aug 2018 06:20:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=qYLBrg++T+fJV7XWGXyOYoMVhePvg7gYt2B0QruNvk8=; b=F8X0POkZMrmMzN/BI3/lGhmUVHDoFbLl3d1g6PJekJjFW+esTKnrdwUmE9XsdqAd9L HJwhB/pEWpgixmyqozaO3W1YMoztR0Q+72ryJ6OVtqCnb1c8ont3n3Dg4UvduQfjGRtY pAfzaZELn5qJFLJWgUO03uZnHmRolsRYPS/9rLDtdiN5k2azAhYQkGQLO9U+xO6AlMJZ C3EVEmBjNdtfJsdhvjQ9KZbArVPySQFm+aHIP1KvBeOpWt5H2NO/noRtCnaeDJXYyvDn f+QayU31agyhl3QT1TIlfkbrN7FX9VB0Fq7CUpt9oxHylTMlqVt3+QwpU1VkjxJZCVyB Oqkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=qYLBrg++T+fJV7XWGXyOYoMVhePvg7gYt2B0QruNvk8=; b=FNXwc8ZtHiVVu9O49ra0OiOBu+8ZT5INBzQEoUfoYCIXOY2t8Bq8qDQb+stdp3J+vB ArV9iH/xbYExyM+27Fd0p7N0oRvf68sc3uLRPjckF2qRY0RQfEcFexB/ILcKhZkSDtPE uGfIE8mdEJ1CCnw6U34Q5MfWJnZlXS9BA6/q05Pe7TYjUcruia2PvDZryZxbMFUbbBL8 VghTUG7oZP2v28xhBcvRYInNWk1JicsCI5Q/AKgbmfSGBSzS2aRrhASXzV+PsPya0I4j JB0neEKdaCwDnPgCFFAGwubNnVwogG8oNjbudufdo1whRM3HhbT2lpKWFlKcvI7c6e/C PVgQ== X-Gm-Message-State: APzg51D+6EZ+QiLjc22uCEF5vxIaNJWGdmRJtEcR7JGIkt+3EqBzT3Ge OOLdMWKsOXx+xPA1el4OoXCLJRBO9kI= X-Google-Smtp-Source: ANB0Vda9vGThVXxhKSzKdVetLUYQlU+7R9YUSC60qgu7cnewKmvCKiIwwnT7ISijKuMGUZimEtvKOA== X-Received: by 2002:a50:ab15:: with SMTP id s21-v6mr2465931edc.133.1535462454351; Tue, 28 Aug 2018 06:20:54 -0700 (PDT) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id q24-v6sm568561edd.73.2018.08.28.06.20.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 06:20:53 -0700 (PDT) Subject: Re: ZFS (ARC) performance regression in r321610 To: freebsd-fs@freebsd.org References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> <20180828105048.d5a4133c8a932eff900bd758@yamagi.org> <20180828110543.7cc7fa65a0e04e29a06a32b1@yamagi.org> From: Steven Hartland Message-ID: <93ad65ca-e1d5-0275-b8c2-bc8a55e66e0c@multiplay.co.uk> Date: Tue, 28 Aug 2018 14:20:53 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 13:20:56 -0000 On 28/08/2018 14:10, Lev Serebryakov wrote: > On 28.08.2018 12:05, Yamagi Burmeister wrote: > > >>> ZFS on Linux exposes zfs_abd_scatter_enabled through a module >>> parameter. What about turning it into a sysctl? Something like this: >> Ups, there sneaked an unrelated edit into the patch. New version here: > I think, you need to create Phabricator review ("change") and push it > HARD, or patch will be lost and never go to sources :-( > I'll try and pick this up if no else gets to it first.     Regards     Steve From owner-freebsd-fs@freebsd.org Tue Aug 28 14:17:52 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A214108D8E0 for ; Tue, 28 Aug 2018 14:17:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BB94085DA8 for ; Tue, 28 Aug 2018 14:17:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 7D3A7108D8DF; Tue, 28 Aug 2018 14:17:51 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BE63108D8DE for ; Tue, 28 Aug 2018 14:17:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D79D85DA4 for ; Tue, 28 Aug 2018 14:17:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4377A21E09 for ; Tue, 28 Aug 2018 14:17:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7SEHo7u020670 for ; Tue, 28 Aug 2018 14:17:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7SEHoLs020667 for fs@FreeBSD.org; Tue, 28 Aug 2018 14:17:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Tue, 28 Aug 2018 14:17:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 14:17:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 --- Comment #9 from Conrad Meyer --- (In reply to Ali Abdallah from comment #8) Yeah, with zsh I end up taking backups. I rely on ^R history a lot to work quickly, so losing it is painful. Here's a really ugly kludge I use (saves= a new copy any time .zhistory is modified): https://github.com/cemeyer/backup_zhistory --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 28 15:18:27 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 480DB108F54E for ; Tue, 28 Aug 2018 15:18:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D923389771 for ; Tue, 28 Aug 2018 15:18:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 9E6CE108F54D; Tue, 28 Aug 2018 15:18:26 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D6BE108F54C for ; Tue, 28 Aug 2018 15:18:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2BECE89768 for ; Tue, 28 Aug 2018 15:18:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5B61122660 for ; Tue, 28 Aug 2018 15:18:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7SFIPvX084165 for ; Tue, 28 Aug 2018 15:18:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7SFIPpm084158 for fs@FreeBSD.org; Tue, 28 Aug 2018 15:18:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Tue, 28 Aug 2018 15:18:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 15:18:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 --- Comment #4 from commit-hook@freebsd.org --- A commit references this bug: Author: markj Date: Tue Aug 28 15:18:14 UTC 2018 New revision: 338350 URL: https://svnweb.freebsd.org/changeset/base/338350 Log: Add missing endpwent() and endgrent() calls to nfsuserd(8). PR: 230937 Submitted by: Peter Eriksson Reviewed by: rmacklem Approved by: re (gjb) MFC after: 1 week Changes: head/usr.sbin/nfsuserd/nfsuserd.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 29 09:42:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3449F1083945 for ; Wed, 29 Aug 2018 09:42:43 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CAD8584B13 for ; Wed, 29 Aug 2018 09:42:42 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id:To: Subject:From:Date:Sender:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=gQ0cCpy08VB5wPPYQ2OY1l3mbA+TFF2efbb4lCoQqq0=; b=J7yLZpmss2iHOM9Ct0HWbIVkra rVSpcGRELBeycnQTriaGOoUPeh51Ej+zOs6KjEgQTDYOP1zz4snBfRmFHMUu0qbzraiIxZXcAWqTB alD1FUHRfqroSXWuIWeDMHHkahUyuYhtnYqUFFFw0T23IaujD8iV4PZBjKbr1WbIVaWY=; Received: from [10.10.10.33] (helo=frv33.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1fuwzh-000I4O-Mi for freebsd-fs@freebsd.org; Wed, 29 Aug 2018 12:42:33 +0300 Date: Wed, 29 Aug 2018 12:42:33 +0300 From: Paul Subject: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size To: freebsd-fs@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> Received: from devgs@ukr.net by frv33.fwdcdn.com; Wed, 29 Aug 2018 12:42:33 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 09:42:43 -0000 Hello team, It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() target is reached after MRU is evicted current code continues evicting MFU. Before said commit, on the step prior to MFU eviction, target value was recalculated as: target = arc_size - arc_c; arc_size here is a global variable that was being updated accordingly, during MRU eviction, hence this expression, resulted in zero or negative target if MRU eviction was enough to reach the original goal. Modern version uses cached value of arc_size, called asize: target = asize - arc_c; Because asize stays constant during execution of whole body of arc_adjust() it means that above expression will always be evaluated to value > 0, causing MFU to be evicted every time, even if MRU eviction has reached the goal already. Because of the difference in nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC to dramatic numbers. Servers that run the version of FreeBSD prior to the issue have this picture of ARC: ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G Other As you can see, MFU dominates. This is a nature of our workload: we have a considerably small dataset that we use constantly and repeatedly; and a large dataset that we use rarely. But on the modern version of FreeBSD picture is dramatically different: ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G Other This leads to a much heavier burden on the disk sub-system. Commit that introduced a bug: https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233cf1f92b8 Best regards, Paul From owner-freebsd-fs@freebsd.org Wed Aug 29 09:45:57 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 004221083B24; Wed, 29 Aug 2018 09:45:56 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from connect.ultra-secure.de (connect.ultra-secure.de [88.198.71.201]) by mx1.freebsd.org (Postfix) with ESMTP id 313E384C73; Wed, 29 Aug 2018 09:45:55 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (Haraka outbound); Wed, 29 Aug 2018 11:45:49 +0200 Authentication-Results: connect.ultra-secure.de; auth=pass (login); spf=none smtp.mailfrom=ultra-secure.de Received-SPF: None (connect.ultra-secure.de: domain of ultra-secure.de does not designate 127.0.0.10 as permitted sender) receiver=connect.ultra-secure.de; identity=mailfrom; client-ip=127.0.0.10; helo=connect.ultra-secure.de; envelope-from= Received: from connect.ultra-secure.de (webmail [127.0.0.10]) by connect.ultra-secure.de (Haraka/2.6.2-toaster) with ESMTPSA id D9C3E06D-0A03-4E0E-BF2A-36955DEE3DEE.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=AES256-SHA verify=NO); Wed, 29 Aug 2018 11:45:46 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 29 Aug 2018 11:45:45 +0200 From: rainer@ultra-secure.de To: lev@freebsd.org Cc: Yamagi Burmeister , freebsd-fs@freebsd.org, MacCrest@gmail.com, owner-freebsd-fs@freebsd.org Subject: Re: ZFS (ARC) performance regression in r321610 In-Reply-To: <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> Message-ID: <1c816966fbe698f0497df231139f9a05@ultra-secure.de> X-Sender: rainer@ultra-secure.de User-Agent: Roundcube Webmail/1.2.0 X-Haraka-GeoIP: --, , NaNkm X-Haraka-GeoIP-Received: X-Haraka-p0f: os="undefined undefined" link_type="undefined" distance=undefined total_conn=undefined shared_ip=Y X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on spamassassin X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Haraka-Karma: score: 6, good: 678, bad: 0, connections: 684, history: 678, pass:all_good, relaying X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 09:45:57 -0000 Am 2018-08-27 16:19, schrieb Lev Serebryakov: > On 27.08.2018 16:47, Yamagi Burmeister wrote: > >> With this program I was able to bisect the source and identify commit >> r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the >> culprit: >> https://svnweb.freebsd.org/base?view=revision&revision=321610 > You could try to set "zfs_abd_scatter_enabled = 0" via kgdb and repeat > tests to be sure, that this code is a problem. > > BTW, it is not clear, that FreeBSD needs this at all. Hi, how exactly do I do this? Best Regards Rainer From owner-freebsd-fs@freebsd.org Wed Aug 29 10:14:27 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4965510842CF; Wed, 29 Aug 2018 10:14:27 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BF98D859C3; Wed, 29 Aug 2018 10:14:26 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id w7TAEP4d082783; Wed, 29 Aug 2018 12:14:25 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 47AABABB; Wed, 29 Aug 2018 12:14:25 +0200 (CEST) Subject: Re: ZFS (ARC) performance regression in r321610 To: rainer@ultra-secure.de Cc: freebsd-fs@freebsd.org, owner-freebsd-fs@freebsd.org References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <1c816966fbe698f0497df231139f9a05@ultra-secure.de> From: Harry Schmalzbauer Organization: OmniLAN Message-ID: <3530981a-7a44-6cf8-6fcd-3b44fd364cb7@omnilan.de> Date: Wed, 29 Aug 2018 12:14:25 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <1c816966fbe698f0497df231139f9a05@ultra-secure.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Greylist: ACL 130 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Wed, 29 Aug 2018 12:14:25 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 10:14:27 -0000 Am 29.08.2018 um 11:45 schrieb rainer@ultra-secure.de: > Am 2018-08-27 16:19, schrieb Lev Serebryakov: >> On 27.08.2018 16:47, Yamagi Burmeister wrote: >> >>> With this program I was able to bisect the source and identify commit >>> r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the >>> culprit: >>> https://svnweb.freebsd.org/base?view=revision&revision=321610 >>  You could try to set "zfs_abd_scatter_enabled = 0" via kgdb and repeat >> tests to be sure, that this code is a problem. >> >>  BTW, it is not clear, that FreeBSD needs this at all. > > > Hi, > > how exactly do I do this? > Maybe this is more/easier applicaple for you: https://svnweb.freebsd.org/changeset/base/338365 Can't really answer your question, but should help getting the desired result :-) -harry From owner-freebsd-fs@freebsd.org Wed Aug 29 11:20:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1E92108579F; Wed, 29 Aug 2018 11:20:01 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 95A6E87322; Wed, 29 Aug 2018 11:20:01 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [212.48.125.108] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fuyVt-000Lht-EI; Wed, 29 Aug 2018 13:19:59 +0200 Date: Wed, 29 Aug 2018 13:19:40 +0200 From: Yamagi Burmeister To: rainer@ultra-secure.de Cc: lev@freebsd.org, freebsd-fs@freebsd.org, MacCrest@gmail.com, owner-freebsd-fs@freebsd.org Subject: Re: ZFS (ARC) performance regression in r321610 Message-Id: <20180829131940.7ca6ccb23035eee6808119fb@yamagi.org> In-Reply-To: <1c816966fbe698f0497df231139f9a05@ultra-secure.de> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <1c816966fbe698f0497df231139f9a05@ultra-secure.de> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Wed__29_Aug_2018_13_19_40_+0200_BZT9N58_+8NNCZMT" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 11:20:02 -0000 --Signature=_Wed__29_Aug_2018_13_19_40_+0200_BZT9N58_+8NNCZMT Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, my patch was committed this morning. If you're on 12-CURRENT you can just set vfs.zfs.abd_scatter_enabled to 0. On other branches the easiest way to make the change permanent is to edit sys/cddl/contrib/opensolaris/uts/common/fszfs/abd.c and change zfs_abd_scatter_enabled (on 11.2 it's in line 141) from B_TRUE to B_FALSE. Then rebuild the kernel, reboot and you're done. Regards, Yamagi On Wed, 29 Aug 2018 11:45:45 +0200 rainer@ultra-secure.de wrote: > Am 2018-08-27 16:19, schrieb Lev Serebryakov: > > On 27.08.2018 16:47, Yamagi Burmeister wrote: > >=20 > >> With this program I was able to bisect the source and identify commit > >> r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the=20 > >> culprit: > >> https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 > > You could try to set "zfs_abd_scatter_enabled =3D 0" via kgdb and repe= at > > tests to be sure, that this code is a problem. > >=20 > > BTW, it is not clear, that FreeBSD needs this at all. >=20 >=20 > Hi, >=20 > how exactly do I do this? >=20 >=20 >=20 > Best Regards > Rainer --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Wed__29_Aug_2018_13_19_40_+0200_BZT9N58_+8NNCZMT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluGgU0ACgkQ6xRy5x1Q JRXqdw/6AzHoyR1N5zV+K9FVlEfytIN5KWTLgKh6S+N24RUQtoUzm0ElMuKupBPr 4vAnpC4U7ZDxPvN1h0wEiDSlfPSTOEV9a46v3xZ4FEdi8RFulOcskTvPRIdQ8rxK 3GquuJZjXo9O7hglDGpWd1BHr7Az1BaR3lASAtpeqw7vMwPe6GVwn+A7dvWZaAAq dSZLL2NppZSXq9HrAtOwF8Yq4sYpIWNqLjeTmrXPSadaGhFoCy48bprOBkY0faXY Tbevg5poK7CAtIU6UdryZ5j8zrApLVw8994OI7WwXSDUaT2VwY1U1ivPB2QVCcqf gE+mhrpfPh+YO895yY4gIGiqEyc7FPQUO6Yk3SX8xWEaR43TOm6bZEGCJUBz51zN mvkxjj/WhDjBKpE9390JLKOdmNh0F9XYI8gG/PRAfJonm+FSe/s682u3H/BWwowW qljN5uQbXANgPWCjXOUiMYvel+Typyv3kQJXHOh7amvpdq/1wkNi7APUymmFtbv2 y759s5Qb5TWfhFvyp3wp0NRk6sz2J9FV0gl9Jprw+CHd4RiFOK4iOttz+K7vNgHB C5yDi4m4cA+GLlOGs91NFIABBk1P1Yl100hAi3VYiSdpGRB0Qs2pAOd+7IoN8Tfc YQgEaWk4GLPsgaICVtVqcYT3Aqkxx4IyYjOvqgBnxHormW0469A= =Z0vH -----END PGP SIGNATURE----- --Signature=_Wed__29_Aug_2018_13_19_40_+0200_BZT9N58_+8NNCZMT-- From owner-freebsd-fs@freebsd.org Wed Aug 29 20:08:17 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C39DD1094A59 for ; Wed, 29 Aug 2018 20:08:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 68C347F787 for ; Wed, 29 Aug 2018 20:08:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 227D91094A58; Wed, 29 Aug 2018 20:08:17 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A7EB1094A56 for ; Wed, 29 Aug 2018 20:08:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A9D197F782 for ; Wed, 29 Aug 2018 20:08:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id D160611A6D for ; Wed, 29 Aug 2018 20:08:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7TK8FI7085103 for ; Wed, 29 Aug 2018 20:08:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7TK8F7H085102 for fs@FreeBSD.org; Wed, 29 Aug 2018 20:08:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Wed, 29 Aug 2018 20:08:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 20:08:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |linimon@FreeBSD.org Version|CURRENT |11.0-STABLE --- Comment #6 from Mark Linimon --- Is this PR still relevant? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 29 21:22:13 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEB131096CF9 for ; Wed, 29 Aug 2018 21:22:12 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C17582A91 for ; Wed, 29 Aug 2018 21:22:12 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pl1-x643.google.com with SMTP id b12-v6so2846124plr.8 for ; Wed, 29 Aug 2018 14:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=reTQoFlYVhrTC2Ga2IUfoy/mDBXmpfIu5DeaKjWZdnI=; b=HVYKrdJBsgt34/RtVhP7rPLSy6nNne8/9jOk6RJum4WXkNwSj9Dc6uta+8lDG8Prba XjdvWg0CKL6K0t/wEGzdto+K1M5ZoTmJVs5gMpK/DPv0CyvceIc/nFjzA3rdfzEClOH1 fCASJ+lfvToGCfwStP11R0VJWCe6EM28LQfnMxGJPQY5M2zzmDub9rj+41WxXLdw1q5L +v+9LDyYA0zZ0d95t+/21I/61HYC0NhcqEB1BFvnXLRHK1j2u9blCuDrRZ+ADO3LYM47 AArFFesLZj18hNYYGpYPX7EMAv5jl7luBW/3+aLAxhxUmvKtb4ejLt+o1lxqFusDpMfw VTRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=reTQoFlYVhrTC2Ga2IUfoy/mDBXmpfIu5DeaKjWZdnI=; b=nLxhSoccwg+RxxOrJKAyy67UJH4ShknFq/UMoDqSFCDu1/mXGyOe3RocaililrPOHT j103+XNDr/8JjW4k4api4tGyH68KSAXDvS/stsndUx3BuwVyuyN7XTPcqGaYKZ9omEHl IUwsP9Ti0wEYDA8r609geArYFvKWdocVDhgJEPTFT3FciyKH8NYCHUaknD7Z6hmk2lbf 5vr8EqD1MtvxPvhmNcUDKPn0lCaAAZ9xzSaeG7Ktrcnlf/YumkqVgC8g7MIc1Q3Zlmq0 TABKGo3LPtZx9Z0LICFqh65WVbWn63dPiJIE4Da9R2xnX7wkHaKTdUzF4GZV5ryusiXR eo8w== X-Gm-Message-State: APzg51Bbf1ZBWPasvuSXD7VXlnrX0k14hJfd8yYjOxwu77tDsAGi5GUX tpdCK44hNTIZ+zey06SC8M8= X-Google-Smtp-Source: ANB0VdbQVr8KfExd3MsDhp5unsxZwCC861y8gyt2kuHn2Qj2MBAtz2SYPFmOXVjw1OovCYrtZxcRug== X-Received: by 2002:a17:902:b688:: with SMTP id c8-v6mr7488879pls.114.1535577731425; Wed, 29 Aug 2018 14:22:11 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id v19-v6sm8481426pgn.94.2018.08.29.14.22.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Aug 2018 14:22:10 -0700 (PDT) Sender: Mark Johnston Date: Wed, 29 Aug 2018 17:22:07 -0400 From: Mark Johnston To: Paul Cc: freebsd-fs@freebsd.org, developer@open-zfs.org Subject: Re: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Message-ID: <20180829212207.GF2709@raichu> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 21:22:13 -0000 On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > Hello team, > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > target is reached after MRU is evicted current code continues evicting MFU. Before said > commit, on the step prior to MFU eviction, target value was recalculated as: > > target = arc_size - arc_c; > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > hence this expression, resulted in zero or negative target if MRU eviction was enough > to reach the original goal. > > Modern version uses cached value of arc_size, called asize: > > target = asize - arc_c; > > Because asize stays constant during execution of whole body of arc_adjust() it means that > above expression will always be evaluated to value > 0, causing MFU to be evicted every > time, even if MRU eviction has reached the goal already. Because of the difference in > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > to dramatic numbers. Hi Paul, Your analysis does seem right to me. I cc'ed the openzfs mailing list so that an actual ZFS expert can chime in; it looks like this behaviour is consistent between FreeBSD, illumos and ZoL. Have you already tried the obvious "fix" of subtracting total_evicted from the MFU target? > Servers that run the version of FreeBSD prior to the issue have this picture of ARC: > > ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G Other > > As you can see, MFU dominates. This is a nature of our workload: we have a considerably > small dataset that we use constantly and repeatedly; and a large dataset that we use > rarely. > > But on the modern version of FreeBSD picture is dramatically different: > > ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G Other > > This leads to a much heavier burden on the disk sub-system. > > > Commit that introduced a bug: > https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233cf1f92b8 From owner-freebsd-fs@freebsd.org Wed Aug 29 22:03:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A40A81097CA8 for ; Wed, 29 Aug 2018 22:03:33 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44B96840E1 for ; Wed, 29 Aug 2018 22:03:33 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: by mail-pf1-x441.google.com with SMTP id l9-v6so2880401pff.9 for ; Wed, 29 Aug 2018 15:03:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=richardelling.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=6L9BM6DqgWpDlLqacQ2FtbXQSEif8XfJkVkcBejCu0I=; b=ZO5ckVfUdb4c2izpqlRyLUGgvIawW+9VzRlkIGjEGbbMKTAIFLJSPxcVTM+VpLUxRu W+rcVs0iKsOMAQNJDtQbUTD7YSfMwtAbYQQMya6pDt/LdpBHqj2ewr1hzIFdkf7RMFcX gcgRH/7LjQfzmZ4XI8x2kd8IFcq3NpewkyNho= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=6L9BM6DqgWpDlLqacQ2FtbXQSEif8XfJkVkcBejCu0I=; b=AmTeZQCTwHzY7vy1bg52Zf5YW05xvhPDzV/NBjmAcQSOuYl2ZE5wJMt1ZIUbpVHjQ5 GDyK1ALL3yKyD1KCCr28ZItsxK2Qk6WNocMUJpMdgmeA0Lw9hzb2EyzaTqnphBNx9tX2 m1rtxWv1t7KYPDSQqwSxlGjAxYOKmDLAh6K+rdLnMHJJeKSRXcV34uRQreKDi1KLQEiE PTNU+axRqeu11H2vaFWFUHsvtZwBuFyf84RmZnbFhHyN5gbSx2iUSJmas3QNYXwX9uu+ 98vje7oPonOy8OibES7wXPnhPtVFVQms9vk4oiLPnliipgZpk+OVL3xlJ/9dlaFKH2qa 2pKA== X-Gm-Message-State: APzg51CA87wg/MBS6cy/SRlhxkmob4Zm+ksmYoJkPG/1FkFbMEr4xkNw 4ZVyp5eWVX/qcF7/U5sJQvzrvg== X-Google-Smtp-Source: ANB0VdYaIHFKPeFn4Qs98shVrsSJqZ4WEEkgnW7RoEV11wPC1ydAOz2En5MOF3EL//QLFumVgYl1nQ== X-Received: by 2002:a63:5fc8:: with SMTP id t191-v6mr7228851pgb.183.1535580211449; Wed, 29 Aug 2018 15:03:31 -0700 (PDT) Received: from [192.168.0.138] ([172.58.19.49]) by smtp.gmail.com with ESMTPSA id t21-v6sm8236877pfi.73.2018.08.29.15.03.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Aug 2018 15:03:30 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: [developer] Re: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size From: Richard Elling In-Reply-To: <20180829212207.GF2709@raichu> Date: Wed, 29 Aug 2018 15:03:27 -0700 Cc: Paul , freebsd-fs@freebsd.org, developer@open-zfs.org Content-Transfer-Encoding: quoted-printable Message-Id: <6569BAF8-1AC4-4CB4-A384-E5C8EFF129D8@richardelling.com> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> To: openzfs-developer X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 22:03:33 -0000 Thanks for passing this along, Mark. Comments embedded > On Aug 29, 2018, at 2:22 PM, Mark Johnston wrote: >=20 > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: >> Hello team, >>=20 >>=20 >> It seems like a commit on Mar 23 introduced a bug: if during = execution of arc_adjust() >> target is reached after MRU is evicted current code continues = evicting MFU. Before said >> commit, on the step prior to MFU eviction, target value was = recalculated as: arc_size is hot, so it was broken up into per-cpu counters and asize is = now a snapshot of the sum of the counters... >>=20 >> target =3D arc_size - arc_c; >>=20 >> arc_size here is a global variable that was being updated = accordingly, during MRU eviction, >> hence this expression, resulted in zero or negative target if MRU = eviction was enough >> to reach the original goal. >>=20 >> Modern version uses cached value of arc_size, called asize: >>=20 >> target =3D asize - arc_c; >>=20 >> Because asize stays constant during execution of whole body of = arc_adjust() it means that >> above expression will always be evaluated to value > 0, causing MFU = to be evicted every=20 >> time, even if MRU eviction has reached the goal already. Because of = the difference in=20 >> nature of MFU and MRU, globally it leads to eventual reduction of = amount of MFU in ARC=20 >> to dramatic numbers. >=20 > Hi Paul, >=20 > Your analysis does seem right to me. I cc'ed the openzfs mailing list > so that an actual ZFS expert can chime in; it looks like this = behaviour > is consistent between FreeBSD, illumos and ZoL. Agree. In the pre-aggsum code, arc_size would have changed after the MRU = adjustment. Now it does not. I have at least one correlation to this occuring in a = repeatable test that I can run on my ZoL test machine (when it is finished punishing some = other code). >=20 > Have you already tried the obvious "fix" of subtracting total_evicted > from the MFU target? ameta also needs to be re-aggsummed after the MRU adjustments. -- richard >=20 >> Servers that run the version of FreeBSD prior to the issue have this = picture of ARC: >>=20 >> ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G = Other >>=20 >> As you can see, MFU dominates. This is a nature of our workload: we = have a considerably=20 >> small dataset that we use constantly and repeatedly; and a large = dataset that we use >> rarely. >>=20 >> But on the modern version of FreeBSD picture is dramatically = different:=20 >>=20 >> ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G = Other >>=20 >> This leads to a much heavier burden on the disk sub-system. >>=20 >>=20 >> Commit that introduced a bug:=20 >> = https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233= cf1f92b8 >=20 > ------------------------------------------ > openzfs: openzfs-developer > Permalink: = https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M8152dc243= 0a5ea4e625ad564 > Delivery options: = https://openzfs.topicbox.com/groups/developer/subscription From owner-freebsd-fs@freebsd.org Thu Aug 30 06:55:31 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D23710967D4 for ; Thu, 30 Aug 2018 06:55:31 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB4D274B5E for ; Thu, 30 Aug 2018 06:55:30 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-Id:Cc:To:Subject:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=d2dgnriTyupDsC8f+s4uefxDlXwti3HKnwsimEuQNE8=; b=iC44mcJd6LvcLNyUGE9ZhZSs0J W9o5xNfIgkteyowXCPUAcj1H41/Zr2z2q/Ii3dcXz2Cem01wOznbQvyEAybvsZ5MSwRMuOI3eSSSC 9gkDLUDFFpYZ5LYGAEE+PJm6dyx5plqY3eHffkXFUAeoIvdWgKoIl2zZGfX5EI/VLsXg=; Received: from [10.10.10.33] (helo=frv33.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1fvGrX-000LtD-UQ for freebsd-fs@freebsd.org; Thu, 30 Aug 2018 09:55:27 +0300 Date: Thu, 30 Aug 2018 09:55:27 +0300 From: Paul Subject: Re[2]: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size To: Mark Johnston Cc: freebsd-fs@freebsd.org, developer@open-zfs.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1535611885.624706680.h132ex07@frv33.fwdcdn.com> In-Reply-To: <20180829212207.GF2709@raichu> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> X-Reply-Action: reply Received: from devgs@ukr.net by frv33.fwdcdn.com; Thu, 30 Aug 2018 09:55:27 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 06:55:31 -0000 30 August 2018, 00:22:14, by "Mark Johnston" : > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > > Hello team, > > > > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > > target is reached after MRU is evicted current code continues evicting MFU. Before said > > commit, on the step prior to MFU eviction, target value was recalculated as: > > > > target = arc_size - arc_c; > > > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > > hence this expression, resulted in zero or negative target if MRU eviction was enough > > to reach the original goal. > > > > Modern version uses cached value of arc_size, called asize: > > > > target = asize - arc_c; > > > > Because asize stays constant during execution of whole body of arc_adjust() it means that > > above expression will always be evaluated to value > 0, causing MFU to be evicted every > > time, even if MRU eviction has reached the goal already. Because of the difference in > > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > > to dramatic numbers. > > Hi Paul, > > Your analysis does seem right to me. I cc'ed the openzfs mailing list > so that an actual ZFS expert can chime in; it looks like this behaviour > is consistent between FreeBSD, illumos and ZoL. > > Have you already tried the obvious "fix" of subtracting total_evicted > from the MFU target? We are going to apply the asize patch (plus the ameta, as suggested by Richard) and reboot one of our production servers this night or the following. Then we have to wait a few days and observer the ARC behaviour. > > > Servers that run the version of FreeBSD prior to the issue have this picture of ARC: > > > > ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G Other > > > > As you can see, MFU dominates. This is a nature of our workload: we have a considerably > > small dataset that we use constantly and repeatedly; and a large dataset that we use > > rarely. > > > > But on the modern version of FreeBSD picture is dramatically different: > > > > ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G Other > > > > This leads to a much heavier burden on the disk sub-system. > > > > > > Commit that introduced a bug: > > https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233cf1f92b8 From owner-freebsd-fs@freebsd.org Thu Aug 30 14:18:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09BC1109FA86 for ; Thu, 30 Aug 2018 14:18:02 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 92E568293E for ; Thu, 30 Aug 2018 14:18:01 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 55B71109FA84; Thu, 30 Aug 2018 14:18:01 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43E64109FA82; Thu, 30 Aug 2018 14:18:01 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9FBC82935; Thu, 30 Aug 2018 14:18:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg1-x530.google.com with SMTP id e2-v6so3957573pgv.5; Thu, 30 Aug 2018 07:18:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=q7FfFeqErpEpzYgveGC3dap/DfHGhnXNu9I/9Z5osDM=; b=HDilkKECKDgJw29qIbsw65XuIgIpsgKj7seI6GyvvpZ2E6ZleZVGFsK3PspiZ+lzyo hVBjRrf0s2v+Wts0UQn+81GYSXU+rU63ghIxIpUr0tLncPTSjJFnR8mmtoGqKiA5xDHV ldb2TwDfw3xYad9NAO3LDDQc49bmE0wXhg/++77X+S/OA5UFmmdzG1RKjeFzOuIWuzQU Spg6ciVOLf+qnqd+YSOGrQECvksvq+eutv+LaC44wIuaN0nnaSLKeL5LZ6ID/zCSAyei xDSCbKsBxh2WOjc5c/LJMjRhfaYvcCQh5J33dfUyG1wTQGzPV4pZxrSQU+XsjdueVdwW wEKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=q7FfFeqErpEpzYgveGC3dap/DfHGhnXNu9I/9Z5osDM=; b=gH7tY1XA88RpYwTBPO6+Ov4jlIey9nH2D5z3zZQI8So8oznx9NC7y2ZSz3G1riXRMX Gn5abCBfN+hiSxMyraJC/+JnMiQMaXmWNXhtRa/1GXLb1+J7Sx1TVaDNx1Ef17SeyWEq 8BfKPKQodt5oT9Tlt+wA8TOZ2EdWgLNilRkwOOyLSwh4gJwRISnE0V8wmOodG2SoJ0hL IiK8QJFBCN4cUIHlhTCfwmZHWSLFAaNcxmukX+CKBAfFreYkf4llfyrVZFg0QEtfYT+S s8F3FBui0B/lPAB9/8uiKuiOWEXiiKJiL/OM6wmXNzOoayiOkwcgUjYC+/UPHx3JNf80 n22Q== X-Gm-Message-State: APzg51CZ5kXkJfX5isWlLz2AWbXDhSZ2FQ/iJ/6LajYsm1/oWWre0k6H xlCk4XAF9RQVVMpQTFT8hxd1v8LstFg= X-Google-Smtp-Source: ANB0VdYCrdWL15La2vAe7ddXknsAX8xoOplZ39aTNNHaHn8rd+Eilh8cwY6VLQTa2J6LzXItUd25oQ== X-Received: by 2002:a62:6a01:: with SMTP id f1-v6mr10675346pfc.156.1535638679818; Thu, 30 Aug 2018 07:17:59 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id e7-v6sm9781603pgc.55.2018.08.30.07.17.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:17:58 -0700 (PDT) Sender: Mark Johnston Date: Thu, 30 Aug 2018 10:17:56 -0400 From: Mark Johnston To: Alexander Leidinger Cc: fs@freebsd.org, current@freebsd.org Subject: Re: old top and new -current: missing arcstat sysctl Message-ID: <20180830141756.GC15740@raichu> References: <20180828084015.Horde.JUeG828X_SDlJjeJdfmH7pV@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180828084015.Horde.JUeG828X_SDlJjeJdfmH7pV@webmail.leidinger.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 14:18:02 -0000 On Tue, Aug 28, 2018 at 08:40:15AM +0200, Alexander Leidinger wrote: > Hi, > > top reports missing sysctl kstat.zfs.misc.arcstats.other_size for > 12.0-alpha3 with a top from an old-ish -current. > > Is/will this be handled via a compat-11 sysctl (my kernel is without > compat-xx), or did this slip through? This should be fixed by r338395. From owner-freebsd-fs@freebsd.org Thu Aug 30 14:40:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0982010A05A8 for ; Thu, 30 Aug 2018 14:40:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C43883E4F for ; Thu, 30 Aug 2018 14:40:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pf1-x444.google.com with SMTP id k21-v6so3966045pff.11 for ; Thu, 30 Aug 2018 07:40:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=jOeYtncPj+q7rD29er3ysomxyCDKrHmZNmoj97Ukbg8=; b=XpEzblbtfTajM6wW21hAKrM9uN6QQOQERebGqYnJgV1YR9r11wy3AN1GBdQaxqwqpJ kDzVIcx+qajViIk8nJpUv7eRnUAz3sKiFR+YJdumSgyrlD9t78yP12BmfTK9C1y5wY2p aCFOnQM5X2P6lytAXPNSf8tUCtCgW2YR59CAo8qck2XA8npqItBY2tEGNVLXd6361Tu9 5QDMW92MmHyH4N3aLrsS2PskrTlXWSoD7Fa05y/8PLsbGYkNoaZ9UDzOfQI9DIWHdiCu b3HHLYK/tGJKAfB2bNfOKz8z6/r7kofG5YgAi3vOoALxwNpTv9EAjZwfnG6TcubIhEOX AC5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=jOeYtncPj+q7rD29er3ysomxyCDKrHmZNmoj97Ukbg8=; b=hU3TrfcEI0wWcIJs5I5PWVScJHSbLLfXH4FEh8lkmSaLKuuaTqGiFmoxJ4UHsN46Hr SVneO8Kp1K9u8YVl0+VSOvcdNSW+9OrwqOKT6f2MKdbb4DUCV5D919zK9+xBmo+CNpdb n9bbCzfzbRuDjKZyZkYnReHlXAMOysWvcqvrwqIz0ALThnVkW3jssop3L5KNa1iiXMX1 IHkcQLysCYpvjSZWGayr1+0dz2JeHnnbAOMCmINAoMkVH1fZ9PVN4lQPHGyIjtJUvBz/ WbZxMNZ6Ty6zxgal4X2hK2opUm72t/G3kzM5ZVzTHzYshalfsACImdgDpJfKZnbj9RHm Ndcw== X-Gm-Message-State: APzg51ByHqZGIiRd+lK5d59c6SW95KD6DnHDy0zKUvLCuwCFlMMwZ7uB bS0vc4sqj7eRi8mkUhhN0hM= X-Google-Smtp-Source: ANB0VdZszNhjqsFqlKgpKX9QV5diz1a1LJgyFtwV2DEDm+xqUE+q9CVVQ5Hbs3/l3TeDzfZkUTrAJA== X-Received: by 2002:aa7:860b:: with SMTP id p11-v6mr10760285pfn.247.1535640008468; Thu, 30 Aug 2018 07:40:08 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id r87-v6sm12477270pfb.1.2018.08.30.07.40.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:40:07 -0700 (PDT) Sender: Mark Johnston Date: Thu, 30 Aug 2018 10:40:04 -0400 From: Mark Johnston To: Paul Cc: freebsd-fs@freebsd.org, developer@open-zfs.org Subject: Re: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Message-ID: <20180830144004.GD15740@raichu> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1535611885.624706680.h132ex07@frv33.fwdcdn.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 14:40:10 -0000 On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: > 30 August 2018, 00:22:14, by "Mark Johnston" : > > > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > > > Hello team, > > > > > > > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > > > target is reached after MRU is evicted current code continues evicting MFU. Before said > > > commit, on the step prior to MFU eviction, target value was recalculated as: > > > > > > target = arc_size - arc_c; > > > > > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > > > hence this expression, resulted in zero or negative target if MRU eviction was enough > > > to reach the original goal. > > > > > > Modern version uses cached value of arc_size, called asize: > > > > > > target = asize - arc_c; > > > > > > Because asize stays constant during execution of whole body of arc_adjust() it means that > > > above expression will always be evaluated to value > 0, causing MFU to be evicted every > > > time, even if MRU eviction has reached the goal already. Because of the difference in > > > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > > > to dramatic numbers. > > > > Hi Paul, > > > > Your analysis does seem right to me. I cc'ed the openzfs mailing list > > so that an actual ZFS expert can chime in; it looks like this behaviour > > is consistent between FreeBSD, illumos and ZoL. > > > > Have you already tried the obvious "fix" of subtracting total_evicted > > from the MFU target? > > We are going to apply the asize patch (plus the ameta, as suggested by Richard) and reboot > one of our production servers this night or the following. Just to be explicit, are you testing something equivalent to the patch at the end of this email? > Then we have to wait a few days and observer the ARC behaviour. Thanks! Please let us know how it goes: we're preparing to release FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon as possible. diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c index 1387925c4607..882c04dba50a 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c @@ -4446,6 +4446,12 @@ arc_adjust(void) arc_adjust_impl(arc_mru, 0, target, ARC_BUFC_METADATA); } + /* + * Re-sum ARC stats after the first round of evictions. + */ + asize = aggsum_value(&arc_size); + ameta = aggsum_value(&arc_meta_used); + /* * Adjust MFU size * From owner-freebsd-fs@freebsd.org Thu Aug 30 15:08:03 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52CFF10A10ED for ; Thu, 30 Aug 2018 15:08:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DFC008500A for ; Thu, 30 Aug 2018 15:08:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id A7C9910A10EC; Thu, 30 Aug 2018 15:08:02 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9696210A10EB for ; Thu, 30 Aug 2018 15:08:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 34BE885009 for ; Thu, 30 Aug 2018 15:08:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8435E16251 for ; Thu, 30 Aug 2018 15:08:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7UF81WH079899 for ; Thu, 30 Aug 2018 15:08:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7UF81lt079888 for fs@FreeBSD.org; Thu, 30 Aug 2018 15:08:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230937] nfsuserd - missing endpwent() and endgrent() before daemon() & fork() Date: Thu, 30 Aug 2018 15:08:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: markj@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 15:08:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230937 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fs@FreeBSD.org |markj@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 30 17:13:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B0A910A451F for ; Thu, 30 Aug 2018 17:13:46 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B4458A935 for ; Thu, 30 Aug 2018 17:13:46 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: by mail-pf1-x441.google.com with SMTP id u24-v6so4157406pfn.13 for ; Thu, 30 Aug 2018 10:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=richardelling.com; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=EGSGA8Ru8xsRnAULkIZALCQayEYQapw0SVxJNwOXmOo=; b=RkdJRt/6Y3xrxQPLrqWvf/vuI/4TXsGaQvvVYC84mmtjvgqyZFhh0cyZqQVkha+ebr CV/Yb2R44T0aLwqOr2Hvym8QmOPYyaMWaM3AnNxE57AcZKcWLnsLVf8z4VVCcI0/tkRA 7FbXzr32SdXWaj4Or7rhN3yg4+ckSVFFWyE6s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=EGSGA8Ru8xsRnAULkIZALCQayEYQapw0SVxJNwOXmOo=; b=FVtzcKafe+039TGayQLdwSrby1oJjzl3VfazBrm8FizE9aXvjiAwKFPo7Z0m4mcNbj gTIiX2hhAVagT9+OsgzWLbAtBwTE9xCRYobgnEF6BsmngSc89uccoiErJ0OxUhfOV/f+ rnMliXZngZb4R8G6s5rTKjZXyc+gKadFVksLOchJ10Ikm6hrAy9REDEFQkGw0DR9u7X8 GHOLslI/iSX6MbTLv43fjYAlDFQH/AwnBcfMH1b8hbLnz2gfFBVO5woVni2lviSxY6MR j230fTDmCvIAMcn3/lfu39QYdOKabRI6L/XMdLHTiemcSiAZU3DOhSQFYY3rbhHqYTyK /dmA== X-Gm-Message-State: APzg51Bu3Vgya7w+tJHsnDtryelFT/OKjAFekP/jCEIQ52nVqk9VqKlD cLfuVOxOCxUHsCN+rJLAdTWZc7DfE9Y= X-Google-Smtp-Source: ANB0VdbJiujtgZ9YSohffPdPwF6FaOh+bHi/PMolSeOSGko9jmDZLlbBvqLnM5y3EZwxoIaQei8fyA== X-Received: by 2002:a63:d09:: with SMTP id c9-v6mr10737930pgl.314.1535649224914; Thu, 30 Aug 2018 10:13:44 -0700 (PDT) Received: from [172.20.10.3] ([172.56.41.147]) by smtp.gmail.com with ESMTPSA id b18-v6sm15196060pgk.15.2018.08.30.10.13.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 10:13:44 -0700 (PDT) From: Richard Elling Message-Id: <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: [developer] Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Date: Thu, 30 Aug 2018 10:13:41 -0700 In-Reply-To: <20180830144004.GD15740@raichu> Cc: Paul , freebsd-fs@freebsd.org, developer@open-zfs.org To: openzfs-developer References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> <20180830144004.GD15740@raichu> X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 17:13:47 -0000 Hi Mark,=20 yes, this is the change I've tested on ZoL. It is a trivial, low-risk = change that is needed to restore the=20 previous behaviour. -- richard > On Aug 30, 2018, at 7:40 AM, Mark Johnston wrote: >=20 > On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: >> 30 August 2018, 00:22:14, by "Mark Johnston" : >>=20 >>> On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: >>>> Hello team, >>>>=20 >>>>=20 >>>> It seems like a commit on Mar 23 introduced a bug: if during = execution of arc_adjust() >>>> target is reached after MRU is evicted current code continues = evicting MFU. Before said >>>> commit, on the step prior to MFU eviction, target value was = recalculated as: >>>>=20 >>>> target =3D arc_size - arc_c; >>>>=20 >>>> arc_size here is a global variable that was being updated = accordingly, during MRU eviction, >>>> hence this expression, resulted in zero or negative target if MRU = eviction was enough >>>> to reach the original goal. >>>>=20 >>>> Modern version uses cached value of arc_size, called asize: >>>>=20 >>>> target =3D asize - arc_c; >>>>=20 >>>> Because asize stays constant during execution of whole body of = arc_adjust() it means that >>>> above expression will always be evaluated to value > 0, causing MFU = to be evicted every=20 >>>> time, even if MRU eviction has reached the goal already. Because of = the difference in=20 >>>> nature of MFU and MRU, globally it leads to eventual reduction of = amount of MFU in ARC=20 >>>> to dramatic numbers. >>>=20 >>> Hi Paul, >>>=20 >>> Your analysis does seem right to me. I cc'ed the openzfs mailing = list >>> so that an actual ZFS expert can chime in; it looks like this = behaviour >>> is consistent between FreeBSD, illumos and ZoL. >>>=20 >>> Have you already tried the obvious "fix" of subtracting = total_evicted >>> from the MFU target? >>=20 >> We are going to apply the asize patch (plus the ameta, as suggested = by Richard) and reboot=20 >> one of our production servers this night or the following. >=20 > Just to be explicit, are you testing something equivalent to the patch > at the end of this email? >=20 >> Then we have to wait a few days and observer the ARC behaviour. >=20 > Thanks! Please let us know how it goes: we're preparing to release > FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon = as > possible. >=20 > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c = b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > index 1387925c4607..882c04dba50a 100644 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > @@ -4446,6 +4446,12 @@ arc_adjust(void) > arc_adjust_impl(arc_mru, 0, target, = ARC_BUFC_METADATA); > } >=20 > + /* > + * Re-sum ARC stats after the first round of evictions. > + */ > + asize =3D aggsum_value(&arc_size); > + ameta =3D aggsum_value(&arc_meta_used); > + > /* > * Adjust MFU size > * >=20 > ------------------------------------------ > openzfs: openzfs-developer > Permalink: = https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M1c45cd091= 14d2ce2e8c9dd26 = > Delivery options: = https://openzfs.topicbox.com/groups/developer/subscription = From owner-freebsd-fs@freebsd.org Thu Aug 30 21:27:37 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C3CAEF9378 for ; Thu, 30 Aug 2018 21:27:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3ED157354D for ; Thu, 30 Aug 2018 21:27:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id EF614EF9377; Thu, 30 Aug 2018 21:27:36 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD5E3EF9374 for ; Thu, 30 Aug 2018 21:27:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5AC6973547 for ; Thu, 30 Aug 2018 21:27:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 822D019798 for ; Thu, 30 Aug 2018 21:27:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7ULRZCq016829 for ; Thu, 30 Aug 2018 21:27:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7ULRZcL016828 for fs@FreeBSD.org; Thu, 30 Aug 2018 21:27:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Thu, 30 Aug 2018 21:27:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 21:27:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #10 from Dimitry Andric --- Unfortunately I had a crash again, this time with the clang700-import branc= h at r338392, which has been sync'd with head as of r338391, so after r338193. Unread portion of the kernel message buffer: shared lock of (lockmgr) ufs @ /usr/src/sys/kern/vfs_lookup.c:671 while exclusively locked from /usr/src/sys/kern/vfs_subr.c:2590 panic: excl->share cpuid =3D 3 time =3D 1535629184 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000059c= 0e0 vpanic() at vpanic+0x1b4/frame 0xfffffe000059c140 panic() at panic+0x43/frame 0xfffffe000059c1a0 witness_checkorder() at witness_checkorder+0xbc4/frame 0xfffffe000059c230 lockmgr_slock_hard() at lockmgr_slock_hard+0x6e/frame 0xfffffe000059c2c0 __lockmgr_args() at __lockmgr_args+0x758/frame 0xfffffe000059c360 ffs_lock() at ffs_lock+0xa5/frame 0xfffffe000059c3b0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x92/frame 0xfffffe000059c3e0 _vn_lock() at _vn_lock+0x65/frame 0xfffffe000059c440 lookup() at lookup+0x100/frame 0xfffffe000059c4e0 namei() at namei+0x4bd/frame 0xfffffe000059c5a0 unp_connectat() at unp_connectat+0x155/frame 0xfffffe000059c830 soconnectat() at soconnectat+0xe8/frame 0xfffffe000059c880 clnt_vc_create() at clnt_vc_create+0x349/frame 0xfffffe000059c9c0 clnt_reconnect_call() at clnt_reconnect_call+0x20b/frame 0xfffffe000059ca70 newnfs_request() at newnfs_request+0x8fc/frame 0xfffffe000059cbe0 nfsrv_getuser() at nfsrv_getuser+0x129/frame 0xfffffe000059cd80 nfsrv_getgrpscred() at nfsrv_getgrpscred+0xca/frame 0xfffffe000059cdd0 nfsd_excred() at nfsd_excred+0x10e/frame 0xfffffe000059cdf0 nfsrvd_dorpc() at nfsrvd_dorpc+0x130f/frame 0xfffffe000059cfd0 nfssvc_program() at nfssvc_program+0x527/frame 0xfffffe000059d190 svc_run_internal() at svc_run_internal+0x9f9/frame 0xfffffe000059d2d0 svc_run() at svc_run+0x1ee/frame 0xfffffe000059d330 nfsrvd_nfsd() at nfsrvd_nfsd+0x356/frame 0xfffffe000059d490 nfssvc_nfsd() at nfssvc_nfsd+0x57a/frame 0xfffffe000059d960 sys_nfssvc() at sys_nfssvc+0xcf/frame 0xfffffe000059d980 amd64_syscall() at amd64_syscall+0x28a/frame 0xfffffe000059dab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe000059dab0 --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip =3D 0x8002dee8a, rsp =3D 0x7fffffffe518, rbp =3D 0x7fffffffe7b0 --- KDB: enter: panic __curthread () at ./machine/pcpu.h:230 230 ./machine/pcpu.h: No such file or directory. (kgdb) bt #0 __curthread () at ./machine/pcpu.h:230 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:366 #2 0xffffffff8044f16b in db_dump (dummy=3D, dummy2=3D, dummy3=3D, dummy4=3D) at /usr/src/sys/ddb/db_command.c:574 #3 0xffffffff8044ef39 in db_command (last_cmdp=3D, cmd_table=3D, dopager=3D1) at /usr/src/sys/ddb/db_command.c:= 481 #4 0xffffffff8044ecb4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534 #5 0xffffffff80451e6f in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:252 #6 0xffffffff80be47f5 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:693 #7 0xffffffff8106d0c5 in trap (frame=3D0xfffffe000059c010) at /usr/src/sys/amd64/amd64/trap.c:619 #8 #9 kdb_enter (why=3D0xffffffff812f5de0 "panic", msg=3D) at /usr/src/sys/kern/subr_kdb.c:479 #10 0xffffffff80b9c5c1 in vpanic (fmt=3D, ap=3D0xfffffe00005= 9c180) at /usr/src/sys/kern/kern_shutdown.c:861 #11 0xffffffff80b9c353 in panic (fmt=3D0xffffffff81e8e998 "B\267+\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:799 #12 0xffffffff80c04114 in witness_checkorder (lock=3D, flags=3D, file=3D, line=3D671, interlock=3D) at /usr/src/sys/kern/subr_witness.c:1176 #13 0xffffffff80b70fae in lockmgr_slock_hard (lk=3D, flags=3D2106368, ilk=3D0xfffff8000422c278, file=3D, line=3D<= optimized out>, lwa=3D) at /usr/src/sys/kern/kern_lock.c:567 #14 0xffffffff80b71f48 in __lockmgr_args (lk=3D, flags=3D, ilk=3D0xfffff8000422c278, wmesg=3D, pri=3D, timo=3D, file=3D0xffffffff812f19eb "/usr/src/sys/kern/vfs_lookup.c", line=3D671)= at /usr/src/sys/kern/kern_lock.c:1195 #15 0xffffffff80eb46e5 in _lockmgr_args (lk=3D0xfffff8000422c248, flags=3D2= 106368, ilk=3D, wmesg=3D, prio=3D0, timo=3D0, file=3D= , line=3D18) at /usr/src/sys/sys/lockmgr.h:104 #16 ffs_lock (ap=3D0xfffffe000059c3f0) at /usr/src/sys/ufs/ffs/ffs_vnops.c:= 428 #17 0xffffffff811ec2c2 in VOP_LOCK1_APV (vop=3D0xffffffff81b635e0 , a=3D0xfffffe000059c3f0) at vnode_if.c:2087 #18 0xffffffff80c7e645 in VOP_LOCK1 (vp=3D, flags=3D2106368, file=3D0xffffffff812f19eb "/usr/src/sys/kern/vfs_lookup.c", line=3D671) at ./vnode_if.h:859 #19 _vn_lock (vp=3D0xfffff8000422c1e0, flags=3D2106368, file=3D, line=3D) at /usr/src/sys/kern/vfs_vnops.c:1531 #20 0xffffffff80c613c0 in lookup (ndp=3D0xfffffe000059c5b0) at /usr/src/sys/kern/vfs_lookup.c:669 #21 0xffffffff80c60ead in namei (ndp=3D0xfffffe000059c5b0) at /usr/src/sys/kern/vfs_lookup.c:450 #22 0xffffffff80c431b5 in unp_connectat (fd=3D, so=3D, nam=3D, td=3D0xfffff80003316580) at /usr/src/sys/kern/uipc_usrreq.c:1554 #23 0xffffffff80c358b8 in soconnectat (fd=3D, so=3D, nam=3D0x1030000, td=3D0xffffffff80c0327a ) at /usr/src/sys/kern/uipc_socket.c:1230 #24 0xffffffff80e507a9 in clnt_vc_create (so=3D0xfffff80004cb96d0, raddr=3D0xfffff800041c7420, prog=3D553713921, vers=3D1, sendsz=3D4096, recv= sz=3D4096, intrflag=3D0) at /usr/src/sys/rpc/clnt_vc.c:159 #25 0xffffffff80e4fa3b in clnt_reconnect_connect (cl=3D0xfffff8000386f5c0) = at /usr/src/sys/rpc/clnt_rc.c:193 #26 clnt_reconnect_call (cl=3D0xfffff8000386f5c0, ext=3D0xfffffe000059cac0,= proc=3D1, args=3D0xfffff8000480a200, resultsp=3D, utimeout=3D...) at /usr/src/sys/rpc/clnt_rc.c:265 #27 0xffffffff80a65cac in newnfs_request (nd=3D0xfffffe000059cc38, nmp=3D0x= 0, clp=3D0x0, nrp=3D0xffffffff82080298 , vp=3D0x0, td=3D0x= 0, cred=3D0xfffff80032c74700, prog=3D553713921, vers=3D1, retsum=3D0x0, toplevel=3D0, xidp=3D0x0, dssep=3D0x0) at /usr/src/sys/fs/nfs/nfs_commonkrpc.c:818 #28 0xffffffff80a6fa79 in nfsrv_getuser (procnum=3D1, uid=3D, gid=3D, name=3D0x0, p=3D0x0) at /usr/src/sys/fs/nfs/nfs_commonsubs.c:3616 #29 0xffffffff80a6fb9a in nfsrv_getgrpscred (oldcred=3D0xfffff80032c74800) = at /usr/src/sys/fs/nfs/nfs_commonsubs.c:3150 #30 0xffffffff80ad16be in nfsd_excred (nd=3D0xfffffe000059d008, exp=3D, credanon=3D0xfffffe000059bfd0) at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:2932 #31 0xffffffff80aaa79f in nfsrvd_compound (nd=3D, isdgram=3D, tag=3D0x10 , taglen=3D, minorvers=3D, p=3D) at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:1008 #32 nfsrvd_dorpc (nd=3D0xfffffe000059d008, isdgram=3D0, tag=3D0x10 , taglen=3D7, minorvers=3D2, p=3D0xfffff80003= 316580) at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:560 #33 0xffffffff80abe597 in nfs_proc (xid=3D, xprt=3D, nd=3D, rpp=3D) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:387 #34 nfssvc_program (rqst=3D0xfffff80096076800, xprt=3D0xfffff8000369a800) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:272 #35 0xffffffff80e57789 in svc_executereq (rqstp=3D) at /usr/src/sys/rpc/svc.c:1031 #36 svc_run_internal (grp=3D, ismaster=3D1) at /usr/src/sys/rpc/svc.c:1306 #37 0xffffffff80e56cce in svc_run (pool=3D) at /usr/src/sys/rpc/svc.c:1385 #38 0xffffffff80abebf6 in nfsrvd_nfsd (td=3D, args=3D0xfffffe000059d520) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:542 #39 0xffffffff80ad4b0a in nfssvc_nfsd (td=3D0xfffff80003316580, uap=3D) at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:3451 #40 0xffffffff80e3572f in sys_nfssvc (td=3D0xfffff80003316580, uap=3D0xfffff80003316940) at /usr/src/sys/nfs/nfs_nfssvc.c:111 #41 0xffffffff8106de6a in syscallenter (td=3D0xfffff80003316580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #42 amd64_syscall (td=3D0xfffff80003316580, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1043 #43 #44 0x00000008002dee8a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffe518 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 30 21:33:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1AFCEF961A for ; Thu, 30 Aug 2018 21:33:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 64AFA738EF for ; Thu, 30 Aug 2018 21:33:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 20E9BEF9619; Thu, 30 Aug 2018 21:33:45 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F9F0EF9617 for ; Thu, 30 Aug 2018 21:33:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A9DCA738ED for ; Thu, 30 Aug 2018 21:33:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 0F4D6198F2 for ; Thu, 30 Aug 2018 21:33:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7ULXh5G031445 for ; Thu, 30 Aug 2018 21:33:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7ULXhnj031444 for fs@FreeBSD.org; Thu, 30 Aug 2018 21:33:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Thu, 30 Aug 2018 21:33:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 21:33:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #11 from Dimitry Andric --- Created attachment 196722 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D196722&action= =3Dedit Text dump of newnfs_request -> witness panic 2 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 30 22:29:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA8DBEFA998 for ; Thu, 30 Aug 2018 22:29:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4E60475441 for ; Thu, 30 Aug 2018 22:29:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 09FDDEFA997; Thu, 30 Aug 2018 22:29:10 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECC1FEFA996 for ; Thu, 30 Aug 2018 22:29:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 907737543D for ; Thu, 30 Aug 2018 22:29:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B793C19FEB for ; Thu, 30 Aug 2018 22:29:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7UMT8eY059156 for ; Thu, 30 Aug 2018 22:29:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7UMT8oB059154 for fs@FreeBSD.org; Thu, 30 Aug 2018 22:29:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Thu, 30 Aug 2018 22:29:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 22:29:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #12 from Rick Macklem --- Are you sure you replaced the nfsuserd binary in /usr/sbin after the post-r338193 upgrade? (It was changes to nfsuserd.c that got reverted, not anything in the kernel.) rick --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 31 00:34:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEA9CEFD228 for ; Fri, 31 Aug 2018 00:34:44 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: from malone.genomecenter.ucdavis.edu (malone.genomecenter.ucdavis.edu [128.120.243.192]) by mx1.freebsd.org (Postfix) with ESMTP id A0751798A7 for ; Fri, 31 Aug 2018 00:34:44 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: by malone.genomecenter.ucdavis.edu (Postfix, from userid 20141) id D843620189; Thu, 30 Aug 2018 17:34:36 -0700 (PDT) Date: Thu, 30 Aug 2018 17:34:36 -0700 From: "M. Casper Lewis" To: freebsd-fs@freebsd.org Subject: Failing ZFS log devices/panic Message-ID: <20180831003436.GW1473@genomecenter.ucdavis.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 00:34:45 -0000 Greetings, We are having an issue with stability problems on one of our ZFS fileservers. The system will run fine for a few days, but gradually report the log the log devices as failing, and then eventually panic. After several rounds of this, we finally removed the log devices and the machine has not panicked since. We have tried several different types of SSD (both datacenter and non) and the issue happens with all of them. When queried with the vendor tools, the drives all report themselves healthy, and after a reboot they all report healthy as well. The same SSDs are serving as cache devices without issue. This is FreeBSD 11.2-RELEASE-p2 #2 r337991 Here is a backtrace: KDB: stack backtrace: #0 0xffffffff80b3d3c7 at kdb_backtrace+0x67 #1 0xffffffff80af6a37 at vpanic+0x177 #2 0xffffffff80af68b3 at panic+0x43 #3 0xffffffff80deabea at vm_fault_hold+0x244a #4 0xffffffff80de8755 at vm_fault+0x75 #5 0xffffffff80f7810c at trap_pfault+0x14c #6 0xffffffff80f777d7 at trap+0x2c7 #7 0xffffffff80f5740c at calltrap+0x8 #8 0xffffffff823442b9 at zfs_log_write+0x169 #9 0xffffffff82350a30 at zfs_freebsd_write+0xb50 #10 0xffffffff810faea3 at VOP_WRITE_APV+0x103 #11 0xffffffff80a32ffb at nfsvno_write+0x12b #12 0xffffffff80a2af45 at nfsrvd_write+0x4a5 #13 0xffffffff80a1866b at nfsrvd_dorpc+0x11bb #14 0xffffffff80a287e7 at nfssvc_program+0x557 #15 0xffffffff80d6bcd9 at svc_run_internal+0xe09 #16 0xffffffff80d6c18b at svc_thread_start+0xb #17 0xffffffff80aba073 at fork_exit+0x83 Any suggestions on what to try next? We are at a loss as to why the devices are being marked failed when they clearly are not. -- M. Casper Lewis | mclewis@ucdavis.edu Systems Administrator | Voice: (530) 754-7978 Genome Center | University of California, Davis | From owner-freebsd-fs@freebsd.org Fri Aug 31 00:40:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6AC10EFD391 for ; Fri, 31 Aug 2018 00:40:33 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from mail.agc.net.au (mail.evps.com.au [27.121.114.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E2A0C799A8 for ; Fri, 31 Aug 2018 00:40:31 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id E74974719F72; Fri, 31 Aug 2018 10:40:07 +1000 (AEST) Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 6dfm6I1FfCpX; Fri, 31 Aug 2018 10:40:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id 513884719F76; Fri, 31 Aug 2018 10:40:04 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.agc.net.au Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id JkvRchB4nQ2J; Fri, 31 Aug 2018 10:40:03 +1000 (AEST) Received: from mail.agc.net.au (mail.agc.net.au [27.121.114.102]) by mail.agc.net.au (Postfix) with ESMTP id EBA134719F72; Fri, 31 Aug 2018 10:40:01 +1000 (AEST) Date: Fri, 31 Aug 2018 10:39:59 +1000 (AEST) From: Grant Gray To: "M. Casper Lewis" Cc: freebsd-fs@freebsd.org Message-ID: <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> In-Reply-To: <20180831003436.GW1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> Subject: Re: Failing ZFS log devices/panic MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [27.121.114.102] X-Mailer: Zimbra 8.8.8_GA_1728 (ZimbraWebClient - FF62 (Linux)/8.8.8_GA_1703) Thread-Topic: Failing ZFS log devices/panic Thread-Index: 0YTtrsCb3C/haCUbtFppmqe16fLqwg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 00:40:33 -0000 SAS? SATA? Make/model? HBA? Are you mixing SAS and SATA on the same bus? Pool configuration? ----- On 31 Aug, 2018, at 10:34 AM, M. Casper Lewis mclewis@genomecenter.ucdavis.edu wrote: > Greetings, > > We are having an issue with stability problems on one of our ZFS fileservers. > The system will run fine for a few days, but gradually report the log > the log devices as failing, and then eventually panic. After several rounds > of this, we finally removed the log devices and the machine has not > panicked since. > > We have tried several different types of SSD (both datacenter and non) and > the issue happens with all of them. When queried with the vendor tools, the > drives all report themselves healthy, and after a reboot they all report > healthy as well. > > The same SSDs are serving as cache devices without issue. > > This is FreeBSD 11.2-RELEASE-p2 #2 r337991 > > Here is a backtrace: > > KDB: stack backtrace: > #0 0xffffffff80b3d3c7 at kdb_backtrace+0x67 > #1 0xffffffff80af6a37 at vpanic+0x177 > #2 0xffffffff80af68b3 at panic+0x43 > #3 0xffffffff80deabea at vm_fault_hold+0x244a > #4 0xffffffff80de8755 at vm_fault+0x75 > #5 0xffffffff80f7810c at trap_pfault+0x14c > #6 0xffffffff80f777d7 at trap+0x2c7 > #7 0xffffffff80f5740c at calltrap+0x8 > #8 0xffffffff823442b9 at zfs_log_write+0x169 > #9 0xffffffff82350a30 at zfs_freebsd_write+0xb50 > #10 0xffffffff810faea3 at VOP_WRITE_APV+0x103 > #11 0xffffffff80a32ffb at nfsvno_write+0x12b > #12 0xffffffff80a2af45 at nfsrvd_write+0x4a5 > #13 0xffffffff80a1866b at nfsrvd_dorpc+0x11bb > #14 0xffffffff80a287e7 at nfssvc_program+0x557 > #15 0xffffffff80d6bcd9 at svc_run_internal+0xe09 > #16 0xffffffff80d6c18b at svc_thread_start+0xb > #17 0xffffffff80aba073 at fork_exit+0x83 > > Any suggestions on what to try next? We are at a loss as to why the devices > are being marked failed when they clearly are not. > > -- > M. Casper Lewis | mclewis@ucdavis.edu > Systems Administrator | Voice: (530) 754-7978 > Genome Center | > University of California, Davis | > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Fri Aug 31 01:16:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02711EFE404 for ; Fri, 31 Aug 2018 01:16:08 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: from malone.genomecenter.ucdavis.edu (malone.genomecenter.ucdavis.edu [128.120.243.192]) by mx1.freebsd.org (Postfix) with ESMTP id 87C987AE33 for ; Fri, 31 Aug 2018 01:16:08 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: by malone.genomecenter.ucdavis.edu (Postfix, from userid 20141) id E2DB620189; Thu, 30 Aug 2018 18:16:07 -0700 (PDT) Date: Thu, 30 Aug 2018 18:16:07 -0700 From: "M. Casper Lewis" To: Grant Gray Cc: freebsd-fs@freebsd.org Subject: Re: Failing ZFS log devices/panic Message-ID: <20180831011607.GX1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 01:16:09 -0000 On Fri, Aug 31, 2018 at 10:39:59AM +1000, Grant Gray wrote: > SAS? SATA? Make/model? HBA? SATA. Currently Intel SSDSC2BB24 but like I said, we've tried several. HBA is a Supermicro AOC-S3008L-L8I (reports itself as a LSI3008-IT). Firmware has been updated to latest on all controllers. We've purchased an LSI 9300-8i and scheduled downtime to install it, to see if we can rule out a bad controller. > Are you mixing SAS and SATA on the same bus? No, the pool is on separate HBAs. Only the cache and log devices are attached to this HBA. > Pool configuration? 8 11-disk raidz3s. We've tried various numbers and configurations of log devices, anywhere from 4 to 8 devices, both mirrored and unmirrored. -- M. Casper Lewis | mclewis@ucdavis.edu Systems Administrator | Voice: (530) 754-7978 Genome Center | University of California, Davis | From owner-freebsd-fs@freebsd.org Fri Aug 31 01:35:29 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1EF97EFEA87 for ; Fri, 31 Aug 2018 01:35:29 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from mail.agc.net.au (mail.evps.com.au [27.121.114.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 832917B921 for ; Fri, 31 Aug 2018 01:35:27 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id C3A034719F72; Fri, 31 Aug 2018 11:35:15 +1000 (AEST) Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id vK0QaaZ7nOR1; Fri, 31 Aug 2018 11:35:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id 2508A4719F76; Fri, 31 Aug 2018 11:35:13 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.agc.net.au Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Jz1k2IIdZmtw; Fri, 31 Aug 2018 11:35:13 +1000 (AEST) Received: from mail.agc.net.au (mail.agc.net.au [27.121.114.102]) by mail.agc.net.au (Postfix) with ESMTP id E0AE84719F72; Fri, 31 Aug 2018 11:35:12 +1000 (AEST) Date: Fri, 31 Aug 2018 11:35:12 +1000 (AEST) From: Grant Gray To: "M. Casper Lewis" Cc: freebsd-fs Message-ID: <1121631966.257797.1535679312447.JavaMail.zimbra@grantgray.id.au> In-Reply-To: <20180831011607.GX1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> <20180831011607.GX1473@genomecenter.ucdavis.edu> Subject: Re: Failing ZFS log devices/panic MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [27.121.114.102] X-Mailer: Zimbra 8.8.8_GA_1728 (ZimbraWebClient - FF62 (Linux)/8.8.8_GA_1703) Thread-Topic: Failing ZFS log devices/panic Thread-Index: Y9NH0oaWIVPIVIcEYmh0PDvqJFoPqg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 01:35:29 -0000 I'm going to defer to people with more experience with this hardware combination than myself, but I will say I've had compatibility issues with SATA SSD's on LSI SAS controllers in the past. In my case this manifested as the SSD's disappearing off the bus and not returning. I've currently got an issue between some HGST SATA disks and a SAS3008 HBA where mixing SAS and SATA disks on the same port results in intermittent I/O errors on the HGST SATA disks, but not other SAS devices. If feasible, it may be useful to move the SSD's onto a plain SATA controller as a diagnostic step. ----- On 31 Aug, 2018, at 11:16 AM, M. Casper Lewis mclewis@genomecenter.ucdavis.edu wrote: > On Fri, Aug 31, 2018 at 10:39:59AM +1000, Grant Gray wrote: >> SAS? SATA? Make/model? HBA? > > SATA. Currently Intel SSDSC2BB24 but like I said, we've tried several. > > HBA is a Supermicro AOC-S3008L-L8I (reports itself as a LSI3008-IT). > > Firmware has been updated to latest on all controllers. > > We've purchased an LSI 9300-8i and scheduled downtime to install it, to > see if we can rule out a bad controller. > >> Are you mixing SAS and SATA on the same bus? > > No, the pool is on separate HBAs. Only the cache and log devices are > attached to this HBA. > >> Pool configuration? > > 8 11-disk raidz3s. We've tried various numbers and configurations of > log devices, anywhere from 4 to 8 devices, both mirrored and unmirrored. > > -- > M. Casper Lewis | mclewis@ucdavis.edu > Systems Administrator | Voice: (530) 754-7978 > Genome Center | > University of California, Davis | From owner-freebsd-fs@freebsd.org Fri Aug 31 02:58:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA2CFF6E9EE for ; Fri, 31 Aug 2018 02:58:06 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: from malone.genomecenter.ucdavis.edu (malone.genomecenter.ucdavis.edu [128.120.243.192]) by mx1.freebsd.org (Postfix) with ESMTP id 74BF97DF8F for ; Fri, 31 Aug 2018 02:58:06 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: by malone.genomecenter.ucdavis.edu (Postfix, from userid 20141) id C93F920189; Thu, 30 Aug 2018 19:58:05 -0700 (PDT) Date: Thu, 30 Aug 2018 19:58:05 -0700 From: "M. Casper Lewis" To: Grant Gray Cc: freebsd-fs@freebsd.org Subject: Re: Failing ZFS log devices/panic Message-ID: <20180831025805.GY1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> <20180831011607.GX1473@genomecenter.ucdavis.edu> <1121631966.257797.1535679312447.JavaMail.zimbra@grantgray.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1121631966.257797.1535679312447.JavaMail.zimbra@grantgray.id.au> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 02:58:07 -0000 On Fri, Aug 31, 2018 at 11:35:12AM +1000, Grant Gray wrote: > I'm going to defer to people with more experience with this hardware > combination than myself, but I will say I've had compatibility issues with > SATA SSD's on LSI SAS controllers in the past. In my case this manifested > as the SSD's disappearing off the bus and not returning. And after a reboot they are back? Interesting. This is not what we are seeing, rather an accumulation of errors until the device is faulted, despite both SMART and the vendor utility reporting a healthy drive. > I've currently got an issue between some HGST SATA disks and a SAS3008 HBA > where mixing SAS and SATA disks on the same port results in intermittent > I/O errors on the HGST SATA disks, but not other SAS devices. This sounds like what we are seeing, but we're not mixing SAS and SATA. > If feasible, it may be useful to move the SSD's onto a plain SATA controller > as a diagnostic step. Certainly a step to consider if our controller swap does not improve the situation. That said, this is a production fileserver with about half a petabyte of Important Data(tm) on it, so experimentation is not exactly the soup of the day. Which is to say we'd like to take this machine down as infrequently as possible. Diagnostic steps sans downtime are preferable. -- M. Casper Lewis | mclewis@ucdavis.edu Systems Administrator | Voice: (530) 754-7978 Genome Center | University of California, Davis | From owner-freebsd-fs@freebsd.org Fri Aug 31 03:04:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15A8DF6EF7C for ; Fri, 31 Aug 2018 03:04:07 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: from malone.genomecenter.ucdavis.edu (malone.genomecenter.ucdavis.edu [128.120.243.192]) by mx1.freebsd.org (Postfix) with ESMTP id A97617E669 for ; Fri, 31 Aug 2018 03:04:06 +0000 (UTC) (envelope-from mclewis@genomecenter.ucdavis.edu) Received: by malone.genomecenter.ucdavis.edu (Postfix, from userid 20141) id 2D1F820189; Thu, 30 Aug 2018 20:04:06 -0700 (PDT) Date: Thu, 30 Aug 2018 20:04:06 -0700 From: "M. Casper Lewis" To: Grant Gray Cc: freebsd-fs@freebsd.org Subject: Re: Failing ZFS log devices/panic Message-ID: <20180831030405.GZ1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> <20180831011607.GX1473@genomecenter.ucdavis.edu> <1121631966.257797.1535679312447.JavaMail.zimbra@grantgray.id.au> <20180831025805.GY1473@genomecenter.ucdavis.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180831025805.GY1473@genomecenter.ucdavis.edu> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 03:04:07 -0000 On Thu, Aug 30, 2018 at 07:58:05PM -0700, M. Casper Lewis wrote: > On Fri, Aug 31, 2018 at 11:35:12AM +1000, Grant Gray wrote: > > I've currently got an issue between some HGST SATA disks and a SAS3008 HBA > > where mixing SAS and SATA disks on the same port results in intermittent > > I/O errors on the HGST SATA disks, but not other SAS devices. > > This sounds like what we are seeing, but we're not mixing SAS and SATA. I forgot to add: Why are the log devices showing problems and not the cache devices? Same drives, same bus. Different access pattern, sure. -- M. Casper Lewis | mclewis@ucdavis.edu Systems Administrator | Voice: (530) 754-7978 Genome Center | University of California, Davis | From owner-freebsd-fs@freebsd.org Fri Aug 31 03:04:18 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B641F6EFA2 for ; Fri, 31 Aug 2018 03:04:18 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from mail.agc.net.au (mail.evps.com.au [27.121.114.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B5787E687 for ; Fri, 31 Aug 2018 03:04:15 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id 9B02C4719F72; Fri, 31 Aug 2018 13:04:02 +1000 (AEST) Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id cJH3anAG_D19; Fri, 31 Aug 2018 13:04:00 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id A23FF4719F76; Fri, 31 Aug 2018 13:04:00 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.agc.net.au Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id kCLKO_IxByGJ; Fri, 31 Aug 2018 13:04:00 +1000 (AEST) Received: from mail.agc.net.au (mail.agc.net.au [27.121.114.102]) by mail.agc.net.au (Postfix) with ESMTP id 3E7394719F72; Fri, 31 Aug 2018 13:04:00 +1000 (AEST) Date: Fri, 31 Aug 2018 13:03:59 +1000 (AEST) From: Grant Gray To: "M. Casper Lewis" Cc: freebsd-fs Message-ID: <19839688.258331.1535684639446.JavaMail.zimbra@grantgray.id.au> In-Reply-To: <20180831025805.GY1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> <20180831011607.GX1473@genomecenter.ucdavis.edu> <1121631966.257797.1535679312447.JavaMail.zimbra@grantgray.id.au> <20180831025805.GY1473@genomecenter.ucdavis.edu> Subject: Re: Failing ZFS log devices/panic MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [27.121.114.102] X-Mailer: Zimbra 8.8.8_GA_1728 (ZimbraWebClient - FF62 (Linux)/8.8.8_GA_1703) Thread-Topic: Failing ZFS log devices/panic Thread-Index: 73tMnzJaCuhk7iuZvfvopQEgOg683w== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 03:04:18 -0000 ----- On 31 Aug, 2018, at 12:58 PM, M. Casper Lewis mclewis@genomecenter.ucdavis.edu wrote: > On Fri, Aug 31, 2018 at 11:35:12AM +1000, Grant Gray wrote: >> I'm going to defer to people with more experience with this hardware >> combination than myself, but I will say I've had compatibility issues with >> SATA SSD's on LSI SAS controllers in the past. In my case this manifested >> as the SSD's disappearing off the bus and not returning. > > And after a reboot they are back? Yep, they would work for 2-3 days at a time before falling off the bus. Can you post any specific I/O errors from your logs? > > Interesting. This is not what we are seeing, rather an accumulation of > errors until the device is faulted, despite both SMART and the vendor > utility reporting a healthy drive. > >> I've currently got an issue between some HGST SATA disks and a SAS3008 HBA >> where mixing SAS and SATA disks on the same port results in intermittent >> I/O errors on the HGST SATA disks, but not other SAS devices. > > This sounds like what we are seeing, but we're not mixing SAS and SATA. > >> If feasible, it may be useful to move the SSD's onto a plain SATA controller >> as a diagnostic step. > > Certainly a step to consider if our controller swap does not improve the > situation. That said, this is a production fileserver with about half a > petabyte of Important Data(tm) on it, so experimentation is not exactly > the soup of the day. Which is to say we'd like to take this machine down > as infrequently as possible. Diagnostic steps sans downtime are > preferable. In that case, it sounds like you need a similarly configured machine you can use to reproduce and resolve the issue in parallel with production work loads. One further point of consideration in the suitability of these SSD's for write-intensive workloads; it isn't too difficult to overwhelm an SSD with synchronous writes, potentially leading to timeouts. Have you experimented with any write-biased SSD's? > > -- > M. Casper Lewis | mclewis@ucdavis.edu > Systems Administrator | Voice: (530) 754-7978 > Genome Center | > University of California, Davis | From owner-freebsd-fs@freebsd.org Fri Aug 31 03:08:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CBDEF71178 for ; Fri, 31 Aug 2018 03:08:59 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from mail.agc.net.au (mail.evps.com.au [27.121.114.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E42F47E81D for ; Fri, 31 Aug 2018 03:08:58 +0000 (UTC) (envelope-from grant@grantgray.id.au) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id 19D714719F72; Fri, 31 Aug 2018 13:08:48 +1000 (AEST) Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id z60UFYnpdyzb; Fri, 31 Aug 2018 13:08:46 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.agc.net.au (Postfix) with ESMTP id 87DA54719F76; Fri, 31 Aug 2018 13:08:46 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.agc.net.au Received: from mail.agc.net.au ([127.0.0.1]) by localhost (mail.agc.net.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XnjN0zJLwQiT; Fri, 31 Aug 2018 13:08:46 +1000 (AEST) Received: from mail.agc.net.au (mail.agc.net.au [27.121.114.102]) by mail.agc.net.au (Postfix) with ESMTP id 452254719F72; Fri, 31 Aug 2018 13:08:46 +1000 (AEST) Date: Fri, 31 Aug 2018 13:08:46 +1000 (AEST) From: Grant Gray To: "M. Casper Lewis" Cc: freebsd-fs Message-ID: <61612853.258505.1535684926024.JavaMail.zimbra@grantgray.id.au> In-Reply-To: <20180831030405.GZ1473@genomecenter.ucdavis.edu> References: <20180831003436.GW1473@genomecenter.ucdavis.edu> <707525919.257415.1535675999891.JavaMail.zimbra@grantgray.id.au> <20180831011607.GX1473@genomecenter.ucdavis.edu> <1121631966.257797.1535679312447.JavaMail.zimbra@grantgray.id.au> <20180831025805.GY1473@genomecenter.ucdavis.edu> <20180831030405.GZ1473@genomecenter.ucdavis.edu> Subject: Re: Failing ZFS log devices/panic MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [27.121.114.102] X-Mailer: Zimbra 8.8.8_GA_1728 (ZimbraWebClient - FF62 (Linux)/8.8.8_GA_1703) Thread-Topic: Failing ZFS log devices/panic Thread-Index: zZmJfkBzzBxrnI9k3QDj3oFyx7Anmw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 03:08:59 -0000 ----- On 31 Aug, 2018, at 1:04 PM, M. Casper Lewis mclewis@genomecenter.ucdavis.edu wrote: > On Thu, Aug 30, 2018 at 07:58:05PM -0700, M. Casper Lewis wrote: >> On Fri, Aug 31, 2018 at 11:35:12AM +1000, Grant Gray wrote: >> > I've currently got an issue between some HGST SATA disks and a SAS3008 HBA >> > where mixing SAS and SATA disks on the same port results in intermittent >> > I/O errors on the HGST SATA disks, but not other SAS devices. >> >> This sounds like what we are seeing, but we're not mixing SAS and SATA. > > I forgot to add: Why are the log devices showing problems and not the > cache devices? Same drives, same bus. Different access pattern, sure. > > -- > M. Casper Lewis | mclewis@ucdavis.edu > Systems Administrator | Voice: (530) 754-7978 > Genome Center | > University of California, Davis | It could be a function of synchronous write load. Log devices must be coherent. Cache devices can back-fill (I don't know how the ZFS algorithm works, but I imagine pages will continue to hit the L1 ARC before and during demotion to the L2 ARC). From owner-freebsd-fs@freebsd.org Fri Aug 31 07:05:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CCD3F75DC8 for ; Fri, 31 Aug 2018 07:05:20 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D2FD185211 for ; Fri, 31 Aug 2018 07:05:19 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-Id:Cc:To:Subject:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=dV5AD2cO8sIY6y2VnOwBI0VYdJgusN1EpltFKgiLM/k=; b=cDuSHsqcQE8rr73/LsqD7pKcwa k2mRSVfSWr9ndtGt79vN8wKtZUD/LlasrUawus7rnWp64Fe4fW8PFMcLKYEaFfFtcr5vCH9Jll/rP dJ5w3wKMVSjp8l+X3f5genqifM52CjSWt3JJMbQVHr5HvKU0T3EJVciwEQMkvgre8lhE=; Received: from [10.10.10.33] (helo=frv33.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1fvdUU-000G2q-Pa for freebsd-fs@freebsd.org; Fri, 31 Aug 2018 10:05:10 +0300 Date: Fri, 31 Aug 2018 10:05:10 +0300 From: Paul Subject: Re[2]: [developer] Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size To: Richard Elling Cc: openzfs-developer , freebsd-fs@freebsd.org, developer@open-zfs.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1535697386.721828038.3yjli8h7@frv33.fwdcdn.com> In-Reply-To: <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> <20180830144004.GD15740@raichu> <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> X-Reply-Action: reply Received: from devgs@ukr.net by frv33.fwdcdn.com; Fri, 31 Aug 2018 10:05:10 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 07:05:20 -0000 I can confirm that the old behaviour is restored. Though it's only 16 hours in we can see a stable, healthy grown of MFU, unlike before the patch. Most importantly the MRU is now pushed below the arc_p limit, that was never the case before patch. Just as an example, the current ARC picture is as such: ARC: 360G Total, 179G MFU, 158G MRU, 361M Anon, 3398M Header, 20G Other And yes, our version of patch was exactly what Mark Johnston has assumed. 30 August 2018, 20:13:47, by "Richard Elling" : > Hi Mark, > yes, this is the change I've tested on ZoL. It is a trivial, low-risk change that is needed to restore the > previous behaviour. -- richard > > > On Aug 30, 2018, at 7:40 AM, Mark Johnston wrote: > > On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: > > > 30 August 2018, 00:22:14, by "Mark Johnston" : > > > > > > > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > > > > > Hello team, > > > > > > > > > > > > > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > > > > > target is reached after MRU is evicted current code continues evicting MFU. Before said > > > > > commit, on the step prior to MFU eviction, target value was recalculated as: > > > > > > > > > > target = arc_size - arc_c; > > > > > > > > > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > > > > > hence this expression, resulted in zero or negative target if MRU eviction was enough > > > > > to reach the original goal. > > > > > > > > > > Modern version uses cached value of arc_size, called asize: > > > > > > > > > > target = asize - arc_c; > > > > > > > > > > Because asize stays constant during execution of whole body of arc_adjust() it means that > > > > > above expression will always be evaluated to value > 0, causing MFU to be evicted every > > > > > time, even if MRU eviction has reached the goal already. Because of the difference in > > > > > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > > > > > to dramatic numbers. > > > > > > > > Hi Paul, > > > > > > > > Your analysis does seem right to me. I cc'ed the openzfs mailing list > > > > so that an actual ZFS expert can chime in; it looks like this behaviour > > > > is consistent between FreeBSD, illumos and ZoL. > > > > > > > > Have you already tried the obvious "fix" of subtracting total_evicted > > > > from the MFU target? > > > > > > We are going to apply the asize patch (plus the ameta, as suggested by Richard) and reboot > > > one of our production servers this night or the following. > > > > Just to be explicit, are you testing something equivalent to the patch > > at the end of this email? > > > > > Then we have to wait a few days and observer the ARC behaviour. > > > > Thanks! Please let us know how it goes: we're preparing to release > > FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon as > > possible. > > > > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > index 1387925c4607..882c04dba50a 100644 > > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > @@ -4446,6 +4446,12 @@ arc_adjust(void) > > arc_adjust_impl(arc_mru, 0, target, ARC_BUFC_METADATA); > > } > > > > + /* > > + * Re-sum ARC stats after the first round of evictions. > > + */ > > + asize = aggsum_value(&arc_size); > > + ameta = aggsum_value(&arc_meta_used); > > + > > /* > > * Adjust MFU size > > * > > > > ------------------------------------------ > > openzfs: openzfs-developer > > Permalink: https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M1c45cd09114d2ce2e8c9dd26 > > Delivery options: https://openzfs.topicbox.com/groups/developer/subscription From owner-freebsd-fs@freebsd.org Fri Aug 31 12:29:32 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F7BDF7F546 for ; Fri, 31 Aug 2018 12:29:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CC0DF709F1 for ; Fri, 31 Aug 2018 12:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8CFA1F7F545; Fri, 31 Aug 2018 12:29:31 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69C74F7F544 for ; Fri, 31 Aug 2018 12:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 05A1A709EC for ; Fri, 31 Aug 2018 12:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 405D7214DE for ; Fri, 31 Aug 2018 12:29:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7VCTUdl048416 for ; Fri, 31 Aug 2018 12:29:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7VCTUDL048415 for fs@FreeBSD.org; Fri, 31 Aug 2018 12:29:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Fri, 31 Aug 2018 12:29:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugzilla.freebsd@omnilan.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 12:29:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 --- Comment #7 from Harald Schmalzbauer --- My ZFS setups suffer from an upstream ACL inheritance bug: https://www.illumos.org/issues/9722 wich is referring https://www.illumos.org/issues/8984 The latter is marked closed/solved but it is only partially solved, like the newer issue describes. I had to give up using samba for environments where NTFS ACLs are crucial. I also gave up deploying new NFSv4 ACL setups. All previous setups were damaged by the mentioned ZFS inheritance bugs. I remember many questions to be clarified regarding d and D flags (and in respect to NTFS conventions also R and W and C), but lost almost all experi= ence results. I can't tell anything about the test failure status! I just can tell that I'm observing issues regading ACL/permission mappings. Time for me to look into tests... I'm really interested in polishing our ACL implementation, but I'm lacking a good portion of standards knowledge (posix) and my lousy C skills require someone else to fix the ZFS inheritance bug first. Symlinks are one major issue to discuss regarding inheritance on ZFS =E2=80= =93 that was/is another big problem in my setups. I still have my private ACL setups (partially damaged), which is a collecti= on of special usage scenarios (e.g. users can add files and directories, but c= an't delete anything, even not their own; or shared home directory, which suffers from chmod(2) fallout due to the ACL). So it's my pleasure to test any changes and put some light in uncommon but usefuls corner cases ;-) And I'm willing to do my best to catch up with posix knowledge, but unfortunately I can't afford spending much time :-( My personal estimation was that some very skilled persons would need at lea= st a week to decide/implement and test a new/improved ACL mapping mode. My needs aren't covered by the current ACL/permission mappings =E2=80=93 no= t for NFSv4 only environments and additional CIFS/NTFS bridges were mutual destructive. Thanks, -harry --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 31 15:33:50 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8F9AFCE5F8 for ; Fri, 31 Aug 2018 15:33:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 69CEB7869E for ; Fri, 31 Aug 2018 15:33:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 27C0CFCE5F7; Fri, 31 Aug 2018 15:33:50 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14A95FCE5F6 for ; Fri, 31 Aug 2018 15:33:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A2BB678699 for ; Fri, 31 Aug 2018 15:33:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id EC7A422F29 for ; Fri, 31 Aug 2018 15:33:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7VFXmHf089906 for ; Fri, 31 Aug 2018 15:33:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7VFXm2m089904 for fs@FreeBSD.org; Fri, 31 Aug 2018 15:33:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Fri, 31 Aug 2018 15:33:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 15:33:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 --- Comment #8 from Edward Tomasz Napierala --- Lack of POSIX knowledge shouldn't be a problem, as most of the NFSv4 ACL semantics is not covered by any kind of standard. That's actually a large = part of the problem: it's hard to fix something if we don't know what the 'fixed' state should be. Would it be possible for you to set up a system with a FreeBSD version from before the ZFS ACL changes (before r299448, if I'm reading this right), and= see if the Samba problems are fixed? If they are, perhaps we could just revert= the ZFS ACL changes. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 31 17:13:35 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1864EFD0EB9 for ; Fri, 31 Aug 2018 17:13:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 99CE17C3F2 for ; Fri, 31 Aug 2018 17:13:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 58B7CFD0EB8; Fri, 31 Aug 2018 17:13:34 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36D7FFD0EB7 for ; Fri, 31 Aug 2018 17:13:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C18407C3EA for ; Fri, 31 Aug 2018 17:13:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id EF44123D45 for ; Fri, 31 Aug 2018 17:13:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7VHDWAH035277 for ; Fri, 31 Aug 2018 17:13:32 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7VHDWLs035276 for fs@FreeBSD.org; Fri, 31 Aug 2018 17:13:32 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Fri, 31 Aug 2018 17:13:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugzilla.freebsd@omnilan.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 17:13:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 --- Comment #9 from Harald Schmalzbauer --- (In reply to Edward Tomasz Napierala from comment #8) Unfortunately samba is an extra bunch of issues. The problems which made me stop using samba for distinct jobs in windows environments were not only/directly related to ACLs. I know that some of them have been fixed meanwhile, but I know of others wh= ich are new and unresolved (at least on FreeBSD), so even if you end up with a satisfying NFSv4 ACL overhaul/FreeBSD-standardization, this was only a small part making samba usable in production environments =E2=80=93 which is not = the goal of this problem report (still have to analyze a trace about WindowsServerBackup failing, compared to windows native implementation of SMB2_02, SMB2_10 and another version, but couldn't find time and have to start over again at next attempt =E2=80=93 which probably will never happen because a few hours were= n't enough for me to isolate the crucial area last time). Since it's about a design job rather than fixing, it's best to look at exis= ting conventions for other type of ACLs on other operating systems =E2=80=93 Lin= ux and XFS comes to my mind and of course NTFS on Windows. JFS on AIX was more POSIXA= CL like, missing the majority of the NFSv4/NTFS flags. Stearing the unclear semantics and convention into a NTFS friendly direction would be desireable. But I never read any posixacl standard also :-( That said, I also missed reading any (official) NTFSv5 documents and I don't even know how the inheritance is done in windows. The NTFS versions I start= ed with (NT4) didn't provide inheritance and since then I did observations onl= y. I'd need to setup a complete artificial test environment with at least 3 windows versions, since I currently have absolutely no idea which combinati= on does anything else wrong but ACL related stuff... Let me think about it over the weekend. I planed to abuse one cold-standby system for extended iflib, if_vlan(4) and ctld(8) tests before 12-release, since there are untracked issues as well (btw. the ctld(8) on 11.2 and Server2016-initiators turns out to be jumbo frames related). Let me start materializing this test environment and if things run smooth, = I'll extend the FreeBSD machine to serve a r299448 bhyve VM. For NFSv4 ACLs we'= re not bound to real hardware as far as I can imagine at the moment (while ifl= ib needs...) The ESXi Hypervisor is yet to setup too, and undusting the equipment I have= in mind will most likely discover new problems, and this will bring me into big time troubles.=20 Sorry for my hesitation, but I don't want to promise anything I can't do du= e to real job interference... But I'll try hard and come back if I made first progress. Thanks, -harry P.S.: samba gained a vfs_freebsd module, which is an attempt to fix various problems which never affected me as far as I understood. Haven't found any documentation about this module, so I need to look into that code to make s= ure there's no adaptor to sysutils/libsunacl disturbing test cases. For this reading my C skills should suffice. --=20 You are receiving this mail because: You are the assignee for the bug.=