From owner-svn-src-vendor@freebsd.org Sun Jun 2 04:23:59 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A6D515AA79B; Sun, 2 Jun 2019 04:23:59 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D54B68A502; Sun, 2 Jun 2019 04:23:58 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A912F21E6F; Sun, 2 Jun 2019 04:23:58 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x524Nwgt063194; Sun, 2 Jun 2019 04:23:58 GMT (envelope-from imp@FreeBSD.org) Received: (from imp@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x524NvQX063185; Sun, 2 Jun 2019 04:23:57 GMT (envelope-from imp@FreeBSD.org) Message-Id: <201906020423.x524NvQX063185@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: imp set sender to imp@FreeBSD.org using -f From: Warner Losh Date: Sun, 2 Jun 2019 04:23:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348505 - in vendor/one-true-awk/dist: . bugs-fixed X-SVN-Group: vendor X-SVN-Commit-Author: imp X-SVN-Commit-Paths: in vendor/one-true-awk/dist: . bugs-fixed X-SVN-Commit-Revision: 348505 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: D54B68A502 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2019 04:23:59 -0000 Author: imp Date: Sun Jun 2 04:23:56 2019 New Revision: 348505 URL: https://svnweb.freebsd.org/changeset/base/348505 Log: Import latest one-true-awk from upstream Import git hash 4189ef5d from https://github.com/onetrueawk/awk.git as there's not been a release in a while. Upstream one-true-awk woke-up! Time to catch up. This may also revert FreeBSD changes that we'd placed in the vendor branch in anticipation of their inclusion in upstream. That's not yet the case, and these will be resolved in the merge. See FIXES for a complete list of bugs fixed (starting with the Jun 7, 2018 entry). Added: vendor/one-true-awk/dist/ChangeLog vendor/one-true-awk/dist/LICENSE vendor/one-true-awk/dist/REGRESS (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/ vendor/one-true-awk/dist/bugs-fixed/README vendor/one-true-awk/dist/bugs-fixed/a-format.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/a-format.bad vendor/one-true-awk/dist/bugs-fixed/a-format.ok vendor/one-true-awk/dist/bugs-fixed/concat-assign-same.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/concat-assign-same.bad vendor/one-true-awk/dist/bugs-fixed/concat-assign-same.ok vendor/one-true-awk/dist/bugs-fixed/decr-NF.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/decr-NF.bad vendor/one-true-awk/dist/bugs-fixed/decr-NF.ok vendor/one-true-awk/dist/bugs-fixed/fmt-overflow.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/fmt-overflow.ok vendor/one-true-awk/dist/bugs-fixed/fs-overflow.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/getline-numeric.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/getline-numeric.bad vendor/one-true-awk/dist/bugs-fixed/getline-numeric.in (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/getline-numeric.ok vendor/one-true-awk/dist/bugs-fixed/missing-precision.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/missing-precision.ok vendor/one-true-awk/dist/bugs-fixed/negative-nf.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/negative-nf.ok vendor/one-true-awk/dist/bugs-fixed/nf-self-assign.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/nf-self-assign.bad vendor/one-true-awk/dist/bugs-fixed/nf-self-assign.ok vendor/one-true-awk/dist/bugs-fixed/numeric-fs.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/numeric-fs.ok vendor/one-true-awk/dist/bugs-fixed/numeric-output-seps.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/numeric-output-seps.bad vendor/one-true-awk/dist/bugs-fixed/numeric-output-seps.ok vendor/one-true-awk/dist/bugs-fixed/numeric-rs.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/numeric-rs.bad vendor/one-true-awk/dist/bugs-fixed/numeric-rs.ok vendor/one-true-awk/dist/bugs-fixed/numeric-subsep.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/numeric-subsep.bad (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/numeric-subsep.ok vendor/one-true-awk/dist/bugs-fixed/ofs-rebuild.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/ofs-rebuild.bad vendor/one-true-awk/dist/bugs-fixed/ofs-rebuild.ok vendor/one-true-awk/dist/bugs-fixed/space.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/space.bad vendor/one-true-awk/dist/bugs-fixed/space.ok vendor/one-true-awk/dist/bugs-fixed/split-fs-from-array.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/split-fs-from-array.ok vendor/one-true-awk/dist/bugs-fixed/string-conv.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/string-conv.bad vendor/one-true-awk/dist/bugs-fixed/string-conv.ok vendor/one-true-awk/dist/bugs-fixed/subsep-overflow.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/subsep-overflow.ok vendor/one-true-awk/dist/bugs-fixed/system-status.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/system-status.bad vendor/one-true-awk/dist/bugs-fixed/system-status.ok vendor/one-true-awk/dist/bugs-fixed/unary-plus.awk (contents, props changed) vendor/one-true-awk/dist/bugs-fixed/unary-plus.bad vendor/one-true-awk/dist/bugs-fixed/unary-plus.ok vendor/one-true-awk/dist/proctab.c (contents, props changed) Modified: vendor/one-true-awk/dist/FIXES vendor/one-true-awk/dist/awk.1 vendor/one-true-awk/dist/awk.h vendor/one-true-awk/dist/awkgram.y vendor/one-true-awk/dist/b.c vendor/one-true-awk/dist/lex.c vendor/one-true-awk/dist/lib.c vendor/one-true-awk/dist/main.c vendor/one-true-awk/dist/makefile vendor/one-true-awk/dist/maketab.c vendor/one-true-awk/dist/parse.c vendor/one-true-awk/dist/proto.h vendor/one-true-awk/dist/run.c vendor/one-true-awk/dist/tran.c Added: vendor/one-true-awk/dist/ChangeLog ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/one-true-awk/dist/ChangeLog Sun Jun 2 04:23:56 2019 (r348505) @@ -0,0 +1,245 @@ +2019-05-29 Arnold D. Robbins + + * lib.c (isclvar): Remove check for additional '=' after + first one. No longer needed. + +2019-01-26 Arnold D. Robbins + + * main.c (version): Updated. + +2019-01-25 Arnold D. Robbins + + * run.c (awkgetline): Check for numeric value in all getline + variants. See the numeric-getline.* files in bugs-fixed directory. + +2018-08-29 Arnold D. Robbins + + * REGRESS: Check for existence of a.out. If not there, run + make. Enable core dumps for T.arnold system status test + to work on MacOS X. + +2018-08-22 Arnold D. Robbins + + * awktest.tar (testdir/T.expr): Fix test for unary plus. + +2018-08-22 Arnold D. Robbins + + * REGRESS: Extract tests if necessary, set PATH to include '.'. + * regdir/beebe.tar (Makefile): Fix longwrds test to prefix + sort with LC_ALL=C. + * awktest.tar: Updated from fixed test suite, directory + it extracts is now called 'testdir' to match what's in top-level + REGRESS script. + * regdir: Removed, as Brian wants to keep the test suite in + the tar file. + +2018-08-22 Arnold D. Robbins + + * FIXES, lib.c, run.c, makefile, main.c: Merge from Brian's tree. + * REGRESS: New file, from Brian. + * awktest.tar: Restored from Brian's tree. + +2018-08-22 Arnold D. Robbins + + * awkgram.y (UPLUS): New token. In the grammar, call op1() + with it. + * maketab.c (proc): Add entry for UPLUS. + * run.c (arith): Handle UPLUS. + * main.c (version): Updated. + * bugs-fixed/unary-plus.awk, bugs-fixed/unary-plus.bad, + bugs-fixed/unary-plus.ok: New files. + +2018-08-10 Arnold D. Robbins + + * TODO: Updated. + * awk.1: Improve use of macros, add some additional explanation + in a few places, alphabetize list of variables. + +2018-08-08 Arnold D. Robbins + + * awk.h (Cell): Add new field `fmt' to track xFMT value used + for a string conversion. + [CONVC, CONVO]: New flag macros. + * bugs-fixed/README: Updated. + * bugs-fixed/string-conv.awk, bugs-fixed/string-conv.bad, + bugs-fixed/string-conv.ok: New files. + * main.c (version): Updated. + * proto.h (flags2str): Add declaration. + * tran.c (setfval): Clear CONVC and CONVO flags and set vp->fmt + to NULL. + (setsval): Ditto. Add large comment and new code to manage + correct conversion of number to string based on various flags + and the value of vp->fmt. The idea is to not convert again + if xFMT is the same as before and we're doing the same conversion. + Otherwise, clear the old flags, set the new, and reconvert. + (flags2str): New function. For debug prints and for use from a debugger. + +2018-08-05 Arnold D. Robbins + + Fix filename conflicts in regdir where the only difference was + in letter case. This caused problems on Windows systems. + + * regdir/Compare.T1: Renamed from regdir/Compare.T. + * regdir/t.delete0: Renamed from regdir/t.delete. + * regdir/t.getline1: Renamed from regdir/t.getline. + * regdir/t.redir1: Renamed from regdir/t.redir. + * regdir/t.split1: Renamed from regdir/t.split. + * regdir/t.sub0: Renamed from regdir/t.sub. + * regdir/REGRESS: Adjusted. + +2018-08-04 Arnold D. Robbins + + With scalpel, tweasers, magnifying glass and bated breath, + borrow code from the NetBSD version of nawk to fix the years-old + bug whereby decrementing the value of NF did not change the + record. + + * lib.c (fldbld): Set donerec to 1 when done. + (setlastfld): New function. + * proto.h (setlastfld): Add declaration. + * run.c (copycell): Make code smarter about flags (from NetBSD code). + * tran.c (setfree): New function. + * tran.c (setfval): Normalize negative zero to positive zero. + If setting NF, clear donerec and call setlastfld(). + (setsval): Remove call to save_old_OFS(). If setting OFS, call + recbld(). If setting NF, clear donerec and call setlastfld(). + + As part of the process, revert OFS-related changes of 2018-05-22: + + * awk.h (saveOFS, saveOFSlen, save_old_OFS): Remove declarations. + * lib.c (recbld): Use *OFS instead of saveOFS. + * run.c (saveOFS, saveOFSlen, save_old_OFS): Remove. + * tran.c (syminit): Remove initialization of saveOFS and saveOFSlen. + + General stuff that goes along with all this: + + * bugs-fixed/README: Updated. + * bugs-fixed/decr-NF.awk, bugs-fixed/decr-NF.bad, + bugs-fixed/decr-NF.ok: New files. + * main.c (version): Updated. + * regdir/README.TESTS: Fix awk book title. + * regdir/T.misc: Revise test to match fixed code. + * run.c (format): Increase size of buffer used for %a test. (Unrelated + to NF or OFS, but fixes a compiler complaint.) + +2018-06-07 Arnold D. Robbins + + * regdir/beebe.tar: Fix longwrds.ok so that the test will pass. + The file was incorrectly sorted. + +2018-06-06 Arnold D. Robbins + + * regdir/T.lilly: Fix the bug again in the second instance + of the code. Thanks to BWK for pointing this out. + +2018-05-31 Arnold D. Robbins + + * regdir/T.lilly: Fix a syntax error and ordering bug + in creating the 'foo' file. + +2018-05-23 Arnold D. Robbins + + * awk.1: Remove standalone 'awk' at the top of file, it messed up + the formatting. Arrange built-in variable list in alphabetical + order. + +2018-05-23 Arnold D. Robbins + + * main.c (version): Add my email address and a date so that + users can tell this isn't straight BWK awk. + * README.md: Minor updates. + * TODO: Updated. + +2018-05-22 Arnold D. Robbins + + Add POSIX-required formats %a and %A. + + * run.c (format): Check for %a support in C library. If there, + allow %a and %A as valid formats. + * TODO: Updated. + * bugs-fixed/README: Updated. + * bugs-fixed/a-format.awk, bugs-fixed/a-format.bad, + bugs-fixed/a-format.ok: New files. + +2018-05-22 Arnold D. Robbins + + * FIXES: Restored a line from a much earlier version that + apparently got lost when the dates were reordered. + * TODO: Updated. + +2018-05-22 Arnold D. Robbins + + * README.md: New file. + +2018-05-22 Arnold D. Robbins + + * regdir/echo.c, regdir/time.c: Minor fixes to compile without + warning on current GCC / Linux. + +2018-05-22 Arnold D. Robbins + + * TODO: New file. + +2018-05-22 Arnold D. Robbins + + * makefile (gitadd, gitpush): Remove these targets. They + should not be automated and were incorrect for things that + would be done regularly. + +2018-05-22 Arnold D. Robbins + + Fix nawk so that [[:blank:]] only matches space and tab instead + of any whitespace character, originally made May 10, 2018. + See bugs-fixed/space.awk. + + This appears to have been a thinko on Brian's part. + + * b.c (charclasses): Use xisblank() function for [[:blank:]]. + * bugs-fixed/README: Updated. + * bugs-fixed/space.awk, bugs-fixed/space.bad, + bugs-fixed/space.ok: New files. + +2018-05-22 Arnold D. Robbins + + * .gitignore: New file. + +2018-05-22 Arnold D. Robbins + + Fix nawk to provide reasonable exit status for system(), + a la gawk, originally made March 12, 2016. See + bugs-fixed/system-status.awk. + + * run.c (bltin): For FSYSTEM, use the macros defined for wait(2) + to produce a reasonable exit value, instead of doing a floating-point + division by 256. + * awk.1: Document the return status values. + * bugs-fixed/README: Updated. + * bugs-fixed/system-status.awk, bugs-fixed/system-status.bad, + bugs-fixed/system-status.ok: New files. + +2018-05-22 Arnold D. Robbins + + Bug fix with respect to rebuilding a record, originally + made August 19, 2014. See bugs-fixed/ofs-rebuild.awk. + + * awk.h (saveOFS, saveOFSlen): Declare new variables. + * lib.c (recbld): Use them when rebuilding the record. + * run.c (saveOFS, saveOFSlen): Define new variables. + (save_old_OFS): New function to save OFS aside. + * tran.c (syminit): Initialize saveOFS and saveOFSlen. + (setsval): If setting a field, call save_old_OFS(). + * bugs-fixed/README, bugs-fixed/ofs-rebuild.awk, + bugs-fixed/ofs-rebuild.bad, bugs-fixed/ofs-rebuild.ok: New files. + +2018-05-22 Arnold D. Robbins + + * makefile (YACC): Use bison. + +2018-05-22 Arnold D. Robbins + + * ChangeLog: Created. + * regdir: Created. Based on contents of awktest.a. + * .gitattributes: Created, to preserve CR LF in regdir/t.crlf. + * awktest.a: Removed. + * regdir/T.gawk, regdir/T.latin1: Updated from awktest.tar. + * awktest.tar: Removed. Modified: vendor/one-true-awk/dist/FIXES ============================================================================== --- vendor/one-true-awk/dist/FIXES Sun Jun 2 04:04:21 2019 (r348504) +++ vendor/one-true-awk/dist/FIXES Sun Jun 2 04:23:56 2019 (r348505) @@ -25,6 +25,113 @@ THIS SOFTWARE. This file lists all bug fixes, changes, etc., made since the AWK book was sent to the printers in August, 1987. +May 29,2019: + Fix check for command line arguments to no longer require that + first character after '=' not be another '='. Reverts change of + August 11, 1989. Thanks to GitHub user Jamie Landeg Jones for + pointing out the issue; from Issue #38. + +Apr 7, 2019: + Update awktest.tar(p.50) to use modern options to sort. Needed + for Android development. Thanks to GitHub user mohd-akram (Mohamed + Akram). From Comment #33. + +Mar 12, 2019: + Added very simplistic support for cross-compiling in the + makefile. We are NOT going to go in the direction of the + autotools, though. Thanks to GitHub user nee-san for + the basic change. (Merged from PR #34.) + +Mar 5, 2019: + Added support for POSIX-standard interval expressions (a.k.a. + bounds, a.k.a. repetition expressions) in regular expressions, + backported (via NetBSD) from Apple awk-24 (20070501). + Thanks to Martijn Dekker for the port. + (Merged from PR #30.) + +Mar 3, 2019: + Merge PRs as follows: + #12: Avoid undefined behaviour when using ctype(3) functions in + relex(). Thanks to GitHub user iamleot. + #31: Make getline handle numeric strings, and update FIXES. Thanks + to GitHub user arnoldrobbins + #32: maketab: support build systems with read-only source. Thanks + to GitHub user enh. + +Jan 25, 2019: + Make getline handle numeric strings properly in all cases. + (Thanks, Arnold.) + +Jan 21, 2019: + Merged a number of small fixes from GitHub pull requests. + Thanks to GitHub users Arnold Robbins (arnoldrobbins), + Cody Mello (melloc) and Christoph Junghans (junghans). + PR numbers: 13-21, 23, 24, 27. + +Oct 25, 2018: + Added test in maketab.c to prevent generating a proctab entry + for YYSTYPE_IS_DEFINED. It was harmless but some gcc settings + generated a warning message. Thanks to Nan Xiao for report. + +Aug 27, 2018: + Disallow '$' in printf formats; arguments evaluated in order + and printed in order. + + Added some casts to silence warnings on debugging printfs. + (Thanks, Arnold.) + +Aug 23, 2018: + A long list of fixes courtesy of Arnold Robbins, + to whom profound thanks. + + 1. ofs-rebuild: OFS value used to rebuild the record was incorrect. + Fixed August 19, 2014. Revised fix August 2018. + + 2. system-status: Instead of a floating-point division by 256, use + the wait(2) macros to create a reasonable exit status. + Fixed March 12, 2016. + + 3. space: Use provided xisblank() function instead of ispace() for + matching [[:blank:]]. + + 4. a-format: Add POSIX standard %a and %A to supported formats. Check + at runtime that this format is available. + + 5. decr-NF: Decrementing NF did not change $0. This is a decades-old + bug. There are interactions with the old and new value of OFS as well. + Most of the fix came from the NetBSD awk. + + 6. string-conv: String conversions of scalars were sticky. Once a + conversion to string happened, even with OFMT, that value was used until + a new numeric value was assigned, even if OFMT differed from CONVFMT, + and also if CONVFMT changed. + + 7. unary-plus: Unary plus on a string constant returned the string. + Instead, it should convert the value to numeric and give that value. + + Also added Arnold's tests for these to awktest.tar as T.arnold. + +Aug 15, 2018: + fixed mangled awktest.tar (thanks, Arnold), posted all + current (very minor) fixes to github / onetrueawk + +Jun 7, 2018: + (yes, a long layoff) + Updated some broken tests (beebe.tar, T.lilly) + [thanks to Arnold Robbins] + +Mar 26, 2015: + buffer overflow in error reporting; thanks to tobias ulmer + and john-mark gurney for spotting it and the fix. + +Feb 4, 2013: + cleaned up a handful of tests that didn't seem to actually + test for correct behavior: T.latin1, T.gawk. + +Jan 5, 2013: + added ,NULL initializer to static Cells in run.c; not really + needed but cleaner. Thanks to Michael Bombardieri. + Dec 20, 2012: fiddled makefile to get correct yacc and bison flags. pick yacc (linux) or bison (mac) as necessary. @@ -492,6 +599,8 @@ May 12, 1998: Mar 12, 1998: added -V to print version number and die. + +[notify dave kerns, dkerns@dacsoup.ih.lucent.com] Feb 11, 1998: subtle silent bug in lex.c: if the program ended with a number Added: vendor/one-true-awk/dist/LICENSE ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/one-true-awk/dist/LICENSE Sun Jun 2 04:23:56 2019 (r348505) @@ -0,0 +1,23 @@ +/**************************************************************** +Copyright (C) Lucent Technologies 1997 +All Rights Reserved + +Permission to use, copy, modify, and distribute this software and +its documentation for any purpose and without fee is hereby +granted, provided that the above copyright notice appear in all +copies and that both that the copyright notice and this +permission notice and warranty disclaimer appear in supporting +documentation, and that the name Lucent Technologies or any of +its entities not be used in advertising or publicity pertaining +to distribution of the software without specific, written prior +permission. + +LUCENT DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. +IN NO EVENT SHALL LUCENT OR ANY OF ITS ENTITIES BE LIABLE FOR ANY +SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER +IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, +ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF +THIS SOFTWARE. +****************************************************************/ Added: vendor/one-true-awk/dist/REGRESS ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/one-true-awk/dist/REGRESS Sun Jun 2 04:23:56 2019 (r348505) @@ -0,0 +1,35 @@ +#! /bin/sh + +case `uname` in +CYGWIN) EXE=a.exe ;; +*) EXE=a.out ;; +esac + +if [ ! -f $EXE ] +then + make || exit 1 +fi + +if [ -d testdir ] +then + true # do nothing +elif [ -f awktest.tar ] +then + echo extracting testdir + tar -xpf awktest.tar +else + echo $0: No testdir directory and no awktest.tar to extract it from! >&2 + exit 1 +fi + +cd testdir +pwd +PATH=.:$PATH +export PATH +if (ulimit -c unlimited > /dev/null 2>&1) +then + # Workaround broken default on MacOS X + ulimit -c unlimited +fi + +REGRESS Modified: vendor/one-true-awk/dist/awk.1 ============================================================================== --- vendor/one-true-awk/dist/awk.1 Sun Jun 2 04:04:21 2019 (r348504) +++ vendor/one-true-awk/dist/awk.1 Sun Jun 2 04:23:56 2019 (r348505) @@ -7,7 +7,6 @@ .fi .ft 1 .. -awk .TH AWK 1 .CT 1 files prog_other .SH NAME @@ -36,7 +35,7 @@ awk \- pattern-directed scanning and processing langua scans each input .I file for lines that match any of a set of patterns specified literally in -.IR prog +.I prog or in one or more files specified as .B \-f @@ -53,7 +52,7 @@ The file name .B \- means the standard input. Any -.IR file +.I file of the form .I var=value is treated as an assignment, not a filename, @@ -70,12 +69,12 @@ any number of options may be present. The .B \-F -.IR fs +.I fs option defines the input field separator to be the regular expression -.IR fs. +.IR fs . .PP An input line is normally made up of fields separated by white space, -or by regular expression +or by the regular expression .BR FS . The fields are denoted .BR $1 , @@ -87,7 +86,7 @@ If .BR FS is null, the input line is split into one field per character. .PP -A pattern-action statement has the form +A pattern-action statement has the form: .IP .IB pattern " { " action " } .PP @@ -101,7 +100,7 @@ An action is a sequence of statements. A statement can be one of the following: .PP .EX -.ta \w'\f(CWdelete array[expression]'u +.ta \w'\f(CWdelete array[expression]\fR'u .RS .nf .ft CW @@ -145,7 +144,7 @@ The operators are also available in expressions. Variables may be scalars, array elements (denoted -.IB x [ i ] ) +.IB x [ i ] \fR) or fields. Variables are initialized to the null string. Array subscripts may be any string, @@ -161,11 +160,11 @@ The .B print statement prints its arguments on the standard output (or on a file if -.BI > file +.BI > " file or -.BI >> file +.BI >> " file is present or on a pipe if -.BI | cmd +.BI | " cmd is present), separated by the current output field separator, and terminated by the output record separator. .I file @@ -176,9 +175,10 @@ identical string values in different statements denote the same open file. The .B printf -statement formats its expression list according to the format +statement formats its expression list according to the +.I format (see -.IR printf (3)) . +.IR printf (3)). The built-in function .BI close( expr ) closes the file or pipe @@ -189,13 +189,13 @@ flushes any buffered output for the file or pipe .IR expr . .PP The mathematical functions +.BR atan2 , +.BR cos , .BR exp , .BR log , -.BR sqrt , .BR sin , -.BR cos , and -.BR atan2 +.B sqrt are built in. Other built-in functions: .TF length @@ -203,7 +203,8 @@ Other built-in functions: .B length the length of its argument taken as a string, -or of +number of elements in an array for an array argument, +or length of .B $0 if no argument. .TP @@ -218,14 +219,18 @@ and returns the previous seed. .B int truncates to an integer value .TP -.BI substr( s , " m" , " n\fB) +\fBsubstr(\fIs\fB, \fIm\fR [\fB, \fIn\^\fR]\fB)\fR the .IR n -character substring of .I s that begins at position -.IR m +.I m counted from 1. +If no +.IR m , +use the rest of the string +.I .TP .BI index( s , " t" ) the position in @@ -246,14 +251,14 @@ and .B RLENGTH are set to the position and length of the matched string. .TP -.BI split( s , " a" , " fs\fB) +\fBsplit(\fIs\fB, \fIa \fR[\fB, \fIfs\^\fR]\fB)\fR splits the string .I s into array elements -.IB a [1] , -.IB a [2] , +.IB a [1] \fR, +.IB a [2] \fR, \&..., -.IB a [ n ] , +.IB a [ n ] \fR, and returns .IR n . The separation is done with the regular expression @@ -266,7 +271,7 @@ is not given. An empty string as field separator splits the string into one array element per character. .TP -.BI sub( r , " t" , " s\fB) +\fBsub(\fIr\fB, \fIt \fR[, \fIs\^\fR]\fB) substitutes .I t for the first occurrence of the regular expression @@ -279,7 +284,7 @@ is not given, .B $0 is used. .TP -.B gsub +\fBgsub(\fIr\fB, \fIt \fR[, \fIs\^\fR]\fB) same as .B sub except that all occurrences of the regular expression @@ -289,18 +294,28 @@ and .B gsub return the number of replacements. .TP -.BI sprintf( fmt , " expr" , " ...\fB ) +.BI sprintf( fmt , " expr" , " ...\fB) the string resulting from formatting .I expr ... according to the .IR printf (3) format -.I fmt +.IR fmt . .TP .BI system( cmd ) executes .I cmd -and returns its exit status +and returns its exit status. This will be \-1 upon error, +.IR cmd 's +exit status upon a normal exit, +256 + +.I sig +upon death-by-signal, where +.I sig +is the number of the murdering signal, +or 512 + +.I sig +if there was a core dump. .TP .BI tolower( str ) returns a copy of @@ -321,7 +336,7 @@ sets .B $0 to the next input record from the current input file; .B getline -.BI < file +.BI < " file sets .B $0 to the next record from @@ -359,7 +374,7 @@ Isolated regular expressions in a pattern apply to the entire line. Regular expressions may also occur in relational expressions, using the operators -.BR ~ +.B ~ and .BR !~ . .BI / re / @@ -383,8 +398,12 @@ A relational expression is one of the following: .br .BI ( expr , expr,... ") in " array-name .PP -where a relop is any of the six relational operators in C, -and a matchop is either +where a +.I relop +is any of the six relational operators in C, +and a +.I matchop +is either .B ~ (matches) or @@ -405,57 +424,68 @@ and after the last. and .B END do not combine with other patterns. +They may appear multiple times in a program and execute +in the order they are read by +.IR awk . .PP Variable names with special meanings: .TF FILENAME .TP +.B ARGC +argument count, assignable. +.TP +.B ARGV +argument array, assignable; +non-null members are taken as filenames. +.TP .B CONVFMT conversion format used when converting numbers (default -.BR "%.6g" ) +.BR "%.6g" ). .TP +.B ENVIRON +array of environment variables; subscripts are names. +.TP +.B FILENAME +the name of the current input file. +.TP +.B FNR +ordinal number of the current record in the current file. +.TP .B FS regular expression used to separate fields; also settable by option -.BI \-F fs. +.BI \-F fs\fR. .TP .BR NF -number of fields in the current record +number of fields in the current record. .TP .B NR -ordinal number of the current record +ordinal number of the current record. .TP -.B FNR -ordinal number of the current record in the current file +.B OFMT +output format for numbers (default +.BR "%.6g" ). .TP -.B FILENAME -the name of the current input file -.TP -.B RS -input record separator (default newline) -.TP .B OFS -output field separator (default blank) +output field separator (default space). .TP .B ORS -output record separator (default newline) +output record separator (default newline). .TP -.B OFMT -output format for numbers (default -.BR "%.6g" ) +.B RLENGTH +the length of a string matched by +.BR match . .TP -.B SUBSEP -separates multiple subscripts (default 034) +.B RS +input record separator (default newline). .TP -.B ARGC -argument count, assignable +.B RSTART +the start position of a string matched by +.BR match . .TP -.B ARGV -argument array, assignable; -non-null members are taken as filenames -.TP -.B ENVIRON -array of environment variables; subscripts are names. +.B SUBSEP +separates multiple subscripts (default 034). .PD .PP Functions may be defined (at the position of a pattern-action statement) thus: @@ -486,7 +516,7 @@ BEGIN { FS = ",[ \et]*|[ \et]+" } .EE .ns .IP -Same, with input fields separated by comma and/or blanks and tabs. +Same, with input fields separated by comma and/or spaces and tabs. .PP .EX .nf @@ -512,13 +542,13 @@ BEGIN { # Simulate echo(1) .fi .EE .SH SEE ALSO +.IR grep (1), .IR lex (1), .IR sed (1) .br A. V. Aho, B. W. Kernighan, P. J. Weinberger, -.I -The AWK Programming Language, -Addison-Wesley, 1988. ISBN 0-201-07981-X +.IR "The AWK Programming Language" , +Addison-Wesley, 1988. ISBN 0-201-07981-X. .SH BUGS There are no explicit conversions between numbers and strings. To force an expression to be treated as a number add 0 to it; @@ -527,3 +557,5 @@ to force it to be treated as a string concatenate .br The scope rules for variables in functions are a botch; the syntax is worse. +.br +Only eight-bit characters sets are handled correctly. Modified: vendor/one-true-awk/dist/awk.h ============================================================================== --- vendor/one-true-awk/dist/awk.h Sun Jun 2 04:04:21 2019 (r348504) +++ vendor/one-true-awk/dist/awk.h Sun Jun 2 04:23:56 2019 (r348505) @@ -81,7 +81,8 @@ typedef struct Cell { char *nval; /* name, for variables only */ char *sval; /* string value */ Awkfloat fval; /* value as number */ - int tval; /* type info: STR|NUM|ARR|FCN|FLD|CON|DONTFREE */ + int tval; /* type info: STR|NUM|ARR|FCN|FLD|CON|DONTFREE|CONVC|CONVO */ + char *fmt; /* CONVFMT/OFMT value used to convert from number */ struct Cell *cnext; /* ptr to next if chained */ } Cell; @@ -96,9 +97,14 @@ extern Array *symtab; extern Cell *nrloc; /* NR */ extern Cell *fnrloc; /* FNR */ +extern Cell *fsloc; /* FS */ extern Cell *nfloc; /* NF */ +extern Cell *ofsloc; /* OFS */ +extern Cell *orsloc; /* ORS */ +extern Cell *rsloc; /* RS */ extern Cell *rstartloc; /* RSTART */ extern Cell *rlengthloc; /* RLENGTH */ +extern Cell *subseploc; /* SUBSEP */ /* Cell.tval values: */ #define NUM 01 /* number value is valid */ @@ -109,6 +115,8 @@ extern Cell *rlengthloc; /* RLENGTH */ #define FCN 040 /* this is a function name */ #define FLD 0100 /* this is a field $1, $2, ... */ #define REC 0200 /* this is $0 */ +#define CONVC 0400 /* string was converted from number via CONVFMT */ +#define CONVO 01000 /* string was converted from number via OFMT */ /* function types */ Modified: vendor/one-true-awk/dist/awkgram.y ============================================================================== --- vendor/one-true-awk/dist/awkgram.y Sun Jun 2 04:04:21 2019 (r348504) +++ vendor/one-true-awk/dist/awkgram.y Sun Jun 2 04:23:56 2019 (r348505) @@ -86,7 +86,7 @@ Node *arglist = 0; /* list of args for current functio %left CAT %left '+' '-' %left '*' '/' '%' -%left NOT UMINUS +%left NOT UMINUS UPLUS %right POWER %right DECR INCR %left INDIRECT @@ -357,7 +357,7 @@ term: | term '%' term { $$ = op2(MOD, $1, $3); } | term POWER term { $$ = op2(POWER, $1, $3); } | '-' term %prec UMINUS { $$ = op1(UMINUS, $2); } - | '+' term %prec UMINUS { $$ = $2; } + | '+' term %prec UMINUS { $$ = op1(UPLUS, $2); } | NOT term %prec UMINUS { $$ = op1(NOT, notnull($2)); } | BLTIN '(' ')' { $$ = op2(BLTIN, itonp($1), rectonode()); } | BLTIN '(' patlist ')' { $$ = op2(BLTIN, itonp($1), $3); } Modified: vendor/one-true-awk/dist/b.c ============================================================================== --- vendor/one-true-awk/dist/b.c Sun Jun 2 04:04:21 2019 (r348504) +++ vendor/one-true-awk/dist/b.c Sun Jun 2 04:23:56 2019 (r348505) @@ -27,6 +27,7 @@ THIS SOFTWARE. #define DEBUG #include +#include #include #include #include @@ -65,6 +66,11 @@ int rlxval; static uschar *rlxstr; static uschar *prestr; /* current position in current re */ static uschar *lastre; /* origin of last re */ +static uschar *lastatom; /* origin of last Atom */ +static uschar *starttok; +static uschar *basestr; /* starts with original, replaced during + repetition processing */ +static uschar *firstbasestr; static int setcnt; static int poscnt; @@ -82,11 +88,11 @@ fa *makedfa(const char *s, int anchor) /* returns dfa fa *pfa; static int now = 1; - if (setvec == NULL) { /* first time through any RE */ + if (setvec == 0) { /* first time through any RE */ maxsetvec = MAXLIN; setvec = (int *) malloc(maxsetvec * sizeof(int)); tmpset = (int *) malloc(maxsetvec * sizeof(int)); - if (setvec == NULL || tmpset == NULL) + if (setvec == 0 || tmpset == 0) overflo("out of space initializing makedfa"); } @@ -124,6 +130,8 @@ fa *mkdfa(const char *s, int anchor) /* does the real Node *p, *p1; fa *f; + firstbasestr = (uschar *) s; + basestr = firstbasestr; p = reparse(s); p1 = op2(CAT, op2(STAR, op2(ALL, NIL, NIL), NIL), p); /* put ALL STAR in front of reg. exp. */ @@ -137,7 +145,7 @@ fa *mkdfa(const char *s, int anchor) /* does the real f->accept = poscnt-1; /* penter has computed number of positions in re */ cfoll(f, p1); /* set up follow sets */ freetr(p1); - if ((f->posns[0] = (int *) calloc(*(f->re[0].lfollow), sizeof(int))) == NULL) + if ((f->posns[0] = (int *) calloc(1, *(f->re[0].lfollow)*sizeof(int))) == NULL) overflo("out of space in makedfa"); if ((f->posns[1] = (int *) calloc(1, sizeof(int))) == NULL) overflo("out of space in makedfa"); @@ -145,6 +153,10 @@ fa *mkdfa(const char *s, int anchor) /* does the real f->initstat = makeinit(f, anchor); f->anchor = anchor; f->restr = (uschar *) tostring(s); + if (firstbasestr != basestr) { + if (basestr) + xfree(basestr); + } return f; } @@ -157,7 +169,7 @@ int makeinit(fa *f, int anchor) f->reset = 0; k = *(f->re[0].lfollow); xfree(f->posns[2]); - if ((f->posns[2] = (int *) calloc(k+1, sizeof(int))) == NULL) + if ((f->posns[2] = (int *) calloc(1, (k+1)*sizeof(int))) == NULL) overflo("out of space in makeinit"); for (i=0; i <= k; i++) { (f->posns[2])[i] = (f->re[0].lfollow)[i]; @@ -290,11 +302,11 @@ char *cclenter(const char *argp) /* add a character cl int i, c, c2; uschar *p = (uschar *) argp; uschar *op, *bp; - static uschar *buf = NULL; + static uschar *buf = 0; static int bufsz = 100; op = p; - if (buf == NULL && (buf = (uschar *) malloc(bufsz)) == NULL) + if (buf == 0 && (buf = (uschar *) malloc(bufsz)) == NULL) FATAL("out of space for character class [%.10s...] 1", p); bp = buf; for (i = 0; (c = *p++) != 0; ) { @@ -350,14 +362,14 @@ void cfoll(fa *f, Node *v) /* enter follow set of each maxsetvec *= 4; setvec = (int *) realloc(setvec, maxsetvec * sizeof(int)); tmpset = (int *) realloc(tmpset, maxsetvec * sizeof(int)); - if (setvec == NULL || tmpset == NULL) + if (setvec == 0 || tmpset == 0) overflo("out of space in cfoll()"); } for (i = 0; i <= f->accept; i++) setvec[i] = 0; setcnt = 0; follow(v); /* computes setvec and setcnt */ - if ((p = (int *) calloc(setcnt+1, sizeof(int))) == NULL) + if ((p = (int *) calloc(1, (setcnt+1)*sizeof(int))) == NULL) overflo("out of space building follow set"); f->re[info(v)].lfollow = p; *p = setcnt; @@ -391,7 +403,7 @@ int first(Node *p) /* collects initially active leaves maxsetvec *= 4; setvec = (int *) realloc(setvec, maxsetvec * sizeof(int)); tmpset = (int *) realloc(tmpset, maxsetvec * sizeof(int)); - if (setvec == NULL || tmpset == NULL) + if (setvec == 0 || tmpset == 0) overflo("out of space in first()"); } if (type(p) == EMPTYRE) { @@ -531,7 +543,7 @@ int pmatch(fa *f, const char *p0) /* longest match, fo for (i = 2; i <= f->curstat; i++) xfree(f->posns[i]); k = *f->posns[0]; - if ((f->posns[2] = (int *) calloc(k+1, sizeof(int))) == NULL) + if ((f->posns[2] = (int *) calloc(1, (k+1)*sizeof(int))) == NULL) overflo("out of space in pmatch"); for (i = 0; i <= k; i++) (f->posns[2])[i] = (f->posns[0])[i]; @@ -588,7 +600,7 @@ int nematch(fa *f, const char *p0) /* non-empty match, for (i = 2; i <= f->curstat; i++) xfree(f->posns[i]); k = *f->posns[0]; - if ((f->posns[2] = (int *) calloc(k+1, sizeof(int))) == NULL) + if ((f->posns[2] = (int *) calloc(1, (k+1)*sizeof(int))) == NULL) overflo("out of state space"); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sun Jun 2 04:28:11 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE96B15AA911; Sun, 2 Jun 2019 04:28:11 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 51FB58A699; Sun, 2 Jun 2019 04:28:11 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0C84421E74; Sun, 2 Jun 2019 04:28:11 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x524SAj9063411; Sun, 2 Jun 2019 04:28:10 GMT (envelope-from imp@FreeBSD.org) Received: (from imp@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x524SAmN063410; Sun, 2 Jun 2019 04:28:10 GMT (envelope-from imp@FreeBSD.org) Message-Id: <201906020428.x524SAmN063410@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: imp set sender to imp@FreeBSD.org using -f From: Warner Losh Date: Sun, 2 Jun 2019 04:28:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348506 - vendor/one-true-awk/4189ef5d X-SVN-Group: vendor X-SVN-Commit-Author: imp X-SVN-Commit-Paths: vendor/one-true-awk/4189ef5d X-SVN-Commit-Revision: 348506 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 51FB58A699 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2019 04:28:11 -0000 Author: imp Date: Sun Jun 2 04:28:10 2019 New Revision: 348506 URL: https://svnweb.freebsd.org/changeset/base/348506 Log: Record the vendor import of git has 4189ef5d from https://github.com/onetrueawk/awk.git Added: vendor/one-true-awk/4189ef5d/ - copied from r348505, vendor/one-true-awk/dist/ From owner-svn-src-vendor@freebsd.org Mon Jun 3 15:19:25 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6DBF15B44C0; Mon, 3 Jun 2019 15:19:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E44C85875; Mon, 3 Jun 2019 15:19:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E483F7EBA; Mon, 3 Jun 2019 15:19:23 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53FJNtv067036; Mon, 3 Jun 2019 15:19:23 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53FJNK1067035; Mon, 3 Jun 2019 15:19:23 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031519.x53FJNK1067035@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 15:19:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348534 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 348534 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 6E44C85875 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.966,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 15:19:25 -0000 Author: mav Date: Mon Jun 3 15:19:23 2019 New Revision: 348534 URL: https://svnweb.freebsd.org/changeset/base/348534 Log: 9616 Bogus error when attempting to set property on read-only pool illumos/illumos-gate@f62db44dbcda5dd786bb821f1e6fd3ca2e6d4391 Reviewed by: Paul Dagnelie Reviewed by: Matt Ahrens Approved by: Robert Mustacchi Author: Andrew Stormont Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Mon Jun 3 15:19:11 2019 (r348533) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Mon Jun 3 15:19:23 2019 (r348534) @@ -30,7 +30,7 @@ * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 Nexenta Systems, Inc. * Copyright 2016 Igor Kozhukhov - * Copyright 2017 RackTop Systems. + * Copyright 2017-2018 RackTop Systems. */ #include @@ -1808,13 +1808,18 @@ zfs_prop_set_list(zfs_handle_t *zhp, nvlist_t *props) ret = zfs_ioctl(hdl, ZFS_IOC_SET_PROP, &zc); if (ret != 0) { + if (zc.zc_nvlist_dst_filled == B_FALSE) { + (void) zfs_standard_error(hdl, errno, errbuf); + goto error; + } + /* Get the list of unset properties back and report them. */ nvlist_t *errorprops = NULL; if (zcmd_read_dst_nvlist(hdl, &zc, &errorprops) != 0) goto error; - for (nvpair_t *elem = nvlist_next_nvpair(nvl, NULL); + for (nvpair_t *elem = nvlist_next_nvpair(errorprops, NULL); elem != NULL; - elem = nvlist_next_nvpair(nvl, elem)) { + elem = nvlist_next_nvpair(errorprops, elem)) { zfs_prop_t prop = zfs_name_to_prop(nvpair_name(elem)); zfs_setprop_error(hdl, prop, errno, errbuf); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 15:23:23 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB03C15B47C4; Mon, 3 Jun 2019 15:23:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8718F85E4C; Mon, 3 Jun 2019 15:23:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 42DF68070; Mon, 3 Jun 2019 15:23:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53FNMGd072016; Mon, 3 Jun 2019 15:23:22 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53FNM8D072015; Mon, 3 Jun 2019 15:23:22 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031523.x53FNM8D072015@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 15:23:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348535 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348535 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 8718F85E4C X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.967,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 15:23:23 -0000 Author: mav Date: Mon Jun 3 15:23:21 2019 New Revision: 348535 URL: https://svnweb.freebsd.org/changeset/base/348535 Log: 9677 panic from zio_write_gang_block() when creating dump device on fragmented rpool illumos/illumos-gate@7341a7de4f0489193e0cfe11049a7bcf1596a4db Reviewed by: Matt Ahrens Reviewed by: George Wilson Reviewed by: Prashanth Sreenivasa Approved by: Robert Mustacchi Author: Brad Lewis Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Mon Jun 3 15:19:23 2019 (r348534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Mon Jun 3 15:23:21 2019 (r348535) @@ -2192,7 +2192,13 @@ zio_write_gang_member_ready(zio_t *zio) static void zio_write_gang_done(zio_t *zio) { - abd_put(zio->io_abd); + /* + * The io_abd field will be NULL for a zio with no data. The io_flags + * will initially have the ZIO_FLAG_NODATA bit flag set, but we can't + * check for it here as it is cleared in zio_ready. + */ + if (zio->io_abd != NULL) + abd_put(zio->io_abd); } static int @@ -2213,11 +2219,12 @@ zio_write_gang_block(zio_t *pio) int gbh_copies = MIN(copies + 1, spa_max_replication(spa)); zio_prop_t zp; int error; + boolean_t has_data = !(pio->io_flags & ZIO_FLAG_NODATA); int flags = METASLAB_HINTBP_FAVOR | METASLAB_GANG_HEADER; if (pio->io_flags & ZIO_FLAG_IO_ALLOCATING) { ASSERT(pio->io_priority == ZIO_PRIORITY_ASYNC_WRITE); - ASSERT(!(pio->io_flags & ZIO_FLAG_NODATA)); + ASSERT(has_data); flags |= METASLAB_ASYNC_ALLOC; VERIFY(refcount_held(&mc->mc_alloc_slots[pio->io_allocator], @@ -2241,7 +2248,7 @@ zio_write_gang_block(zio_t *pio) if (error) { if (pio->io_flags & ZIO_FLAG_IO_ALLOCATING) { ASSERT(pio->io_priority == ZIO_PRIORITY_ASYNC_WRITE); - ASSERT(!(pio->io_flags & ZIO_FLAG_NODATA)); + ASSERT(has_data); /* * If we failed to allocate the gang block header then @@ -2294,14 +2301,15 @@ zio_write_gang_block(zio_t *pio) zp.zp_nopwrite = B_FALSE; zio_t *cio = zio_write(zio, spa, txg, &gbh->zg_blkptr[g], - abd_get_offset(pio->io_abd, pio->io_size - resid), lsize, - lsize, &zp, zio_write_gang_member_ready, NULL, NULL, + has_data ? abd_get_offset(pio->io_abd, pio->io_size - + resid) : NULL, lsize, lsize, &zp, + zio_write_gang_member_ready, NULL, NULL, zio_write_gang_done, &gn->gn_child[g], pio->io_priority, ZIO_GANG_CHILD_FLAGS(pio), &pio->io_bookmark); if (pio->io_flags & ZIO_FLAG_IO_ALLOCATING) { ASSERT(pio->io_priority == ZIO_PRIORITY_ASYNC_WRITE); - ASSERT(!(pio->io_flags & ZIO_FLAG_NODATA)); + ASSERT(has_data); /* * Gang children won't throttle but we should From owner-svn-src-vendor@freebsd.org Mon Jun 3 15:25:14 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB67A15B48EE; Mon, 3 Jun 2019 15:25:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DC7B86128; Mon, 3 Jun 2019 15:25:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 593D88072; Mon, 3 Jun 2019 15:25:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53FPDq1072208; Mon, 3 Jun 2019 15:25:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53FPDLD072207; Mon, 3 Jun 2019 15:25:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031525.x53FPDLD072207@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 15:25:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348537 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348537 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 7DC7B86128 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.966,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 15:25:14 -0000 Author: mav Date: Mon Jun 3 15:25:12 2019 New Revision: 348537 URL: https://svnweb.freebsd.org/changeset/base/348537 Log: 8601 memory leak in get_special_prop() illumos/illumos-gate@e19b450bec203d8be04447ea476d7a86b36d63a1 Reviewed by: Serapheim Dimitropoulos Reviewed by: Sara Hartse Reviewed by: Pavel Zakharov Reviewed by: Matt Ahrens Reviewed by: Robert Mustacchi Approved by: Dan McDonald Author: John Gallagher Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c Mon Jun 3 15:23:37 2019 (r348536) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c Mon Jun 3 15:25:12 2019 (r348537) @@ -14,7 +14,7 @@ */ /* - * Copyright (c) 2016 by Delphix. All rights reserved. + * Copyright (c) 2016, 2017 by Delphix. All rights reserved. */ #include "lua.h" @@ -422,16 +422,21 @@ get_special_prop(lua_State *state, dsl_dataset_t *ds, case ZFS_PROP_INCONSISTENT: numval = dsl_get_inconsistent(ds); break; - case ZFS_PROP_RECEIVE_RESUME_TOKEN: - VERIFY3U(strlcpy(strval, get_receive_resume_stats_impl(ds), - ZAP_MAXVALUELEN), <, ZAP_MAXVALUELEN); + case ZFS_PROP_RECEIVE_RESUME_TOKEN: { + char *token = get_receive_resume_stats_impl(ds); + VERIFY3U(strlcpy(strval, token, ZAP_MAXVALUELEN), <, + ZAP_MAXVALUELEN); + strfree(token); if (strcmp(strval, "") == 0) { - VERIFY3U(strlcpy(strval, get_child_receive_stats(ds), - ZAP_MAXVALUELEN), <, ZAP_MAXVALUELEN); + token = get_child_receive_stats(ds); + VERIFY3U(strlcpy(strval, token, ZAP_MAXVALUELEN), <, + ZAP_MAXVALUELEN); + strfree(token); if (strcmp(strval, "") == 0) error = ENOENT; } break; + } case ZFS_PROP_VOLSIZE: ASSERT(ds_type == ZFS_TYPE_VOLUME); error = dmu_objset_from_ds(ds, &os); From owner-svn-src-vendor@freebsd.org Mon Jun 3 16:57:08 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9E7915B7052; Mon, 3 Jun 2019 16:57:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A2598A346; Mon, 3 Jun 2019 16:57:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 651A98F76; Mon, 3 Jun 2019 16:57:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53Gv7l2019849; Mon, 3 Jun 2019 16:57:07 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53Gv6ik019846; Mon, 3 Jun 2019 16:57:06 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031657.x53Gv6ik019846@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 16:57:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348548 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 348548 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 8A2598A346 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 16:57:08 -0000 Author: mav Date: Mon Jun 3 16:57:06 2019 New Revision: 348548 URL: https://svnweb.freebsd.org/changeset/base/348548 Log: 9617 too-frequent TXG sync causes excessive write inflation illumos/illumos-gate@7928f4baf4ab3230557eb6289be68aa7a3003f38 Reviewed by: Serapheim Dimitropoulos Reviewed by: Brad Lewis Reviewed by: George Wilson Reviewed by: Andrew Stormont Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c Mon Jun 3 16:51:07 2019 (r348547) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c Mon Jun 3 16:57:06 2019 (r348548) @@ -103,9 +103,11 @@ uint64_t zfs_dirty_data_max_max = 4ULL * 1024 * 1024 * int zfs_dirty_data_max_percent = 10; /* - * If there is at least this much dirty data, push out a txg. + * If there's at least this much dirty data (as a percentage of + * zfs_dirty_data_max), push out a txg. This should be less than + * zfs_vdev_async_write_active_min_dirty_percent. */ -uint64_t zfs_dirty_data_sync = 64 * 1024 * 1024; +uint64_t zfs_dirty_data_sync_pct = 20; /* * Once there is this amount of dirty data, the dmu_tx_delay() will kick in @@ -824,10 +826,12 @@ dsl_pool_need_dirty_delay(dsl_pool_t *dp) { uint64_t delay_min_bytes = zfs_dirty_data_max * zfs_delay_min_dirty_percent / 100; + uint64_t dirty_min_bytes = + zfs_dirty_data_max * zfs_dirty_data_sync_pct / 100; boolean_t rv; mutex_enter(&dp->dp_lock); - if (dp->dp_dirty_total > zfs_dirty_data_sync) + if (dp->dp_dirty_total > dirty_min_bytes) txg_kick(dp); rv = (dp->dp_dirty_total > delay_min_bytes); mutex_exit(&dp->dp_lock); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h Mon Jun 3 16:51:07 2019 (r348547) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h Mon Jun 3 16:57:06 2019 (r348548) @@ -53,7 +53,7 @@ struct dsl_scan; extern uint64_t zfs_dirty_data_max; extern uint64_t zfs_dirty_data_max_max; -extern uint64_t zfs_dirty_data_sync; +extern uint64_t zfs_dirty_data_sync_pct; extern int zfs_dirty_data_max_percent; extern int zfs_delay_min_dirty_percent; extern uint64_t zfs_delay_scale; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Mon Jun 3 16:51:07 2019 (r348547) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Mon Jun 3 16:57:06 2019 (r348548) @@ -484,6 +484,8 @@ txg_sync_thread(void *arg) uint64_t timeout = zfs_txg_timeout * hz; uint64_t timer; uint64_t txg; + uint64_t dirty_min_bytes = + zfs_dirty_data_max * zfs_dirty_data_sync_pct / 100; /* * We sync when we're scanning, there's someone waiting @@ -495,7 +497,7 @@ txg_sync_thread(void *arg) !tx->tx_exiting && timer > 0 && tx->tx_synced_txg >= tx->tx_sync_txg_waiting && !txg_has_quiesced_to_sync(dp) && - dp->dp_dirty_total < zfs_dirty_data_sync) { + dp->dp_dirty_total < dirty_min_bytes) { dprintf("waiting; tx_synced=%llu waiting=%llu dp=%p\n", tx->tx_synced_txg, tx->tx_sync_txg_waiting, dp); txg_thread_wait(tx, &cpr, &tx->tx_sync_more_cv, timer); From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:01:17 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ED6215B7544; Mon, 3 Jun 2019 17:01:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 420398A91D; Mon, 3 Jun 2019 17:01:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EE04E8FB9; Mon, 3 Jun 2019 17:01:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53H1GOs021588; Mon, 3 Jun 2019 17:01:16 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53H1GJX021587; Mon, 3 Jun 2019 17:01:16 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031701.x53H1GJX021587@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:01:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348550 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348550 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 420398A91D X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:01:17 -0000 Author: mav Date: Mon Jun 3 17:01:16 2019 New Revision: 348550 URL: https://svnweb.freebsd.org/changeset/base/348550 Log: 1700 Add SCSI UNMAP support illumos/illumos-gate@047c81d31d0f571d6652e97143cd15281de61e84 Reviewed by: Jerry Jelinek Reviewed by: Garrett D'Amore Reviewed by: Igor Kozhukhov Approved by: Dan McDonald Author: Saso Kiselkov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Mon Jun 3 16:59:56 2019 (r348549) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Mon Jun 3 17:01:16 2019 (r348550) @@ -23,7 +23,7 @@ * * Portions Copyright 2010 Robert Milkowski * - * Copyright 2011 Nexenta Systems, Inc. All rights reserved. + * Copyright 2017 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. * Copyright (c) 2014 Integros [integros.com] @@ -89,6 +89,7 @@ #include #include #include +#include #include "zfs_namecheck.h" @@ -1778,44 +1779,64 @@ zvol_ioctl(dev_t dev, int cmd, intptr_t arg, int flag, case DKIOCFREE: { - dkioc_free_t df; + dkioc_free_list_t *dfl; dmu_tx_t *tx; if (!zvol_unmap_enabled) break; - if (ddi_copyin((void *)arg, &df, sizeof (df), flag)) { - error = SET_ERROR(EFAULT); - break; + if (!(flag & FKIOCTL)) { + error = dfl_copyin((void *)arg, &dfl, flag, KM_SLEEP); + if (error != 0) + break; + } else { + dfl = (dkioc_free_list_t *)arg; + ASSERT3U(dfl->dfl_num_exts, <=, DFL_COPYIN_MAX_EXTS); + if (dfl->dfl_num_exts > DFL_COPYIN_MAX_EXTS) { + error = SET_ERROR(EINVAL); + break; + } } - /* - * Apply Postel's Law to length-checking. If they overshoot, - * just blank out until the end, if there's a need to blank - * out anything. - */ - if (df.df_start >= zv->zv_volsize) - break; /* No need to do anything... */ - mutex_exit(&zfsdev_state_lock); - rl = zfs_range_lock(&zv->zv_znode, df.df_start, df.df_length, - RL_WRITER); - tx = dmu_tx_create(zv->zv_objset); - dmu_tx_mark_netfree(tx); - error = dmu_tx_assign(tx, TXG_WAIT); - if (error != 0) { - dmu_tx_abort(tx); - } else { - zvol_log_truncate(zv, tx, df.df_start, - df.df_length, B_TRUE); - dmu_tx_commit(tx); - error = dmu_free_long_range(zv->zv_objset, ZVOL_OBJ, - df.df_start, df.df_length); - } + for (int i = 0; i < dfl->dfl_num_exts; i++) { + uint64_t start = dfl->dfl_exts[i].dfle_start, + length = dfl->dfl_exts[i].dfle_length, + end = start + length; - zfs_range_unlock(rl); + /* + * Apply Postel's Law to length-checking. If they + * overshoot, just blank out until the end, if there's + * a need to blank out anything. + */ + if (start >= zv->zv_volsize) + continue; /* No need to do anything... */ + if (end > zv->zv_volsize) { + end = DMU_OBJECT_END; + length = end - start; + } + rl = zfs_range_lock(&zv->zv_znode, start, length, + RL_WRITER); + tx = dmu_tx_create(zv->zv_objset); + error = dmu_tx_assign(tx, TXG_WAIT); + if (error != 0) { + dmu_tx_abort(tx); + } else { + zvol_log_truncate(zv, tx, start, length, + B_TRUE); + dmu_tx_commit(tx); + error = dmu_free_long_range(zv->zv_objset, + ZVOL_OBJ, start, length); + } + + zfs_range_unlock(rl); + + if (error != 0) + break; + } + /* * If the write-cache is disabled, 'sync' property * is set to 'always', or if the caller is asking for @@ -1827,9 +1848,12 @@ zvol_ioctl(dev_t dev, int cmd, intptr_t arg, int flag, if ((error == 0) && zvol_unmap_sync_enabled && (!(zv->zv_flags & ZVOL_WCE) || (zv->zv_objset->os_sync == ZFS_SYNC_ALWAYS) || - (df.df_flags & DF_WAIT_SYNC))) { + (dfl->dfl_flags & DF_WAIT_SYNC))) { zil_commit(zv->zv_zilog, ZVOL_OBJ); } + + if (!(flag & FKIOCTL)) + dfl_free(dfl); return (error); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:02:36 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 748CC15B781E; Mon, 3 Jun 2019 17:02:36 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B5968ACA4; Mon, 3 Jun 2019 17:02:36 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EA93D912F; Mon, 3 Jun 2019 17:02:35 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53H2ZQg024737; Mon, 3 Jun 2019 17:02:35 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53H2ZN2024736; Mon, 3 Jun 2019 17:02:35 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031702.x53H2ZN2024736@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:02:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348551 - vendor-sys/illumos/dist/uts/common/fs/zfs/sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs/sys X-SVN-Commit-Revision: 348551 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 1B5968ACA4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:02:36 -0000 Author: mav Date: Mon Jun 3 17:02:35 2019 New Revision: 348551 URL: https://svnweb.freebsd.org/changeset/base/348551 Log: 9862 fix typo in comment in vdev_impl.h illumos/illumos-gate@84927f52bd837f6e4882a19e43fd026f1828d910 Reviewed by: Matthew Ahrens Reviewed by: Brian Behlendorf Approved by: Robert Mustacchi Author: Allan Jude Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Mon Jun 3 17:01:16 2019 (r348550) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Mon Jun 3 17:02:35 2019 (r348551) @@ -297,7 +297,7 @@ struct vdev { /* * The queue depth parameters determine how many async writes are - * still pending (i.e. allocated by net yet issued to disk) per + * still pending (i.e. allocated but not yet issued to disk) per * top-level (vdev_async_write_queue_depth) and the maximum allowed * (vdev_max_async_write_queue_depth). These values only apply to * top-level vdevs. From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:06:03 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90B7D15B7919; Mon, 3 Jun 2019 17:06:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 329048AEB1; Mon, 3 Jun 2019 17:06:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 233D49132; Mon, 3 Jun 2019 17:06:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53H62N7024934; Mon, 3 Jun 2019 17:06:02 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53H629V024933; Mon, 3 Jun 2019 17:06:02 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031706.x53H629V024933@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:06:02 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348552 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 348552 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 329048AEB1 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:06:03 -0000 Author: mav Date: Mon Jun 3 17:06:02 2019 New Revision: 348552 URL: https://svnweb.freebsd.org/changeset/base/348552 Log: 9682 page fault in dsl_async_clone_destroy() while opening pool illumos/illumos-gate@ade2c82828f0dca1f46919aa1bd936ea1a5a0047 Reviewed by: Brad Lewis Reviewed by: Matt Ahrens Reviewed by: Sara Hartse Approved by: Robert Mustacchi Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Mon Jun 3 17:02:35 2019 (r348551) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Mon Jun 3 17:06:02 2019 (r348552) @@ -3862,8 +3862,17 @@ spa_load_impl(spa_t *spa, spa_import_type_t type, char */ spa_history_log_version(spa, "open"); + spa_restart_removal(spa); + spa_spawn_aux_threads(spa); + /* * Delete any inconsistent datasets. + * + * Note: + * Since we may be issuing deletes for clones here, + * we make sure to do so after we've spawned all the + * auxiliary threads above (from which the livelist + * deletion zthr is part of). */ (void) dmu_objset_find(spa_name(spa), dsl_destroy_inconsistent, NULL, DS_FIND_CHILDREN); @@ -3872,10 +3881,6 @@ spa_load_impl(spa_t *spa, spa_import_type_t type, char * Clean up any stale temporary dataset userrefs. */ dsl_pool_clean_tmp_userrefs(spa->spa_dsl_pool); - - spa_restart_removal(spa); - - spa_spawn_aux_threads(spa); spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); vdev_initialize_restart(spa->spa_root_vdev); From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:06:04 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16FE215B791E; Mon, 3 Jun 2019 17:06:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AE3E78AEB2; Mon, 3 Jun 2019 17:06:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 888FC9133; Mon, 3 Jun 2019 17:06:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53H63Zb024940; Mon, 3 Jun 2019 17:06:03 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53H632X024939; Mon, 3 Jun 2019 17:06:03 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031706.x53H632X024939@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:06:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348552 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 348552 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: AE3E78AEB2 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:06:04 -0000 Author: mav Date: Mon Jun 3 17:06:02 2019 New Revision: 348552 URL: https://svnweb.freebsd.org/changeset/base/348552 Log: 9682 page fault in dsl_async_clone_destroy() while opening pool illumos/illumos-gate@ade2c82828f0dca1f46919aa1bd936ea1a5a0047 Reviewed by: Brad Lewis Reviewed by: Matt Ahrens Reviewed by: Sara Hartse Approved by: Robert Mustacchi Author: Serapheim Dimitropoulos Modified: vendor/illumos/dist/cmd/zdb/zdb.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Mon Jun 3 17:02:35 2019 (r348551) +++ vendor/illumos/dist/cmd/zdb/zdb.c Mon Jun 3 17:06:02 2019 (r348552) @@ -2245,10 +2245,11 @@ dump_dir(objset_t *os) dmu_objset_name(os, osname); (void) printf("Dataset %s [%s], ID %llu, cr_txg %llu, " - "%s, %llu objects%s\n", + "%s, %llu objects%s%s\n", osname, type, (u_longlong_t)dmu_objset_id(os), (u_longlong_t)dds.dds_creation_txg, - numbuf, (u_longlong_t)usedobjs, blkbuf); + numbuf, (u_longlong_t)usedobjs, blkbuf, + (dds.dds_inconsistent) ? " (inconsistent)" : ""); if (zopt_objects != 0) { for (i = 0; i < zopt_objects; i++) From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:08:50 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B48615B7AC4; Mon, 3 Jun 2019 17:08:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 120AC8B1D9; Mon, 3 Jun 2019 17:08:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DDD8A9137; Mon, 3 Jun 2019 17:08:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53H8nDE025086; Mon, 3 Jun 2019 17:08:49 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53H8n2S025085; Mon, 3 Jun 2019 17:08:49 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031708.x53H8n2S025085@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:08:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348553 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 348553 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 120AC8B1D9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:08:50 -0000 Author: mav Date: Mon Jun 3 17:08:49 2019 New Revision: 348553 URL: https://svnweb.freebsd.org/changeset/base/348553 Log: 9681 ztest failure in spa_history_log_internal due to spa_rename() illumos/illumos-gate@6aee0ad76969eb0027131b3a338f2d94ae86f728 Reviewed by: Prakash Surya Reviewed by: Serapheim Dimitropoulos Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/ztest/ztest.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Mon Jun 3 17:06:02 2019 (r348552) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Mon Jun 3 17:08:49 2019 (r348553) @@ -222,9 +222,6 @@ * vdev state is protected by spa_vdev_state_enter() / spa_vdev_state_exit(). * Like spa_vdev_enter/exit, these are convenience wrappers -- the actual * locking is, always, based on spa_namespace_lock and spa_config_lock[]. - * - * spa_rename() is also implemented within this file since it requires - * manipulation of the namespace. */ static avl_tree_t spa_namespace_avl; @@ -1331,56 +1328,6 @@ spa_deactivate_mos_feature(spa_t *spa, const char *fea { if (nvlist_remove_all(spa->spa_label_features, feature) == 0) vdev_config_dirty(spa->spa_root_vdev); -} - -/* - * Rename a spa_t. - */ -int -spa_rename(const char *name, const char *newname) -{ - spa_t *spa; - int err; - - /* - * Lookup the spa_t and grab the config lock for writing. We need to - * actually open the pool so that we can sync out the necessary labels. - * It's OK to call spa_open() with the namespace lock held because we - * allow recursive calls for other reasons. - */ - mutex_enter(&spa_namespace_lock); - if ((err = spa_open(name, &spa, FTAG)) != 0) { - mutex_exit(&spa_namespace_lock); - return (err); - } - - spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); - - avl_remove(&spa_namespace_avl, spa); - (void) strlcpy(spa->spa_name, newname, sizeof (spa->spa_name)); - avl_add(&spa_namespace_avl, spa); - - /* - * Sync all labels to disk with the new names by marking the root vdev - * dirty and waiting for it to sync. It will pick up the new pool name - * during the sync. - */ - vdev_config_dirty(spa->spa_root_vdev); - - spa_config_exit(spa, SCL_ALL, FTAG); - - txg_wait_synced(spa->spa_dsl_pool, 0); - - /* - * Sync the updated config cache. - */ - spa_write_cachefile(spa, B_FALSE, B_TRUE); - - spa_close(spa, FTAG); - - mutex_exit(&spa_namespace_lock); - - return (0); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Mon Jun 3 17:06:02 2019 (r348552) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Mon Jun 3 17:08:49 2019 (r348553) @@ -821,7 +821,6 @@ extern void spa_load_note(spa_t *spa, const char *fmt, extern void spa_activate_mos_feature(spa_t *spa, const char *feature, dmu_tx_t *tx); extern void spa_deactivate_mos_feature(spa_t *spa, const char *feature); -extern int spa_rename(const char *oldname, const char *newname); extern spa_t *spa_by_guid(uint64_t pool_guid, uint64_t device_guid); extern boolean_t spa_guid_exists(uint64_t pool_guid, uint64_t device_guid); extern char *spa_strdup(const char *); From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:08:50 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0C2F15B7AC8; Mon, 3 Jun 2019 17:08:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67A8D8B1DA; Mon, 3 Jun 2019 17:08:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 418869138; Mon, 3 Jun 2019 17:08:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53H8oHY025092; Mon, 3 Jun 2019 17:08:50 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53H8oGr025091; Mon, 3 Jun 2019 17:08:50 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031708.x53H8oGr025091@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:08:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348553 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 348553 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 67A8D8B1DA X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:08:51 -0000 Author: mav Date: Mon Jun 3 17:08:49 2019 New Revision: 348553 URL: https://svnweb.freebsd.org/changeset/base/348553 Log: 9681 ztest failure in spa_history_log_internal due to spa_rename() illumos/illumos-gate@6aee0ad76969eb0027131b3a338f2d94ae86f728 Reviewed by: Prakash Surya Reviewed by: Serapheim Dimitropoulos Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor/illumos/dist/cmd/ztest/ztest.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Mon Jun 3 17:06:02 2019 (r348552) +++ vendor/illumos/dist/cmd/ztest/ztest.c Mon Jun 3 17:08:49 2019 (r348553) @@ -334,7 +334,6 @@ ztest_func_t ztest_spa_create_destroy; ztest_func_t ztest_fault_inject; ztest_func_t ztest_ddt_repair; ztest_func_t ztest_dmu_snapshot_hold; -ztest_func_t ztest_spa_rename; ztest_func_t ztest_scrub; ztest_func_t ztest_dsl_dataset_promote_busy; ztest_func_t ztest_vdev_attach_detach; @@ -379,7 +378,6 @@ ztest_info_t ztest_info[] = { { ztest_ddt_repair, 1, &zopt_sometimes }, { ztest_dmu_snapshot_hold, 1, &zopt_sometimes }, { ztest_reguid, 1, &zopt_rarely }, - { ztest_spa_rename, 1, &zopt_rarely }, { ztest_scrub, 1, &zopt_rarely }, { ztest_spa_upgrade, 1, &zopt_rarely }, { ztest_dsl_dataset_promote_busy, 1, &zopt_rarely }, @@ -5420,59 +5418,6 @@ ztest_reguid(ztest_ds_t *zd, uint64_t id) VERIFY3U(load, ==, spa_load_guid(spa)); } -/* - * Rename the pool to a different name and then rename it back. - */ -/* ARGSUSED */ -void -ztest_spa_rename(ztest_ds_t *zd, uint64_t id) -{ - char *oldname, *newname; - spa_t *spa; - - rw_enter(&ztest_name_lock, RW_WRITER); - - oldname = ztest_opts.zo_pool; - newname = umem_alloc(strlen(oldname) + 5, UMEM_NOFAIL); - (void) strcpy(newname, oldname); - (void) strcat(newname, "_tmp"); - - /* - * Do the rename - */ - VERIFY3U(0, ==, spa_rename(oldname, newname)); - - /* - * Try to open it under the old name, which shouldn't exist - */ - VERIFY3U(ENOENT, ==, spa_open(oldname, &spa, FTAG)); - - /* - * Open it under the new name and make sure it's still the same spa_t. - */ - VERIFY3U(0, ==, spa_open(newname, &spa, FTAG)); - - ASSERT(spa == ztest_spa); - spa_close(spa, FTAG); - - /* - * Rename it back to the original - */ - VERIFY3U(0, ==, spa_rename(newname, oldname)); - - /* - * Make sure it can still be opened - */ - VERIFY3U(0, ==, spa_open(oldname, &spa, FTAG)); - - ASSERT(spa == ztest_spa); - spa_close(spa, FTAG); - - umem_free(newname, strlen(newname) + 1); - - rw_exit(&ztest_name_lock); -} - static vdev_t * ztest_random_concrete_vdev_leaf(vdev_t *vd) { @@ -6525,7 +6470,6 @@ main(int argc, char **argv) ztest_shared_callstate_t *zc; char timebuf[100]; char numbuf[NN_NUMBUF_SZ]; - spa_t *spa; char *cmd; boolean_t hasalt; char *fd_data_str = getenv("ZTEST_FD_DATA"); @@ -6699,24 +6643,6 @@ main(int argc, char **argv) } (void) printf("\n"); } - - /* - * It's possible that we killed a child during a rename test, - * in which case we'll have a 'ztest_tmp' pool lying around - * instead of 'ztest'. Do a blind rename in case this happened. - */ - kernel_init(FREAD); - if (spa_open(ztest_opts.zo_pool, &spa, FTAG) == 0) { - spa_close(spa, FTAG); - } else { - char tmpname[ZFS_MAX_DATASET_NAME_LEN]; - kernel_fini(); - kernel_init(FREAD | FWRITE); - (void) snprintf(tmpname, sizeof (tmpname), "%s_tmp", - ztest_opts.zo_pool); - (void) spa_rename(tmpname, ztest_opts.zo_pool); - } - kernel_fini(); ztest_run_zdb(ztest_opts.zo_pool); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:11:27 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD6C715B7BCF; Mon, 3 Jun 2019 17:11:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 706008B522; Mon, 3 Jun 2019 17:11:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 31C19918B; Mon, 3 Jun 2019 17:11:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53HBRNx027052; Mon, 3 Jun 2019 17:11:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53HBQBg027050; Mon, 3 Jun 2019 17:11:26 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031711.x53HBQBg027050@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:11:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348554 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348554 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 706008B522 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:11:28 -0000 Author: mav Date: Mon Jun 3 17:11:26 2019 New Revision: 348554 URL: https://svnweb.freebsd.org/changeset/base/348554 Log: 9688 aggsum_fini leaks memory illumos/illumos-gate@29bf2d68bef208274f5a54a14cc80c4a8cb76f53 Reviewed by: Serapheim Dimitropoulos Reviewed by: Matt Ahrens Reviewed by: Prashanth Sreenivasa Reviewed by: Jorgen Lundman Reviewed by: Igor Kozhukhov Approved by: Robert Mustacchi Author: Paul Dagnelie Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/aggsum.c vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/aggsum.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/aggsum.c Mon Jun 3 17:08:49 2019 (r348553) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/aggsum.c Mon Jun 3 17:11:26 2019 (r348554) @@ -13,7 +13,7 @@ * CDDL HEADER END */ /* - * Copyright (c) 2017 by Delphix. All rights reserved. + * Copyright (c) 2017, 2018 by Delphix. All rights reserved. */ #include @@ -99,6 +99,7 @@ aggsum_fini(aggsum_t *as) { for (int i = 0; i < as->as_numbuckets; i++) mutex_destroy(&as->as_buckets[i].asc_lock); + kmem_free(as->as_buckets, as->as_numbuckets * sizeof (aggsum_bucket_t)); mutex_destroy(&as->as_lock); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Mon Jun 3 17:08:49 2019 (r348553) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Mon Jun 3 17:11:26 2019 (r348554) @@ -6159,6 +6159,14 @@ arc_state_fini(void) multilist_destroy(arc_mru_ghost->arcs_list[ARC_BUFC_DATA]); multilist_destroy(arc_mfu->arcs_list[ARC_BUFC_DATA]); multilist_destroy(arc_mfu_ghost->arcs_list[ARC_BUFC_DATA]); + + aggsum_fini(&arc_meta_used); + aggsum_fini(&arc_size); + aggsum_fini(&astat_data_size); + aggsum_fini(&astat_metadata_size); + aggsum_fini(&astat_hdr_size); + aggsum_fini(&astat_other_size); + aggsum_fini(&astat_l2_hdr_size); } uint64_t @@ -6328,8 +6336,13 @@ arc_fini(void) mutex_destroy(&arc_adjust_lock); cv_destroy(&arc_adjust_waiters_cv); - arc_state_fini(); + /* + * buf_fini() must proceed arc_state_fini() because buf_fin() may + * trigger the release of kmem magazines, which can callback to + * arc_space_return() which accesses aggsums freed in act_state_fini(). + */ buf_fini(); + arc_state_fini(); ASSERT0(arc_loaned_bytes); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:13:36 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0237815B7DDF; Mon, 3 Jun 2019 17:13:36 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9AE2D8B85E; Mon, 3 Jun 2019 17:13:35 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7276D92E3; Mon, 3 Jun 2019 17:13:35 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53HDZbq030591; Mon, 3 Jun 2019 17:13:35 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53HDZSa030590; Mon, 3 Jun 2019 17:13:35 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031713.x53HDZSa030590@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:13:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348555 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348555 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 9AE2D8B85E X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:13:36 -0000 Author: mav Date: Mon Jun 3 17:13:35 2019 New Revision: 348555 URL: https://svnweb.freebsd.org/changeset/base/348555 Log: 9690 metaslab of vdev with no space maps was flushed during removal illumos/illumos-gate@4e75ba682600b2bf19d158577d528c3db65050e8 Reviewed by: Matt Ahrens Reviewed by: Brad Lewis Approved by: Robert Mustacchi Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Mon Jun 3 17:11:26 2019 (r348554) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Mon Jun 3 17:13:35 2019 (r348555) @@ -2823,11 +2823,11 @@ vdev_destroy_spacemaps(vdev_t *vd, dmu_tx_t *tx) } static void -vdev_remove_empty(vdev_t *vd, uint64_t txg) +vdev_remove_empty_log(vdev_t *vd, uint64_t txg) { spa_t *spa = vd->vdev_spa; - dmu_tx_t *tx; + ASSERT(vd->vdev_islog); ASSERT(vd == vd->vdev_top); ASSERT3U(txg, ==, spa_syncing_txg(spa)); @@ -2871,13 +2871,14 @@ vdev_remove_empty(vdev_t *vd, uint64_t txg) ASSERT0(mg->mg_histogram[i]); } - tx = dmu_tx_create_assigned(spa_get_dsl(spa), txg); - vdev_destroy_spacemaps(vd, tx); + dmu_tx_t *tx = dmu_tx_create_assigned(spa_get_dsl(spa), txg); - if (vd->vdev_islog && vd->vdev_top_zap != 0) { + vdev_destroy_spacemaps(vd, tx); + if (vd->vdev_top_zap != 0) { vdev_destroy_unlink_zap(vd, vd->vdev_top_zap, tx); vd->vdev_top_zap = 0; } + dmu_tx_commit(tx); } @@ -2949,14 +2950,11 @@ vdev_sync(vdev_t *vd, uint64_t txg) vdev_dtl_sync(lvd, txg); /* - * Remove the metadata associated with this vdev once it's empty. - * Note that this is typically used for log/cache device removal; - * we don't empty toplevel vdevs when removing them. But if - * a toplevel happens to be emptied, this is not harmful. + * If this is an empty log device being removed, destroy the + * metadata associated with it. */ - if (vd->vdev_stat.vs_alloc == 0 && vd->vdev_removing) { - vdev_remove_empty(vd, txg); - } + if (vd->vdev_islog && vd->vdev_stat.vs_alloc == 0 && vd->vdev_removing) + vdev_remove_empty_log(vd, txg); (void) txg_list_add(&spa->spa_vdev_txg_list, vd, TXG_CLEAN(txg)); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 17:14:55 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 685EC15B7E77; Mon, 3 Jun 2019 17:14:55 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 08CE08B9A8; Mon, 3 Jun 2019 17:14:55 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D101892E5; Mon, 3 Jun 2019 17:14:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53HEs6d030685; Mon, 3 Jun 2019 17:14:54 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53HEs5Z030684; Mon, 3 Jun 2019 17:14:54 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031714.x53HEs5Z030684@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 17:14:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348556 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348556 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 08CE08B9A8 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 17:14:55 -0000 Author: mav Date: Mon Jun 3 17:14:54 2019 New Revision: 348556 URL: https://svnweb.freebsd.org/changeset/base/348556 Log: 9738 9112 broke third block copy allocations within one metaslab group illumos/illumos-gate@b86e7e3f0e50748bb5bb5cc91632d72ff17f08dd Reviewed by: Paul Dagnelie Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Alexander Motin Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Mon Jun 3 17:13:35 2019 (r348555) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Mon Jun 3 17:14:54 2019 (r348556) @@ -3080,7 +3080,6 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ metaslab_t *msp = NULL; uint64_t offset = -1ULL; uint64_t activation_weight; - boolean_t tertiary = B_FALSE; activation_weight = METASLAB_WEIGHT_PRIMARY; for (int i = 0; i < d; i++) { @@ -3089,7 +3088,7 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ activation_weight = METASLAB_WEIGHT_SECONDARY; } else if (activation_weight == METASLAB_WEIGHT_SECONDARY && DVA_GET_VDEV(&dva[i]) == mg->mg_vd->vdev_id) { - tertiary = B_TRUE; + activation_weight = METASLAB_WEIGHT_CLAIM; break; } } @@ -3098,10 +3097,8 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ * If we don't have enough metaslabs active to fill the entire array, we * just use the 0th slot. */ - if (mg->mg_ms_ready < mg->mg_allocators * 2) { - tertiary = B_FALSE; + if (mg->mg_ms_ready < mg->mg_allocators * 3) allocator = 0; - } ASSERT3U(mg->mg_vd->vdev_ms_count, >=, 2); @@ -3127,7 +3124,7 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ msp = mg->mg_primaries[allocator]; was_active = B_TRUE; } else if (activation_weight == METASLAB_WEIGHT_SECONDARY && - mg->mg_secondaries[allocator] != NULL && !tertiary) { + mg->mg_secondaries[allocator] != NULL) { msp = mg->mg_secondaries[allocator]; was_active = B_TRUE; } else { @@ -3170,7 +3167,8 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ continue; } - if (msp->ms_weight & METASLAB_WEIGHT_CLAIM) { + if (msp->ms_weight & METASLAB_WEIGHT_CLAIM && + activation_weight != METASLAB_WEIGHT_CLAIM) { metaslab_passivate(msp, msp->ms_weight & ~METASLAB_WEIGHT_CLAIM); mutex_exit(&msp->ms_lock); From owner-svn-src-vendor@freebsd.org Mon Jun 3 19:09:41 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3210E15BAD9F; Mon, 3 Jun 2019 19:09:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6E9E6A52F; Mon, 3 Jun 2019 19:09:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 97D35A55C; Mon, 3 Jun 2019 19:09:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53J9e1v090305; Mon, 3 Jun 2019 19:09:40 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53J9el1090302; Mon, 3 Jun 2019 19:09:40 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031909.x53J9el1090302@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 19:09:40 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348568 - in vendor/illumos/dist: cmd/zfs man/man1m X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor/illumos/dist: cmd/zfs man/man1m X-SVN-Commit-Revision: 348568 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: C6E9E6A52F X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.963,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 19:09:41 -0000 Author: mav Date: Mon Jun 3 19:09:39 2019 New Revision: 348568 URL: https://svnweb.freebsd.org/changeset/base/348568 Log: 9466 add JSON output support to channel programs illumos/illumos-gate@5267591016146502784860802129b16dab6f135c Reviewed by: Matt Ahrens Reviewed by: Serapheim Dimitropoulos Reviewed by: Sara Hartse Reviewed by: John Kennedy Approved by: Dan McDonald Author: Alek Pinchuk Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/man/man1m/zfs-program.1m vendor/illumos/dist/man/man1m/zfs.1m Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c ============================================================================== --- vendor/illumos/dist/cmd/zfs/zfs_main.c Mon Jun 3 19:03:24 2019 (r348567) +++ vendor/illumos/dist/cmd/zfs/zfs_main.c Mon Jun 3 19:09:39 2019 (r348568) @@ -29,6 +29,7 @@ * Copyright (c) 2014 Integros [integros.com] * Copyright 2016 Igor Kozhukhov . * Copyright 2016 Nexenta Systems, Inc. + * Copyright (c) 2018 Datto Inc. */ #include @@ -335,7 +336,7 @@ get_usage(zfs_help_t idx) case HELP_BOOKMARK: return (gettext("\tbookmark \n")); case HELP_CHANNEL_PROGRAM: - return (gettext("\tprogram [-n] [-t ] " + return (gettext("\tprogram [-jn] [-t ] " "[-m ] " "[lua args...]\n")); } @@ -7072,12 +7073,12 @@ zfs_do_channel_program(int argc, char **argv) nvlist_t *outnvl; uint64_t instrlimit = ZCP_DEFAULT_INSTRLIMIT; uint64_t memlimit = ZCP_DEFAULT_MEMLIMIT; - boolean_t sync_flag = B_TRUE; + boolean_t sync_flag = B_TRUE, json_output = B_FALSE; zpool_handle_t *zhp; /* check options */ while (-1 != - (c = getopt(argc, argv, "nt:(instr-limit)m:(memory-limit)"))) { + (c = getopt(argc, argv, "jnt:(instr-limit)m:(memory-limit)"))) { switch (c) { case 't': case 'm': { @@ -7119,6 +7120,10 @@ zfs_do_channel_program(int argc, char **argv) sync_flag = B_FALSE; break; } + case 'j': { + json_output = B_TRUE; + break; + } case '?': (void) fprintf(stderr, gettext("invalid option '%c'\n"), optopt); @@ -7237,11 +7242,14 @@ zfs_do_channel_program(int argc, char **argv) gettext("Channel program execution failed:\n%s\n"), errstring); } else { - (void) printf("Channel program fully executed "); - if (nvlist_empty(outnvl)) { - (void) printf("with no return value.\n"); + if (json_output) { + (void) nvlist_print_json(stdout, outnvl); + } else if (nvlist_empty(outnvl)) { + (void) fprintf(stdout, gettext("Channel program fully " + "executed and did not produce output.\n")); } else { - (void) printf("with return value:\n"); + (void) fprintf(stdout, gettext("Channel program fully " + "executed and produced output:\n")); dump_nvlist(outnvl, 4); } } Modified: vendor/illumos/dist/man/man1m/zfs-program.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zfs-program.1m Mon Jun 3 19:03:24 2019 (r348567) +++ vendor/illumos/dist/man/man1m/zfs-program.1m Mon Jun 3 19:09:39 2019 (r348568) @@ -9,6 +9,7 @@ .\" .\" .\" Copyright (c) 2016, 2017 by Delphix. All rights reserved. +.\" Copyright (c) 2018 Datto Inc. .\" .Dd January 21, 2016 .Dt ZFS-PROGRAM 1M @@ -18,7 +19,7 @@ .Nd executes ZFS channel programs .Sh SYNOPSIS .Cm "zfs program" -.Op Fl n +.Op Fl jn .Op Fl t Ar instruction-limit .Op Fl m Ar memory-limit .Ar pool @@ -46,6 +47,11 @@ will be run on and any attempts to access or modify other pools will cause an error. .Sh OPTIONS .Bl -tag -width "-t" +.It Fl j +Display channel program output in JSON format. +When this flag is specified and standard output is empty - +channel program encountered an error. +The details of such an error will be printed to standard error in plain text. .It Fl n Executes a read-only channel program, which runs faster. The program cannot change on-disk state by calling functions from the Modified: vendor/illumos/dist/man/man1m/zfs.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zfs.1m Mon Jun 3 19:03:24 2019 (r348567) +++ vendor/illumos/dist/man/man1m/zfs.1m Mon Jun 3 19:09:39 2019 (r348568) @@ -28,6 +28,7 @@ .\" Copyright (c) 2014 Integros [integros.com] .\" Copyright 2017 Nexenta Systems, Inc. .\" Copyright 2018 Joyent, Inc. +.\" Copyright (c) 2018 Datto Inc. .\" .Dd Feb 10, 2018 .Dt ZFS 1M @@ -271,7 +272,7 @@ .Ar snapshot Ar snapshot Ns | Ns Ar filesystem .Nm .Cm program -.Op Fl n +.Op Fl jn .Op Fl t Ar timeout .Op Fl m Ar memory_limit .Ar pool script @@ -3472,7 +3473,7 @@ Display the path's inode change time as the first colu .It Xo .Nm .Cm program -.Op Fl n +.Op Fl jn .Op Fl t Ar timeout .Op Fl m Ar memory_limit .Ar pool script @@ -3493,6 +3494,11 @@ Channel programs may only be run with root privileges. For full documentation of the ZFS channel program interface, see the manual page for .Bl -tag -width "" +.It Fl j +Display channel program output in JSON format. +When this flag is specified and standard output is empty - +channel program encountered an error. +The details of such an error will be printed to standard error in plain text. .It Fl n Executes a read-only channel program, which runs faster. The program cannot change on-disk state by calling functions from From owner-svn-src-vendor@freebsd.org Mon Jun 3 19:20:11 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C6F815BB1AC; Mon, 3 Jun 2019 19:20:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 238B76AE88; Mon, 3 Jun 2019 19:20:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EFEEDA712; Mon, 3 Jun 2019 19:20:10 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53JKAiG095809; Mon, 3 Jun 2019 19:20:10 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53JKAZ9095808; Mon, 3 Jun 2019 19:20:10 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031920.x53JKAZ9095808@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 19:20:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348572 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 348572 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 238B76AE88 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.963,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 19:20:11 -0000 Author: mav Date: Mon Jun 3 19:20:10 2019 New Revision: 348572 URL: https://svnweb.freebsd.org/changeset/base/348572 Log: 9880 Race in ZFS parallel mount illumos/illumos-gate@bc4c0ff1343a311cc24933908ac6c4455af09031 Reviewed by: Jason King Reviewed by: Sebastien Roy Approved by: Joshua M. Clulow Author: Andy Fiddaman Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Mon Jun 3 19:19:35 2019 (r348571) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Mon Jun 3 19:20:10 2019 (r348572) @@ -26,6 +26,7 @@ * Copyright 2016 Igor Kozhukhov * Copyright 2017 Joyent, Inc. * Copyright 2017 RackTop Systems. + * Copyright 2018 OmniOS Community Edition (OmniOSce) Association. */ /* @@ -1142,19 +1143,28 @@ zfs_iter_cb(zfs_handle_t *zhp, void *data) /* * Sort comparator that compares two mountpoint paths. We sort these paths so * that subdirectories immediately follow their parents. This means that we - * effectively treat the '/' character as the lowest value non-nul char. An - * example sorted list using this comparator would look like: + * effectively treat the '/' character as the lowest value non-nul char. + * Since filesystems from non-global zones can have the same mountpoint + * as other filesystems, the comparator sorts global zone filesystems to + * the top of the list. This means that the global zone will traverse the + * filesystem list in the correct order and can stop when it sees the + * first zoned filesystem. In a non-global zone, only the delegated + * filesystems are seen. * + * An example sorted list using this comparator would look like: + * * /foo * /foo/bar * /foo/bar/baz * /foo/baz * /foo.bar + * /foo (NGZ1) + * /foo (NGZ2) * * The mounting code depends on this ordering to deterministically iterate * over filesystems in order to spawn parallel mount tasks. */ -int +static int mountpoint_cmp(const void *arga, const void *argb) { zfs_handle_t *const *zap = arga; @@ -1166,7 +1176,15 @@ mountpoint_cmp(const void *arga, const void *argb) const char *a = mounta; const char *b = mountb; boolean_t gota, gotb; + uint64_t zoneda, zonedb; + zoneda = zfs_prop_get_int(za, ZFS_PROP_ZONED); + zonedb = zfs_prop_get_int(zb, ZFS_PROP_ZONED); + if (zoneda && !zonedb) + return (1); + if (!zoneda && zonedb) + return (-1); + gota = (zfs_get_type(za) == ZFS_TYPE_FILESYSTEM); if (gota) { verify(zfs_prop_get(za, ZFS_PROP_MOUNTPOINT, mounta, @@ -1379,6 +1397,8 @@ void zfs_foreach_mountpoint(libzfs_handle_t *hdl, zfs_handle_t **handles, size_t num_handles, zfs_iter_f func, void *data, boolean_t parallel) { + zoneid_t zoneid = getzoneid(); + /* * The ZFS_SERIAL_MOUNT environment variable is an undocumented * variable that can be used as a convenience to do a/b comparison @@ -1414,6 +1434,14 @@ zfs_foreach_mountpoint(libzfs_handle_t *hdl, zfs_handl */ for (int i = 0; i < num_handles; i = non_descendant_idx(handles, num_handles, i)) { + /* + * Since the mountpoints have been sorted so that the zoned + * filesystems are at the end, a zoned filesystem seen from + * the global zone means that we're done. + */ + if (zoneid == GLOBAL_ZONEID && + zfs_prop_get_int(handles[i], ZFS_PROP_ZONED)) + break; zfs_dispatch_mount(hdl, handles, num_handles, i, func, data, tq); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 19:24:41 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE88415BB762; Mon, 3 Jun 2019 19:24:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C1956B513; Mon, 3 Jun 2019 19:24:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 22EE5A8B5; Mon, 3 Jun 2019 19:24:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53JOepf000638; Mon, 3 Jun 2019 19:24:40 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53JOekq000637; Mon, 3 Jun 2019 19:24:40 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031924.x53JOekq000637@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 19:24:40 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348573 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348573 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 5C1956B513 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.963,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 19:24:41 -0000 Author: mav Date: Mon Jun 3 19:24:40 2019 New Revision: 348573 URL: https://svnweb.freebsd.org/changeset/base/348573 Log: 9993 zil writes can get delayed in zio pipeline illumos/illumos-gate@2258ad0b755b24a55c6173b1e6bb6188389f72dd Reviewed by: Prakash Surya Reviewed by: Brad Lewis Reviewed by: Matt Ahrens Approved by: Dan McDonald Author: George Wilson Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Mon Jun 3 19:20:10 2019 (r348572) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Mon Jun 3 19:24:40 2019 (r348573) @@ -1497,7 +1497,8 @@ zio_taskq_dispatch(zio_t *zio, zio_taskq_type_t q, boo * If this is a high priority I/O, then use the high priority taskq if * available. */ - if (zio->io_priority == ZIO_PRIORITY_NOW && + if ((zio->io_priority == ZIO_PRIORITY_NOW || + zio->io_priority == ZIO_PRIORITY_SYNC_WRITE) && spa->spa_zio_taskq[t][q + 1].stqs_count != 0) q++; From owner-svn-src-vendor@freebsd.org Mon Jun 3 19:53:40 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B08CF15BC3E4; Mon, 3 Jun 2019 19:53:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52B3D6C8DD; Mon, 3 Jun 2019 19:53:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F4009ADCA; Mon, 3 Jun 2019 19:53:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53Jrd97016469; Mon, 3 Jun 2019 19:53:39 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53JrdsH016468; Mon, 3 Jun 2019 19:53:39 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906031953.x53JrdsH016468@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 19:53:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348576 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348576 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 52B3D6C8DD X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 19:53:41 -0000 Author: mav Date: Mon Jun 3 19:53:39 2019 New Revision: 348576 URL: https://svnweb.freebsd.org/changeset/base/348576 Log: 9963 Seperate tunable for disabling ZIL vdev flush illumos/illumos-gate@f8fdf6812567ce1a3b76a6bee2ee3f04bd3fcaca Reviewed by: Matt Ahrens Reviewed by: Brad Lewis Reviewed by: Patrick Mooney Approved by: Dan McDonald Author: Prakash Surya Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Mon Jun 3 19:37:14 2019 (r348575) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Mon Jun 3 19:53:39 2019 (r348576) @@ -38,6 +38,13 @@ #include /* + * Tunable parameter for debugging or performance analysis. Setting this + * will cause pool corruption on power loss if a volatile out-of-order + * write cache is enabled. + */ +boolean_t zfs_nocacheflush = B_FALSE; + +/* * Virtual device vector for disks. */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Mon Jun 3 19:37:14 2019 (r348575) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Mon Jun 3 19:53:39 2019 (r348576) @@ -95,11 +95,12 @@ int zfs_commit_timeout_pct = 5; int zil_replay_disable = 0; /* - * Tunable parameter for debugging or performance analysis. Setting - * zfs_nocacheflush will cause corruption on power loss if a volatile - * out-of-order write cache is enabled. + * Disable the DKIOCFLUSHWRITECACHE commands that are normally sent to + * the disk(s) by the ZIL after an LWB write has completed. Setting this + * will cause ZIL corruption on power loss if a volatile out-of-order + * write cache is enabled. */ -boolean_t zfs_nocacheflush = B_FALSE; +boolean_t zil_nocacheflush = B_FALSE; /* * Limit SLOG write size per commit executed with synchronous priority. @@ -991,7 +992,7 @@ zil_lwb_add_block(lwb_t *lwb, const blkptr_t *bp) int ndvas = BP_GET_NDVAS(bp); int i; - if (zfs_nocacheflush) + if (zil_nocacheflush) return; mutex_enter(&lwb->lwb_vdev_lock); @@ -1015,7 +1016,7 @@ zil_lwb_add_txg(lwb_t *lwb, uint64_t txg) /* * This function is a called after all VDEVs associated with a given lwb * write have completed their DKIOCFLUSHWRITECACHE command; or as soon - * as the lwb write completes, if "zfs_nocacheflush" is set. + * as the lwb write completes, if "zil_nocacheflush" is set. * * The intention is for this function to be called as soon as the * contents of an lwb are considered "stable" on disk, and will survive From owner-svn-src-vendor@freebsd.org Mon Jun 3 20:18:04 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB6BB15BCAE5; Mon, 3 Jun 2019 20:18:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A1826D6D2; Mon, 3 Jun 2019 20:18:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 750A8B12F; Mon, 3 Jun 2019 20:18:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53KI3De029317; Mon, 3 Jun 2019 20:18:03 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53KI20i029312; Mon, 3 Jun 2019 20:18:02 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906032018.x53KI20i029312@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 20:18:02 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348578 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 348578 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 9A1826D6D2 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 20:18:04 -0000 Author: mav Date: Mon Jun 3 20:18:02 2019 New Revision: 348578 URL: https://svnweb.freebsd.org/changeset/base/348578 Log: 9962 zil_commit should omit cache thrash illumos/illumos-gate@cab3a55e158118937e07d059c46f1bc14d1f254d Reviewed by: Matt Ahrens Reviewed by: Brad Lewis Reviewed by: Patrick Mooney Reviewed by: Jerry Jelinek Approved by: Joshua M. Clulow Author: Prakash Surya Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/ztest/ztest.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Mon Jun 3 20:05:43 2019 (r348577) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Mon Jun 3 20:18:02 2019 (r348578) @@ -1706,7 +1706,16 @@ dmu_sync_done(zio_t *zio, arc_buf_t *buf, void *varg) dmu_sync_arg_t *dsa = varg; dbuf_dirty_record_t *dr = dsa->dsa_dr; dmu_buf_impl_t *db = dr->dr_dbuf; + zgd_t *zgd = dsa->dsa_zgd; + /* + * Record the vdev(s) backing this blkptr so they can be flushed after + * the writes for the lwb have completed. + */ + if (zio->io_error == 0) { + zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); + } + mutex_enter(&db->db_mtx); ASSERT(dr->dt.dl.dr_override_state == DR_IN_DMU_SYNC); if (zio->io_error == 0) { @@ -1756,13 +1765,22 @@ dmu_sync_late_arrival_done(zio_t *zio) blkptr_t *bp = zio->io_bp; dmu_sync_arg_t *dsa = zio->io_private; blkptr_t *bp_orig = &zio->io_bp_orig; + zgd_t *zgd = dsa->dsa_zgd; - if (zio->io_error == 0 && !BP_IS_HOLE(bp)) { - ASSERT(!(zio->io_flags & ZIO_FLAG_NOPWRITE)); - ASSERT(BP_IS_HOLE(bp_orig) || !BP_EQUAL(bp, bp_orig)); - ASSERT(zio->io_bp->blk_birth == zio->io_txg); - ASSERT(zio->io_txg > spa_syncing_txg(zio->io_spa)); - zio_free(zio->io_spa, zio->io_txg, zio->io_bp); + if (zio->io_error == 0) { + /* + * Record the vdev(s) backing this blkptr so they can be + * flushed after the writes for the lwb have completed. + */ + zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); + + if (!BP_IS_HOLE(bp)) { + ASSERT(!(zio->io_flags & ZIO_FLAG_NOPWRITE)); + ASSERT(BP_IS_HOLE(bp_orig) || !BP_EQUAL(bp, bp_orig)); + ASSERT(zio->io_bp->blk_birth == zio->io_txg); + ASSERT(zio->io_txg > spa_syncing_txg(zio->io_spa)); + zio_free(zio->io_spa, zio->io_txg, zio->io_bp); + } } dmu_tx_commit(dsa->dsa_tx); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h Mon Jun 3 20:05:43 2019 (r348577) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h Mon Jun 3 20:18:02 2019 (r348578) @@ -48,10 +48,11 @@ extern "C" { * via zil_lwb_write_issue(). Again, the zilog's "zl_issuer_lock" must * be held when making this transition. * - * After the lwb's zio completes, and the vdev's are flushed, the lwb - * will transition into the "done" state via zil_lwb_write_done(). When - * transitioning from "issued" to "done", the zilog's "zl_lock" must be - * held, *not* the "zl_issuer_lock". + * After the lwb's write zio completes, it transitions into the "write + * done" state via zil_lwb_write_done(); and then into the "flush done" + * state via zil_lwb_flush_vdevs_done(). When transitioning from + * "issued" to "write done", and then from "write done" to "flush done", + * the zilog's "zl_lock" must be held, *not* the "zl_issuer_lock". * * The zilog's "zl_issuer_lock" can become heavily contended in certain * workloads, so we specifically avoid acquiring that lock when @@ -68,13 +69,14 @@ extern "C" { * "zl_issuer_lock" will prevent a concurrent thread from transitioning * that lwb to the "issued" state. Likewise, if an lwb is already in the * "issued" state, holding the "zl_lock" will prevent a concurrent - * thread from transitioning that lwb to the "done" state. + * thread from transitioning that lwb to the "write done" state. */ typedef enum { LWB_STATE_CLOSED, LWB_STATE_OPENED, LWB_STATE_ISSUED, - LWB_STATE_DONE, + LWB_STATE_WRITE_DONE, + LWB_STATE_FLUSH_DONE, LWB_NUM_STATES } lwb_state_t; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Mon Jun 3 20:05:43 2019 (r348577) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Mon Jun 3 20:18:02 2019 (r348578) @@ -1015,6 +1015,7 @@ zfs_write(vnode_t *vp, uio_t *uio, int ioflag, cred_t return (0); } +/* ARGSUSED */ void zfs_get_done(zgd_t *zgd, int error) { @@ -1032,9 +1033,6 @@ zfs_get_done(zgd_t *zgd, int error) */ VN_RELE_ASYNC(ZTOV(zp), dsl_pool_vnrele_taskq(dmu_objset_pool(os))); - if (error == 0 && zgd->zgd_bp) - zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); - kmem_free(zgd, sizeof (zgd_t)); } @@ -1158,11 +1156,7 @@ zfs_get_data(void *arg, lr_write_t *lr, char *buf, str * TX_WRITE2 relies on the data previously * written by the TX_WRITE that caused * EALREADY. We zero out the BP because - * it is the old, currently-on-disk BP, - * so there's no need to zio_flush() its - * vdevs (flushing would needlesly hurt - * performance, and doesn't work on - * indirect vdevs). + * it is the old, currently-on-disk BP. */ zgd->zgd_bp = NULL; BP_ZERO(bp); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Mon Jun 3 20:05:43 2019 (r348577) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Mon Jun 3 20:18:02 2019 (r348578) @@ -549,7 +549,7 @@ zil_free_lwb(zilog_t *zilog, lwb_t *lwb) ASSERT3P(lwb->lwb_root_zio, ==, NULL); ASSERT3U(lwb->lwb_max_txg, <=, spa_syncing_txg(zilog->zl_spa)); ASSERT(lwb->lwb_state == LWB_STATE_CLOSED || - lwb->lwb_state == LWB_STATE_DONE); + lwb->lwb_state == LWB_STATE_FLUSH_DONE); /* * Clear the zilog's field to indicate this lwb is no longer @@ -961,7 +961,8 @@ zil_commit_waiter_link_lwb(zil_commit_waiter_t *zcw, l ASSERT3P(zcw->zcw_lwb, ==, NULL); ASSERT3P(lwb, !=, NULL); ASSERT(lwb->lwb_state == LWB_STATE_OPENED || - lwb->lwb_state == LWB_STATE_ISSUED); + lwb->lwb_state == LWB_STATE_ISSUED || + lwb->lwb_state == LWB_STATE_WRITE_DONE); list_insert_tail(&lwb->lwb_waiters, zcw); zcw->zcw_lwb = lwb; @@ -1007,6 +1008,42 @@ zil_lwb_add_block(lwb_t *lwb, const blkptr_t *bp) mutex_exit(&lwb->lwb_vdev_lock); } +static void +zil_lwb_flush_defer(lwb_t *lwb, lwb_t *nlwb) +{ + avl_tree_t *src = &lwb->lwb_vdev_tree; + avl_tree_t *dst = &nlwb->lwb_vdev_tree; + void *cookie = NULL; + zil_vdev_node_t *zv; + + ASSERT3S(lwb->lwb_state, ==, LWB_STATE_WRITE_DONE); + ASSERT3S(nlwb->lwb_state, !=, LWB_STATE_WRITE_DONE); + ASSERT3S(nlwb->lwb_state, !=, LWB_STATE_FLUSH_DONE); + + /* + * While 'lwb' is at a point in its lifetime where lwb_vdev_tree does + * not need the protection of lwb_vdev_lock (it will only be modified + * while holding zilog->zl_lock) as its writes and those of its + * children have all completed. The younger 'nlwb' may be waiting on + * future writes to additional vdevs. + */ + mutex_enter(&nlwb->lwb_vdev_lock); + /* + * Tear down the 'lwb' vdev tree, ensuring that entries which do not + * exist in 'nlwb' are moved to it, freeing any would-be duplicates. + */ + while ((zv = avl_destroy_nodes(src, &cookie)) != NULL) { + avl_index_t where; + + if (avl_find(dst, zv, &where) == NULL) { + avl_insert(dst, zv, where); + } else { + kmem_free(zv, sizeof (*zv)); + } + } + mutex_exit(&nlwb->lwb_vdev_lock); +} + void zil_lwb_add_txg(lwb_t *lwb, uint64_t txg) { @@ -1014,9 +1051,13 @@ zil_lwb_add_txg(lwb_t *lwb, uint64_t txg) } /* - * This function is a called after all VDEVs associated with a given lwb + * This function is a called after all vdevs associated with a given lwb * write have completed their DKIOCFLUSHWRITECACHE command; or as soon - * as the lwb write completes, if "zil_nocacheflush" is set. + * as the lwb write completes, if "zil_nocacheflush" is set. Further, + * all "previous" lwb's will have completed before this function is + * called; i.e. this function is called for all previous lwbs before + * it's called for "this" lwb (enforced via zio the dependencies + * configured in zil_lwb_set_zio_dependency()). * * The intention is for this function to be called as soon as the * contents of an lwb are considered "stable" on disk, and will survive @@ -1053,8 +1094,10 @@ zil_lwb_flush_vdevs_done(zio_t *zio) zilog->zl_last_lwb_latency = gethrtime() - lwb->lwb_issued_timestamp; lwb->lwb_root_zio = NULL; - lwb->lwb_state = LWB_STATE_DONE; + ASSERT3S(lwb->lwb_state, ==, LWB_STATE_WRITE_DONE); + lwb->lwb_state = LWB_STATE_FLUSH_DONE; + if (zilog->zl_last_lwb_opened == lwb) { /* * Remember the highest committed log sequence number @@ -1094,14 +1137,17 @@ zil_lwb_flush_vdevs_done(zio_t *zio) } /* - * This is called when an lwb write completes. This means, this specific - * lwb was written to disk, and all dependent lwb have also been - * written to disk. - * - * At this point, a DKIOCFLUSHWRITECACHE command hasn't been issued to - * the VDEVs involved in writing out this specific lwb. The lwb will be - * "done" once zil_lwb_flush_vdevs_done() is called, which occurs in the - * zio completion callback for the lwb's root zio. + * This is called when an lwb's write zio completes. The callback's + * purpose is to issue the DKIOCFLUSHWRITECACHE commands for the vdevs + * in the lwb's lwb_vdev_tree. The tree will contain the vdevs involved + * in writing out this specific lwb's data, and in the case that cache + * flushes have been deferred, vdevs involved in writing the data for + * previous lwbs. The writes corresponding to all the vdevs in the + * lwb_vdev_tree will have completed by the time this is called, due to + * the zio dependencies configured in zil_lwb_set_zio_dependency(), + * which takes deferred flushes into account. The lwb will be "done" + * once zil_lwb_flush_vdevs_done() is called, which occurs in the zio + * completion callback for the lwb's root zio. */ static void zil_lwb_write_done(zio_t *zio) @@ -1112,6 +1158,7 @@ zil_lwb_write_done(zio_t *zio) avl_tree_t *t = &lwb->lwb_vdev_tree; void *cookie = NULL; zil_vdev_node_t *zv; + lwb_t *nlwb; ASSERT3S(spa_config_held(spa, SCL_STATE, RW_READER), !=, 0); @@ -1125,10 +1172,11 @@ zil_lwb_write_done(zio_t *zio) abd_put(zio->io_abd); - ASSERT3S(lwb->lwb_state, ==, LWB_STATE_ISSUED); - mutex_enter(&zilog->zl_lock); + ASSERT3S(lwb->lwb_state, ==, LWB_STATE_ISSUED); + lwb->lwb_state = LWB_STATE_WRITE_DONE; lwb->lwb_write_zio = NULL; + nlwb = list_next(&zilog->zl_lwb_list, lwb); mutex_exit(&zilog->zl_lock); if (avl_numnodes(t) == 0) @@ -1147,6 +1195,27 @@ zil_lwb_write_done(zio_t *zio) return; } + /* + * If this lwb does not have any threads waiting for it to + * complete, we want to defer issuing the DKIOCFLUSHWRITECACHE + * command to the vdevs written to by "this" lwb, and instead + * rely on the "next" lwb to handle the DKIOCFLUSHWRITECACHE + * command for those vdevs. Thus, we merge the vdev tree of + * "this" lwb with the vdev tree of the "next" lwb in the list, + * and assume the "next" lwb will handle flushing the vdevs (or + * deferring the flush(s) again). + * + * This is a useful performance optimization, especially for + * workloads with lots of async write activity and few sync + * write and/or fsync activity, as it has the potential to + * coalesce multiple flush commands to a vdev into one. + */ + if (list_head(&lwb->lwb_waiters) == NULL && nlwb != NULL) { + zil_lwb_flush_defer(lwb, nlwb); + ASSERT(avl_is_empty(&lwb->lwb_vdev_tree)); + return; + } + while ((zv = avl_destroy_nodes(t, &cookie)) != NULL) { vdev_t *vd = vdev_lookup_top(spa, zv->zv_vdev); if (vd != NULL) @@ -1155,6 +1224,73 @@ zil_lwb_write_done(zio_t *zio) } } +static void +zil_lwb_set_zio_dependency(zilog_t *zilog, lwb_t *lwb) +{ + lwb_t *last_lwb_opened = zilog->zl_last_lwb_opened; + + ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock)); + ASSERT(MUTEX_HELD(&zilog->zl_lock)); + + /* + * The zilog's "zl_last_lwb_opened" field is used to build the + * lwb/zio dependency chain, which is used to preserve the + * ordering of lwb completions that is required by the semantics + * of the ZIL. Each new lwb zio becomes a parent of the + * "previous" lwb zio, such that the new lwb's zio cannot + * complete until the "previous" lwb's zio completes. + * + * This is required by the semantics of zil_commit(); the commit + * waiters attached to the lwbs will be woken in the lwb zio's + * completion callback, so this zio dependency graph ensures the + * waiters are woken in the correct order (the same order the + * lwbs were created). + */ + if (last_lwb_opened != NULL && + last_lwb_opened->lwb_state != LWB_STATE_FLUSH_DONE) { + ASSERT(last_lwb_opened->lwb_state == LWB_STATE_OPENED || + last_lwb_opened->lwb_state == LWB_STATE_ISSUED || + last_lwb_opened->lwb_state == LWB_STATE_WRITE_DONE); + + ASSERT3P(last_lwb_opened->lwb_root_zio, !=, NULL); + zio_add_child(lwb->lwb_root_zio, + last_lwb_opened->lwb_root_zio); + + /* + * If the previous lwb's write hasn't already completed, + * we also want to order the completion of the lwb write + * zios (above, we only order the completion of the lwb + * root zios). This is required because of how we can + * defer the DKIOCFLUSHWRITECACHE commands for each lwb. + * + * When the DKIOCFLUSHWRITECACHE commands are defered, + * the previous lwb will rely on this lwb to flush the + * vdevs written to by that previous lwb. Thus, we need + * to ensure this lwb doesn't issue the flush until + * after the previous lwb's write completes. We ensure + * this ordering by setting the zio parent/child + * relationship here. + * + * Without this relationship on the lwb's write zio, + * it's possible for this lwb's write to complete prior + * to the previous lwb's write completing; and thus, the + * vdevs for the previous lwb would be flushed prior to + * that lwb's data being written to those vdevs (the + * vdevs are flushed in the lwb write zio's completion + * handler, zil_lwb_write_done()). + */ + if (last_lwb_opened->lwb_state != LWB_STATE_WRITE_DONE) { + ASSERT(last_lwb_opened->lwb_state == LWB_STATE_OPENED || + last_lwb_opened->lwb_state == LWB_STATE_ISSUED); + + ASSERT3P(last_lwb_opened->lwb_write_zio, !=, NULL); + zio_add_child(lwb->lwb_write_zio, + last_lwb_opened->lwb_write_zio); + } + } +} + + /* * This function's purpose is to "open" an lwb such that it is ready to * accept new itxs being committed to it. To do this, the lwb's zio @@ -1199,33 +1335,8 @@ zil_lwb_write_open(zilog_t *zilog, lwb_t *lwb) lwb->lwb_state = LWB_STATE_OPENED; mutex_enter(&zilog->zl_lock); - - /* - * The zilog's "zl_last_lwb_opened" field is used to - * build the lwb/zio dependency chain, which is used to - * preserve the ordering of lwb completions that is - * required by the semantics of the ZIL. Each new lwb - * zio becomes a parent of the "previous" lwb zio, such - * that the new lwb's zio cannot complete until the - * "previous" lwb's zio completes. - * - * This is required by the semantics of zil_commit(); - * the commit waiters attached to the lwbs will be woken - * in the lwb zio's completion callback, so this zio - * dependency graph ensures the waiters are woken in the - * correct order (the same order the lwbs were created). - */ - lwb_t *last_lwb_opened = zilog->zl_last_lwb_opened; - if (last_lwb_opened != NULL && - last_lwb_opened->lwb_state != LWB_STATE_DONE) { - ASSERT(last_lwb_opened->lwb_state == LWB_STATE_OPENED || - last_lwb_opened->lwb_state == LWB_STATE_ISSUED); - ASSERT3P(last_lwb_opened->lwb_root_zio, !=, NULL); - zio_add_child(lwb->lwb_root_zio, - last_lwb_opened->lwb_root_zio); - } + zil_lwb_set_zio_dependency(zilog, lwb); zilog->zl_last_lwb_opened = lwb; - mutex_exit(&zilog->zl_lock); } @@ -1926,7 +2037,8 @@ zil_prune_commit_list(zilog_t *zilog) mutex_enter(&zilog->zl_lock); lwb_t *last_lwb = zilog->zl_last_lwb_opened; - if (last_lwb == NULL || last_lwb->lwb_state == LWB_STATE_DONE) { + if (last_lwb == NULL || + last_lwb->lwb_state == LWB_STATE_FLUSH_DONE) { /* * All of the itxs this waiter was waiting on * must have already completed (or there were @@ -2007,7 +2119,8 @@ zil_process_commit_list(zilog_t *zilog) lwb = zil_create(zilog); } else { ASSERT3S(lwb->lwb_state, !=, LWB_STATE_ISSUED); - ASSERT3S(lwb->lwb_state, !=, LWB_STATE_DONE); + ASSERT3S(lwb->lwb_state, !=, LWB_STATE_WRITE_DONE); + ASSERT3S(lwb->lwb_state, !=, LWB_STATE_FLUSH_DONE); } while (itx = list_head(&zilog->zl_itx_commit_list)) { @@ -2109,7 +2222,8 @@ zil_process_commit_list(zilog_t *zilog) ASSERT(list_is_empty(&nolwb_waiters)); ASSERT3P(lwb, !=, NULL); ASSERT3S(lwb->lwb_state, !=, LWB_STATE_ISSUED); - ASSERT3S(lwb->lwb_state, !=, LWB_STATE_DONE); + ASSERT3S(lwb->lwb_state, !=, LWB_STATE_WRITE_DONE); + ASSERT3S(lwb->lwb_state, !=, LWB_STATE_FLUSH_DONE); /* * At this point, the ZIL block pointed at by the "lwb" @@ -2230,7 +2344,8 @@ zil_commit_waiter_timeout(zilog_t *zilog, zil_commit_w * acquiring it when it's not necessary to do so. */ if (lwb->lwb_state == LWB_STATE_ISSUED || - lwb->lwb_state == LWB_STATE_DONE) + lwb->lwb_state == LWB_STATE_WRITE_DONE || + lwb->lwb_state == LWB_STATE_FLUSH_DONE) return; /* @@ -2278,7 +2393,8 @@ zil_commit_waiter_timeout(zilog_t *zilog, zil_commit_w * more details on the lwb states, and locking requirements. */ if (lwb->lwb_state == LWB_STATE_ISSUED || - lwb->lwb_state == LWB_STATE_DONE) + lwb->lwb_state == LWB_STATE_WRITE_DONE || + lwb->lwb_state == LWB_STATE_FLUSH_DONE) goto out; ASSERT3S(lwb->lwb_state, ==, LWB_STATE_OPENED); @@ -2451,7 +2567,8 @@ zil_commit_waiter(zilog_t *zilog, zil_commit_waiter_t IMPLY(lwb != NULL, lwb->lwb_state == LWB_STATE_ISSUED || - lwb->lwb_state == LWB_STATE_DONE); + lwb->lwb_state == LWB_STATE_WRITE_DONE || + lwb->lwb_state == LWB_STATE_FLUSH_DONE); cv_wait(&zcw->zcw_cv, &zcw->zcw_lock); } } @@ -3093,13 +3210,13 @@ zil_suspend(const char *osname, void **cookiep) * to disk before proceeding. If we used zil_commit instead, it * would just call txg_wait_synced(), because zl_suspend is set. * txg_wait_synced() doesn't wait for these lwb's to be - * LWB_STATE_DONE before returning. + * LWB_STATE_FLUSH_DONE before returning. */ zil_commit_impl(zilog, 0); /* - * Now that we've ensured all lwb's are LWB_STATE_DONE, we use - * txg_wait_synced() to ensure the data from the zilog has + * Now that we've ensured all lwb's are LWB_STATE_FLUSH_DONE, we + * use txg_wait_synced() to ensure the data from the zilog has * migrated to the main pool before calling zil_destroy(). */ txg_wait_synced(zilog->zl_dmu_pool, 0); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Mon Jun 3 20:05:43 2019 (r348577) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Mon Jun 3 20:18:02 2019 (r348578) @@ -973,6 +973,7 @@ zvol_close(dev_t dev, int flag, int otyp, cred_t *cr) return (error); } +/* ARGSUSED */ static void zvol_get_done(zgd_t *zgd, int error) { @@ -980,9 +981,6 @@ zvol_get_done(zgd_t *zgd, int error) dmu_buf_rele(zgd->zgd_db, zgd); zfs_range_unlock(zgd->zgd_rl); - - if (error == 0 && zgd->zgd_bp) - zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); kmem_free(zgd, sizeof (zgd_t)); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 20:18:04 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5939115BCAE6; Mon, 3 Jun 2019 20:18:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EDD0D6D6D3; Mon, 3 Jun 2019 20:18:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C5BEEB130; Mon, 3 Jun 2019 20:18:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53KI3EX029323; Mon, 3 Jun 2019 20:18:03 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53KI36Q029322; Mon, 3 Jun 2019 20:18:03 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906032018.x53KI36Q029322@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 20:18:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348578 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 348578 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: EDD0D6D6D3 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.975,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 20:18:04 -0000 Author: mav Date: Mon Jun 3 20:18:02 2019 New Revision: 348578 URL: https://svnweb.freebsd.org/changeset/base/348578 Log: 9962 zil_commit should omit cache thrash illumos/illumos-gate@cab3a55e158118937e07d059c46f1bc14d1f254d Reviewed by: Matt Ahrens Reviewed by: Brad Lewis Reviewed by: Patrick Mooney Reviewed by: Jerry Jelinek Approved by: Joshua M. Clulow Author: Prakash Surya Modified: vendor/illumos/dist/cmd/ztest/ztest.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Mon Jun 3 20:05:43 2019 (r348577) +++ vendor/illumos/dist/cmd/ztest/ztest.c Mon Jun 3 20:18:02 2019 (r348578) @@ -1840,6 +1840,7 @@ zil_replay_func_t *ztest_replay_vector[TX_MAX_TYPE] = * ZIL get_data callbacks */ +/* ARGSUSED */ static void ztest_get_done(zgd_t *zgd, int error) { @@ -1851,9 +1852,6 @@ ztest_get_done(zgd_t *zgd, int error) ztest_range_unlock(zgd->zgd_rl); ztest_object_unlock(zd, object); - - if (error == 0 && zgd->zgd_bp) - zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); umem_free(zgd, sizeof (*zgd)); } From owner-svn-src-vendor@freebsd.org Mon Jun 3 20:39:14 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6C3A15BD433; Mon, 3 Jun 2019 20:39:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4EBEE6E58B; Mon, 3 Jun 2019 20:39:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E9EA8B47C; Mon, 3 Jun 2019 20:39:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53KdDKR039662; Mon, 3 Jun 2019 20:39:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53KdDiN039661; Mon, 3 Jun 2019 20:39:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906032039.x53KdDiN039661@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 20:39:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348580 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 348580 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4EBEE6E58B X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.975,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 20:39:14 -0000 Author: mav Date: Mon Jun 3 20:39:13 2019 New Revision: 348580 URL: https://svnweb.freebsd.org/changeset/base/348580 Log: 9559 zfs diff handles files on delete queue in fromsnap poorly illumos/illumos-gate@20633e304b57bc98f70fdb194081b7023adf527b Reviewed by: Joshua M. Clulow Approved by: Richard Lowe Author: Paul Dagnelie Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Mon Jun 3 20:24:40 2019 (r348579) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Mon Jun 3 20:39:13 2019 (r348580) @@ -22,7 +22,7 @@ /* * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2015 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2015, 2017 by Delphix. All rights reserved. + * Copyright (c) 2015, 2018 by Delphix. All rights reserved. * Copyright 2016 Joyent, Inc. * Copyright 2016 Igor Kozhukhov */ @@ -49,7 +49,7 @@ #include "libzfs_impl.h" #define ZDIFF_SNAPDIR "/.zfs/snapshot/" -#define ZDIFF_SHARESDIR "/.zfs/shares/" +#define ZDIFF_SHARESDIR "/.zfs/shares/" #define ZDIFF_PREFIX "zfs-diff-%d" #define ZDIFF_ADDED '+' @@ -359,12 +359,12 @@ describe_free(FILE *fp, differ_info_t *di, uint64_t ob if (get_stats_for_obj(di, di->fromsnap, object, namebuf, maxlen, &sb) != 0) { - /* Let it slide, if in the delete queue on from side */ - if (di->zerr == ENOENT && sb.zs_links == 0) { - di->zerr = 0; - return (0); - } return (-1); + } + /* Don't print if in the delete queue on from side */ + if (di->zerr == ESTALE) { + di->zerr = 0; + return (0); } print_file(fp, di, ZDIFF_REMOVED, namebuf, &sb); From owner-svn-src-vendor@freebsd.org Mon Jun 3 20:45:49 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8049315BD82B; Mon, 3 Jun 2019 20:45:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2751C6ECD5; Mon, 3 Jun 2019 20:45:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 011BAB635; Mon, 3 Jun 2019 20:45:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53KjmNK044953; Mon, 3 Jun 2019 20:45:48 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53Kjm0p044951; Mon, 3 Jun 2019 20:45:48 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906032045.x53Kjm0p044951@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 20:45:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348583 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 348583 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 2751C6ECD5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.975,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 20:45:49 -0000 Author: mav Date: Mon Jun 3 20:45:48 2019 New Revision: 348583 URL: https://svnweb.freebsd.org/changeset/base/348583 Log: 9847 leaking dd_clones (DMU_OT_DSL_CLONES) objects illumos/illumos-gate@17fb938fd6cdce3ff1bb47dafda0774f742249a3 Reviewed by: George Wilson Reviewed by: Serapheim Dimitropoulos Reviewed by: Brian Behlendorf Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Mon Jun 3 20:45:32 2019 (r348582) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Mon Jun 3 20:45:48 2019 (r348583) @@ -786,6 +786,8 @@ dsl_dir_destroy_sync(uint64_t ddobj, dmu_tx_t *tx) VERIFY0(zap_destroy(mos, dsl_dir_phys(dd)->dd_child_dir_zapobj, tx)); VERIFY0(zap_destroy(mos, dsl_dir_phys(dd)->dd_props_zapobj, tx)); + if (dsl_dir_phys(dd)->dd_clones != 0) + VERIFY0(zap_destroy(mos, dsl_dir_phys(dd)->dd_clones, tx)); VERIFY0(dsl_deleg_destroy(mos, dsl_dir_phys(dd)->dd_deleg_zapobj, tx)); VERIFY0(zap_remove(mos, dsl_dir_phys(dd->dd_parent)->dd_child_dir_zapobj, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Mon Jun 3 20:45:32 2019 (r348582) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Mon Jun 3 20:45:48 2019 (r348583) @@ -279,7 +279,6 @@ vdev_indirect_mapping_entry_for_offset_or_next(vdev_in B_TRUE)); } - void vdev_indirect_mapping_close(vdev_indirect_mapping_t *vim) { From owner-svn-src-vendor@freebsd.org Mon Jun 3 20:45:50 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3C3515BD830; Mon, 3 Jun 2019 20:45:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 79D656ECD6; Mon, 3 Jun 2019 20:45:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 54CC3B636; Mon, 3 Jun 2019 20:45:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53KjnYo044959; Mon, 3 Jun 2019 20:45:49 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53KjnN3044958; Mon, 3 Jun 2019 20:45:49 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906032045.x53KjnN3044958@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 20:45:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348583 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 348583 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 79D656ECD6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 20:45:50 -0000 Author: mav Date: Mon Jun 3 20:45:48 2019 New Revision: 348583 URL: https://svnweb.freebsd.org/changeset/base/348583 Log: 9847 leaking dd_clones (DMU_OT_DSL_CLONES) objects illumos/illumos-gate@17fb938fd6cdce3ff1bb47dafda0774f742249a3 Reviewed by: George Wilson Reviewed by: Serapheim Dimitropoulos Reviewed by: Brian Behlendorf Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor/illumos/dist/cmd/zdb/zdb.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Mon Jun 3 20:45:32 2019 (r348582) +++ vendor/illumos/dist/cmd/zdb/zdb.c Mon Jun 3 20:45:48 2019 (r348583) @@ -63,6 +63,7 @@ #include #include #include +#include #include #include #undef verify @@ -111,6 +112,7 @@ uint64_t max_inflight = 1000; static int leaked_objects = 0; static void snprintf_blkptr_compact(char *, size_t, const blkptr_t *); +static void mos_obj_refd(uint64_t); /* * These libumem hooks provide a reasonable set of defaults for the allocator's @@ -1565,6 +1567,8 @@ dump_dsl_dir(objset_t *os, uint64_t object, void *data DO(CHILD_RSRV); DO(REFRSRV); #undef DO + (void) printf("\t\tclones = %llu\n", + (u_longlong_t)dd->dd_clones); } /*ARGSUSED*/ @@ -1747,6 +1751,33 @@ dump_full_bpobj(bpobj_t *bpo, const char *name, int in } static void +bpobj_count_refd(bpobj_t *bpo) +{ + mos_obj_refd(bpo->bpo_object); + + if (bpo->bpo_havesubobj && bpo->bpo_phys->bpo_subobjs != 0) { + mos_obj_refd(bpo->bpo_phys->bpo_subobjs); + for (uint64_t i = 0; i < bpo->bpo_phys->bpo_num_subobjs; i++) { + uint64_t subobj; + bpobj_t subbpo; + int error; + VERIFY0(dmu_read(bpo->bpo_os, + bpo->bpo_phys->bpo_subobjs, + i * sizeof (subobj), sizeof (subobj), &subobj, 0)); + error = bpobj_open(&subbpo, bpo->bpo_os, subobj); + if (error != 0) { + (void) printf("ERROR %u while trying to open " + "subobj id %llu\n", + error, (u_longlong_t)subobj); + continue; + } + bpobj_count_refd(&subbpo); + bpobj_close(&subbpo); + } + } +} + +static void dump_deadlist(dsl_deadlist_t *dl) { dsl_deadlist_entry_t *dle; @@ -1754,7 +1785,24 @@ dump_deadlist(dsl_deadlist_t *dl) char bytes[32]; char comp[32]; char uncomp[32]; + uint64_t empty_bpobj = + dmu_objset_spa(dl->dl_os)->spa_dsl_pool->dp_empty_bpobj; + /* force the tree to be loaded */ + dsl_deadlist_space_range(dl, 0, UINT64_MAX, &unused, &unused, &unused); + + if (dl->dl_oldfmt) { + if (dl->dl_bpobj.bpo_object != empty_bpobj) + bpobj_count_refd(&dl->dl_bpobj); + } else { + mos_obj_refd(dl->dl_object); + for (dle = avl_first(&dl->dl_tree); dle; + dle = AVL_NEXT(&dl->dl_tree, dle)) { + if (dle->dle_bpobj.bpo_object != empty_bpobj) + bpobj_count_refd(&dle->dle_bpobj); + } + } + /* make sure nicenum has enough space */ CTASSERT(sizeof (bytes) >= NN_NUMBUF_SZ); CTASSERT(sizeof (comp) >= NN_NUMBUF_SZ); @@ -1779,9 +1827,6 @@ dump_deadlist(dsl_deadlist_t *dl) (void) printf("\n"); - /* force the tree to be loaded */ - dsl_deadlist_space_range(dl, 0, UINT64_MAX, &unused, &unused, &unused); - for (dle = avl_first(&dl->dl_tree); dle; dle = AVL_NEXT(&dl->dl_tree, dle)) { if (dump_opt['d'] >= 5) { @@ -1796,7 +1841,6 @@ dump_deadlist(dsl_deadlist_t *dl) (void) printf("mintxg %llu -> obj %llu\n", (longlong_t)dle->dle_mintxg, (longlong_t)dle->dle_bpobj.bpo_object); - } } } @@ -2193,6 +2237,30 @@ dump_object(objset_t *os, uint64_t object, int verbosi dmu_buf_rele(db, FTAG); } +static void +count_dir_mos_objects(dsl_dir_t *dd) +{ + mos_obj_refd(dd->dd_object); + mos_obj_refd(dsl_dir_phys(dd)->dd_child_dir_zapobj); + mos_obj_refd(dsl_dir_phys(dd)->dd_deleg_zapobj); + mos_obj_refd(dsl_dir_phys(dd)->dd_props_zapobj); + mos_obj_refd(dsl_dir_phys(dd)->dd_clones); +} + +static void +count_ds_mos_objects(dsl_dataset_t *ds) +{ + mos_obj_refd(ds->ds_object); + mos_obj_refd(dsl_dataset_phys(ds)->ds_next_clones_obj); + mos_obj_refd(dsl_dataset_phys(ds)->ds_props_obj); + mos_obj_refd(dsl_dataset_phys(ds)->ds_userrefs_obj); + mos_obj_refd(dsl_dataset_phys(ds)->ds_snapnames_zapobj); + + if (!dsl_dataset_is_snapshot(ds)) { + count_dir_mos_objects(ds->ds_dir); + } +} + static const char *objset_types[DMU_OST_NUMTYPES] = { "NONE", "META", "ZPL", "ZVOL", "OTHER", "ANY" }; @@ -2270,6 +2338,7 @@ dump_dir(objset_t *os) (void) printf("ds_remap_deadlist:\n"); dump_deadlist(&ds->ds_remap_deadlist); } + count_ds_mos_objects(ds); } if (verbosity < 2) @@ -4429,7 +4498,162 @@ verify_checkpoint(spa_t *spa) return (error); } +/* ARGSUSED */ static void +mos_leaks_cb(void *arg, uint64_t start, uint64_t size) +{ + for (uint64_t i = start; i < size; i++) { + (void) printf("MOS object %llu referenced but not allocated\n", + (u_longlong_t)i); + } +} + +static range_tree_t *mos_refd_objs; + +static void +mos_obj_refd(uint64_t obj) +{ + if (obj != 0 && mos_refd_objs != NULL) + range_tree_add(mos_refd_objs, obj, 1); +} + +static void +mos_leak_vdev(vdev_t *vd) +{ + mos_obj_refd(vd->vdev_dtl_object); + mos_obj_refd(vd->vdev_ms_array); + mos_obj_refd(vd->vdev_top_zap); + mos_obj_refd(vd->vdev_indirect_config.vic_births_object); + mos_obj_refd(vd->vdev_indirect_config.vic_mapping_object); + mos_obj_refd(vd->vdev_leaf_zap); + if (vd->vdev_checkpoint_sm != NULL) + mos_obj_refd(vd->vdev_checkpoint_sm->sm_object); + if (vd->vdev_indirect_mapping != NULL) { + mos_obj_refd(vd->vdev_indirect_mapping-> + vim_phys->vimp_counts_object); + } + if (vd->vdev_obsolete_sm != NULL) + mos_obj_refd(vd->vdev_obsolete_sm->sm_object); + + for (uint64_t m = 0; m < vd->vdev_ms_count; m++) { + metaslab_t *ms = vd->vdev_ms[m]; + mos_obj_refd(space_map_object(ms->ms_sm)); + } + + for (uint64_t c = 0; c < vd->vdev_children; c++) { + mos_leak_vdev(vd->vdev_child[c]); + } +} + +static int +dump_mos_leaks(spa_t *spa) +{ + int rv = 0; + objset_t *mos = spa->spa_meta_objset; + dsl_pool_t *dp = spa->spa_dsl_pool; + + /* Visit and mark all referenced objects in the MOS */ + + mos_obj_refd(DMU_POOL_DIRECTORY_OBJECT); + mos_obj_refd(spa->spa_pool_props_object); + mos_obj_refd(spa->spa_config_object); + mos_obj_refd(spa->spa_ddt_stat_object); + mos_obj_refd(spa->spa_feat_desc_obj); + mos_obj_refd(spa->spa_feat_enabled_txg_obj); + mos_obj_refd(spa->spa_feat_for_read_obj); + mos_obj_refd(spa->spa_feat_for_write_obj); + mos_obj_refd(spa->spa_history); + mos_obj_refd(spa->spa_errlog_last); + mos_obj_refd(spa->spa_errlog_scrub); + mos_obj_refd(spa->spa_all_vdev_zaps); + mos_obj_refd(spa->spa_dsl_pool->dp_bptree_obj); + mos_obj_refd(spa->spa_dsl_pool->dp_tmp_userrefs_obj); + mos_obj_refd(spa->spa_dsl_pool->dp_scan->scn_phys.scn_queue_obj); + bpobj_count_refd(&spa->spa_deferred_bpobj); + mos_obj_refd(dp->dp_empty_bpobj); + bpobj_count_refd(&dp->dp_obsolete_bpobj); + bpobj_count_refd(&dp->dp_free_bpobj); + mos_obj_refd(spa->spa_l2cache.sav_object); + mos_obj_refd(spa->spa_spares.sav_object); + + mos_obj_refd(spa->spa_condensing_indirect_phys. + scip_next_mapping_object); + mos_obj_refd(spa->spa_condensing_indirect_phys. + scip_prev_obsolete_sm_object); + if (spa->spa_condensing_indirect_phys.scip_next_mapping_object != 0) { + vdev_indirect_mapping_t *vim = + vdev_indirect_mapping_open(mos, + spa->spa_condensing_indirect_phys.scip_next_mapping_object); + mos_obj_refd(vim->vim_phys->vimp_counts_object); + vdev_indirect_mapping_close(vim); + } + + if (dp->dp_origin_snap != NULL) { + dsl_dataset_t *ds; + + dsl_pool_config_enter(dp, FTAG); + VERIFY0(dsl_dataset_hold_obj(dp, + dsl_dataset_phys(dp->dp_origin_snap)->ds_next_snap_obj, + FTAG, &ds)); + count_ds_mos_objects(ds); + dump_deadlist(&ds->ds_deadlist); + dsl_dataset_rele(ds, FTAG); + dsl_pool_config_exit(dp, FTAG); + + count_ds_mos_objects(dp->dp_origin_snap); + dump_deadlist(&dp->dp_origin_snap->ds_deadlist); + } + count_dir_mos_objects(dp->dp_mos_dir); + if (dp->dp_free_dir != NULL) + count_dir_mos_objects(dp->dp_free_dir); + if (dp->dp_leak_dir != NULL) + count_dir_mos_objects(dp->dp_leak_dir); + + mos_leak_vdev(spa->spa_root_vdev); + + for (uint64_t class = 0; class < DDT_CLASSES; class++) { + for (uint64_t type = 0; type < DDT_TYPES; type++) { + for (uint64_t cksum = 0; + cksum < ZIO_CHECKSUM_FUNCTIONS; cksum++) { + ddt_t *ddt = spa->spa_ddt[cksum]; + mos_obj_refd(ddt->ddt_object[type][class]); + } + } + } + + /* + * Visit all allocated objects and make sure they are referenced. + */ + uint64_t object = 0; + while (dmu_object_next(mos, &object, B_FALSE, 0) == 0) { + if (range_tree_contains(mos_refd_objs, object, 1)) { + range_tree_remove(mos_refd_objs, object, 1); + } else { + dmu_object_info_t doi; + const char *name; + dmu_object_info(mos, object, &doi); + if (doi.doi_type & DMU_OT_NEWTYPE) { + dmu_object_byteswap_t bswap = + DMU_OT_BYTESWAP(doi.doi_type); + name = dmu_ot_byteswap[bswap].ob_name; + } else { + name = dmu_ot[doi.doi_type].ot_name; + } + + (void) printf("MOS object %llu (%s) leaked\n", + (u_longlong_t)object, name); + rv = 2; + } + } + (void) range_tree_walk(mos_refd_objs, mos_leaks_cb, NULL); + if (!range_tree_is_empty(mos_refd_objs)) + rv = 2; + range_tree_vacate(mos_refd_objs, NULL, NULL); + range_tree_destroy(mos_refd_objs); + return (rv); +} + +static void dump_zpool(spa_t *spa) { dsl_pool_t *dp = spa_get_dsl(spa); @@ -4460,7 +4684,9 @@ dump_zpool(spa_t *spa) dump_metaslab_groups(spa); if (dump_opt['d'] || dump_opt['i']) { + mos_refd_objs = range_tree_create(NULL, NULL); dump_dir(dp->dp_meta_objset); + if (dump_opt['d'] >= 3) { dsl_pool_t *dp = spa->spa_dsl_pool; dump_full_bpobj(&spa->spa_deferred_bpobj, @@ -4487,6 +4713,9 @@ dump_zpool(spa_t *spa) (void) dmu_objset_find(spa_name(spa), dump_one_dir, NULL, DS_FIND_SNAPSHOTS | DS_FIND_CHILDREN); + if (rc == 0 && !dump_opt['L']) + rc = dump_mos_leaks(spa); + for (spa_feature_t f = 0; f < SPA_FEATURES; f++) { uint64_t refcount; @@ -4516,6 +4745,7 @@ dump_zpool(spa_t *spa) rc = verify_device_removal_feature_counts(spa); } } + if (rc == 0 && (dump_opt['b'] || dump_opt['c'])) rc = dump_block_stats(spa); From owner-svn-src-vendor@freebsd.org Mon Jun 3 20:52:21 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E460015BDBB5; Mon, 3 Jun 2019 20:52:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 89F816F402; Mon, 3 Jun 2019 20:52:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 64127B7CA; Mon, 3 Jun 2019 20:52:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x53KqKhh048062; Mon, 3 Jun 2019 20:52:20 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x53KqKvX048061; Mon, 3 Jun 2019 20:52:20 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906032052.x53KqKvX048061@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 3 Jun 2019 20:52:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348585 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 348585 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 89F816F402 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 20:52:21 -0000 Author: mav Date: Mon Jun 3 20:52:19 2019 New Revision: 348585 URL: https://svnweb.freebsd.org/changeset/base/348585 Log: 9683 Allow bypassing devid in vdev_disk_open() illumos/illumos-gate@6fe4f3002ca33af5ab7123ed78d81899dadf2fbb Reviewed by: Sara Hartse Reviewed by: George Wilson Approved by: Dan McDonald Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Mon Jun 3 20:49:20 2019 (r348584) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Mon Jun 3 20:52:19 2019 (r348585) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2016 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright 2016 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2013 Joyent, Inc. All rights reserved. */ @@ -57,6 +57,21 @@ typedef struct vdev_disk_ldi_cb { ldi_callback_id_t lcb_id; } vdev_disk_ldi_cb_t; +/* + * Bypass the devid when opening a disk vdev. + * There have been issues where the devids of several devices were shuffled, + * causing pool open failures. Note, that this flag is intended to be used + * for pool recovery only. + * + * Note that if a pool is imported with the devids bypassed, all its vdevs will + * cease storing devid information permanently. In practice, the devid is rarely + * useful as vdev paths do not tend to change unless the hardware is + * reconfigured. That said, if the paths do change and a pool fails to open + * automatically at boot, a simple zpool import should re-scan the paths and fix + * the issue. + */ +boolean_t vdev_disk_bypass_devid = B_FALSE; + static void vdev_disk_alloc(vdev_t *vd) { @@ -313,6 +328,16 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * dvd = vd->vdev_tsd; /* + * Allow bypassing the devid. + */ + if (vd->vdev_devid != NULL && vdev_disk_bypass_devid) { + vdev_dbgmsg(vd, "vdev_disk_open, devid %s bypassed", + vd->vdev_devid); + spa_strfree(vd->vdev_devid); + vd->vdev_devid = NULL; + } + + /* * When opening a disk device, we want to preserve the user's original * intent. We always want to open the device by the path the user gave * us, even if it is one of multiple paths to the same device. But we @@ -374,6 +399,19 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * if (error == 0 && vd->vdev_devid != NULL && ldi_get_devid(dvd->vd_lh, &devid) == 0) { if (ddi_devid_compare(devid, dvd->vd_devid) != 0) { + /* + * A mismatch here is unexpected, log it. + */ + char *devid_str = ddi_devid_str_encode(devid, + dvd->vd_minor); + vdev_dbgmsg(vd, "vdev_disk_open: devid " + "mismatch: %s != %s", vd->vdev_devid, + devid_str); + cmn_err(CE_NOTE, "vdev_disk_open %s: devid " + "mismatch: %s != %s", vd->vdev_path, + vd->vdev_devid, devid_str); + ddi_devid_str_free(devid_str); + error = SET_ERROR(EINVAL); (void) ldi_close(dvd->vd_lh, spa_mode(spa), kcred); @@ -397,6 +435,10 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * if (error != 0 && vd->vdev_devid != NULL) { error = ldi_open_by_devid(dvd->vd_devid, dvd->vd_minor, spa_mode(spa), kcred, &dvd->vd_lh, zfs_li); + if (error != 0) { + vdev_dbgmsg(vd, "Failed to open by devid (%s)", + vd->vdev_devid); + } } /* @@ -443,6 +485,9 @@ vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t * vd_devid = ddi_devid_str_encode(devid, dvd->vd_minor); vdev_dbgmsg(vd, "vdev_disk_open: update devid from " "'%s' to '%s'", vd->vdev_devid, vd_devid); + cmn_err(CE_NOTE, "vdev_disk_open %s: update devid " + "from '%s' to '%s'", vd->vdev_path != NULL ? + vd->vdev_path : "?", vd->vdev_devid, vd_devid); spa_strfree(vd->vdev_devid); vd->vdev_devid = spa_strdup(vd_devid); ddi_devid_str_free(vd_devid); From owner-svn-src-vendor@freebsd.org Tue Jun 4 00:42:53 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 371B915C2991; Tue, 4 Jun 2019 00:42:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D3315770E0; Tue, 4 Jun 2019 00:42:52 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AFFC8E061; Tue, 4 Jun 2019 00:42:52 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x540gq0D070941; Tue, 4 Jun 2019 00:42:52 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x540gpNl070935; Tue, 4 Jun 2019 00:42:51 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906040042.x540gpNl070935@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 4 Jun 2019 00:42:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348596 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 348596 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: D3315770E0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.974,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2019 00:42:53 -0000 Author: mav Date: Tue Jun 4 00:42:51 2019 New Revision: 348596 URL: https://svnweb.freebsd.org/changeset/base/348596 Log: 9689 zfs range lock code should not be zpl-specific illumos/illumos-gate@7931524763ef94dc16989451dddd206563d03bb4 Reviewed by: Serapheim Dimitropoulos Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_rlock.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/ztest/ztest.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Tue Jun 4 00:01:37 2019 (r348595) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Tue Jun 4 00:42:51 2019 (r348596) @@ -75,6 +75,7 @@ struct nvlist; struct arc_buf; struct zio_prop; struct sa_handle; +struct locked_range; typedef struct objset objset_t; typedef struct dmu_tx dmu_tx_t; @@ -941,7 +942,7 @@ typedef struct zgd { struct lwb *zgd_lwb; struct blkptr *zgd_bp; dmu_buf_t *zgd_db; - struct rl *zgd_rl; + struct locked_range *zgd_lr; void *zgd_private; } zgd_t; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_rlock.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_rlock.h Tue Jun 4 00:01:37 2019 (r348595) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_rlock.h Tue Jun 4 00:42:51 2019 (r348596) @@ -22,6 +22,9 @@ * Copyright 2006 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ +/* + * Copyright (c) 2018 by Delphix. All rights reserved. + */ #ifndef _SYS_FS_ZFS_RLOCK_H #define _SYS_FS_ZFS_RLOCK_H @@ -30,54 +33,44 @@ extern "C" { #endif -#ifdef _KERNEL - -#include - typedef enum { RL_READER, RL_WRITER, RL_APPEND -} rl_type_t; +} rangelock_type_t; -typedef struct rl { - znode_t *r_zp; /* znode this lock applies to */ - avl_node_t r_node; /* avl node link */ - uint64_t r_off; /* file range offset */ - uint64_t r_len; /* file range length */ - uint_t r_cnt; /* range reference count in tree */ - rl_type_t r_type; /* range type */ - kcondvar_t r_wr_cv; /* cv for waiting writers */ - kcondvar_t r_rd_cv; /* cv for waiting readers */ - uint8_t r_proxy; /* acting for original range */ - uint8_t r_write_wanted; /* writer wants to lock this range */ - uint8_t r_read_wanted; /* reader wants to lock this range */ -} rl_t; +struct locked_range; -/* - * Lock a range (offset, length) as either shared (RL_READER) - * or exclusive (RL_WRITER or RL_APPEND). RL_APPEND is a special type that - * is converted to RL_WRITER that specified to lock from the start of the - * end of file. Returns the range lock structure. - */ -rl_t *zfs_range_lock(znode_t *zp, uint64_t off, uint64_t len, rl_type_t type); +typedef void (rangelock_cb_t)(struct locked_range *, void *); -/* Unlock range and destroy range lock structure. */ -void zfs_range_unlock(rl_t *rl); +typedef struct rangelock { + avl_tree_t rl_tree; /* contains locked_range_t */ + kmutex_t rl_lock; + rangelock_cb_t *rl_cb; + void *rl_arg; +} rangelock_t; -/* - * Reduce range locked as RW_WRITER from whole file to specified range. - * Asserts the whole file was previously locked. - */ -void zfs_range_reduce(rl_t *rl, uint64_t off, uint64_t len); +typedef struct locked_range { + rangelock_t *lr_rangelock; /* rangelock that this lock applies to */ + avl_node_t lr_node; /* avl node link */ + uint64_t lr_offset; /* file range offset */ + uint64_t lr_length; /* file range length */ + uint_t lr_count; /* range reference count in tree */ + rangelock_type_t lr_type; /* range type */ + kcondvar_t lr_write_cv; /* cv for waiting writers */ + kcondvar_t lr_read_cv; /* cv for waiting readers */ + uint8_t lr_proxy; /* acting for original range */ + uint8_t lr_write_wanted; /* writer wants to lock this range */ + uint8_t lr_read_wanted; /* reader wants to lock this range */ +} locked_range_t; -/* - * AVL comparison function used to order range locks - * Locks are ordered on the start offset of the range. - */ -int zfs_range_compare(const void *arg1, const void *arg2); +void rangelock_init(rangelock_t *, rangelock_cb_t *, void *); +void rangelock_fini(rangelock_t *); -#endif /* _KERNEL */ +locked_range_t *rangelock_enter(rangelock_t *, + uint64_t, uint64_t, rangelock_type_t); +void rangelock_exit(locked_range_t *); +void rangelock_reduce(locked_range_t *, uint64_t, uint64_t); #ifdef __cplusplus } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h Tue Jun 4 00:01:37 2019 (r348595) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h Tue Jun 4 00:42:51 2019 (r348596) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2016 Nexenta Systems, Inc. All rights reserved. */ @@ -39,6 +39,7 @@ #include #include #include +#include #endif #include #include @@ -60,8 +61,8 @@ extern "C" { #define ZFS_APPENDONLY 0x0000004000000000 #define ZFS_NODUMP 0x0000008000000000 #define ZFS_OPAQUE 0x0000010000000000 -#define ZFS_AV_QUARANTINED 0x0000020000000000 -#define ZFS_AV_MODIFIED 0x0000040000000000 +#define ZFS_AV_QUARANTINED 0x0000020000000000 +#define ZFS_AV_MODIFIED 0x0000040000000000 #define ZFS_REPARSE 0x0000080000000000 #define ZFS_OFFLINE 0x0000100000000000 #define ZFS_SPARSE 0x0000200000000000 @@ -81,8 +82,8 @@ extern "C" { */ #define ZFS_XATTR 0x1 /* is an extended attribute */ #define ZFS_INHERIT_ACE 0x2 /* ace has inheritable ACEs */ -#define ZFS_ACL_TRIVIAL 0x4 /* files ACL is trivial */ -#define ZFS_ACL_OBJ_ACE 0x8 /* ACL has CMPLX Object ACE */ +#define ZFS_ACL_TRIVIAL 0x4 /* files ACL is trivial */ +#define ZFS_ACL_OBJ_ACE 0x8 /* ACL has CMPLX Object ACE */ #define ZFS_ACL_PROTECTED 0x10 /* ACL protected */ #define ZFS_ACL_DEFAULTED 0x20 /* ACL should be defaulted */ #define ZFS_ACL_AUTO_INHERIT 0x40 /* ACL should be inherited */ @@ -176,8 +177,7 @@ typedef struct znode { krwlock_t z_parent_lock; /* parent lock for directories */ krwlock_t z_name_lock; /* "master" lock for dirent locks */ zfs_dirlock_t *z_dirlocks; /* directory entry lock list */ - kmutex_t z_range_lock; /* protects changes to z_range_avl */ - avl_tree_t z_range_avl; /* avl tree of file range locks */ + rangelock_t z_rangelock; /* file range locks */ uint8_t z_unlinked; /* file has been unlinked */ uint8_t z_atime_dirty; /* atime needs to be synced */ uint8_t z_zn_prefetch; /* Prefetch znodes? */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c Tue Jun 4 00:01:37 2019 (r348595) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c Tue Jun 4 00:42:51 2019 (r348596) @@ -23,7 +23,7 @@ * Use is subject to license terms. */ /* - * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. */ /* @@ -34,9 +34,9 @@ * Interface * --------- * Defined in zfs_rlock.h but essentially: - * rl = zfs_range_lock(zp, off, len, lock_type); - * zfs_range_unlock(rl); - * zfs_range_reduce(rl, off, len); + * lr = rangelock_enter(zp, off, len, lock_type); + * rangelock_reduce(lr, off, len); // optional + * rangelock_exit(lr); * * AVL tree * -------- @@ -46,9 +46,10 @@ * * Common case * ----------- - * The (hopefully) usual case is of no overlaps or contention for - * locks. On entry to zfs_lock_range() a rl_t is allocated; the tree - * searched that finds no overlap, and *this* rl_t is placed in the tree. + * The (hopefully) usual case is of no overlaps or contention for locks. On + * entry to rangelock_enter(), a locked_range_t is allocated; the tree + * searched that finds no overlap, and *this* locked_range_t is placed in the + * tree. * * Overlaps/Reference counting/Proxy locks * --------------------------------------- @@ -87,67 +88,89 @@ * * Grow block handling * ------------------- - * ZFS supports multiple block sizes currently upto 128K. The smallest + * ZFS supports multiple block sizes, up to 16MB. The smallest * block size is used for the file which is grown as needed. During this * growth all other writers and readers must be excluded. * So if the block size needs to be grown then the whole file is * exclusively locked, then later the caller will reduce the lock - * range to just the range to be written using zfs_reduce_range. + * range to just the range to be written using rangelock_reduce(). */ +#include #include /* + * AVL comparison function used to order range locks + * Locks are ordered on the start offset of the range. + */ +static int +rangelock_compare(const void *arg1, const void *arg2) +{ + const locked_range_t *rl1 = arg1; + const locked_range_t *rl2 = arg2; + + if (rl1->lr_offset > rl2->lr_offset) + return (1); + if (rl1->lr_offset < rl2->lr_offset) + return (-1); + return (0); +} + +/* + * The callback is invoked when acquiring a RL_WRITER or RL_APPEND lock. + * It must convert RL_APPEND to RL_WRITER (starting at the end of the file), + * and may increase the range that's locked for RL_WRITER. + */ +void +rangelock_init(rangelock_t *rl, rangelock_cb_t *cb, void *arg) +{ + mutex_init(&rl->rl_lock, NULL, MUTEX_DEFAULT, NULL); + avl_create(&rl->rl_tree, rangelock_compare, + sizeof (locked_range_t), offsetof(locked_range_t, lr_node)); + rl->rl_cb = cb; + rl->rl_arg = arg; +} + +void +rangelock_fini(rangelock_t *rl) +{ + mutex_destroy(&rl->rl_lock); + avl_destroy(&rl->rl_tree); +} + +/* * Check if a write lock can be grabbed, or wait and recheck until available. */ static void -zfs_range_lock_writer(znode_t *zp, rl_t *new) +rangelock_enter_writer(rangelock_t *rl, locked_range_t *new) { - avl_tree_t *tree = &zp->z_range_avl; - rl_t *rl; + avl_tree_t *tree = &rl->rl_tree; + locked_range_t *lr; avl_index_t where; - uint64_t end_size; - uint64_t off = new->r_off; - uint64_t len = new->r_len; + uint64_t orig_off = new->lr_offset; + uint64_t orig_len = new->lr_length; + rangelock_type_t orig_type = new->lr_type; for (;;) { /* - * Range locking is also used by zvol and uses a - * dummied up znode. However, for zvol, we don't need to - * append or grow blocksize, and besides we don't have - * a "sa" data or z_zfsvfs - so skip that processing. - * - * Yes, this is ugly, and would be solved by not handling - * grow or append in range lock code. If that was done then - * we could make the range locking code generically available - * to other non-zfs consumers. + * Call callback which can modify new->r_off,len,type. + * Note, the callback is used by the ZPL to handle appending + * and changing blocksizes. It isn't needed for zvols. */ - if (zp->z_vnode) { /* caller is ZPL */ - /* - * If in append mode pick up the current end of file. - * This is done under z_range_lock to avoid races. - */ - if (new->r_type == RL_APPEND) - new->r_off = zp->z_size; - - /* - * If we need to grow the block size then grab the whole - * file range. This is also done under z_range_lock to - * avoid races. - */ - end_size = MAX(zp->z_size, new->r_off + len); - if (end_size > zp->z_blksz && (!ISP2(zp->z_blksz) || - zp->z_blksz < zp->z_zfsvfs->z_max_blksz)) { - new->r_off = 0; - new->r_len = UINT64_MAX; - } + if (rl->rl_cb != NULL) { + rl->rl_cb(new, rl->rl_arg); } /* + * If the type was APPEND, the callback must convert it to + * WRITER. + */ + ASSERT3U(new->lr_type, ==, RL_WRITER); + + /* * First check for the usual case of no locks */ if (avl_numnodes(tree) == 0) { - new->r_type = RL_WRITER; /* convert to writer */ avl_add(tree, new); return; } @@ -155,31 +178,33 @@ zfs_range_lock_writer(znode_t *zp, rl_t *new) /* * Look for any locks in the range. */ - rl = avl_find(tree, new, &where); - if (rl) + lr = avl_find(tree, new, &where); + if (lr != NULL) goto wait; /* already locked at same offset */ - rl = (rl_t *)avl_nearest(tree, where, AVL_AFTER); - if (rl && (rl->r_off < new->r_off + new->r_len)) + lr = (locked_range_t *)avl_nearest(tree, where, AVL_AFTER); + if (lr != NULL && + lr->lr_offset < new->lr_offset + new->lr_length) goto wait; - rl = (rl_t *)avl_nearest(tree, where, AVL_BEFORE); - if (rl && rl->r_off + rl->r_len > new->r_off) + lr = (locked_range_t *)avl_nearest(tree, where, AVL_BEFORE); + if (lr != NULL && + lr->lr_offset + lr->lr_length > new->lr_offset) goto wait; - new->r_type = RL_WRITER; /* convert possible RL_APPEND */ avl_insert(tree, new, where); return; wait: - if (!rl->r_write_wanted) { - cv_init(&rl->r_wr_cv, NULL, CV_DEFAULT, NULL); - rl->r_write_wanted = B_TRUE; + if (!lr->lr_write_wanted) { + cv_init(&lr->lr_write_cv, NULL, CV_DEFAULT, NULL); + lr->lr_write_wanted = B_TRUE; } - cv_wait(&rl->r_wr_cv, &zp->z_range_lock); + cv_wait(&lr->lr_write_cv, &rl->rl_lock); /* reset to original */ - new->r_off = off; - new->r_len = len; + new->lr_offset = orig_off; + new->lr_length = orig_len; + new->lr_type = orig_type; } } @@ -187,29 +212,29 @@ wait: * If this is an original (non-proxy) lock then replace it by * a proxy and return the proxy. */ -static rl_t * -zfs_range_proxify(avl_tree_t *tree, rl_t *rl) +static locked_range_t * +rangelock_proxify(avl_tree_t *tree, locked_range_t *lr) { - rl_t *proxy; + locked_range_t *proxy; - if (rl->r_proxy) - return (rl); /* already a proxy */ + if (lr->lr_proxy) + return (lr); /* already a proxy */ - ASSERT3U(rl->r_cnt, ==, 1); - ASSERT(rl->r_write_wanted == B_FALSE); - ASSERT(rl->r_read_wanted == B_FALSE); - avl_remove(tree, rl); - rl->r_cnt = 0; + ASSERT3U(lr->lr_count, ==, 1); + ASSERT(lr->lr_write_wanted == B_FALSE); + ASSERT(lr->lr_read_wanted == B_FALSE); + avl_remove(tree, lr); + lr->lr_count = 0; /* create a proxy range lock */ - proxy = kmem_alloc(sizeof (rl_t), KM_SLEEP); - proxy->r_off = rl->r_off; - proxy->r_len = rl->r_len; - proxy->r_cnt = 1; - proxy->r_type = RL_READER; - proxy->r_proxy = B_TRUE; - proxy->r_write_wanted = B_FALSE; - proxy->r_read_wanted = B_FALSE; + proxy = kmem_alloc(sizeof (locked_range_t), KM_SLEEP); + proxy->lr_offset = lr->lr_offset; + proxy->lr_length = lr->lr_length; + proxy->lr_count = 1; + proxy->lr_type = RL_READER; + proxy->lr_proxy = B_TRUE; + proxy->lr_write_wanted = B_FALSE; + proxy->lr_read_wanted = B_FALSE; avl_add(tree, proxy); return (proxy); @@ -219,29 +244,27 @@ zfs_range_proxify(avl_tree_t *tree, rl_t *rl) * Split the range lock at the supplied offset * returning the *front* proxy. */ -static rl_t * -zfs_range_split(avl_tree_t *tree, rl_t *rl, uint64_t off) +static locked_range_t * +rangelock_split(avl_tree_t *tree, locked_range_t *lr, uint64_t off) { - rl_t *front, *rear; + ASSERT3U(lr->lr_length, >, 1); + ASSERT3U(off, >, lr->lr_offset); + ASSERT3U(off, <, lr->lr_offset + lr->lr_length); + ASSERT(lr->lr_write_wanted == B_FALSE); + ASSERT(lr->lr_read_wanted == B_FALSE); - ASSERT3U(rl->r_len, >, 1); - ASSERT3U(off, >, rl->r_off); - ASSERT3U(off, <, rl->r_off + rl->r_len); - ASSERT(rl->r_write_wanted == B_FALSE); - ASSERT(rl->r_read_wanted == B_FALSE); - /* create the rear proxy range lock */ - rear = kmem_alloc(sizeof (rl_t), KM_SLEEP); - rear->r_off = off; - rear->r_len = rl->r_off + rl->r_len - off; - rear->r_cnt = rl->r_cnt; - rear->r_type = RL_READER; - rear->r_proxy = B_TRUE; - rear->r_write_wanted = B_FALSE; - rear->r_read_wanted = B_FALSE; + locked_range_t *rear = kmem_alloc(sizeof (locked_range_t), KM_SLEEP); + rear->lr_offset = off; + rear->lr_length = lr->lr_offset + lr->lr_length - off; + rear->lr_count = lr->lr_count; + rear->lr_type = RL_READER; + rear->lr_proxy = B_TRUE; + rear->lr_write_wanted = B_FALSE; + rear->lr_read_wanted = B_FALSE; - front = zfs_range_proxify(tree, rl); - front->r_len = off - rl->r_off; + locked_range_t *front = rangelock_proxify(tree, lr); + front->lr_length = off - lr->lr_offset; avl_insert_here(tree, rear, front, AVL_AFTER); return (front); @@ -251,28 +274,27 @@ zfs_range_split(avl_tree_t *tree, rl_t *rl, uint64_t o * Create and add a new proxy range lock for the supplied range. */ static void -zfs_range_new_proxy(avl_tree_t *tree, uint64_t off, uint64_t len) +rangelock_new_proxy(avl_tree_t *tree, uint64_t off, uint64_t len) { - rl_t *rl; - - ASSERT(len); - rl = kmem_alloc(sizeof (rl_t), KM_SLEEP); - rl->r_off = off; - rl->r_len = len; - rl->r_cnt = 1; - rl->r_type = RL_READER; - rl->r_proxy = B_TRUE; - rl->r_write_wanted = B_FALSE; - rl->r_read_wanted = B_FALSE; - avl_add(tree, rl); + ASSERT(len != 0); + locked_range_t *lr = kmem_alloc(sizeof (locked_range_t), KM_SLEEP); + lr->lr_offset = off; + lr->lr_length = len; + lr->lr_count = 1; + lr->lr_type = RL_READER; + lr->lr_proxy = B_TRUE; + lr->lr_write_wanted = B_FALSE; + lr->lr_read_wanted = B_FALSE; + avl_add(tree, lr); } static void -zfs_range_add_reader(avl_tree_t *tree, rl_t *new, rl_t *prev, avl_index_t where) +rangelock_add_reader(avl_tree_t *tree, locked_range_t *new, + locked_range_t *prev, avl_index_t where) { - rl_t *next; - uint64_t off = new->r_off; - uint64_t len = new->r_len; + locked_range_t *next; + uint64_t off = new->lr_offset; + uint64_t len = new->lr_length; /* * prev arrives either: @@ -281,37 +303,37 @@ zfs_range_add_reader(avl_tree_t *tree, rl_t *new, rl_t * range may overlap with the new range * - null, if there were no ranges starting before the new one */ - if (prev) { - if (prev->r_off + prev->r_len <= off) { + if (prev != NULL) { + if (prev->lr_offset + prev->lr_length <= off) { prev = NULL; - } else if (prev->r_off != off) { + } else if (prev->lr_offset != off) { /* * convert to proxy if needed then * split this entry and bump ref count */ - prev = zfs_range_split(tree, prev, off); + prev = rangelock_split(tree, prev, off); prev = AVL_NEXT(tree, prev); /* move to rear range */ } } - ASSERT((prev == NULL) || (prev->r_off == off)); + ASSERT((prev == NULL) || (prev->lr_offset == off)); - if (prev) + if (prev != NULL) next = prev; else - next = (rl_t *)avl_nearest(tree, where, AVL_AFTER); + next = avl_nearest(tree, where, AVL_AFTER); - if (next == NULL || off + len <= next->r_off) { + if (next == NULL || off + len <= next->lr_offset) { /* no overlaps, use the original new rl_t in the tree */ avl_insert(tree, new, where); return; } - if (off < next->r_off) { + if (off < next->lr_offset) { /* Add a proxy for initial range before the overlap */ - zfs_range_new_proxy(tree, off, next->r_off - off); + rangelock_new_proxy(tree, off, next->lr_offset - off); } - new->r_cnt = 0; /* will use proxies in tree */ + new->lr_count = 0; /* will use proxies in tree */ /* * We now search forward through the ranges, until we go past the end * of the new range. For each entry we make it a proxy if it @@ -319,47 +341,51 @@ zfs_range_add_reader(avl_tree_t *tree, rl_t *new, rl_t * gaps between the ranges then we create a new proxy range. */ for (prev = NULL; next; prev = next, next = AVL_NEXT(tree, next)) { - if (off + len <= next->r_off) + if (off + len <= next->lr_offset) break; - if (prev && prev->r_off + prev->r_len < next->r_off) { + if (prev != NULL && prev->lr_offset + prev->lr_length < + next->lr_offset) { /* there's a gap */ - ASSERT3U(next->r_off, >, prev->r_off + prev->r_len); - zfs_range_new_proxy(tree, prev->r_off + prev->r_len, - next->r_off - (prev->r_off + prev->r_len)); + ASSERT3U(next->lr_offset, >, + prev->lr_offset + prev->lr_length); + rangelock_new_proxy(tree, + prev->lr_offset + prev->lr_length, + next->lr_offset - + (prev->lr_offset + prev->lr_length)); } - if (off + len == next->r_off + next->r_len) { + if (off + len == next->lr_offset + next->lr_length) { /* exact overlap with end */ - next = zfs_range_proxify(tree, next); - next->r_cnt++; + next = rangelock_proxify(tree, next); + next->lr_count++; return; } - if (off + len < next->r_off + next->r_len) { + if (off + len < next->lr_offset + next->lr_length) { /* new range ends in the middle of this block */ - next = zfs_range_split(tree, next, off + len); - next->r_cnt++; + next = rangelock_split(tree, next, off + len); + next->lr_count++; return; } - ASSERT3U(off + len, >, next->r_off + next->r_len); - next = zfs_range_proxify(tree, next); - next->r_cnt++; + ASSERT3U(off + len, >, next->lr_offset + next->lr_length); + next = rangelock_proxify(tree, next); + next->lr_count++; } /* Add the remaining end range. */ - zfs_range_new_proxy(tree, prev->r_off + prev->r_len, - (off + len) - (prev->r_off + prev->r_len)); + rangelock_new_proxy(tree, prev->lr_offset + prev->lr_length, + (off + len) - (prev->lr_offset + prev->lr_length)); } /* * Check if a reader lock can be grabbed, or wait and recheck until available. */ static void -zfs_range_lock_reader(znode_t *zp, rl_t *new) +rangelock_enter_reader(rangelock_t *rl, locked_range_t *new) { - avl_tree_t *tree = &zp->z_range_avl; - rl_t *prev, *next; + avl_tree_t *tree = &rl->rl_tree; + locked_range_t *prev, *next; avl_index_t where; - uint64_t off = new->r_off; - uint64_t len = new->r_len; + uint64_t off = new->lr_offset; + uint64_t len = new->lr_length; /* * Look for any writer locks in the range. @@ -367,21 +393,22 @@ zfs_range_lock_reader(znode_t *zp, rl_t *new) retry: prev = avl_find(tree, new, &where); if (prev == NULL) - prev = (rl_t *)avl_nearest(tree, where, AVL_BEFORE); + prev = (locked_range_t *)avl_nearest(tree, where, AVL_BEFORE); /* * Check the previous range for a writer lock overlap. */ - if (prev && (off < prev->r_off + prev->r_len)) { - if ((prev->r_type == RL_WRITER) || (prev->r_write_wanted)) { - if (!prev->r_read_wanted) { - cv_init(&prev->r_rd_cv, NULL, CV_DEFAULT, NULL); - prev->r_read_wanted = B_TRUE; + if (prev && (off < prev->lr_offset + prev->lr_length)) { + if ((prev->lr_type == RL_WRITER) || (prev->lr_write_wanted)) { + if (!prev->lr_read_wanted) { + cv_init(&prev->lr_read_cv, + NULL, CV_DEFAULT, NULL); + prev->lr_read_wanted = B_TRUE; } - cv_wait(&prev->r_rd_cv, &zp->z_range_lock); + cv_wait(&prev->lr_read_cv, &rl->rl_lock); goto retry; } - if (off + len < prev->r_off + prev->r_len) + if (off + len < prev->lr_offset + prev->lr_length) goto got_lock; } @@ -389,70 +416,71 @@ retry: * Search through the following ranges to see if there's * write lock any overlap. */ - if (prev) + if (prev != NULL) next = AVL_NEXT(tree, prev); else - next = (rl_t *)avl_nearest(tree, where, AVL_AFTER); - for (; next; next = AVL_NEXT(tree, next)) { - if (off + len <= next->r_off) + next = (locked_range_t *)avl_nearest(tree, where, AVL_AFTER); + for (; next != NULL; next = AVL_NEXT(tree, next)) { + if (off + len <= next->lr_offset) goto got_lock; - if ((next->r_type == RL_WRITER) || (next->r_write_wanted)) { - if (!next->r_read_wanted) { - cv_init(&next->r_rd_cv, NULL, CV_DEFAULT, NULL); - next->r_read_wanted = B_TRUE; + if ((next->lr_type == RL_WRITER) || (next->lr_write_wanted)) { + if (!next->lr_read_wanted) { + cv_init(&next->lr_read_cv, + NULL, CV_DEFAULT, NULL); + next->lr_read_wanted = B_TRUE; } - cv_wait(&next->r_rd_cv, &zp->z_range_lock); + cv_wait(&next->lr_read_cv, &rl->rl_lock); goto retry; } - if (off + len <= next->r_off + next->r_len) + if (off + len <= next->lr_offset + next->lr_length) goto got_lock; } got_lock: /* * Add the read lock, which may involve splitting existing - * locks and bumping ref counts (r_cnt). + * locks and bumping ref counts (r_count). */ - zfs_range_add_reader(tree, new, prev, where); + rangelock_add_reader(tree, new, prev, where); } /* - * Lock a range (offset, length) as either shared (RL_READER) - * or exclusive (RL_WRITER). Returns the range lock structure - * for later unlocking or reduce range (if entire file - * previously locked as RL_WRITER). + * Lock a range (offset, length) as either shared (RL_READER) or exclusive + * (RL_WRITER or RL_APPEND). If RL_APPEND is specified, rl_cb() will convert + * it to a RL_WRITER lock (with the offset at the end of the file). Returns + * the range lock structure for later unlocking (or reduce range if the + * entire file is locked as RL_WRITER). */ -rl_t * -zfs_range_lock(znode_t *zp, uint64_t off, uint64_t len, rl_type_t type) +locked_range_t * +rangelock_enter(rangelock_t *rl, uint64_t off, uint64_t len, + rangelock_type_t type) { - rl_t *new; - ASSERT(type == RL_READER || type == RL_WRITER || type == RL_APPEND); - new = kmem_alloc(sizeof (rl_t), KM_SLEEP); - new->r_zp = zp; - new->r_off = off; + locked_range_t *new = kmem_alloc(sizeof (locked_range_t), KM_SLEEP); + new->lr_rangelock = rl; + new->lr_offset = off; if (len + off < off) /* overflow */ len = UINT64_MAX - off; - new->r_len = len; - new->r_cnt = 1; /* assume it's going to be in the tree */ - new->r_type = type; - new->r_proxy = B_FALSE; - new->r_write_wanted = B_FALSE; - new->r_read_wanted = B_FALSE; + new->lr_length = len; + new->lr_count = 1; /* assume it's going to be in the tree */ + new->lr_type = type; + new->lr_proxy = B_FALSE; + new->lr_write_wanted = B_FALSE; + new->lr_read_wanted = B_FALSE; - mutex_enter(&zp->z_range_lock); + mutex_enter(&rl->rl_lock); if (type == RL_READER) { /* * First check for the usual case of no locks */ - if (avl_numnodes(&zp->z_range_avl) == 0) - avl_add(&zp->z_range_avl, new); + if (avl_numnodes(&rl->rl_tree) == 0) + avl_add(&rl->rl_tree, new); else - zfs_range_lock_reader(zp, new); + rangelock_enter_reader(rl, new); } else - zfs_range_lock_writer(zp, new); /* RL_WRITER or RL_APPEND */ - mutex_exit(&zp->z_range_lock); + rangelock_enter_writer(rl, new); /* RL_WRITER or RL_APPEND */ + mutex_exit(&rl->rl_lock); return (new); } @@ -460,10 +488,9 @@ zfs_range_lock(znode_t *zp, uint64_t off, uint64_t len * Unlock a reader lock */ static void -zfs_range_unlock_reader(znode_t *zp, rl_t *remove) +rangelock_exit_reader(rangelock_t *rl, locked_range_t *remove) { - avl_tree_t *tree = &zp->z_range_avl; - rl_t *rl, *next = NULL; + avl_tree_t *tree = &rl->rl_tree; uint64_t len; /* @@ -473,133 +500,118 @@ zfs_range_unlock_reader(znode_t *zp, rl_t *remove) * removed from the tree and replaced by proxies (one or * more ranges mapping to the entire range). */ - if (remove->r_cnt == 1) { + if (remove->lr_count == 1) { avl_remove(tree, remove); - if (remove->r_write_wanted) { - cv_broadcast(&remove->r_wr_cv); - cv_destroy(&remove->r_wr_cv); + if (remove->lr_write_wanted) { + cv_broadcast(&remove->lr_write_cv); + cv_destroy(&remove->lr_write_cv); } - if (remove->r_read_wanted) { - cv_broadcast(&remove->r_rd_cv); - cv_destroy(&remove->r_rd_cv); + if (remove->lr_read_wanted) { + cv_broadcast(&remove->lr_read_cv); + cv_destroy(&remove->lr_read_cv); } } else { - ASSERT0(remove->r_cnt); - ASSERT0(remove->r_write_wanted); - ASSERT0(remove->r_read_wanted); + ASSERT0(remove->lr_count); + ASSERT0(remove->lr_write_wanted); + ASSERT0(remove->lr_read_wanted); /* * Find start proxy representing this reader lock, * then decrement ref count on all proxies * that make up this range, freeing them as needed. */ - rl = avl_find(tree, remove, NULL); - ASSERT(rl); - ASSERT(rl->r_cnt); - ASSERT(rl->r_type == RL_READER); - for (len = remove->r_len; len != 0; rl = next) { - len -= rl->r_len; - if (len) { - next = AVL_NEXT(tree, rl); - ASSERT(next); - ASSERT(rl->r_off + rl->r_len == next->r_off); - ASSERT(next->r_cnt); - ASSERT(next->r_type == RL_READER); + locked_range_t *lr = avl_find(tree, remove, NULL); + ASSERT3P(lr, !=, NULL); + ASSERT3U(lr->lr_count, !=, 0); + ASSERT3U(lr->lr_type, ==, RL_READER); + locked_range_t *next = NULL; + for (len = remove->lr_length; len != 0; lr = next) { + len -= lr->lr_length; + if (len != 0) { + next = AVL_NEXT(tree, lr); + ASSERT3P(next, !=, NULL); + ASSERT3U(lr->lr_offset + lr->lr_length, ==, + next->lr_offset); + ASSERT3U(next->lr_count, !=, 0); + ASSERT3U(next->lr_type, ==, RL_READER); } - rl->r_cnt--; - if (rl->r_cnt == 0) { - avl_remove(tree, rl); - if (rl->r_write_wanted) { - cv_broadcast(&rl->r_wr_cv); - cv_destroy(&rl->r_wr_cv); + lr->lr_count--; + if (lr->lr_count == 0) { + avl_remove(tree, lr); + if (lr->lr_write_wanted) { + cv_broadcast(&lr->lr_write_cv); + cv_destroy(&lr->lr_write_cv); } - if (rl->r_read_wanted) { - cv_broadcast(&rl->r_rd_cv); - cv_destroy(&rl->r_rd_cv); + if (lr->lr_read_wanted) { + cv_broadcast(&lr->lr_read_cv); + cv_destroy(&lr->lr_read_cv); } - kmem_free(rl, sizeof (rl_t)); + kmem_free(lr, sizeof (locked_range_t)); } } } - kmem_free(remove, sizeof (rl_t)); + kmem_free(remove, sizeof (locked_range_t)); } /* * Unlock range and destroy range lock structure. */ void -zfs_range_unlock(rl_t *rl) +rangelock_exit(locked_range_t *lr) { - znode_t *zp = rl->r_zp; + rangelock_t *rl = lr->lr_rangelock; - ASSERT(rl->r_type == RL_WRITER || rl->r_type == RL_READER); - ASSERT(rl->r_cnt == 1 || rl->r_cnt == 0); - ASSERT(!rl->r_proxy); + ASSERT(lr->lr_type == RL_WRITER || lr->lr_type == RL_READER); + ASSERT(lr->lr_count == 1 || lr->lr_count == 0); + ASSERT(!lr->lr_proxy); - mutex_enter(&zp->z_range_lock); - if (rl->r_type == RL_WRITER) { + mutex_enter(&rl->rl_lock); + if (lr->lr_type == RL_WRITER) { /* writer locks can't be shared or split */ - avl_remove(&zp->z_range_avl, rl); - mutex_exit(&zp->z_range_lock); - if (rl->r_write_wanted) { - cv_broadcast(&rl->r_wr_cv); - cv_destroy(&rl->r_wr_cv); + avl_remove(&rl->rl_tree, lr); + mutex_exit(&rl->rl_lock); + if (lr->lr_write_wanted) { + cv_broadcast(&lr->lr_write_cv); + cv_destroy(&lr->lr_write_cv); } - if (rl->r_read_wanted) { - cv_broadcast(&rl->r_rd_cv); - cv_destroy(&rl->r_rd_cv); + if (lr->lr_read_wanted) { + cv_broadcast(&lr->lr_read_cv); + cv_destroy(&lr->lr_read_cv); } - kmem_free(rl, sizeof (rl_t)); + kmem_free(lr, sizeof (locked_range_t)); } else { /* - * lock may be shared, let zfs_range_unlock_reader() + * lock may be shared, let rangelock_exit_reader() * release the lock and free the rl_t */ - zfs_range_unlock_reader(zp, rl); - mutex_exit(&zp->z_range_lock); + rangelock_exit_reader(rl, lr); + mutex_exit(&rl->rl_lock); } } /* * Reduce range locked as RL_WRITER from whole file to specified range. - * Asserts the whole file is exclusivly locked and so there's only one + * Asserts the whole file is exclusively locked and so there's only one * entry in the tree. */ void -zfs_range_reduce(rl_t *rl, uint64_t off, uint64_t len) +rangelock_reduce(locked_range_t *lr, uint64_t off, uint64_t len) { - znode_t *zp = rl->r_zp; + rangelock_t *rl = lr->lr_rangelock; /* Ensure there are no other locks */ - ASSERT(avl_numnodes(&zp->z_range_avl) == 1); - ASSERT(rl->r_off == 0); - ASSERT(rl->r_type == RL_WRITER); - ASSERT(!rl->r_proxy); - ASSERT3U(rl->r_len, ==, UINT64_MAX); - ASSERT3U(rl->r_cnt, ==, 1); + ASSERT3U(avl_numnodes(&rl->rl_tree), ==, 1); + ASSERT3U(lr->lr_offset, ==, 0); + ASSERT3U(lr->lr_type, ==, RL_WRITER); + ASSERT(!lr->lr_proxy); + ASSERT3U(lr->lr_length, ==, UINT64_MAX); + ASSERT3U(lr->lr_count, ==, 1); - mutex_enter(&zp->z_range_lock); - rl->r_off = off; - rl->r_len = len; - mutex_exit(&zp->z_range_lock); - if (rl->r_write_wanted) - cv_broadcast(&rl->r_wr_cv); - if (rl->r_read_wanted) - cv_broadcast(&rl->r_rd_cv); -} - -/* - * AVL comparison function used to order range locks - * Locks are ordered on the start offset of the range. - */ -int -zfs_range_compare(const void *arg1, const void *arg2) -{ - const rl_t *rl1 = arg1; - const rl_t *rl2 = arg2; - - if (rl1->r_off > rl2->r_off) - return (1); - if (rl1->r_off < rl2->r_off) - return (-1); - return (0); + mutex_enter(&rl->rl_lock); + lr->lr_offset = off; + lr->lr_length = len; + mutex_exit(&rl->rl_lock); + if (lr->lr_write_wanted) + cv_broadcast(&lr->lr_write_cv); + if (lr->lr_read_wanted) + cv_broadcast(&lr->lr_read_cv); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Tue Jun 4 00:01:37 2019 (r348595) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Tue Jun 4 00:42:51 2019 (r348596) @@ -513,7 +513,6 @@ zfs_read(vnode_t *vp, uio_t *uio, int ioflag, cred_t * zfsvfs_t *zfsvfs = zp->z_zfsvfs; ssize_t n, nbytes; *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Tue Jun 4 00:42:53 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7011815C2992; Tue, 4 Jun 2019 00:42:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E48E770E3; Tue, 4 Jun 2019 00:42:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DFA39E062; Tue, 4 Jun 2019 00:42:52 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x540gq0c070947; Tue, 4 Jun 2019 00:42:52 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x540gq4v070946; Tue, 4 Jun 2019 00:42:52 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201906040042.x540gq4v070946@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 4 Jun 2019 00:42:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r348596 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 348596 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0E48E770E3 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.969,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2019 00:42:53 -0000 Author: mav Date: Tue Jun 4 00:42:51 2019 New Revision: 348596 URL: https://svnweb.freebsd.org/changeset/base/348596 Log: 9689 zfs range lock code should not be zpl-specific illumos/illumos-gate@7931524763ef94dc16989451dddd206563d03bb4 Reviewed by: Serapheim Dimitropoulos Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor/illumos/dist/cmd/ztest/ztest.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_rlock.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Tue Jun 4 00:01:37 2019 (r348595) +++ vendor/illumos/dist/cmd/ztest/ztest.c Tue Jun 4 00:42:51 2019 (r348596) @@ -237,7 +237,9 @@ typedef struct bufwad { } bufwad_t; /* - * XXX -- fix zfs range locks to be generic so we can use them here. + * It would be better to use a rangelock_t per object. Unfortunately + * the rangelock_t is not a drop-in replacement for rl_t, because we + * still need to map from object ID to rangelock_t. */ typedef enum { RL_READER, @@ -1845,12 +1847,12 @@ static void ztest_get_done(zgd_t *zgd, int error) { ztest_ds_t *zd = zgd->zgd_private; - uint64_t object = zgd->zgd_rl->rl_object; + uint64_t object = ((rl_t *)zgd->zgd_lr)->rl_object; if (zgd->zgd_db) dmu_buf_rele(zgd->zgd_db, zgd); - ztest_range_unlock(zgd->zgd_rl); + ztest_range_unlock((rl_t *)zgd->zgd_lr); ztest_object_unlock(zd, object); umem_free(zgd, sizeof (*zgd)); @@ -1900,8 +1902,8 @@ ztest_get_data(void *arg, lr_write_t *lr, char *buf, s zgd->zgd_private = zd; if (buf != NULL) { /* immediate write */ - zgd->zgd_rl = ztest_range_lock(zd, object, offset, size, - RL_READER); + zgd->zgd_lr = (struct locked_range *)ztest_range_lock(zd, + object, offset, size, RL_READER); error = dmu_read(os, object, offset, size, buf, DMU_READ_NO_PREFETCH); @@ -1915,8 +1917,8 @@ ztest_get_data(void *arg, lr_write_t *lr, char *buf, s offset = 0; } - zgd->zgd_rl = ztest_range_lock(zd, object, offset, size, - RL_READER); + zgd->zgd_lr = (struct locked_range *)ztest_range_lock(zd, + object, offset, size, RL_READER); error = dmu_buf_hold(os, object, offset, zgd, &db, DMU_READ_NO_PREFETCH);