From owner-svn-src-vendor@freebsd.org Sun Aug 11 06:22:11 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0C73AB0D45; Sun, 11 Aug 2019 06:22:11 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 465pmL72Rfz4cKQ; Sun, 11 Aug 2019 06:22:10 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D2F0AA030; Sun, 11 Aug 2019 06:22:10 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7B6MAPP052184; Sun, 11 Aug 2019 06:22:10 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7B6M6PV052161; Sun, 11 Aug 2019 06:22:06 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201908110622.x7B6M6PV052161@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Sun, 11 Aug 2019 06:22:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r350853 - vendor/bzip2/dist X-SVN-Group: vendor X-SVN-Commit-Author: delphij X-SVN-Commit-Paths: vendor/bzip2/dist X-SVN-Commit-Revision: 350853 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Aug 2019 06:22:11 -0000 Author: delphij Date: Sun Aug 11 06:22:06 2019 New Revision: 350853 URL: https://svnweb.freebsd.org/changeset/base/350853 Log: Vendor import bzip2 -- upgrade to version 1.0.8. Modified: vendor/bzip2/dist/CHANGES vendor/bzip2/dist/LICENSE vendor/bzip2/dist/Makefile vendor/bzip2/dist/Makefile-libbz2_so vendor/bzip2/dist/README vendor/bzip2/dist/README.COMPILATION.PROBLEMS vendor/bzip2/dist/blocksort.c vendor/bzip2/dist/bzip2.1 vendor/bzip2/dist/bzip2.c vendor/bzip2/dist/bzip2recover.c vendor/bzip2/dist/bzlib.c vendor/bzip2/dist/bzlib.h vendor/bzip2/dist/bzlib_private.h vendor/bzip2/dist/compress.c vendor/bzip2/dist/crctable.c vendor/bzip2/dist/decompress.c vendor/bzip2/dist/huffman.c vendor/bzip2/dist/randtable.c vendor/bzip2/dist/spewG.c vendor/bzip2/dist/unzcrash.c Modified: vendor/bzip2/dist/CHANGES ============================================================================== --- vendor/bzip2/dist/CHANGES Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/CHANGES Sun Aug 11 06:22:06 2019 (r350853) @@ -2,8 +2,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -338,3 +338,19 @@ Security fix only. Fixes CERT-FI 20469 as it applies * bzip2recover: Fix use after free issue with outFile (CVE-2016-3189) * Make sure nSelectors is not out of range (CVE-2019-12900) + +1.0.8 (13 Jul 19) +~~~~~~~~~~~~~~~~~ + +* Accept as many selectors as the file format allows. + This relaxes the fix for CVE-2019-12900 from 1.0.7 + so that bzip2 allows decompression of bz2 files that + use (too) many selectors again. + +* Fix handling of large (> 4GB) files on Windows. + +* Cleanup of bzdiff and bzgrep scripts so they don't use + any bash extensions and handle multiple archives correctly. + +* There is now a bz2-files testsuite at + https://sourceware.org/git/bzip2-tests.git Modified: vendor/bzip2/dist/LICENSE ============================================================================== --- vendor/bzip2/dist/LICENSE Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/LICENSE Sun Aug 11 06:22:06 2019 (r350853) @@ -2,7 +2,7 @@ -------------------------------------------------------------------------- This program, "bzip2", the associated library "libbzip2", and all -documentation, are copyright (C) 1996-2010 Julian R Seward. All +documentation, are copyright (C) 1996-2019 Julian R Seward. All rights reserved. Redistribution and use in source and binary forms, with or without @@ -37,6 +37,6 @@ NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Julian Seward, jseward@acm.org -bzip2/libbzip2 version 1.0.7 of 27 June 2019 +bzip2/libbzip2 version 1.0.8 of 13 July 2019 -------------------------------------------------------------------------- Modified: vendor/bzip2/dist/Makefile ============================================================================== --- vendor/bzip2/dist/Makefile Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/Makefile Sun Aug 11 06:22:06 2019 (r350853) @@ -2,8 +2,8 @@ # This file is part of bzip2/libbzip2, a program and library for # lossless, block-sorting data compression. # -# bzip2/libbzip2 version 1.0.6 of 6 September 2010 -# Copyright (C) 1996-2010 Julian Seward +# bzip2/libbzip2 version 1.0.8 of 13 July 2019 +# Copyright (C) 1996-2019 Julian Seward # # Please read the WARNING, DISCLAIMER and PATENTS sections in the # README file. @@ -137,7 +137,7 @@ bzip2recover.o: bzip2recover.c distclean: clean rm -f manual.ps manual.html manual.pdf -DISTNAME=bzip2-1.0.7 +DISTNAME=bzip2-1.0.8 dist: check manual rm -f $(DISTNAME) ln -s -f . $(DISTNAME) Modified: vendor/bzip2/dist/Makefile-libbz2_so ============================================================================== --- vendor/bzip2/dist/Makefile-libbz2_so Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/Makefile-libbz2_so Sun Aug 11 06:22:06 2019 (r350853) @@ -1,6 +1,6 @@ # This Makefile builds a shared version of the library, -# libbz2.so.1.0.7, with soname libbz2.so.1.0, +# libbz2.so.1.0.8, with soname libbz2.so.1.0, # at least on x86-Linux (RedHat 7.2), # with gcc-2.96 20000731 (Red Hat Linux 7.1 2.96-98). # Please see the README file for some important info @@ -10,8 +10,8 @@ # This file is part of bzip2/libbzip2, a program and library for # lossless, block-sorting data compression. # -# bzip2/libbzip2 version 1.0.6 of 6 September 2010 -# Copyright (C) 1996-2010 Julian Seward +# bzip2/libbzip2 version 1.0.8 of 13 July 2019 +# Copyright (C) 1996-2019 Julian Seward # # Please read the WARNING, DISCLAIMER and PATENTS sections in the # README file. @@ -35,13 +35,13 @@ OBJS= blocksort.o \ bzlib.o all: $(OBJS) - $(CC) -shared -Wl,-soname -Wl,libbz2.so.1.0 -o libbz2.so.1.0.7 $(OBJS) - $(CC) $(CFLAGS) -o bzip2-shared bzip2.c libbz2.so.1.0.7 + $(CC) -shared -Wl,-soname -Wl,libbz2.so.1.0 -o libbz2.so.1.0.8 $(OBJS) + $(CC) $(CFLAGS) -o bzip2-shared bzip2.c libbz2.so.1.0.8 rm -f libbz2.so.1.0 - ln -s libbz2.so.1.0.7 libbz2.so.1.0 + ln -s libbz2.so.1.0.8 libbz2.so.1.0 clean: - rm -f $(OBJS) bzip2.o libbz2.so.1.0.7 libbz2.so.1.0 bzip2-shared + rm -f $(OBJS) bzip2.o libbz2.so.1.0.8 libbz2.so.1.0 bzip2-shared blocksort.o: blocksort.c $(CC) $(CFLAGS) -c blocksort.c Modified: vendor/bzip2/dist/README ============================================================================== --- vendor/bzip2/dist/README Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/README Sun Aug 11 06:22:06 2019 (r350853) @@ -6,8 +6,8 @@ This version is fully compatible with the previous pub This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. -bzip2/libbzip2 version 1.0.7 of 27 June 2019 -Copyright (C) 1996-2010 Julian Seward +bzip2/libbzip2 version 1.0.8 of 13 July 2019 +Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in this file. @@ -165,8 +165,8 @@ WHAT'S NEW IN 1.0.x ? See the CHANGES file. -I hope you find bzip2 useful. Feel free to contact me at - jseward@acm.org +I hope you find bzip2 useful. Feel free to contact the developers at + bzip2-devel@sourceware.org if you have any suggestions or queries. Many people mailed me with comments, suggestions and patches after the releases of bzip-0.15, bzip-0.21, and bzip2 versions 0.1pl2, 0.9.0, 0.9.5, 1.0.0, 1.0.1, @@ -193,3 +193,4 @@ Cambridge, UK. 10 December 2007 (bzip2, version 1.0.5) 6 Sept 2010 (bzip2, version 1.0.6) 27 June 2019 (bzip2, version 1.0.7) +13 July 2019 (bzip2, version 1.0.8) Modified: vendor/bzip2/dist/README.COMPILATION.PROBLEMS ============================================================================== --- vendor/bzip2/dist/README.COMPILATION.PROBLEMS Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/README.COMPILATION.PROBLEMS Sun Aug 11 06:22:06 2019 (r350853) @@ -2,8 +2,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. -bzip2/libbzip2 version 1.0.7 of 27 June 2019 -Copyright (C) 1996-2010 Julian Seward +bzip2/libbzip2 version 1.0.8 of 13 July 2019 +Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/blocksort.c ============================================================================== --- vendor/bzip2/dist/blocksort.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/blocksort.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/bzip2.1 ============================================================================== --- vendor/bzip2/dist/bzip2.1 Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/bzip2.1 Sun Aug 11 06:22:06 2019 (r350853) @@ -1,7 +1,7 @@ .PU .TH bzip2 1 .SH NAME -bzip2, bunzip2 \- a block-sorting file compressor, v1.0.7 +bzip2, bunzip2 \- a block-sorting file compressor, v1.0.8 .br bzcat \- decompresses files to stdout .br @@ -405,7 +405,7 @@ I/O error messages are not as helpful as they could be tries hard to detect I/O errors and exit cleanly, but the details of what the problem is sometimes seem rather misleading. -This manual page pertains to version 1.0.7 of +This manual page pertains to version 1.0.8 of .I bzip2. Compressed data created by this version is entirely forwards and backwards compatible with the previous public releases, versions Modified: vendor/bzip2/dist/bzip2.c ============================================================================== --- vendor/bzip2/dist/bzip2.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/bzip2.c Sun Aug 11 06:22:06 2019 (r350853) @@ -7,8 +7,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -128,12 +128,12 @@ #if BZ_LCCWIN32 # include # include -# include +# include # define NORETURN /**/ # define PATH_SEP '\\' -# define MY_LSTAT _stat -# define MY_STAT _stat +# define MY_LSTAT _stati64 +# define MY_STAT _stati64 # define MY_S_ISREG(x) ((x) & _S_IFREG) # define MY_S_ISDIR(x) ((x) & _S_IFDIR) @@ -748,8 +748,8 @@ void panic ( const Char* s ) fprintf ( stderr, "\n%s: PANIC -- internal consistency error:\n" "\t%s\n" - "\tThis is a BUG. Please report it to me at:\n" - "\tjseward@acm.org\n", + "\tThis is a BUG. Please report it to:\n" + "\tbzip2-devel@sourceware.org\n", progName, s ); showFileNames(); cleanUpAndFail( 3 ); @@ -829,7 +829,7 @@ void mySIGSEGVorSIGBUScatcher ( IntNative n ) " The user's manual, Section 4.3, has more info on (1) and (2).\n" " \n" " If you suspect this is a bug in bzip2, or are unsure about (1)\n" - " or (2), feel free to report it to me at: jseward@acm.org.\n" + " or (2), feel free to report it to: bzip2-devel@sourceware.org.\n" " Section 4.3 of the user's manual describes the info a useful\n" " bug report should have. If the manual is available on your\n" " system, please try and read it before mailing me. If you don't\n" @@ -852,7 +852,7 @@ void mySIGSEGVorSIGBUScatcher ( IntNative n ) " The user's manual, Section 4.3, has more info on (2) and (3).\n" " \n" " If you suspect this is a bug in bzip2, or are unsure about (2)\n" - " or (3), feel free to report it to me at: jseward@acm.org.\n" + " or (3), feel free to report it to: bzip2-devel@sourceware.org.\n" " Section 4.3 of the user's manual describes the info a useful\n" " bug report should have. If the manual is available on your\n" " system, please try and read it before mailing me. If you don't\n" @@ -1605,7 +1605,7 @@ void license ( void ) "bzip2, a block-sorting file compressor. " "Version %s.\n" " \n" - " Copyright (C) 1996-2010 by Julian Seward.\n" + " Copyright (C) 1996-2019 by Julian Seward.\n" " \n" " This program is free software; you can redistribute it and/or modify\n" " it under the terms set out in the LICENSE file, which is included\n" Modified: vendor/bzip2/dist/bzip2recover.c ============================================================================== --- vendor/bzip2/dist/bzip2recover.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/bzip2recover.c Sun Aug 11 06:22:06 2019 (r350853) @@ -7,8 +7,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -314,7 +314,7 @@ Int32 main ( Int32 argc, Char** argv ) inFileName[0] = outFileName[0] = 0; fprintf ( stderr, - "bzip2recover 1.0.7: extracts blocks from damaged .bz2 files.\n" ); + "bzip2recover 1.0.8: extracts blocks from damaged .bz2 files.\n" ); if (argc != 2) { fprintf ( stderr, "%s: usage is `%s damaged_file_name'.\n", Modified: vendor/bzip2/dist/bzlib.c ============================================================================== --- vendor/bzip2/dist/bzlib.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/bzlib.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -43,12 +43,12 @@ void BZ2_bz__AssertH__fail ( int errcode ) fprintf(stderr, "\n\nbzip2/libbzip2: internal error number %d.\n" "This is a bug in bzip2/libbzip2, %s.\n" - "Please report it to me at: jseward@acm.org. If this happened\n" + "Please report it to: bzip2-devel@sourceware.org. If this happened\n" "when you were using some program which uses libbzip2 as a\n" "component, you should also report this bug to the author(s)\n" "of that program. Please make an effort to report this bug;\n" "timely and accurate bug reports eventually lead to higher\n" - "quality software. Thanks. Julian Seward, 10 December 2007.\n\n", + "quality software. Thanks.\n\n", errcode, BZ2_bzlibVersion() ); Modified: vendor/bzip2/dist/bzlib.h ============================================================================== --- vendor/bzip2/dist/bzlib.h Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/bzlib.h Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/bzlib_private.h ============================================================================== --- vendor/bzip2/dist/bzlib_private.h Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/bzlib_private.h Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -36,7 +36,7 @@ /*-- General stuff. --*/ -#define BZ_VERSION "1.0.7, 27-Jun-2019" +#define BZ_VERSION "1.0.8, 13-Jul-2019" typedef char Char; typedef unsigned char Bool; Modified: vendor/bzip2/dist/compress.c ============================================================================== --- vendor/bzip2/dist/compress.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/compress.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -454,7 +454,7 @@ void sendMTFValues ( EState* s ) AssertH( nGroups < 8, 3002 ); AssertH( nSelectors < 32768 && - nSelectors <= (2 + (900000 / BZ_G_SIZE)), + nSelectors <= BZ_MAX_SELECTORS, 3003 ); Modified: vendor/bzip2/dist/crctable.c ============================================================================== --- vendor/bzip2/dist/crctable.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/crctable.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/decompress.c ============================================================================== --- vendor/bzip2/dist/decompress.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/decompress.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. @@ -287,7 +287,7 @@ Int32 BZ2_decompress ( DState* s ) GET_BITS(BZ_X_SELECTOR_1, nGroups, 3); if (nGroups < 2 || nGroups > BZ_N_GROUPS) RETURN(BZ_DATA_ERROR); GET_BITS(BZ_X_SELECTOR_2, nSelectors, 15); - if (nSelectors < 1 || nSelectors > BZ_MAX_SELECTORS) RETURN(BZ_DATA_ERROR); + if (nSelectors < 1) RETURN(BZ_DATA_ERROR); for (i = 0; i < nSelectors; i++) { j = 0; while (True) { @@ -296,8 +296,14 @@ Int32 BZ2_decompress ( DState* s ) j++; if (j >= nGroups) RETURN(BZ_DATA_ERROR); } - s->selectorMtf[i] = j; + /* Having more than BZ_MAX_SELECTORS doesn't make much sense + since they will never be used, but some implementations might + "round up" the number of selectors, so just ignore those. */ + if (i < BZ_MAX_SELECTORS) + s->selectorMtf[i] = j; } + if (nSelectors > BZ_MAX_SELECTORS) + nSelectors = BZ_MAX_SELECTORS; /*--- Undo the MTF values for the selectors. ---*/ { Modified: vendor/bzip2/dist/huffman.c ============================================================================== --- vendor/bzip2/dist/huffman.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/huffman.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/randtable.c ============================================================================== --- vendor/bzip2/dist/randtable.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/randtable.c Sun Aug 11 06:22:06 2019 (r350853) @@ -8,8 +8,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/spewG.c ============================================================================== --- vendor/bzip2/dist/spewG.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/spewG.c Sun Aug 11 06:22:06 2019 (r350853) @@ -13,8 +13,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. Modified: vendor/bzip2/dist/unzcrash.c ============================================================================== --- vendor/bzip2/dist/unzcrash.c Sat Aug 10 23:08:35 2019 (r350852) +++ vendor/bzip2/dist/unzcrash.c Sun Aug 11 06:22:06 2019 (r350853) @@ -17,8 +17,8 @@ This file is part of bzip2/libbzip2, a program and library for lossless, block-sorting data compression. - bzip2/libbzip2 version 1.0.7 of 27 June 2019 - Copyright (C) 1996-2010 Julian Seward + bzip2/libbzip2 version 1.0.8 of 13 July 2019 + Copyright (C) 1996-2019 Julian Seward Please read the WARNING, DISCLAIMER and PATENTS sections in the README file. From owner-svn-src-vendor@freebsd.org Sun Aug 11 06:22:54 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7751B0F17; Sun, 11 Aug 2019 06:22:54 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 465pnB3l9wz4cSX; Sun, 11 Aug 2019 06:22:54 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 48DEFA173; Sun, 11 Aug 2019 06:22:54 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7B6MscJ052259; Sun, 11 Aug 2019 06:22:54 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7B6MshV052258; Sun, 11 Aug 2019 06:22:54 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201908110622.x7B6MshV052258@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Sun, 11 Aug 2019 06:22:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r350854 - vendor/bzip2/1.0.8 X-SVN-Group: vendor X-SVN-Commit-Author: delphij X-SVN-Commit-Paths: vendor/bzip2/1.0.8 X-SVN-Commit-Revision: 350854 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Aug 2019 06:22:54 -0000 Author: delphij Date: Sun Aug 11 06:22:53 2019 New Revision: 350854 URL: https://svnweb.freebsd.org/changeset/base/350854 Log: "Tag" the Bzip2 1.0.8 vendor import. Added: vendor/bzip2/1.0.8/ - copied from r350853, vendor/bzip2/dist/ From owner-svn-src-vendor@freebsd.org Mon Aug 12 11:27:17 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C417EAFC1F; Mon, 12 Aug 2019 11:27:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 466YTx4nqhz4DNV; Mon, 12 Aug 2019 11:27:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 81DCD25DB2; Mon, 12 Aug 2019 11:27:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7CBRHs7093333; Mon, 12 Aug 2019 11:27:17 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7CBRHeA093332; Mon, 12 Aug 2019 11:27:17 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201908121127.x7CBRHeA093332@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Mon, 12 Aug 2019 11:27:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r350896 - vendor-sys/illumos/dist/common/zfs vendor/illumos/dist/man/man5 X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor/illumos/dist/man/man5 X-SVN-Commit-Revision: 350896 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Aug 2019 11:27:17 -0000 Author: avg Date: Mon Aug 12 11:27:17 2019 New Revision: 350896 URL: https://svnweb.freebsd.org/changeset/base/350896 Log: 6585 sha512, skein, and edonr have an unenforced dependency on extensible dataset illumos/illumos-gate@892586e8a147c02d7f4053cc405229a13e796928 https://github.com/illumos/illumos-gate/commit/892586e8a147c02d7f4053cc405229a13e796928 https://www.illumos.org/issues/6585 In any pool without the extensible dataset feature flag already enabled, creating a dataset with dedup set to use one of the new checksums would result in the following panic as soon as any data was added: panic[cpu0]/thread=ffffff0006761c40: feature_get_refcount(spa, feature, &refcount) != 48 (0x30 != 0x30), file: ../../common/fs/zfs/zfeature.c line 390 ffffff0006761830 fffffffffba8fbdd () ffffff0006761890 zfs:feature_do_action+11a () ffffff00067618c0 zfs:spa_feature_incr+1e () ffffff0006761920 zfs:dmu_object_zapify+b7 () ffffff00067619b0 zfs:dsl_dataset_activate_feature+97 () ffffff0006761a20 zfs:dsl_dataset_sync+ba () ffffff0006761ab0 zfs:dsl_pool_sync+153 () ffffff0006761b70 zfs:spa_sync+26e () ffffff0006761c20 zfs:txg_sync_thread+227 () ffffff0006761c30 unix:thread_start+8 () Inspection showed that feature->fi_feature was 7, which is the value of SPA_FEATURE_EXTENSIBLE_DATASET in the spa_feature enum. Testing shows that the panic can be prevented by explicitly setting extensible dataset as a dependency for the sha512, edonr, and skein feature flags. Alternatively, the new checksums code could possibly be changed to obviate the need for the dependency. Reviewed by: Matthew Ahrens Reviewed by: Richard Laager Approved by: Robert Mustacchi Author: ilovezfs Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.c Mon Aug 12 10:57:56 2019 (r350895) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.c Mon Aug 12 11:27:17 2019 (r350896) @@ -245,18 +245,32 @@ zpool_feature_init(void) "Support for blocks larger than 128KB.", ZFEATURE_FLAG_PER_DATASET, large_blocks_deps); + static const spa_feature_t sha512_deps[] = { + SPA_FEATURE_EXTENSIBLE_DATASET, + SPA_FEATURE_NONE + }; zfeature_register(SPA_FEATURE_SHA512, "org.illumos:sha512", "sha512", "SHA-512/256 hash algorithm.", - ZFEATURE_FLAG_PER_DATASET, NULL); + ZFEATURE_FLAG_PER_DATASET, sha512_deps); + + static const spa_feature_t skein_deps[] = { + SPA_FEATURE_EXTENSIBLE_DATASET, + SPA_FEATURE_NONE + }; zfeature_register(SPA_FEATURE_SKEIN, "org.illumos:skein", "skein", "Skein hash algorithm.", - ZFEATURE_FLAG_PER_DATASET, NULL); + ZFEATURE_FLAG_PER_DATASET, skein_deps); + + static const spa_feature_t edonr_deps[] = { + SPA_FEATURE_EXTENSIBLE_DATASET, + SPA_FEATURE_NONE + }; zfeature_register(SPA_FEATURE_EDONR, "org.illumos:edonr", "edonr", "Edon-R hash algorithm.", - ZFEATURE_FLAG_PER_DATASET, NULL); + ZFEATURE_FLAG_PER_DATASET, edonr_deps); zfeature_register(SPA_FEATURE_DEVICE_REMOVAL, "com.delphix:device_removal", "device_removal", From owner-svn-src-vendor@freebsd.org Mon Aug 12 11:27:18 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 13BCAAFC23; Mon, 12 Aug 2019 11:27:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 466YTx6rw1z4DNX; Mon, 12 Aug 2019 11:27:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CB3F725DB3; Mon, 12 Aug 2019 11:27:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7CBRHM1093339; Mon, 12 Aug 2019 11:27:17 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7CBRHB7093338; Mon, 12 Aug 2019 11:27:17 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201908121127.x7CBRHB7093338@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Mon, 12 Aug 2019 11:27:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r350896 - vendor-sys/illumos/dist/common/zfs vendor/illumos/dist/man/man5 X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor/illumos/dist/man/man5 X-SVN-Commit-Revision: 350896 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Aug 2019 11:27:18 -0000 Author: avg Date: Mon Aug 12 11:27:17 2019 New Revision: 350896 URL: https://svnweb.freebsd.org/changeset/base/350896 Log: 6585 sha512, skein, and edonr have an unenforced dependency on extensible dataset illumos/illumos-gate@892586e8a147c02d7f4053cc405229a13e796928 https://github.com/illumos/illumos-gate/commit/892586e8a147c02d7f4053cc405229a13e796928 https://www.illumos.org/issues/6585 In any pool without the extensible dataset feature flag already enabled, creating a dataset with dedup set to use one of the new checksums would result in the following panic as soon as any data was added: panic[cpu0]/thread=ffffff0006761c40: feature_get_refcount(spa, feature, &refcount) != 48 (0x30 != 0x30), file: ../../common/fs/zfs/zfeature.c line 390 ffffff0006761830 fffffffffba8fbdd () ffffff0006761890 zfs:feature_do_action+11a () ffffff00067618c0 zfs:spa_feature_incr+1e () ffffff0006761920 zfs:dmu_object_zapify+b7 () ffffff00067619b0 zfs:dsl_dataset_activate_feature+97 () ffffff0006761a20 zfs:dsl_dataset_sync+ba () ffffff0006761ab0 zfs:dsl_pool_sync+153 () ffffff0006761b70 zfs:spa_sync+26e () ffffff0006761c20 zfs:txg_sync_thread+227 () ffffff0006761c30 unix:thread_start+8 () Inspection showed that feature->fi_feature was 7, which is the value of SPA_FEATURE_EXTENSIBLE_DATASET in the spa_feature enum. Testing shows that the panic can be prevented by explicitly setting extensible dataset as a dependency for the sha512, edonr, and skein feature flags. Alternatively, the new checksums code could possibly be changed to obviate the need for the dependency. Reviewed by: Matthew Ahrens Reviewed by: Richard Laager Approved by: Robert Mustacchi Author: ilovezfs Modified: vendor/illumos/dist/man/man5/zpool-features.5 Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c Modified: vendor/illumos/dist/man/man5/zpool-features.5 ============================================================================== --- vendor/illumos/dist/man/man5/zpool-features.5 Mon Aug 12 10:57:56 2019 (r350895) +++ vendor/illumos/dist/man/man5/zpool-features.5 Mon Aug 12 11:27:17 2019 (r350896) @@ -543,7 +543,7 @@ filesystems that have ever had their recordsize larger l l . GUID org.illumos:sha512 READ\-ONLY COMPATIBLE no -DEPENDENCIES none +DEPENDENCIES extensible_dataset .TE This feature enables the use of the SHA-512/256 truncated hash algorithm @@ -575,7 +575,7 @@ Booting off of pools utilizing SHA-512/256 is supporte l l . GUID org.illumos:skein READ\-ONLY COMPATIBLE no -DEPENDENCIES none +DEPENDENCIES extensible_dataset .TE This feature enables the use of the Skein hash algorithm for checksum @@ -609,7 +609,7 @@ Booting off of pools using \fBskein\fR is supported. l l . GUID org.illumos:edonr READ\-ONLY COMPATIBLE no -DEPENDENCIES none +DEPENDENCIES extensible_dataset .TE This feature enables the use of the Edon-R hash algorithm for checksum, From owner-svn-src-vendor@freebsd.org Mon Aug 12 12:05:43 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 62FF3B1D78; Mon, 12 Aug 2019 12:05:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 466ZLH2fgxz4H7g; Mon, 12 Aug 2019 12:05:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 20D13264EB; Mon, 12 Aug 2019 12:05:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7CC5hsL017150; Mon, 12 Aug 2019 12:05:43 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7CC5fEI017139; Mon, 12 Aug 2019 12:05:41 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201908121205.x7CC5fEI017139@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Mon, 12 Aug 2019 12:05:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r350898 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb vendor/... X-SVN-Commit-Revision: 350898 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Aug 2019 12:05:43 -0000 Author: avg Date: Mon Aug 12 12:05:40 2019 New Revision: 350898 URL: https://svnweb.freebsd.org/changeset/base/350898 Log: 8423 8199 7432 Implement large_dnode pool feature 8423 Implement large_dnode pool feature 8199 multi-threaded dmu_object_alloc() 7432 Large dnode pool feature llumos/illumos-gate@54811da5ac6b517992fdc173df5d605e4e61fdc0 https://github.com/illumos/illumos-gate/commit/54811da5ac6b517992fdc173df5d605e4e61fdc0 https://www.illumos.org/issues/8423 https://www.illumos.org/issues/8199 https://www.illumos.org/issues/7432 ZoL issues: Improved dnode allocation #6564 Clean up large dnode code #6262 Fix dnode_hold() freeing dnode behavior #8172 Fix dnode allocation race #6414, #6439 Partial: Raw sends must be able to decrease nlevels #6821, #6864 Remove unnecessary txg syncs from receive_object() Closes #7197 Author: Toomas Soome Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/common/zfs/zpool_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/sa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/sa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zap.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/zap.c vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_acl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_replay.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_sa.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zdb/zdb_il.c vendor/illumos/dist/cmd/zstreamdump/zstreamdump.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.c Mon Aug 12 12:05:40 2019 (r350898) @@ -245,6 +245,17 @@ zpool_feature_init(void) "Support for blocks larger than 128KB.", ZFEATURE_FLAG_PER_DATASET, large_blocks_deps); + { + static const spa_feature_t large_dnode_deps[] = { + SPA_FEATURE_EXTENSIBLE_DATASET, + SPA_FEATURE_NONE + }; + zfeature_register(SPA_FEATURE_LARGE_DNODE, + "org.zfsonlinux:large_dnode", "large_dnode", + "Variable on-disk size of dnodes.", + ZFEATURE_FLAG_PER_DATASET, large_dnode_deps); + } + static const spa_feature_t sha512_deps[] = { SPA_FEATURE_EXTENSIBLE_DATASET, SPA_FEATURE_NONE Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.h Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.h Mon Aug 12 12:05:40 2019 (r350898) @@ -53,6 +53,7 @@ typedef enum spa_feature { SPA_FEATURE_BOOKMARKS, SPA_FEATURE_FS_SS_LIMIT, SPA_FEATURE_LARGE_BLOCKS, + SPA_FEATURE_LARGE_DNODE, SPA_FEATURE_SHA512, SPA_FEATURE_SKEIN, SPA_FEATURE_EDONR, Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_prop.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/common/zfs/zfs_prop.c Mon Aug 12 12:05:40 2019 (r350898) @@ -210,6 +210,17 @@ zfs_prop_init(void) { NULL } }; + static zprop_index_t dnsize_table[] = { + { "legacy", ZFS_DNSIZE_LEGACY }, + { "auto", ZFS_DNSIZE_AUTO }, + { "1k", ZFS_DNSIZE_1K }, + { "2k", ZFS_DNSIZE_2K }, + { "4k", ZFS_DNSIZE_4K }, + { "8k", ZFS_DNSIZE_8K }, + { "16k", ZFS_DNSIZE_16K }, + { NULL } + }; + static zprop_index_t redundant_metadata_table[] = { { "all", ZFS_REDUNDANT_METADATA_ALL }, { "most", ZFS_REDUNDANT_METADATA_MOST }, @@ -265,6 +276,10 @@ zfs_prop_init(void) zprop_register_index(ZFS_PROP_LOGBIAS, "logbias", ZFS_LOGBIAS_LATENCY, PROP_INHERIT, ZFS_TYPE_FILESYSTEM | ZFS_TYPE_VOLUME, "latency | throughput", "LOGBIAS", logbias_table); + + zprop_register_index(ZFS_PROP_DNODESIZE, "dnodesize", + ZFS_DNSIZE_LEGACY, PROP_INHERIT, ZFS_TYPE_FILESYSTEM, + "legacy | auto | 1k | 2k | 4k | 8k | 16k", "DNSIZE", dnsize_table); /* inherit index (boolean) properties */ zprop_register_index(ZFS_PROP_ATIME, "atime", 1, PROP_INHERIT, Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zpool_prop.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/common/zfs/zpool_prop.c Mon Aug 12 12:05:40 2019 (r350898) @@ -138,6 +138,8 @@ zpool_prop_init(void) PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_POOL, "MAXBLOCKSIZE"); zprop_register_hidden(ZPOOL_PROP_TNAME, "tname", PROP_TYPE_STRING, PROP_ONETIME, ZFS_TYPE_POOL, "TNAME"); + zprop_register_hidden(ZPOOL_PROP_MAXDNODESIZE, "maxdnodesize", + PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_POOL, "MAXDNODESIZE"); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Mon Aug 12 12:05:40 2019 (r350898) @@ -742,7 +742,6 @@ dbuf_verify(dmu_buf_impl_t *db) ASSERT3U(db->db.db_offset, ==, DMU_BONUS_BLKID); } else if (db->db_blkid == DMU_SPILL_BLKID) { ASSERT(dn != NULL); - ASSERT3U(db->db.db_size, >=, dn->dn_bonuslen); ASSERT0(db->db.db_offset); } else { ASSERT3U(db->db.db_offset, ==, db->db_blkid * db->db.db_size); @@ -995,13 +994,18 @@ dbuf_read_impl(dmu_buf_impl_t *db, zio_t *zio, uint32_ ASSERT(db->db_buf == NULL); if (db->db_blkid == DMU_BONUS_BLKID) { + /* + * The bonus length stored in the dnode may be less than + * the maximum available space in the bonus buffer. + */ int bonuslen = MIN(dn->dn_bonuslen, dn->dn_phys->dn_bonuslen); + int max_bonuslen = DN_SLOTS_TO_BONUSLEN(dn->dn_num_slots); ASSERT3U(bonuslen, <=, db->db.db_size); - db->db.db_data = zio_buf_alloc(DN_MAX_BONUSLEN); - arc_space_consume(DN_MAX_BONUSLEN, ARC_SPACE_OTHER); - if (bonuslen < DN_MAX_BONUSLEN) - bzero(db->db.db_data, DN_MAX_BONUSLEN); + db->db.db_data = zio_buf_alloc(max_bonuslen); + arc_space_consume(max_bonuslen, ARC_SPACE_BONUS); + if (bonuslen < max_bonuslen) + bzero(db->db.db_data, max_bonuslen); if (bonuslen) bcopy(DN_BONUS(dn->dn_phys), db->db.db_data, bonuslen); DB_DNODE_EXIT(db); @@ -1108,9 +1112,11 @@ dbuf_fix_old_data(dmu_buf_impl_t *db, uint64_t txg) ASSERT(dr->dr_txg >= txg - 2); if (db->db_blkid == DMU_BONUS_BLKID) { /* Note that the data bufs here are zio_bufs */ - dr->dt.dl.dr_data = zio_buf_alloc(DN_MAX_BONUSLEN); - arc_space_consume(DN_MAX_BONUSLEN, ARC_SPACE_OTHER); - bcopy(db->db.db_data, dr->dt.dl.dr_data, DN_MAX_BONUSLEN); + dnode_t *dn = DB_DNODE(db); + int bonuslen = DN_SLOTS_TO_BONUSLEN(dn->dn_num_slots); + dr->dt.dl.dr_data = zio_buf_alloc(bonuslen); + arc_space_consume(bonuslen, ARC_SPACE_BONUS); + bcopy(db->db.db_data, dr->dt.dl.dr_data, bonuslen); } else if (refcount_count(&db->db_holds) > db->db_dirtycnt) { int size = arc_buf_size(db->db_buf); arc_buf_contents_t type = DBUF_GET_BUFC_TYPE(db); @@ -2081,10 +2087,13 @@ dbuf_destroy(dmu_buf_impl_t *db) } if (db->db_blkid == DMU_BONUS_BLKID) { - ASSERT(db->db.db_data != NULL); - zio_buf_free(db->db.db_data, DN_MAX_BONUSLEN); - arc_space_return(DN_MAX_BONUSLEN, ARC_SPACE_OTHER); - db->db_state = DB_UNCACHED; + int slots = DB_DNODE(db)->dn_num_slots; + int bonuslen = DN_SLOTS_TO_BONUSLEN(slots); + if (db->db.db_data != NULL) { + zio_buf_free(db->db.db_data, bonuslen); + arc_space_return(bonuslen, ARC_SPACE_BONUS); + db->db_state = DB_UNCACHED; + } } dbuf_clear_data(db); @@ -2188,7 +2197,7 @@ dbuf_findbp(dnode_t *dn, int level, uint64_t blkid, in mutex_enter(&dn->dn_mtx); if (dn->dn_have_spill && (dn->dn_phys->dn_flags & DNODE_FLAG_SPILL_BLKPTR)) - *bpp = &dn->dn_phys->dn_spill; + *bpp = DN_SPILL_BLKPTR(dn->dn_phys); else *bpp = NULL; dbuf_add_ref(dn->dn_dbuf, NULL); @@ -2289,7 +2298,7 @@ dbuf_create(dnode_t *dn, uint8_t level, uint64_t blkid if (blkid == DMU_BONUS_BLKID) { ASSERT3P(parent, ==, dn->dn_dbuf); - db->db.db_size = DN_MAX_BONUSLEN - + db->db.db_size = DN_SLOTS_TO_BONUSLEN(dn->dn_num_slots) - (dn->dn_nblkptr-1) * sizeof (blkptr_t); ASSERT3U(db->db.db_size, >=, dn->dn_bonuslen); db->db.db_offset = DMU_BONUS_BLKID; @@ -3031,7 +3040,7 @@ dbuf_check_blkptr(dnode_t *dn, dmu_buf_impl_t *db) return; if (db->db_blkid == DMU_SPILL_BLKID) { - db->db_blkptr = &dn->dn_phys->dn_spill; + db->db_blkptr = DN_SPILL_BLKPTR(dn->dn_phys); BP_ZERO(db->db_blkptr); return; } @@ -3162,13 +3171,17 @@ dbuf_sync_leaf(dbuf_dirty_record_t *dr, dmu_tx_t *tx) ASSERT(*datap != NULL); ASSERT0(db->db_level); - ASSERT3U(dn->dn_phys->dn_bonuslen, <=, DN_MAX_BONUSLEN); - bcopy(*datap, DN_BONUS(dn->dn_phys), dn->dn_phys->dn_bonuslen); + ASSERT3U(DN_MAX_BONUS_LEN(dn->dn_phys), <=, + DN_SLOTS_TO_BONUSLEN(dn->dn_phys->dn_extra_slots + 1)); + bcopy(*datap, DN_BONUS(dn->dn_phys), + DN_MAX_BONUS_LEN(dn->dn_phys)); DB_DNODE_EXIT(db); if (*datap != db->db.db_data) { - zio_buf_free(*datap, DN_MAX_BONUSLEN); - arc_space_return(DN_MAX_BONUSLEN, ARC_SPACE_OTHER); + int slots = DB_DNODE(db)->dn_num_slots; + int bonuslen = DN_SLOTS_TO_BONUSLEN(slots); + zio_buf_free(*datap, bonuslen); + arc_space_return(bonuslen, ARC_SPACE_BONUS); } db->db_data_pending = NULL; drp = &db->db_last_dirty; @@ -3324,7 +3337,7 @@ dbuf_write_ready(zio_t *zio, arc_buf_t *buf, void *vdb if (db->db_blkid == DMU_SPILL_BLKID) { ASSERT(dn->dn_phys->dn_flags & DNODE_FLAG_SPILL_BLKPTR); ASSERT(!(BP_IS_HOLE(bp)) && - db->db_blkptr == &dn->dn_phys->dn_spill); + db->db_blkptr == DN_SPILL_BLKPTR(dn->dn_phys)); } #endif @@ -3336,11 +3349,17 @@ dbuf_write_ready(zio_t *zio, arc_buf_t *buf, void *vdb mutex_exit(&dn->dn_mtx); if (dn->dn_type == DMU_OT_DNODE) { - dnode_phys_t *dnp = db->db.db_data; - for (i = db->db.db_size >> DNODE_SHIFT; i > 0; - i--, dnp++) { - if (dnp->dn_type != DMU_OT_NONE) + i = 0; + while (i < db->db.db_size) { + dnode_phys_t *dnp = + (void *)(((char *)db->db.db_data) + i); + + i += DNODE_MIN_SIZE; + if (dnp->dn_type != DMU_OT_NONE) { fill++; + i += dnp->dn_extra_slots * + DNODE_MIN_SIZE; + } } } else { if (BP_IS_HOLE(bp)) { @@ -3493,7 +3512,7 @@ dbuf_write_done(zio_t *zio, arc_buf_t *buf, void *vdb) dn = DB_DNODE(db); ASSERT(dn->dn_phys->dn_flags & DNODE_FLAG_SPILL_BLKPTR); ASSERT(!(BP_IS_HOLE(db->db_blkptr)) && - db->db_blkptr == &dn->dn_phys->dn_spill); + db->db_blkptr == DN_SPILL_BLKPTR(dn->dn_phys)); DB_DNODE_EXIT(db); } #endif Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Mon Aug 12 12:05:40 2019 (r350898) @@ -254,7 +254,7 @@ dmu_buf_hold(objset_t *os, uint64_t object, uint64_t o int dmu_bonus_max(void) { - return (DN_MAX_BONUSLEN); + return (DN_OLD_MAX_BONUSLEN); } int @@ -2264,6 +2264,7 @@ dmu_object_info_from_dnode(dnode_t *dn, dmu_object_inf doi->doi_type = dn->dn_type; doi->doi_bonus_type = dn->dn_bonustype; doi->doi_bonus_size = dn->dn_bonuslen; + doi->doi_dnodesize = dn->dn_num_slots << DNODE_SHIFT; doi->doi_indirection = dn->dn_nlevels; doi->doi_checksum = dn->dn_checksum; doi->doi_compress = dn->dn_compress; @@ -2326,9 +2327,21 @@ dmu_object_size_from_db(dmu_buf_t *db_fake, uint32_t * dn = DB_DNODE(db); *blksize = dn->dn_datablksz; - /* add 1 for dnode space */ + /* add in number of slots used for the dnode itself */ *nblk512 = ((DN_USED_BYTES(dn->dn_phys) + SPA_MINBLOCKSIZE/2) >> - SPA_MINBLOCKSHIFT) + 1; + SPA_MINBLOCKSHIFT) + dn->dn_num_slots; + DB_DNODE_EXIT(db); +} + +void +dmu_object_dnsize_from_db(dmu_buf_t *db_fake, int *dnsize) +{ + dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake; + dnode_t *dn; + + DB_DNODE_ENTER(db); + dn = DB_DNODE(db); + *dnsize = dn->dn_num_slots << DNODE_SHIFT; DB_DNODE_EXIT(db); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Mon Aug 12 12:05:40 2019 (r350898) @@ -30,101 +30,227 @@ #include #include #include +#include -uint64_t -dmu_object_alloc_ibs(objset_t *os, dmu_object_type_t ot, int blocksize, - int indirect_blockshift, - dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx) +/* + * Each of the concurrent object allocators will grab + * 2^dmu_object_alloc_chunk_shift dnode slots at a time. The default is to + * grab 128 slots, which is 4 blocks worth. This was experimentally + * determined to be the lowest value that eliminates the measurable effect + * of lock contention from this code path. + */ +int dmu_object_alloc_chunk_shift = 7; + +static uint64_t +dmu_object_alloc_impl(objset_t *os, dmu_object_type_t ot, int blocksize, + int indirect_blockshift, dmu_object_type_t bonustype, int bonuslen, + int dnodesize, dmu_tx_t *tx) { uint64_t object; uint64_t L1_dnode_count = DNODES_PER_BLOCK << (DMU_META_DNODE(os)->dn_indblkshift - SPA_BLKPTRSHIFT); dnode_t *dn = NULL; + int dn_slots = dnodesize >> DNODE_SHIFT; + boolean_t restarted = B_FALSE; + uint64_t *cpuobj = &os->os_obj_next_percpu[CPU_SEQID % + os->os_obj_next_percpu_len]; + int dnodes_per_chunk = 1 << dmu_object_alloc_chunk_shift; + int error; - mutex_enter(&os->os_obj_lock); + if (dn_slots == 0) { + dn_slots = DNODE_MIN_SLOTS; + } else { + ASSERT3S(dn_slots, >=, DNODE_MIN_SLOTS); + ASSERT3S(dn_slots, <=, DNODE_MAX_SLOTS); + } + + /* + * The "chunk" of dnodes that is assigned to a CPU-specific + * allocator needs to be at least one block's worth, to avoid + * lock contention on the dbuf. It can be at most one L1 block's + * worth, so that the "rescan after polishing off a L1's worth" + * logic below will be sure to kick in. + */ + if (dnodes_per_chunk < DNODES_PER_BLOCK) + dnodes_per_chunk = DNODES_PER_BLOCK; + if (dnodes_per_chunk > L1_dnode_count) + dnodes_per_chunk = L1_dnode_count; + + object = *cpuobj; + for (;;) { - object = os->os_obj_next; /* - * Each time we polish off a L1 bp worth of dnodes (2^12 - * objects), move to another L1 bp that's still reasonably - * sparse (at most 1/4 full). Look from the beginning at most - * once per txg, but after that keep looking from here. - * os_scan_dnodes is set during txg sync if enough objects - * have been freed since the previous rescan to justify - * backfilling again. If we can't find a suitable block, just - * keep going from here. - * - * Note that dmu_traverse depends on the behavior that we use - * multiple blocks of the dnode object before going back to - * reuse objects. Any change to this algorithm should preserve - * that property or find another solution to the issues - * described in traverse_visitbp. + * If we finished a chunk of dnodes, get a new one from + * the global allocator. */ + if ((P2PHASE(object, dnodes_per_chunk) == 0) || + (P2PHASE(object + dn_slots - 1, dnodes_per_chunk) < + dn_slots)) { + DNODE_STAT_BUMP(dnode_alloc_next_chunk); + mutex_enter(&os->os_obj_lock); + ASSERT0(P2PHASE(os->os_obj_next_chunk, + dnodes_per_chunk)); + object = os->os_obj_next_chunk; - if (P2PHASE(object, L1_dnode_count) == 0) { - uint64_t offset; - int error; - if (os->os_rescan_dnodes) { - offset = 0; - os->os_rescan_dnodes = B_FALSE; - } else { - offset = object << DNODE_SHIFT; + /* + * Each time we polish off a L1 bp worth of dnodes + * (2^12 objects), move to another L1 bp that's + * still reasonably sparse (at most 1/4 full). Look + * from the beginning at most once per txg. If we + * still can't allocate from that L1 block, search + * for an empty L0 block, which will quickly skip + * to the end of the metadnode if the no nearby L0 + * blocks are empty. This fallback avoids a + * pathology where full dnode blocks containing + * large dnodes appear sparse because they have a + * low blk_fill, leading to many failed allocation + * attempts. In the long term a better mechanism to + * search for sparse metadnode regions, such as + * spacemaps, could be implemented. + * + * os_scan_dnodes is set during txg sync if enough + * objects have been freed since the previous + * rescan to justify backfilling again. + * + * Note that dmu_traverse depends on the behavior + * that we use multiple blocks of the dnode object + * before going back to reuse objects. Any change + * to this algorithm should preserve that property + * or find another solution to the issues described + * in traverse_visitbp. + */ + if (P2PHASE(object, L1_dnode_count) == 0) { + uint64_t offset; + uint64_t blkfill; + int minlvl; + if (os->os_rescan_dnodes) { + offset = 0; + os->os_rescan_dnodes = B_FALSE; + } else { + offset = object << DNODE_SHIFT; + } + blkfill = restarted ? 1 : DNODES_PER_BLOCK >> 2; + minlvl = restarted ? 1 : 2; + restarted = B_TRUE; + error = dnode_next_offset(DMU_META_DNODE(os), + DNODE_FIND_HOLE, &offset, minlvl, + blkfill, 0); + if (error == 0) { + object = offset >> DNODE_SHIFT; + } } - error = dnode_next_offset(DMU_META_DNODE(os), - DNODE_FIND_HOLE, - &offset, 2, DNODES_PER_BLOCK >> 2, 0); - if (error == 0) - object = offset >> DNODE_SHIFT; + /* + * Note: if "restarted", we may find a L0 that + * is not suitably aligned. + */ + os->os_obj_next_chunk = + P2ALIGN(object, dnodes_per_chunk) + + dnodes_per_chunk; + (void) atomic_swap_64(cpuobj, object); + mutex_exit(&os->os_obj_lock); } - os->os_obj_next = ++object; /* + * The value of (*cpuobj) before adding dn_slots is the object + * ID assigned to us. The value afterwards is the object ID + * assigned to whoever wants to do an allocation next. + */ + object = atomic_add_64_nv(cpuobj, dn_slots) - dn_slots; + + /* * XXX We should check for an i/o error here and return * up to our caller. Actually we should pre-read it in * dmu_tx_assign(), but there is currently no mechanism * to do so. */ - (void) dnode_hold_impl(os, object, DNODE_MUST_BE_FREE, - FTAG, &dn); - if (dn) - break; + error = dnode_hold_impl(os, object, DNODE_MUST_BE_FREE, + dn_slots, FTAG, &dn); + if (error == 0) { + rw_enter(&dn->dn_struct_rwlock, RW_WRITER); + /* + * Another thread could have allocated it; check + * again now that we have the struct lock. + */ + if (dn->dn_type == DMU_OT_NONE) { + dnode_allocate(dn, ot, blocksize, 0, + bonustype, bonuslen, dn_slots, tx); + rw_exit(&dn->dn_struct_rwlock); + dmu_tx_add_new_object(tx, dn); + dnode_rele(dn, FTAG); + return (object); + } + rw_exit(&dn->dn_struct_rwlock); + dnode_rele(dn, FTAG); + DNODE_STAT_BUMP(dnode_alloc_race); + } - if (dmu_object_next(os, &object, B_TRUE, 0) == 0) - os->os_obj_next = object - 1; + /* + * Skip to next known valid starting point on error. This + * is the start of the next block of dnodes. + */ + if (dmu_object_next(os, &object, B_TRUE, 0) != 0) { + object = P2ROUNDUP(object + 1, DNODES_PER_BLOCK); + DNODE_STAT_BUMP(dnode_alloc_next_block); + } + (void) atomic_swap_64(cpuobj, object); } - - dnode_allocate(dn, ot, blocksize, indirect_blockshift, - bonustype, bonuslen, tx); - mutex_exit(&os->os_obj_lock); - - dmu_tx_add_new_object(tx, dn); - dnode_rele(dn, FTAG); - - return (object); } uint64_t dmu_object_alloc(objset_t *os, dmu_object_type_t ot, int blocksize, dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx) { - return (dmu_object_alloc_ibs(os, ot, blocksize, 0, - bonustype, bonuslen, tx)); + return (dmu_object_alloc_impl(os, ot, blocksize, 0, bonustype, + bonuslen, 0, tx)); } +uint64_t +dmu_object_alloc_ibs(objset_t *os, dmu_object_type_t ot, int blocksize, + int indirect_blockshift, dmu_object_type_t bonustype, int bonuslen, + dmu_tx_t *tx) +{ + return (dmu_object_alloc_impl(os, ot, blocksize, indirect_blockshift, + bonustype, bonuslen, 0, tx)); +} + +uint64_t +dmu_object_alloc_dnsize(objset_t *os, dmu_object_type_t ot, int blocksize, + dmu_object_type_t bonustype, int bonuslen, int dnodesize, dmu_tx_t *tx) +{ + return (dmu_object_alloc_impl(os, ot, blocksize, 0, bonustype, + bonuslen, dnodesize, tx)); +} + int dmu_object_claim(objset_t *os, uint64_t object, dmu_object_type_t ot, int blocksize, dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx) { + return (dmu_object_claim_dnsize(os, object, ot, blocksize, bonustype, + bonuslen, 0, tx)); +} + +int +dmu_object_claim_dnsize(objset_t *os, uint64_t object, dmu_object_type_t ot, + int blocksize, dmu_object_type_t bonustype, int bonuslen, + int dnodesize, dmu_tx_t *tx) +{ dnode_t *dn; + int dn_slots = dnodesize >> DNODE_SHIFT; int err; + if (dn_slots == 0) + dn_slots = DNODE_MIN_SLOTS; + ASSERT3S(dn_slots, >=, DNODE_MIN_SLOTS); + ASSERT3S(dn_slots, <=, DNODE_MAX_SLOTS); + if (object == DMU_META_DNODE_OBJECT && !dmu_tx_private_ok(tx)) return (SET_ERROR(EBADF)); - err = dnode_hold_impl(os, object, DNODE_MUST_BE_FREE, FTAG, &dn); + err = dnode_hold_impl(os, object, DNODE_MUST_BE_FREE, dn_slots, + FTAG, &dn); if (err) return (err); - dnode_allocate(dn, ot, blocksize, 0, bonustype, bonuslen, tx); + dnode_allocate(dn, ot, blocksize, 0, bonustype, bonuslen, dn_slots, tx); dmu_tx_add_new_object(tx, dn); dnode_rele(dn, FTAG); @@ -136,18 +262,28 @@ int dmu_object_reclaim(objset_t *os, uint64_t object, dmu_object_type_t ot, int blocksize, dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx) { + return (dmu_object_reclaim_dnsize(os, object, ot, blocksize, bonustype, + bonuslen, 0, tx)); +} + +int +dmu_object_reclaim_dnsize(objset_t *os, uint64_t object, dmu_object_type_t ot, + int blocksize, dmu_object_type_t bonustype, int bonuslen, int dnodesize, + dmu_tx_t *tx) +{ dnode_t *dn; + int dn_slots = dnodesize >> DNODE_SHIFT; int err; if (object == DMU_META_DNODE_OBJECT) return (SET_ERROR(EBADF)); - err = dnode_hold_impl(os, object, DNODE_MUST_BE_ALLOCATED, + err = dnode_hold_impl(os, object, DNODE_MUST_BE_ALLOCATED, 0, FTAG, &dn); if (err) return (err); - dnode_reallocate(dn, ot, blocksize, bonustype, bonuslen, tx); + dnode_reallocate(dn, ot, blocksize, bonustype, bonuslen, dn_slots, tx); dnode_rele(dn, FTAG); return (err); @@ -161,7 +297,7 @@ dmu_object_free(objset_t *os, uint64_t object, dmu_tx_ ASSERT(object != DMU_META_DNODE_OBJECT || dmu_tx_private_ok(tx)); - err = dnode_hold_impl(os, object, DNODE_MUST_BE_ALLOCATED, + err = dnode_hold_impl(os, object, DNODE_MUST_BE_ALLOCATED, 0, FTAG, &dn); if (err) return (err); @@ -186,8 +322,53 @@ dmu_object_free(objset_t *os, uint64_t object, dmu_tx_ int dmu_object_next(objset_t *os, uint64_t *objectp, boolean_t hole, uint64_t txg) { - uint64_t offset = (*objectp + 1) << DNODE_SHIFT; + uint64_t offset; + uint64_t start_obj; + struct dsl_dataset *ds = os->os_dsl_dataset; int error; + + if (*objectp == 0) { + start_obj = 1; + } else if (ds && ds->ds_feature_inuse[SPA_FEATURE_LARGE_DNODE]) { + uint64_t i = *objectp + 1; + uint64_t last_obj = *objectp | (DNODES_PER_BLOCK - 1); + dmu_object_info_t doi; + + /* + * Scan through the remaining meta dnode block. The contents + * of each slot in the block are known so it can be quickly + * checked. If the block is exhausted without a match then + * hand off to dnode_next_offset() for further scanning. + */ + while (i <= last_obj) { + error = dmu_object_info(os, i, &doi); + if (error == ENOENT) { + if (hole) { + *objectp = i; + return (0); + } else { + i++; + } + } else if (error == EEXIST) { + i++; + } else if (error == 0) { + if (hole) { + i += doi.doi_dnodesize >> DNODE_SHIFT; + } else { + *objectp = i; + return (0); + } + } else { + return (error); + } + } + + start_obj = i; + } else { + start_obj = *objectp + 1; + } + + offset = start_obj << DNODE_SHIFT; error = dnode_next_offset(DMU_META_DNODE(os), (hole ? DNODE_FIND_HOLE : 0), &offset, 0, DNODES_PER_BLOCK, txg); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Mon Aug 12 12:05:40 2019 (r350898) @@ -140,6 +140,12 @@ dmu_objset_id(objset_t *os) return (ds ? ds->ds_object : 0); } +uint64_t +dmu_objset_dnodesize(objset_t *os) +{ + return (os->os_dnodesize); +} + zfs_sync_type_t dmu_objset_syncprop(objset_t *os) { @@ -270,6 +276,34 @@ redundant_metadata_changed_cb(void *arg, uint64_t newv } static void +dnodesize_changed_cb(void *arg, uint64_t newval) +{ + objset_t *os = arg; + + switch (newval) { + case ZFS_DNSIZE_LEGACY: + os->os_dnodesize = DNODE_MIN_SIZE; + break; + case ZFS_DNSIZE_AUTO: + /* + * Choose a dnode size that will work well for most + * workloads if the user specified "auto". Future code + * improvements could dynamically select a dnode size + * based on observed workload patterns. + */ + os->os_dnodesize = DNODE_MIN_SIZE * 2; + break; + case ZFS_DNSIZE_1K: + case ZFS_DNSIZE_2K: + case ZFS_DNSIZE_4K: + case ZFS_DNSIZE_8K: + case ZFS_DNSIZE_16K: + os->os_dnodesize = newval; + break; + } +} + +static void logbias_changed_cb(void *arg, uint64_t newval) { objset_t *os = arg; @@ -477,6 +511,11 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl zfs_prop_to_name(ZFS_PROP_RECORDSIZE), recordsize_changed_cb, os); } + if (err == 0) { + err = dsl_prop_register(ds, + zfs_prop_to_name(ZFS_PROP_DNODESIZE), + dnodesize_changed_cb, os); + } } if (needlock) dsl_pool_config_exit(dmu_objset_pool(os), FTAG); @@ -496,6 +535,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl os->os_sync = ZFS_SYNC_STANDARD; os->os_primary_cache = ZFS_CACHE_ALL; os->os_secondary_cache = ZFS_CACHE_ALL; + os->os_dnodesize = DNODE_MIN_SIZE; } /* * These properties will be filled in by the logic in zfs_get_zplprop() @@ -524,6 +564,9 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl mutex_init(&os->os_userused_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&os->os_obj_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&os->os_user_ptr_lock, NULL, MUTEX_DEFAULT, NULL); + os->os_obj_next_percpu_len = boot_ncpus; + os->os_obj_next_percpu = kmem_zalloc(os->os_obj_next_percpu_len * + sizeof (os->os_obj_next_percpu[0]), KM_SLEEP); dnode_special_open(os, &os->os_phys->os_meta_dnode, DMU_META_DNODE_OBJECT, &os->os_meta_dnode); @@ -802,6 +845,9 @@ dmu_objset_evict_done(objset_t *os) rw_enter(&os_lock, RW_READER); rw_exit(&os_lock); + kmem_free(os->os_obj_next_percpu, + os->os_obj_next_percpu_len * sizeof (os->os_obj_next_percpu[0])); + mutex_destroy(&os->os_lock); mutex_destroy(&os->os_userused_lock); mutex_destroy(&os->os_obj_lock); @@ -836,8 +882,8 @@ dmu_objset_create_impl(spa_t *spa, dsl_dataset_t *ds, mdn = DMU_META_DNODE(os); - dnode_allocate(mdn, DMU_OT_DNODE, 1 << DNODE_BLOCK_SHIFT, - DN_MAX_INDBLKSHIFT, DMU_OT_NONE, 0, tx); + dnode_allocate(mdn, DMU_OT_DNODE, DNODE_BLOCK_SIZE, DN_MAX_INDBLKSHIFT, + DMU_OT_NONE, 0, DNODE_MIN_SLOTS, tx); /* * We don't want to have to increase the meta-dnode's nlevels @@ -1496,7 +1542,7 @@ do_userquota_update(userquota_cache_t *cache, uint64_t uint64_t user, uint64_t group, boolean_t subtract) { if ((flags & DNODE_FLAG_USERUSED_ACCOUNTED)) { - int64_t delta = DNODE_SIZE + used; + int64_t delta = DNODE_MIN_SIZE + used; if (subtract) delta = -delta; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Mon Aug 12 12:05:40 2019 (r350898) @@ -469,6 +469,7 @@ dump_dnode(dmu_sendarg_t *dsp, uint64_t object, dnode_ drro->drr_bonustype = dnp->dn_bonustype; drro->drr_blksz = dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT; drro->drr_bonuslen = dnp->dn_bonuslen; + drro->drr_dn_slots = dnp->dn_extra_slots + 1; drro->drr_checksumtype = dnp->dn_checksum; drro->drr_compress = dnp->dn_compress; drro->drr_toguid = dsp->dsa_toguid; @@ -621,7 +622,7 @@ do_dump(dmu_sendarg_t *dsa, struct send_block_record * } else if (zb->zb_level > 0 || type == DMU_OT_OBJSET) { return (0); } else if (type == DMU_OT_DNODE) { - int blksz = BP_GET_LSIZE(bp); + int epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT; arc_flags_t aflags = ARC_FLAG_WAIT; arc_buf_t *abuf; @@ -633,8 +634,8 @@ do_dump(dmu_sendarg_t *dsa, struct send_block_record * return (SET_ERROR(EIO)); dnode_phys_t *blk = abuf->b_data; - uint64_t dnobj = zb->zb_blkid * (blksz >> DNODE_SHIFT); - for (int i = 0; i < blksz >> DNODE_SHIFT; i++) { + uint64_t dnobj = zb->zb_blkid * epb; + for (int i = 0; i < epb; i += blk[i].dn_extra_slots + 1) { err = dump_dnode(dsa, dnobj + i, blk + i); if (err != 0) break; @@ -802,6 +803,8 @@ dmu_send_impl(void *tag, dsl_pool_t *dp, dsl_dataset_t if (large_block_ok && to_ds->ds_feature_inuse[SPA_FEATURE_LARGE_BLOCKS]) featureflags |= DMU_BACKUP_FEATURE_LARGE_BLOCKS; + if (to_ds->ds_feature_inuse[SPA_FEATURE_LARGE_DNODE]) + featureflags |= DMU_BACKUP_FEATURE_LARGE_DNODE; if (embedok && spa_feature_is_active(dp->dp_spa, SPA_FEATURE_EMBEDDED_DATA)) { featureflags |= DMU_BACKUP_FEATURE_EMBED_DATA; @@ -1396,11 +1399,15 @@ dmu_recv_begin_check(void *arg, dmu_tx_t *tx) /* * The receiving code doesn't know how to translate large blocks * to smaller ones, so the pool must have the LARGE_BLOCKS - * feature enabled if the stream has LARGE_BLOCKS. + * feature enabled if the stream has LARGE_BLOCKS. Same with + * large dnodes. */ if ((featureflags & DMU_BACKUP_FEATURE_LARGE_BLOCKS) && !spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LARGE_BLOCKS)) return (SET_ERROR(ENOTSUP)); + if ((featureflags & DMU_BACKUP_FEATURE_LARGE_DNODE) && + !spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LARGE_DNODE)) + return (SET_ERROR(ENOTSUP)); error = dsl_dataset_hold(dp, tofs, FTAG, &ds); if (error == 0) { @@ -1605,6 +1612,9 @@ dmu_recv_resume_begin_check(void *arg, dmu_tx_t *tx) dsl_dataset_t *ds; const char *tofs = drba->drba_cookie->drc_tofs; + /* 6 extra bytes for /%recv */ + char recvname[ZFS_MAX_DATASET_NAME_LEN + 6]; + /* already checked */ ASSERT3U(drrb->drr_magic, ==, DMU_BACKUP_MAGIC); ASSERT(featureflags & DMU_BACKUP_FEATURE_RESUMING); @@ -1632,8 +1642,18 @@ dmu_recv_resume_begin_check(void *arg, dmu_tx_t *tx) !spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LZ4_COMPRESS)) return (SET_ERROR(ENOTSUP)); - /* 6 extra bytes for /%recv */ - char recvname[ZFS_MAX_DATASET_NAME_LEN + 6]; + /* + * The receiving code doesn't know how to translate large blocks + * to smaller ones, so the pool must have the LARGE_BLOCKS + * feature enabled if the stream has LARGE_BLOCKS. Same with + * large dnodes. + */ + if ((featureflags & DMU_BACKUP_FEATURE_LARGE_BLOCKS) && + !spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LARGE_BLOCKS)) + return (SET_ERROR(ENOTSUP)); + if ((featureflags & DMU_BACKUP_FEATURE_LARGE_DNODE) && + !spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LARGE_DNODE)) + return (SET_ERROR(ENOTSUP)); (void) snprintf(recvname, sizeof (recvname), "%s/%s", tofs, recv_clone_name); @@ -2024,7 +2044,8 @@ deduce_nblkptr(dmu_object_type_t bonus_type, uint64_t return (1); } else { return (1 + - ((DN_MAX_BONUSLEN - bonus_size) >> SPA_BLKPTRSHIFT)); + ((DN_OLD_MAX_BONUSLEN - + MIN(DN_OLD_MAX_BONUSLEN, bonus_size)) >> SPA_BLKPTRSHIFT)); } } @@ -2082,15 +2103,17 @@ receive_object(struct receive_writer_arg *rwa, struct P2PHASE(drro->drr_blksz, SPA_MINBLOCKSIZE) || drro->drr_blksz < SPA_MINBLOCKSIZE || drro->drr_blksz > spa_maxblocksize(dmu_objset_spa(rwa->os)) || - drro->drr_bonuslen > DN_MAX_BONUSLEN) { + drro->drr_bonuslen > + DN_BONUS_SIZE(spa_maxdnodesize(dmu_objset_spa(rwa->os))) || + drro->drr_dn_slots > + (spa_maxdnodesize(dmu_objset_spa(rwa->os)) >> DNODE_SHIFT)) { return (SET_ERROR(EINVAL)); } err = dmu_object_info(rwa->os, drro->drr_object, &doi); - if (err != 0 && err != ENOENT) + if (err != 0 && err != ENOENT && err != EEXIST) return (SET_ERROR(EINVAL)); - object = err == 0 ? drro->drr_object : DMU_NEW_OBJECT; if (drro->drr_object > rwa->max_object) rwa->max_object = drro->drr_object; @@ -2103,18 +2126,66 @@ receive_object(struct receive_writer_arg *rwa, struct if (err == 0) { int nblkptr; + object = drro->drr_object; + nblkptr = deduce_nblkptr(drro->drr_bonustype, drro->drr_bonuslen); if (drro->drr_blksz != doi.doi_data_block_size || - nblkptr < doi.doi_nblkptr) { + nblkptr < doi.doi_nblkptr || + drro->drr_dn_slots != doi.doi_dnodesize >> DNODE_SHIFT) { err = dmu_free_long_range(rwa->os, drro->drr_object, 0, DMU_OBJECT_END); if (err != 0) return (SET_ERROR(EINVAL)); } + } else if (err == EEXIST) { + /* + * The object requested is currently an interior slot of a + * multi-slot dnode. This will be resolved when the next txg + * is synced out, since the send stream will have told us + * to free this slot when we freed the associated dnode + * earlier in the stream. + */ + txg_wait_synced(dmu_objset_pool(rwa->os), 0); + object = drro->drr_object; + } else { + /* object is free and we are about to allocate a new one */ + object = DMU_NEW_OBJECT; } + /* + * If this is a multi-slot dnode there is a chance that this + * object will expand into a slot that is already used by + * another object from the previous snapshot. We must free + * these objects before we attempt to allocate the new dnode. + */ + if (drro->drr_dn_slots > 1) { + boolean_t need_sync = B_FALSE; + + for (uint64_t slot = drro->drr_object + 1; + slot < drro->drr_object + drro->drr_dn_slots; + slot++) { + dmu_object_info_t slot_doi; + + err = dmu_object_info(rwa->os, slot, &slot_doi); + if (err == ENOENT || err == EEXIST) + continue; + else if (err != 0) + return (err); + + err = dmu_free_long_object(rwa->os, slot); + + if (err != 0) + return (err); + + need_sync = B_TRUE; + } + + if (need_sync) + txg_wait_synced(dmu_objset_pool(rwa->os), 0); + } + tx = dmu_tx_create(rwa->os); dmu_tx_hold_bonus(tx, object); err = dmu_tx_assign(tx, TXG_WAIT); @@ -2125,9 +2196,10 @@ receive_object(struct receive_writer_arg *rwa, struct if (object == DMU_NEW_OBJECT) { /* currently free, want to be allocated */ - err = dmu_object_claim(rwa->os, drro->drr_object, + err = dmu_object_claim_dnsize(rwa->os, drro->drr_object, drro->drr_type, drro->drr_blksz, - drro->drr_bonustype, drro->drr_bonuslen, tx); + drro->drr_bonustype, drro->drr_bonuslen, + drro->drr_dn_slots << DNODE_SHIFT, tx); } else if (drro->drr_type != doi.doi_type || drro->drr_blksz != doi.doi_data_block_size || drro->drr_bonustype != doi.doi_bonus_type || @@ -2179,13 +2251,18 @@ receive_freeobjects(struct receive_writer_arg *rwa, if (drrfo->drr_firstobj + drrfo->drr_numobjs < drrfo->drr_firstobj) return (SET_ERROR(EINVAL)); - for (obj = drrfo->drr_firstobj; + for (obj = drrfo->drr_firstobj == 0 ? 1 : drrfo->drr_firstobj; obj < drrfo->drr_firstobj + drrfo->drr_numobjs && next_err == 0; next_err = dmu_object_next(rwa->os, &obj, FALSE, 0)) { int err; - if (dmu_object_info(rwa->os, obj, NULL) != 0) + err = dmu_object_info(rwa->os, obj, NULL); + if (err == ENOENT) { + obj++; continue; + } else if (err != 0) { + return (err); + } err = dmu_free_long_object(rwa->os, obj); if (err != 0) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c Mon Aug 12 12:05:40 2019 (r350898) @@ -327,13 +327,13 @@ traverse_visitbp(traverse_data_t *td, const dnode_phys goto post; dnode_phys_t *child_dnp = buf->b_data; - for (i = 0; i < epb; i++) { + for (i = 0; i < epb; i += child_dnp[i].dn_extra_slots + 1) { prefetch_dnode_metadata(td, &child_dnp[i], zb->zb_objset, zb->zb_blkid * epb + i); } /* recursively visitbp() blocks below this */ - for (i = 0; i < epb; i++) { + for (i = 0; i < epb; i += child_dnp[i].dn_extra_slots + 1) { err = traverse_dnode(td, &child_dnp[i], zb->zb_objset, zb->zb_blkid * epb + i); if (err != 0) @@ -435,7 +435,7 @@ prefetch_dnode_metadata(traverse_data_t *td, const dno if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) { SET_BOOKMARK(&czb, objset, object, 0, DMU_SPILL_BLKID); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon Aug 12 12:05:44 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5912EB1D7C; Mon, 12 Aug 2019 12:05:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 466ZLJ2C9Mz4H7h; Mon, 12 Aug 2019 12:05:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2EB9C264EC; Mon, 12 Aug 2019 12:05:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7CC5iBx017161; Mon, 12 Aug 2019 12:05:44 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7CC5hlJ017155; Mon, 12 Aug 2019 12:05:43 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201908121205.x7CC5hlJ017155@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Mon, 12 Aug 2019 12:05:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r350898 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zdb vendor/... X-SVN-Commit-Revision: 350898 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Aug 2019 12:05:44 -0000 Author: avg Date: Mon Aug 12 12:05:40 2019 New Revision: 350898 URL: https://svnweb.freebsd.org/changeset/base/350898 Log: 8423 8199 7432 Implement large_dnode pool feature 8423 Implement large_dnode pool feature 8199 multi-threaded dmu_object_alloc() 7432 Large dnode pool feature llumos/illumos-gate@54811da5ac6b517992fdc173df5d605e4e61fdc0 https://github.com/illumos/illumos-gate/commit/54811da5ac6b517992fdc173df5d605e4e61fdc0 https://www.illumos.org/issues/8423 https://www.illumos.org/issues/8199 https://www.illumos.org/issues/7432 ZoL issues: Improved dnode allocation #6564 Clean up large dnode code #6262 Fix dnode_hold() freeing dnode behavior #8172 Fix dnode allocation race #6414, #6439 Partial: Raw sends must be able to decrease nlevels #6821, #6864 Remove unnecessary txg syncs from receive_object() Closes #7197 Author: Toomas Soome Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zdb/zdb_il.c vendor/illumos/dist/cmd/zstreamdump/zstreamdump.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/man/man5/zpool-features.5 Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/common/zfs/zpool_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_traverse.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/sa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/sa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zap.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_znode.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/zap.c vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_acl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_replay.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_sa.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor/illumos/dist/cmd/zdb/zdb.c Mon Aug 12 12:05:40 2019 (r350898) @@ -2108,14 +2108,15 @@ static object_viewer_t *object_viewer[DMU_OT_NUMTYPES }; static void -dump_object(objset_t *os, uint64_t object, int verbosity, int *print_header) +dump_object(objset_t *os, uint64_t object, int verbosity, int *print_header, + uint64_t *dnode_slots_used) { dmu_buf_t *db = NULL; dmu_object_info_t doi; dnode_t *dn; void *bonus = NULL; size_t bsize = 0; - char iblk[32], dblk[32], lsize[32], asize[32], fill[32]; + char iblk[32], dblk[32], lsize[32], asize[32], fill[32], dnsize[32]; char bonus_size[32]; char aux[50]; int error; @@ -2128,9 +2129,9 @@ dump_object(objset_t *os, uint64_t object, int verbosi CTASSERT(sizeof (bonus_size) >= NN_NUMBUF_SZ); if (*print_header) { - (void) printf("\n%10s %3s %5s %5s %5s %5s %6s %s\n", - "Object", "lvl", "iblk", "dblk", "dsize", "lsize", - "%full", "type"); + (void) printf("\n%10s %3s %5s %5s %5s %6s %5s %6s %s\n", + "Object", "lvl", "iblk", "dblk", "dsize", "dnsize", + "lsize", "%full", "type"); *print_header = 0; } @@ -2147,11 +2148,15 @@ dump_object(objset_t *os, uint64_t object, int verbosi } dmu_object_info_from_dnode(dn, &doi); + if (dnode_slots_used != NULL) + *dnode_slots_used = doi.doi_dnodesize / DNODE_MIN_SIZE; + zdb_nicenum(doi.doi_metadata_block_size, iblk, sizeof (iblk)); zdb_nicenum(doi.doi_data_block_size, dblk, sizeof (dblk)); zdb_nicenum(doi.doi_max_offset, lsize, sizeof (lsize)); zdb_nicenum(doi.doi_physical_blocks_512 << 9, asize, sizeof (asize)); zdb_nicenum(doi.doi_bonus_size, bonus_size, sizeof (bonus_size)); + zdb_nicenum(doi.doi_dnodesize, dnsize, sizeof (dnsize)); (void) sprintf(fill, "%6.2f", 100.0 * doi.doi_fill_count * doi.doi_data_block_size / (object == 0 ? DNODES_PER_BLOCK : 1) / doi.doi_max_offset); @@ -2168,13 +2173,14 @@ dump_object(objset_t *os, uint64_t object, int verbosi ZDB_COMPRESS_NAME(doi.doi_compress)); } - (void) printf("%10lld %3u %5s %5s %5s %5s %6s %s%s\n", - (u_longlong_t)object, doi.doi_indirection, iblk, dblk, - asize, lsize, fill, ZDB_OT_NAME(doi.doi_type), aux); + (void) printf("%10" PRIu64 + " %3u %5s %5s %5s %5s %5s %6s %s%s\n", + object, doi.doi_indirection, iblk, dblk, + asize, dnsize, lsize, fill, ZDB_OT_NAME(doi.doi_type), aux); if (doi.doi_bonus_type != DMU_OT_NONE && verbosity > 3) { - (void) printf("%10s %3s %5s %5s %5s %5s %6s %s\n", - "", "", "", "", "", bonus_size, "bonus", + (void) printf("%10s %3s %5s %5s %5s %5s %5s %6s %s\n", + "", "", "", "", "", "", bonus_size, "bonus", ZDB_OT_NAME(doi.doi_bonus_type)); } @@ -2278,6 +2284,9 @@ dump_dir(objset_t *os) int print_header = 1; unsigned i; int error; + uint64_t total_slots_used = 0; + uint64_t max_slot_used = 0; + uint64_t dnode_slots; /* make sure nicenum has enough space */ CTASSERT(sizeof (numbuf) >= NN_NUMBUF_SZ); @@ -2322,7 +2331,7 @@ dump_dir(objset_t *os) if (zopt_objects != 0) { for (i = 0; i < zopt_objects; i++) dump_object(os, zopt_object[i], verbosity, - &print_header); + &print_header, NULL); (void) printf("\n"); return; } @@ -2347,24 +2356,39 @@ dump_dir(objset_t *os) if (BP_IS_HOLE(os->os_rootbp)) return; - dump_object(os, 0, verbosity, &print_header); + dump_object(os, 0, verbosity, &print_header, NULL); object_count = 0; if (DMU_USERUSED_DNODE(os) != NULL && DMU_USERUSED_DNODE(os)->dn_type != 0) { - dump_object(os, DMU_USERUSED_OBJECT, verbosity, &print_header); - dump_object(os, DMU_GROUPUSED_OBJECT, verbosity, &print_header); + dump_object(os, DMU_USERUSED_OBJECT, verbosity, &print_header, + NULL); + dump_object(os, DMU_GROUPUSED_OBJECT, verbosity, &print_header, + NULL); } object = 0; while ((error = dmu_object_next(os, &object, B_FALSE, 0)) == 0) { - dump_object(os, object, verbosity, &print_header); + dump_object(os, object, verbosity, &print_header, &dnode_slots); object_count++; + total_slots_used += dnode_slots; + max_slot_used = object + dnode_slots - 1; } ASSERT3U(object_count, ==, usedobjs); (void) printf("\n"); + (void) printf(" Dnode slots:\n"); + (void) printf("\tTotal used: %10llu\n", + (u_longlong_t)total_slots_used); + (void) printf("\tMax used: %10llu\n", + (u_longlong_t)max_slot_used); + (void) printf("\tPercent empty: %10lf\n", + (double)(max_slot_used - total_slots_used)*100 / + (double)max_slot_used); + + (void) printf("\n"); + if (error != ESRCH) { (void) fprintf(stderr, "dmu_object_next() = %d\n", error); abort(); @@ -2553,7 +2577,7 @@ dump_path_impl(objset_t *os, uint64_t obj, char *name) return (dump_path_impl(os, child_obj, s + 1)); /*FALLTHROUGH*/ case DMU_OT_PLAIN_FILE_CONTENTS: - dump_object(os, child_obj, dump_opt['v'], &header); + dump_object(os, child_obj, dump_opt['v'], &header, NULL); return (0); default: (void) fprintf(stderr, "object %llu has non-file/directory " Modified: vendor/illumos/dist/cmd/zdb/zdb_il.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb_il.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor/illumos/dist/cmd/zdb/zdb_il.c Mon Aug 12 12:05:40 2019 (r350898) @@ -84,13 +84,15 @@ zil_prt_rec_create(zilog_t *zilog, int txtype, void *a } (void) printf("%s%s", tab_prefix, ctime(&crtime)); - (void) printf("%sdoid %llu, foid %llu, mode %llo\n", tab_prefix, - (u_longlong_t)lr->lr_doid, (u_longlong_t)lr->lr_foid, - (longlong_t)lr->lr_mode); - (void) printf("%suid %llu, gid %llu, gen %llu, rdev 0x%llx\n", - tab_prefix, - (u_longlong_t)lr->lr_uid, (u_longlong_t)lr->lr_gid, - (u_longlong_t)lr->lr_gen, (u_longlong_t)lr->lr_rdev); + (void) printf("%sdoid %" PRIu64 ", foid %" PRIu64 ", slots %" PRIu64 + ", mode %" PRIo64 "\n", + tab_prefix, lr->lr_doid, + LR_FOID_GET_OBJ(lr->lr_foid), + LR_FOID_GET_SLOTS(lr->lr_foid), + lr->lr_mode); + (void) printf("%suid %" PRIu64 ", gid %" PRIu64 ", gen %" PRIu64 + ", rdev %#" PRIx64 "\n", + tab_prefix, lr->lr_uid, lr->lr_gid, lr->lr_gen, lr->lr_rdev); } /* ARGSUSED */ Modified: vendor/illumos/dist/cmd/zstreamdump/zstreamdump.c ============================================================================== --- vendor/illumos/dist/cmd/zstreamdump/zstreamdump.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor/illumos/dist/cmd/zstreamdump/zstreamdump.c Mon Aug 12 12:05:40 2019 (r350898) @@ -416,13 +416,15 @@ main(int argc, char *argv[]) drro->drr_toguid = BSWAP_64(drro->drr_toguid); } if (verbose) { - (void) printf("OBJECT object = %llu type = %u " - "bonustype = %u blksz = %u bonuslen = %u\n", - (u_longlong_t)drro->drr_object, + (void) printf("OBJECT object = %" PRIu64 + " type = %u bonustype = %u blksz = %u" + " bonuslen = %u dn_slots = %u\n", + drro->drr_object, drro->drr_type, drro->drr_bonustype, drro->drr_blksz, - drro->drr_bonuslen); + drro->drr_bonuslen, + drro->drr_dn_slots); } if (drro->drr_bonuslen > 0) { (void) ssread(buf, Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Mon Aug 12 11:42:16 2019 (r350897) +++ vendor/illumos/dist/cmd/ztest/ztest.c Mon Aug 12 12:05:40 2019 (r350898) @@ -194,6 +194,7 @@ extern uint64_t zfs_deadman_synctime_ms; extern int metaslab_preload_limit; extern boolean_t zfs_compressed_arc_enabled; extern boolean_t zfs_abd_scatter_enabled; +extern int dmu_object_alloc_chunk_shift; extern boolean_t zfs_force_some_double_word_sm_entries; static ztest_shared_opts_t *ztest_shared_opts; @@ -224,6 +225,7 @@ typedef struct ztest_block_tag { uint64_t bt_magic; uint64_t bt_objset; uint64_t bt_object; + uint64_t bt_dnodesize; uint64_t bt_offset; uint64_t bt_gen; uint64_t bt_txg; @@ -274,6 +276,7 @@ typedef struct ztest_od { dmu_object_type_t od_crtype; uint64_t od_blocksize; uint64_t od_crblocksize; + uint64_t od_crdnodesize; uint64_t od_gen; uint64_t od_crgen; char od_name[ZFS_MAX_DATASET_NAME_LEN]; @@ -320,6 +323,7 @@ static ztest_shared_callstate_t *ztest_shared_callstat ztest_func_t ztest_dmu_read_write; ztest_func_t ztest_dmu_write_parallel; ztest_func_t ztest_dmu_object_alloc_free; +ztest_func_t ztest_dmu_object_next_chunk; ztest_func_t ztest_dmu_commit_callbacks; ztest_func_t ztest_zap; ztest_func_t ztest_zap_parallel; @@ -349,6 +353,7 @@ ztest_func_t ztest_device_removal; ztest_func_t ztest_remap_blocks; ztest_func_t ztest_spa_checkpoint_create_discard; ztest_func_t ztest_initialize; +ztest_func_t ztest_verify_dnode_bt; uint64_t zopt_always = 0ULL * NANOSEC; /* all the time */ uint64_t zopt_incessant = 1ULL * NANOSEC / 10; /* every 1/10 second */ @@ -360,6 +365,7 @@ ztest_info_t ztest_info[] = { { ztest_dmu_read_write, 1, &zopt_always }, { ztest_dmu_write_parallel, 10, &zopt_always }, { ztest_dmu_object_alloc_free, 1, &zopt_always }, + { ztest_dmu_object_next_chunk, 1, &zopt_sometimes }, { ztest_dmu_commit_callbacks, 1, &zopt_always }, { ztest_zap, 30, &zopt_always }, { ztest_zap_parallel, 100, &zopt_always }, @@ -392,7 +398,8 @@ ztest_info_t ztest_info[] = { { ztest_device_removal, 1, &zopt_sometimes }, { ztest_remap_blocks, 1, &zopt_sometimes }, { ztest_spa_checkpoint_create_discard, 1, &zopt_rarely }, - { ztest_initialize, 1, &zopt_sometimes } + { ztest_initialize, 1, &zopt_sometimes }, + { ztest_verify_dnode_bt, 1, &zopt_sometimes } }; #define ZTEST_FUNCS (sizeof (ztest_info) / sizeof (ztest_info_t)) @@ -438,8 +445,8 @@ static spa_t *ztest_spa = NULL; static ztest_ds_t *ztest_ds; static kmutex_t ztest_vdev_lock; -static kmutex_t ztest_checkpoint_lock; static boolean_t ztest_device_removal_active = B_FALSE; +static kmutex_t ztest_checkpoint_lock; /* * The ztest_name_lock protects the pool and dataset namespace used by @@ -1009,6 +1016,36 @@ ztest_random_blocksize(void) } static int +ztest_random_dnodesize(void) +{ + int slots; + int max_slots = spa_maxdnodesize(ztest_spa) >> DNODE_SHIFT; + + if (max_slots == DNODE_MIN_SLOTS) + return (DNODE_MIN_SIZE); + + /* + * Weight the random distribution more heavily toward smaller + * dnode sizes since that is more likely to reflect real-world + * usage. + */ + ASSERT3U(max_slots, >, 4); + switch (ztest_random(10)) { + case 0: + slots = 5 + ztest_random(max_slots - 4); + break; + case 1 ... 4: + slots = 2 + ztest_random(3); + break; + default: + slots = 1; + break; + } + + return (slots << DNODE_SHIFT); +} + +static int ztest_random_ibshift(void) { return (DN_MIN_INDBLKSHIFT + @@ -1285,11 +1322,13 @@ ztest_pattern_match(void *buf, uint64_t size, uint64_t static void ztest_bt_generate(ztest_block_tag_t *bt, objset_t *os, uint64_t object, - uint64_t offset, uint64_t gen, uint64_t txg, uint64_t crtxg) + uint64_t dnodesize, uint64_t offset, uint64_t gen, uint64_t txg, + uint64_t crtxg) { bt->bt_magic = BT_MAGIC; bt->bt_objset = dmu_objset_id(os); bt->bt_object = object; + bt->bt_dnodesize = dnodesize; bt->bt_offset = offset; bt->bt_gen = gen; bt->bt_txg = txg; @@ -1298,11 +1337,13 @@ ztest_bt_generate(ztest_block_tag_t *bt, objset_t *os, static void ztest_bt_verify(ztest_block_tag_t *bt, objset_t *os, uint64_t object, - uint64_t offset, uint64_t gen, uint64_t txg, uint64_t crtxg) + uint64_t dnodesize, uint64_t offset, uint64_t gen, uint64_t txg, + uint64_t crtxg) { ASSERT3U(bt->bt_magic, ==, BT_MAGIC); ASSERT3U(bt->bt_objset, ==, dmu_objset_id(os)); ASSERT3U(bt->bt_object, ==, object); + ASSERT3U(bt->bt_dnodesize, ==, dnodesize); ASSERT3U(bt->bt_offset, ==, offset); ASSERT3U(bt->bt_gen, <=, gen); ASSERT3U(bt->bt_txg, <=, txg); @@ -1324,6 +1365,52 @@ ztest_bt_bonus(dmu_buf_t *db) } /* + * Generate a token to fill up unused bonus buffer space. Try to make + * it unique to the object, generation, and offset to verify that data + * is not getting overwritten by data from other dnodes. + */ +#define ZTEST_BONUS_FILL_TOKEN(obj, ds, gen, offset) \ + (((ds) << 48) | ((gen) << 32) | ((obj) << 8) | (offset)) + +/* + * Fill up the unused bonus buffer region before the block tag with a + * verifiable pattern. Filling the whole bonus area with non-zero data + * helps ensure that all dnode traversal code properly skips the + * interior regions of large dnodes. + */ +void +ztest_fill_unused_bonus(dmu_buf_t *db, void *end, uint64_t obj, + objset_t *os, uint64_t gen) +{ + uint64_t *bonusp; + + ASSERT(IS_P2ALIGNED((char *)end - (char *)db->db_data, 8)); + + for (bonusp = db->db_data; bonusp < (uint64_t *)end; bonusp++) { + uint64_t token = ZTEST_BONUS_FILL_TOKEN(obj, dmu_objset_id(os), + gen, bonusp - (uint64_t *)db->db_data); + *bonusp = token; + } +} + +/* + * Verify that the unused area of a bonus buffer is filled with the + * expected tokens. + */ +void +ztest_verify_unused_bonus(dmu_buf_t *db, void *end, uint64_t obj, + objset_t *os, uint64_t gen) +{ + uint64_t *bonusp; + + for (bonusp = db->db_data; bonusp < (uint64_t *)end; bonusp++) { + uint64_t token = ZTEST_BONUS_FILL_TOKEN(obj, dmu_objset_id(os), + gen, bonusp - (uint64_t *)db->db_data); + VERIFY3U(*bonusp, ==, token); + } +} + +/* * ZIL logging ops */ @@ -1331,7 +1418,7 @@ ztest_bt_bonus(dmu_buf_t *db) #define lrz_blocksize lr_uid #define lrz_ibshift lr_gid #define lrz_bonustype lr_rdev -#define lrz_bonuslen lr_crtime[1] +#define lrz_dnodesize lr_crtime[1] static void ztest_log_create(ztest_ds_t *zd, dmu_tx_t *tx, lr_create_t *lr) @@ -1447,6 +1534,7 @@ ztest_replay_create(void *arg1, void *arg2, boolean_t dmu_tx_t *tx; uint64_t txg; int error = 0; + int bonuslen; if (byteswap) byteswap_uint64_array(lr, sizeof (*lr)); @@ -1469,26 +1557,27 @@ ztest_replay_create(void *arg1, void *arg2, boolean_t return (ENOSPC); ASSERT(dmu_objset_zil(os)->zl_replay == !!lr->lr_foid); + bonuslen = DN_BONUS_SIZE(lr->lrz_dnodesize); if (lr->lrz_type == DMU_OT_ZAP_OTHER) { if (lr->lr_foid == 0) { - lr->lr_foid = zap_create(os, + lr->lr_foid = zap_create_dnsize(os, lr->lrz_type, lr->lrz_bonustype, - lr->lrz_bonuslen, tx); + bonuslen, lr->lrz_dnodesize, tx); } else { - error = zap_create_claim(os, lr->lr_foid, + error = zap_create_claim_dnsize(os, lr->lr_foid, lr->lrz_type, lr->lrz_bonustype, - lr->lrz_bonuslen, tx); + bonuslen, lr->lrz_dnodesize, tx); } } else { if (lr->lr_foid == 0) { - lr->lr_foid = dmu_object_alloc(os, + lr->lr_foid = dmu_object_alloc_dnsize(os, lr->lrz_type, 0, lr->lrz_bonustype, - lr->lrz_bonuslen, tx); + bonuslen, lr->lrz_dnodesize, tx); } else { - error = dmu_object_claim(os, lr->lr_foid, + error = dmu_object_claim_dnsize(os, lr->lr_foid, lr->lrz_type, 0, lr->lrz_bonustype, - lr->lrz_bonuslen, tx); + bonuslen, lr->lrz_dnodesize, tx); } } @@ -1508,7 +1597,9 @@ ztest_replay_create(void *arg1, void *arg2, boolean_t VERIFY3U(0, ==, dmu_bonus_hold(os, lr->lr_foid, FTAG, &db)); bbt = ztest_bt_bonus(db); dmu_buf_will_dirty(db, tx); - ztest_bt_generate(bbt, os, lr->lr_foid, -1ULL, lr->lr_gen, txg, txg); + ztest_bt_generate(bbt, os, lr->lr_foid, lr->lrz_dnodesize, -1ULL, + lr->lr_gen, txg, txg); + ztest_fill_unused_bonus(db, bbt, lr->lr_foid, os, lr->lr_gen); dmu_buf_rele(db, FTAG); VERIFY3U(0, ==, zap_add(os, lr->lr_doid, name, sizeof (uint64_t), 1, @@ -1658,7 +1749,7 @@ ztest_replay_write(void *arg1, void *arg2, boolean_t b VERIFY(dmu_read(os, lr->lr_foid, offset, sizeof (rbt), &rbt, prefetch) == 0); if (rbt.bt_magic == BT_MAGIC) { - ztest_bt_verify(&rbt, os, lr->lr_foid, + ztest_bt_verify(&rbt, os, lr->lr_foid, 0, offset, gen, txg, crtxg); } } @@ -1670,7 +1761,7 @@ ztest_replay_write(void *arg1, void *arg2, boolean_t b * as it was when the write was generated. */ if (zd->zd_zilog->zl_replay) { - ztest_bt_verify(bt, os, lr->lr_foid, offset, + ztest_bt_verify(bt, os, lr->lr_foid, 0, offset, MAX(gen, bt->bt_gen), MAX(txg, lrtxg), bt->bt_crtxg); } @@ -1679,7 +1770,8 @@ ztest_replay_write(void *arg1, void *arg2, boolean_t b * Set the bt's gen/txg to the bonus buffer's gen/txg * so that all of the usual ASSERTs will work. */ - ztest_bt_generate(bt, os, lr->lr_foid, offset, gen, txg, crtxg); + ztest_bt_generate(bt, os, lr->lr_foid, 0, offset, gen, txg, + crtxg); } if (abuf == NULL) { @@ -1751,7 +1843,7 @@ ztest_replay_setattr(void *arg1, void *arg2, boolean_t dmu_tx_t *tx; dmu_buf_t *db; ztest_block_tag_t *bbt; - uint64_t txg, lrtxg, crtxg; + uint64_t txg, lrtxg, crtxg, dnodesize; if (byteswap) byteswap_uint64_array(lr, sizeof (*lr)); @@ -1774,6 +1866,7 @@ ztest_replay_setattr(void *arg1, void *arg2, boolean_t ASSERT3U(bbt->bt_magic, ==, BT_MAGIC); crtxg = bbt->bt_crtxg; lrtxg = lr->lr_common.lrc_txg; + dnodesize = bbt->bt_dnodesize; if (zd->zd_zilog->zl_replay) { ASSERT(lr->lr_size != 0); @@ -1792,7 +1885,7 @@ ztest_replay_setattr(void *arg1, void *arg2, boolean_t /* * Verify that the current bonus buffer is not newer than our txg. */ - ztest_bt_verify(bbt, os, lr->lr_foid, -1ULL, lr->lr_mode, + ztest_bt_verify(bbt, os, lr->lr_foid, dnodesize, -1ULL, lr->lr_mode, MAX(txg, lrtxg), crtxg); dmu_buf_will_dirty(db, tx); @@ -1802,7 +1895,9 @@ ztest_replay_setattr(void *arg1, void *arg2, boolean_t VERIFY0(dmu_set_bonus(db, lr->lr_size, tx)); bbt = ztest_bt_bonus(db); - ztest_bt_generate(bbt, os, lr->lr_foid, -1ULL, lr->lr_mode, txg, crtxg); + ztest_bt_generate(bbt, os, lr->lr_foid, dnodesize, -1ULL, lr->lr_mode, + txg, crtxg); + ztest_fill_unused_bonus(db, bbt, lr->lr_foid, os, bbt->bt_gen); dmu_buf_rele(db, FTAG); @@ -2033,7 +2128,7 @@ ztest_create(ztest_ds_t *zd, ztest_od_t *od, int count lr->lrz_blocksize = od->od_crblocksize; lr->lrz_ibshift = ztest_random_ibshift(); lr->lrz_bonustype = DMU_OT_UINT64_OTHER; - lr->lrz_bonuslen = dmu_bonus_max(); + lr->lrz_dnodesize = od->od_crdnodesize; lr->lr_gen = od->od_crgen; lr->lr_crtime[0] = time(NULL); @@ -2212,7 +2307,8 @@ ztest_io(ztest_ds_t *zd, uint64_t object, uint64_t off switch (io_type) { case ZTEST_IO_WRITE_TAG: - ztest_bt_generate(&wbt, zd->zd_os, object, offset, 0, 0, 0); + ztest_bt_generate(&wbt, zd->zd_os, object, doi.doi_dnodesize, + offset, 0, 0, 0); (void) ztest_write(zd, object, offset, sizeof (wbt), &wbt); break; @@ -2273,13 +2369,15 @@ ztest_io(ztest_ds_t *zd, uint64_t object, uint64_t off */ static void ztest_od_init(ztest_od_t *od, uint64_t id, char *tag, uint64_t index, - dmu_object_type_t type, uint64_t blocksize, uint64_t gen) + dmu_object_type_t type, uint64_t blocksize, uint64_t dnodesize, + uint64_t gen) { od->od_dir = ZTEST_DIROBJ; od->od_object = 0; od->od_crtype = type; od->od_crblocksize = blocksize ? blocksize : ztest_random_blocksize(); + od->od_crdnodesize = dnodesize ? dnodesize : ztest_random_dnodesize(); od->od_crgen = gen; od->od_type = DMU_OT_NONE; @@ -3721,8 +3819,10 @@ ztest_dmu_object_alloc_free(ztest_ds_t *zd, uint64_t i ztest_od_t od[4]; int batchsize = sizeof (od) / sizeof (od[0]); - for (int b = 0; b < batchsize; b++) - ztest_od_init(&od[b], id, FTAG, b, DMU_OT_UINT64_OTHER, 0, 0); + for (int b = 0; b < batchsize; b++) { + ztest_od_init(&od[b], id, FTAG, b, DMU_OT_UINT64_OTHER, + 0, 0, 0); + } /* * Destroy the previous batch of objects, create a new batch, @@ -3737,6 +3837,26 @@ ztest_dmu_object_alloc_free(ztest_ds_t *zd, uint64_t i } /* + * Rewind the global allocator to verify object allocation backfilling. + */ +void +ztest_dmu_object_next_chunk(ztest_ds_t *zd, uint64_t id) +{ + objset_t *os = zd->zd_os; + int dnodes_per_chunk = 1 << dmu_object_alloc_chunk_shift; + uint64_t object; + + /* + * Rewind the global allocator randomly back to a lower object number + * to force backfilling and reclamation of recently freed dnodes. + */ + mutex_enter(&os->os_obj_lock); + object = ztest_random(os->os_obj_next_chunk); + os->os_obj_next_chunk = P2ALIGN(object, dnodes_per_chunk); + mutex_exit(&os->os_obj_lock); +} + +/* * Verify that dmu_{read,write} work as expected. */ void @@ -3782,8 +3902,10 @@ ztest_dmu_read_write(ztest_ds_t *zd, uint64_t id) /* * Read the directory info. If it's the first time, set things up. */ - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, 0, chunksize); - ztest_od_init(&od[1], id, FTAG, 1, DMU_OT_UINT64_OTHER, 0, chunksize); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, 0, 0, + chunksize); + ztest_od_init(&od[1], id, FTAG, 1, DMU_OT_UINT64_OTHER, 0, 0, + chunksize); if (ztest_object_init(zd, od, sizeof (od), B_FALSE) != 0) return; @@ -4052,8 +4174,10 @@ ztest_dmu_read_write_zcopy(ztest_ds_t *zd, uint64_t id /* * Read the directory info. If it's the first time, set things up. */ - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, blocksize, 0); - ztest_od_init(&od[1], id, FTAG, 1, DMU_OT_UINT64_OTHER, 0, chunksize); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, blocksize, + 0, 0); + ztest_od_init(&od[1], id, FTAG, 1, DMU_OT_UINT64_OTHER, 0, 0, + chunksize); if (ztest_object_init(zd, od, sizeof (od), B_FALSE) != 0) return; @@ -4254,7 +4378,8 @@ ztest_dmu_write_parallel(ztest_ds_t *zd, uint64_t id) * to verify that parallel writes to an object -- even to the * same blocks within the object -- doesn't cause any trouble. */ - ztest_od_init(&od[0], ID_PARALLEL, FTAG, 0, DMU_OT_UINT64_OTHER, 0, 0); + ztest_od_init(&od[0], ID_PARALLEL, FTAG, 0, DMU_OT_UINT64_OTHER, + 0, 0, 0); if (ztest_object_init(zd, od, sizeof (od), B_FALSE) != 0) return; @@ -4273,7 +4398,8 @@ ztest_dmu_prealloc(ztest_ds_t *zd, uint64_t id) uint64_t blocksize = ztest_random_blocksize(); void *data; - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, blocksize, 0); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, blocksize, + 0, 0); if (ztest_object_init(zd, od, sizeof (od), !ztest_random(2)) != 0) return; @@ -4319,7 +4445,7 @@ ztest_zap(ztest_ds_t *zd, uint64_t id) int error; char *hc[2] = { "s.acl.h", ".s.open.h.hyLZlg" }; - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_ZAP_OTHER, 0, 0); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_ZAP_OTHER, 0, 0, 0); if (ztest_object_init(zd, od, sizeof (od), !ztest_random(2)) != 0) return; @@ -4451,7 +4577,7 @@ ztest_fzap(ztest_ds_t *zd, uint64_t id) ztest_od_t od[1]; uint64_t object, txg; - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_ZAP_OTHER, 0, 0); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_ZAP_OTHER, 0, 0, 0); if (ztest_object_init(zd, od, sizeof (od), !ztest_random(2)) != 0) return; @@ -4497,7 +4623,8 @@ ztest_zap_parallel(ztest_ds_t *zd, uint64_t id) char name[20], string_value[20]; void *data; - ztest_od_init(&od[0], ID_PARALLEL, FTAG, micro, DMU_OT_ZAP_OTHER, 0, 0); + ztest_od_init(&od[0], ID_PARALLEL, FTAG, micro, DMU_OT_ZAP_OTHER, + 0, 0, 0); if (ztest_object_init(zd, od, sizeof (od), B_FALSE) != 0) return; @@ -4685,7 +4812,7 @@ ztest_dmu_commit_callbacks(ztest_ds_t *zd, uint64_t id uint64_t old_txg, txg; int i, error; - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, 0, 0); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, 0, 0, 0); if (ztest_object_init(zd, od, sizeof (od), B_FALSE) != 0) return; @@ -4799,6 +4926,41 @@ ztest_dmu_commit_callbacks(ztest_ds_t *zd, uint64_t id dmu_tx_commit(tx); } +/* + * Visit each object in the dataset. Verify that its properties + * are consistent what was stored in the block tag when it was created, + * and that its unused bonus buffer space has not been overwritten. + */ +void +ztest_verify_dnode_bt(ztest_ds_t *zd, uint64_t id) +{ + objset_t *os = zd->zd_os; + uint64_t obj; + int err = 0; + + for (obj = 0; err == 0; err = dmu_object_next(os, &obj, FALSE, 0)) { + ztest_block_tag_t *bt = NULL; + dmu_object_info_t doi; + dmu_buf_t *db; + + if (dmu_bonus_hold(os, obj, FTAG, &db) != 0) + continue; + + dmu_object_info_from_db(db, &doi); + if (doi.doi_bonus_size >= sizeof (*bt)) + bt = ztest_bt_bonus(db); + + if (bt && bt->bt_magic == BT_MAGIC) { + ztest_bt_verify(bt, os, obj, doi.doi_dnodesize, + bt->bt_offset, bt->bt_gen, bt->bt_txg, + bt->bt_crtxg); + ztest_verify_unused_bonus(db, bt, obj, os, bt->bt_gen); + } + + dmu_buf_rele(db, FTAG); + } +} + /* ARGSUSED */ void ztest_dsl_prop_get_set(ztest_ds_t *zd, uint64_t id) @@ -5283,7 +5445,8 @@ ztest_ddt_repair(ztest_ds_t *zd, uint64_t id) blocksize = ztest_random_blocksize(); blocksize = MIN(blocksize, 2048); /* because we write so many */ - ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, blocksize, 0); + ztest_od_init(&od[0], id, FTAG, 0, DMU_OT_UINT64_OTHER, blocksize, + 0, 0); if (ztest_object_init(zd, od, sizeof (od), B_FALSE) != 0) return; @@ -6155,7 +6318,7 @@ ztest_freeze(void) numloops++ < ztest_opts.zo_maxloops && metaslab_class_get_alloc(spa_normal_class(spa)) < capacity) { ztest_od_t od; - ztest_od_init(&od, 0, FTAG, 0, DMU_OT_UINT64_OTHER, 0, 0); + ztest_od_init(&od, 0, FTAG, 0, DMU_OT_UINT64_OTHER, 0, 0, 0); VERIFY0(ztest_object_init(zd, &od, sizeof (od), B_FALSE)); ztest_io(zd, od.od_object, ztest_random(ZTEST_RANGE_LOCKS) << SPA_MAXBLOCKSHIFT); Modified: vendor/illumos/dist/man/man5/zpool-features.5 ============================================================================== --- vendor/illumos/dist/man/man5/zpool-features.5 Mon Aug 12 11:42:16 2019 (r350897) +++ vendor/illumos/dist/man/man5/zpool-features.5 Mon Aug 12 12:05:40 2019 (r350898) @@ -533,6 +533,30 @@ set larger than 128KB, and will return to being \fBena filesystems that have ever had their recordsize larger than 128KB are destroyed. .RE +.ne 2 +.na +\fB\fBlarge_dnode\fR\fR +.ad +.RS 4n +.TS +l l . +GUID org.zfsonlinux:large_dnode +READ\-ONLY COMPATIBLE no +DEPENDENCIES extensible_dataset +.TE + +The \fBlarge_dnode\fR feature allows the size of dnodes in a dataset to be +set larger than 512B. + +This feature becomes \fBactive\fR once a dataset contains an object with a +dnode larger than 512B, which occurs as a result of setting the \fBdnodesize\fR +dataset property to a value other than \fBlegacy\fR. The feature will return to +being \fBenabled\fR once all filesystems that have ever contained a dnode larger +than 512B are destroyed. Large dnodes allow more data to be stored in the +bonus buffer, thus potentially improving performance by avoiding the use of +spill blocks. +.RE + .sp .ne 2 .na From owner-svn-src-vendor@freebsd.org Thu Aug 15 15:07:18 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05F29ABFBE; Thu, 15 Aug 2019 15:07:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 468VDP6P2wz3xND; Thu, 15 Aug 2019 15:07:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B8FC31A523; Thu, 15 Aug 2019 15:07:17 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7FF7H7M044991; Thu, 15 Aug 2019 15:07:17 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7FF7HQR044989; Thu, 15 Aug 2019 15:07:17 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201908151507.x7FF7HQR044989@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Thu, 15 Aug 2019 15:07:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r351075 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 351075 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Aug 2019 15:07:18 -0000 Author: avg Date: Thu Aug 15 15:07:17 2019 New Revision: 351075 URL: https://svnweb.freebsd.org/changeset/base/351075 Log: 10406 large_dnode changes broke zfs recv of legacy stream illumos/illumos-gate@811964cd9f1fbae0fc3b93d116269e9b1fca090a https://github.com/illumos/illumos-gate/commit/811964cd9f1fbae0fc3b93d116269e9b1fca090a https://www.illumos.org/issues/10406 The large dnode changes from 8423 caused problems in zfs recv for a legacy stream. This manifests when attempting to mount the received stream, but the problem is in the receive code. We missed the following commit from ZoL which fixes this. commit da2feb42fb5c7a8c1e1cc67f7a880da9d8e97bc2 Author: Tom Caputi Date: Thu Jun 28 17:55:11 2018 -0400 Fix 'zfs recv' of non large_dnode send streams Currently, there is a bug where older send streams without the DMU_BACKUP_FEATURE_LARGE_DNODE flag are not handled correctly. The code in receive_object() fails to handle cases where drro->drr_dn_slots is set to 0, which is always the case when the sending code does not support this feature flag. This patch fixes the issue by ensuring that that a value of 0 is treated as DNODE_MIN_SLOTS. Author: Tom Caputi Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Thu Aug 15 14:57:27 2019 (r351074) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Thu Aug 15 15:07:17 2019 (r351075) @@ -275,6 +275,9 @@ dmu_object_reclaim_dnsize(objset_t *os, uint64_t objec int dn_slots = dnodesize >> DNODE_SHIFT; int err; + if (dn_slots == 0) + dn_slots = DNODE_MIN_SLOTS; + if (object == DMU_META_DNODE_OBJECT) return (SET_ERROR(EBADF)); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Thu Aug 15 14:57:27 2019 (r351074) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Thu Aug 15 15:07:17 2019 (r351075) @@ -2094,6 +2094,8 @@ receive_object(struct receive_writer_arg *rwa, struct dmu_tx_t *tx; uint64_t object; int err; + uint8_t dn_slots = drro->drr_dn_slots != 0 ? + drro->drr_dn_slots : DNODE_MIN_SLOTS; if (drro->drr_type == DMU_OT_NONE || !DMU_OT_IS_VALID(drro->drr_type) || @@ -2105,7 +2107,7 @@ receive_object(struct receive_writer_arg *rwa, struct drro->drr_blksz > spa_maxblocksize(dmu_objset_spa(rwa->os)) || drro->drr_bonuslen > DN_BONUS_SIZE(spa_maxdnodesize(dmu_objset_spa(rwa->os))) || - drro->drr_dn_slots > + dn_slots > (spa_maxdnodesize(dmu_objset_spa(rwa->os)) >> DNODE_SHIFT)) { return (SET_ERROR(EINVAL)); } @@ -2133,7 +2135,7 @@ receive_object(struct receive_writer_arg *rwa, struct if (drro->drr_blksz != doi.doi_data_block_size || nblkptr < doi.doi_nblkptr || - drro->drr_dn_slots != doi.doi_dnodesize >> DNODE_SHIFT) { + dn_slots != doi.doi_dnodesize >> DNODE_SHIFT) { err = dmu_free_long_range(rwa->os, drro->drr_object, 0, DMU_OBJECT_END); if (err != 0) @@ -2160,11 +2162,11 @@ receive_object(struct receive_writer_arg *rwa, struct * another object from the previous snapshot. We must free * these objects before we attempt to allocate the new dnode. */ - if (drro->drr_dn_slots > 1) { + if (dn_slots > 1) { boolean_t need_sync = B_FALSE; for (uint64_t slot = drro->drr_object + 1; - slot < drro->drr_object + drro->drr_dn_slots; + slot < drro->drr_object + dn_slots; slot++) { dmu_object_info_t slot_doi; @@ -2199,7 +2201,7 @@ receive_object(struct receive_writer_arg *rwa, struct err = dmu_object_claim_dnsize(rwa->os, drro->drr_object, drro->drr_type, drro->drr_blksz, drro->drr_bonustype, drro->drr_bonuslen, - drro->drr_dn_slots << DNODE_SHIFT, tx); + dn_slots << DNODE_SHIFT, tx); } else if (drro->drr_type != doi.doi_type || drro->drr_blksz != doi.doi_data_block_size || drro->drr_bonustype != doi.doi_bonus_type || From owner-svn-src-vendor@freebsd.org Thu Aug 15 17:31:11 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E30CEAFF6E; Thu, 15 Aug 2019 17:31:11 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 468YQR5dRzz489t; Thu, 15 Aug 2019 17:31:11 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A31781BEE9; Thu, 15 Aug 2019 17:31:11 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7FHVBT9042413; Thu, 15 Aug 2019 17:31:11 GMT (envelope-from jkim@FreeBSD.org) Received: (from jkim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7FHVBbA042412; Thu, 15 Aug 2019 17:31:11 GMT (envelope-from jkim@FreeBSD.org) Message-Id: <201908151731.x7FHVBbA042412@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jkim set sender to jkim@FreeBSD.org using -f From: Jung-uk Kim Date: Thu, 15 Aug 2019 17:31:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r351091 - vendor-sys/acpica/dist/source/components/executer X-SVN-Group: vendor-sys X-SVN-Commit-Author: jkim X-SVN-Commit-Paths: vendor-sys/acpica/dist/source/components/executer X-SVN-Commit-Revision: 351091 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Aug 2019 17:31:11 -0000 Author: jkim Date: Thu Aug 15 17:31:11 2019 New Revision: 351091 URL: https://svnweb.freebsd.org/changeset/base/351091 Log: Exit interpreter before initializing objects. https://github.com/acpica/acpica/commit/73f6372 Modified: vendor-sys/acpica/dist/source/components/executer/exconfig.c Modified: vendor-sys/acpica/dist/source/components/executer/exconfig.c ============================================================================== --- vendor-sys/acpica/dist/source/components/executer/exconfig.c Thu Aug 15 17:24:10 2019 (r351090) +++ vendor-sys/acpica/dist/source/components/executer/exconfig.c Thu Aug 15 17:31:11 2019 (r351091) @@ -344,7 +344,9 @@ AcpiExLoadTableOp ( /* Complete the initialization/resolution of new objects */ - AcpiNsInitializeObjects (); + AcpiExExitInterpreter(); + AcpiNsInitializeObjects(); + AcpiExEnterInterpreter(); /* Parameter Data (optional) */ From owner-svn-src-vendor@freebsd.org Fri Aug 16 17:03:04 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 37D75A89E0; Fri, 16 Aug 2019 17:03:04 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4698lX1CGFz4SmT; Fri, 16 Aug 2019 17:03:04 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 242B644FA; Fri, 16 Aug 2019 17:03:04 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7GH34cD088548; Fri, 16 Aug 2019 17:03:04 GMT (envelope-from jkim@FreeBSD.org) Received: (from jkim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7GH3172088535; Fri, 16 Aug 2019 17:03:01 GMT (envelope-from jkim@FreeBSD.org) Message-Id: <201908161703.x7GH3172088535@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jkim set sender to jkim@FreeBSD.org using -f From: Jung-uk Kim Date: Fri, 16 Aug 2019 17:03:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r351127 - in vendor-sys/acpica/dist: . generate/unix generate/unix/iasl source/common source/compiler source/components/debugger source/components/events source/components/executer sour... X-SVN-Group: vendor-sys X-SVN-Commit-Author: jkim X-SVN-Commit-Paths: in vendor-sys/acpica/dist: . generate/unix generate/unix/iasl source/common source/compiler source/components/debugger source/components/events source/components/executer source/components/namespace s... X-SVN-Commit-Revision: 351127 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2019 17:03:04 -0000 Author: jkim Date: Fri Aug 16 17:03:01 2019 New Revision: 351127 URL: https://svnweb.freebsd.org/changeset/base/351127 Log: Import ACPICA 20190816. Added: vendor-sys/acpica/dist/source/compiler/aslcompiler.y vendor-sys/acpica/dist/source/compiler/dtcompilerparser.l vendor-sys/acpica/dist/source/compiler/dtcompilerparser.y Modified: vendor-sys/acpica/dist/changes.txt vendor-sys/acpica/dist/generate/unix/Makefile.config vendor-sys/acpica/dist/generate/unix/iasl/Makefile vendor-sys/acpica/dist/source/common/adwalk.c vendor-sys/acpica/dist/source/common/dmrestag.c vendor-sys/acpica/dist/source/common/dmtables.c vendor-sys/acpica/dist/source/common/dmtbdump2.c vendor-sys/acpica/dist/source/compiler/aslcache.c vendor-sys/acpica/dist/source/compiler/aslcodegen.c vendor-sys/acpica/dist/source/compiler/aslcompile.c vendor-sys/acpica/dist/source/compiler/aslcompiler.h vendor-sys/acpica/dist/source/compiler/asldebug.c vendor-sys/acpica/dist/source/compiler/asldefine.h vendor-sys/acpica/dist/source/compiler/aslglobal.h vendor-sys/acpica/dist/source/compiler/aslhelp.c vendor-sys/acpica/dist/source/compiler/asllength.c vendor-sys/acpica/dist/source/compiler/asllistsup.c vendor-sys/acpica/dist/source/compiler/aslload.c vendor-sys/acpica/dist/source/compiler/aslmain.c vendor-sys/acpica/dist/source/compiler/aslmessages.c vendor-sys/acpica/dist/source/compiler/aslmessages.h vendor-sys/acpica/dist/source/compiler/aslnamesp.c vendor-sys/acpica/dist/source/compiler/asloptions.c vendor-sys/acpica/dist/source/compiler/aslparseop.c vendor-sys/acpica/dist/source/compiler/aslstartup.c vendor-sys/acpica/dist/source/compiler/asltransform.c vendor-sys/acpica/dist/source/compiler/asltypes.h vendor-sys/acpica/dist/source/compiler/aslutils.c vendor-sys/acpica/dist/source/compiler/aslxrefout.c vendor-sys/acpica/dist/source/compiler/dtcompile.c vendor-sys/acpica/dist/source/compiler/dtcompiler.h vendor-sys/acpica/dist/source/compiler/dtfield.c vendor-sys/acpica/dist/source/compiler/dtio.c vendor-sys/acpica/dist/source/compiler/dtparser.l vendor-sys/acpica/dist/source/compiler/dttemplate.c vendor-sys/acpica/dist/source/compiler/prmacros.c vendor-sys/acpica/dist/source/compiler/prscan.c vendor-sys/acpica/dist/source/components/debugger/dbhistry.c vendor-sys/acpica/dist/source/components/debugger/dbinput.c vendor-sys/acpica/dist/source/components/debugger/dbmethod.c vendor-sys/acpica/dist/source/components/debugger/dbobject.c vendor-sys/acpica/dist/source/components/debugger/dbstats.c vendor-sys/acpica/dist/source/components/events/evgpe.c vendor-sys/acpica/dist/source/components/events/evxfgpe.c vendor-sys/acpica/dist/source/components/executer/exdump.c vendor-sys/acpica/dist/source/components/namespace/nsaccess.c vendor-sys/acpica/dist/source/components/namespace/nsalloc.c vendor-sys/acpica/dist/source/components/namespace/nsdump.c vendor-sys/acpica/dist/source/components/namespace/nsrepair2.c vendor-sys/acpica/dist/source/components/tables/tbdata.c vendor-sys/acpica/dist/source/components/utilities/utdebug.c vendor-sys/acpica/dist/source/components/utilities/uterror.c vendor-sys/acpica/dist/source/components/utilities/utosi.c vendor-sys/acpica/dist/source/components/utilities/utownerid.c vendor-sys/acpica/dist/source/include/acconfig.h vendor-sys/acpica/dist/source/include/aclocal.h vendor-sys/acpica/dist/source/include/acobject.h vendor-sys/acpica/dist/source/include/acpiosxf.h vendor-sys/acpica/dist/source/include/acpixf.h vendor-sys/acpica/dist/source/include/acstruct.h vendor-sys/acpica/dist/source/include/actypes.h vendor-sys/acpica/dist/source/include/acutils.h vendor-sys/acpica/dist/source/include/platform/acmsvc.h vendor-sys/acpica/dist/source/tools/acpidump/apfiles.c vendor-sys/acpica/dist/source/tools/acpiexec/aeexec.c vendor-sys/acpica/dist/source/tools/examples/examples.c vendor-sys/acpica/dist/tests/templates/Makefile Modified: vendor-sys/acpica/dist/changes.txt ============================================================================== --- vendor-sys/acpica/dist/changes.txt Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/changes.txt Fri Aug 16 17:03:01 2019 (r351127) @@ -1,4 +1,70 @@ ---------------------------------------- +16 August 2019. Summary of changes for version 20190816: + +This release is available at https://acpica.org/downloads + + +1) ACPICA kernel-resident subsystem: + +Modified the OwnerId mechanism to allow for more Owner Ids. The previous +limit was 256 Ids, now it is 4096 Ids. This prevents OWNER_ID_LIMIT +exceptions on machines with a large number of initialization threads, +many CPU cores and nested initialization control methods. + +Introduced acpi_dispatch_gpe() as a wrapper around AcpiEvDetectGpe() for +checking if the given GPE (as represented by a GPE device handle and a +GPE number) is currently active and dispatching it (if that's the case) +outside of interrupt context. + +Table load: exit the interpreter before initializing objects within the +new table This prevents re-acquiring the interpreter lock when loading +tables + +Added the "Windows 2019" string to the _OSI support (version 1903). Jung- +uk Kim + +Macros: removed pointer math on a null pointer. Causes warnings on some +compilers and/or tools. Changed ACPI_TO_POINTER to use ACPI_CAST_PTR +instead of using arithmetic. + +Fully deployed the ACPI_PRINTF_LIKE macro. This macro was not being used +across all "printf-like" internal functions. Also, cleanup all calls to +such functions (both in 32-bit mode and 64-bit mode) now that they are +analyzed by the gcc compiler via ACPI_PRINTF_LIKE. + + +2) iASL Compiler/Disassembler and ACPICA tools: + +iASL: implemented a new data table compiler flex/bison front-end. This +change is internal and is not intended to result in changes to the +compiled code. This new compiler front-end can be invoked using the -tp +option for now, until the old mechanism is removed. + +ASLTS: Implemented a new data table compiler test suite. This test suite +generates all table templates and compile/disassemble/re-compile/binary- +compare each file. + +iASL: return -1 if AML files were not generated due to compiler errors + +iASL: added a warning on use of the now-legacy ASL Processor () keyword. + +iASL: added an error on _UID object declaration that returns a String +within a Processor () declaration. A _UID for a processor must be an +Integer. + +iASL: added a null terminator to name strings that consist only of +multiple parent prefixes (^) + +iASL: added support to compile both ASL and data table files in a single +command. + +Updated the tool generation project files that were recently migrated to +MSVC 2017 to eliminate all new warnings. The new project files appear in +the directory \acpica\generate\msvc2017. This change effectively +deprecates the older project files in \acpica\generate\msvc9. + + +---------------------------------------- 03 July 2019. Summary of changes for version 20190703: Modified: vendor-sys/acpica/dist/generate/unix/Makefile.config ============================================================================== --- vendor-sys/acpica/dist/generate/unix/Makefile.config Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/generate/unix/Makefile.config Fri Aug 16 17:03:01 2019 (r351127) @@ -148,6 +148,14 @@ ACPICA_HEADERS = \ OPT_CFLAGS ?= $(CWARNINGFLAGS) # +# Debug flags +# +ifeq ($(DEBUG),TRUE) +CFLAGS +=-g +LDFLAGS +=-g +endif + +# # Common compiler flags # The _GNU_SOURCE symbol is required for many hosts. # @@ -272,3 +280,4 @@ MFLAGS= -P -I$(ASL_COMPILER) LEX= flex LFLAGS += -i -s +DLFLAGS += -i Modified: vendor-sys/acpica/dist/generate/unix/iasl/Makefile ============================================================================== --- vendor-sys/acpica/dist/generate/unix/iasl/Makefile Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/generate/unix/iasl/Makefile Fri Aug 16 17:03:01 2019 (r351127) @@ -33,12 +33,15 @@ vpath %.c \ HEADERS = \ $(wildcard $(ASL_COMPILER)/*.h)\ $(OBJDIR)/aslcompiler.y.h\ + $(OBJDIR)/dtcompilerparser.y.h\ $(OBJDIR)/dtparser.y.h\ $(OBJDIR)/prparser.y.h OBJECTS = \ $(OBJDIR)/aslcompilerlex.o\ $(OBJDIR)/aslcompilerparse.o\ + $(OBJDIR)/dtcompilerparserlex.o\ + $(OBJDIR)/dtcompilerparserparse.o\ $(OBJDIR)/dtparserlex.o\ $(OBJDIR)/dtparserparse.o\ $(OBJDIR)/prparserlex.o\ @@ -251,11 +254,14 @@ INTERMEDIATES = \ $(OBJDIR)/aslcompilerparse.c\ $(OBJDIR)/dtparserlex.c\ $(OBJDIR)/dtparserparse.c\ + $(OBJDIR)/dtcompilerparserlex.c\ + $(OBJDIR)/dtcompilerparserparse.c\ $(OBJDIR)/prparserlex.c\ $(OBJDIR)/prparserparse.c MISC = \ $(OBJDIR)/aslcompiler.y.h\ + $(OBJDIR)/dtcompilerparser.y.h\ $(OBJDIR)/dtparser.y.h\ $(OBJDIR)/prparser.y.h @@ -322,9 +328,17 @@ $(OBJDIR)/aslcompilerparse.c $(OBJDIR)/aslcompiler.y.h @echo "- " $(YACC) " "$< @$(call safe_yacc,AslCompiler,$<,$@) +$(OBJDIR)/dtcompilerparserlex.c : $(ASL_COMPILER)/dtcompilerparser.l $(OBJDIR)/dtcompilerparser.y.h + @echo "- " $(LEX) " "$< + @$(LEX) $(DLFLAGS) -PDtCompilerParser -o$@ $< + +$(OBJDIR)/dtcompilerparserparse.c $(OBJDIR)/dtcompilerparser.y.h : $(ASL_COMPILER)/dtcompilerparser.y + @echo "- " $(YACC) " "$< + @$(call safe_yacc,DtCompilerParser,$<,$@) + $(OBJDIR)/dtparserlex.c : $(ASL_COMPILER)/dtparser.l $(OBJDIR)/dtparser.y.h @echo "- " $(LEX) " "$< - @$(LEX) $(LFLAGS) -PDtParser -o$@ $< + @$(LEX) $(DLFLAGS) -PDtParser -o$@ $< $(OBJDIR)/dtparserparse.c $(OBJDIR)/dtparser.y.h : $(ASL_COMPILER)/dtparser.y @echo "- " $(YACC) " "$< @@ -349,6 +363,14 @@ $(OBJDIR)/aslcompilerlex.o : $(OBJDIR)/aslcompilerle @$(CC) -c $(CFLAGS) -Wall -Werror -o$@ $< $(OBJDIR)/aslcompilerparse.o : $(OBJDIR)/aslcompilerparse.c + @echo "- " "Intermediate" $< + @$(CC) -c $(CFLAGS) -Wall -Werror -o$@ $< + +$(OBJDIR)/dtcompilerparserlex.o : $(OBJDIR)/dtcompilerparserlex.c + @echo "- " "Intermediate" $< + @$(CC) -c $(CFLAGS) -Wall -Werror -o$@ $< + +$(OBJDIR)/dtcompilerparserparse.o : $(OBJDIR)/dtcompilerparserparse.c @echo "- " "Intermediate" $< @$(CC) -c $(CFLAGS) -Wall -Werror -o$@ $< Modified: vendor-sys/acpica/dist/source/common/adwalk.c ============================================================================== --- vendor-sys/acpica/dist/source/common/adwalk.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/common/adwalk.c Fri Aug 16 17:03:01 2019 (r351127) @@ -480,6 +480,7 @@ AcpiDmDumpDescending ( { ACPI_OP_WALK_INFO *Info = Context; char *Path; + ACPI_STATUS Status; if (!Op) @@ -522,10 +523,18 @@ AcpiDmDumpDescending ( if (Op->Common.Value.String) { - AcpiNsExternalizeName (ACPI_UINT32_MAX, Op->Common.Value.String, + Status = AcpiNsExternalizeName (ACPI_UINT32_MAX, Op->Common.Value.String, NULL, &Path); - AcpiOsPrintf ("%s %p", Path, Op->Common.Node); - ACPI_FREE (Path); + if (ACPI_SUCCESS (Status)) + { + AcpiOsPrintf ("%s %p", Path, Op->Common.Node); + ACPI_FREE (Path); + } + else + { + AcpiOsPrintf ("Could not externalize pathname for node [%4.4s]", + Op->Common.Node->Name.Ascii); + } } else { @@ -830,9 +839,9 @@ AcpiDmLoadDescendingOp ( * 2) Not the root node * 3) Not a node created by Scope */ - - if (!PreDefined && Node != AcpiGbl_RootNode && - Op->Common.AmlOpcode != AML_SCOPE_OP) + if (!PreDefined && + (Node != AcpiGbl_RootNode) && + (Op->Common.AmlOpcode != AML_SCOPE_OP)) { Node->OwnerId = WalkState->OwnerId; } Modified: vendor-sys/acpica/dist/source/common/dmrestag.c ============================================================================== --- vendor-sys/acpica/dist/source/common/dmrestag.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/common/dmrestag.c Fri Aug 16 17:03:01 2019 (r351127) @@ -833,8 +833,12 @@ AcpiGetTagPathname ( /* Internalize the namepath to AML format */ - AcpiNsInternalizeName (Pathname, &InternalPath); + Status = AcpiNsInternalizeName (Pathname, &InternalPath); ACPI_FREE (Pathname); + if (ACPI_FAILURE (Status)) + { + return (NULL); + } /* Update the Op with the symbol */ Modified: vendor-sys/acpica/dist/source/common/dmtables.c ============================================================================== --- vendor-sys/acpica/dist/source/common/dmtables.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/common/dmtables.c Fri Aug 16 17:03:01 2019 (r351127) @@ -443,8 +443,8 @@ AdGetLocalTables ( /* Get the DSDT via table override */ ACPI_MOVE_32_TO_32 (TableHeader.Signature, ACPI_SIG_DSDT); - AcpiOsTableOverride (&TableHeader, &NewTable); - if (!NewTable) + Status = AcpiOsTableOverride (&TableHeader, &NewTable); + if (ACPI_FAILURE (Status) || !NewTable) { fprintf (stderr, "Could not obtain DSDT\n"); return (AE_NO_ACPI_TABLES); Modified: vendor-sys/acpica/dist/source/common/dmtbdump2.c ============================================================================== --- vendor-sys/acpica/dist/source/common/dmtbdump2.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/common/dmtbdump2.c Fri Aug 16 17:03:01 2019 (r351127) @@ -871,7 +871,7 @@ AcpiDmDumpMcfg ( if (Offset + sizeof (ACPI_MCFG_ALLOCATION) > Table->Length) { AcpiOsPrintf ("Warning: there are %u invalid trailing bytes\n", - sizeof (ACPI_MCFG_ALLOCATION) - (Offset - Table->Length)); + (UINT32) sizeof (ACPI_MCFG_ALLOCATION) - (Offset - Table->Length)); return; } Modified: vendor-sys/acpica/dist/source/compiler/aslcache.c ============================================================================== --- vendor-sys/acpica/dist/source/compiler/aslcache.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/compiler/aslcache.c Fri Aug 16 17:03:01 2019 (r351127) @@ -421,7 +421,7 @@ UtDeleteLocalCaches ( DbgPrint (ASL_DEBUG_OUTPUT, "%u ParseOps, Buffer size: %u ops (%u bytes), %u Buffers\n", AslGbl_ParseOpCount, ASL_PARSEOP_CACHE_SIZE, - (sizeof (ACPI_PARSE_OBJECT) * ASL_PARSEOP_CACHE_SIZE), BufferCount); + ((UINT32) sizeof (ACPI_PARSE_OBJECT) * ASL_PARSEOP_CACHE_SIZE), BufferCount); /* Reset cache globals */ @@ -445,7 +445,7 @@ UtDeleteLocalCaches ( DbgPrint (ASL_DEBUG_OUTPUT, "%u Fields, Buffer size: %u fields (%u bytes), %u Buffers\n", AslGbl_FieldCount, ASL_FIELD_CACHE_SIZE, - (sizeof (DT_FIELD) * ASL_FIELD_CACHE_SIZE), BufferCount); + ((UINT32) sizeof (DT_FIELD) * ASL_FIELD_CACHE_SIZE), BufferCount); /* Reset cache globals */ @@ -468,7 +468,7 @@ UtDeleteLocalCaches ( DbgPrint (ASL_DEBUG_OUTPUT, "%u Subtables, Buffer size: %u subtables (%u bytes), %u Buffers\n", AslGbl_SubtableCount, ASL_SUBTABLE_CACHE_SIZE, - (sizeof (DT_SUBTABLE) * ASL_SUBTABLE_CACHE_SIZE), BufferCount); + ((UINT32) sizeof (DT_SUBTABLE) * ASL_SUBTABLE_CACHE_SIZE), BufferCount); /* Reset cache globals */ Modified: vendor-sys/acpica/dist/source/compiler/aslcodegen.c ============================================================================== --- vendor-sys/acpica/dist/source/compiler/aslcodegen.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/compiler/aslcodegen.c Fri Aug 16 17:03:01 2019 (r351127) @@ -618,8 +618,8 @@ CgWriteTableHeader ( Op->Asl.AmlSubtreeLength += strlen (AslGbl_ParseTreeRoot->Asl.Filename) + 3; - CvDbgPrint (" Length: %lu\n", - strlen (AslGbl_ParseTreeRoot->Asl.Filename) + 3); + CvDbgPrint (" Length: %u\n", + (UINT32) strlen (AslGbl_ParseTreeRoot->Asl.Filename) + 3); if (Op->Asl.CommentList) { Modified: vendor-sys/acpica/dist/source/compiler/aslcompile.c ============================================================================== --- vendor-sys/acpica/dist/source/compiler/aslcompile.c Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/compiler/aslcompile.c Fri Aug 16 17:03:01 2019 (r351127) @@ -853,10 +853,11 @@ CmDumpAllEvents ( * ******************************************************************************/ -void +int CmCleanupAndExit ( void) { + int Status = 0; BOOLEAN DeleteAmlFile = FALSE; ASL_GLOBAL_FILE_NODE *CurrentFileNode = AslGbl_FilesList; @@ -915,20 +916,38 @@ CmCleanupAndExit ( UtDisplaySummary (ASL_FILE_STDOUT); /* - * We will delete the AML file if there are errors and the - * force AML output option has not been used. + * Delete the AML file if there are errors and the force AML output option + * (-f) has not been used. + * + * Return -1 as a status of the compiler if no AML files are generated. If + * the AML file is generated in the presence of errors, return 0. In the + * latter case, the errors were ignored by the user so the compilation is + * considered successful. */ - if (AslGbl_ParserErrorDetected || AslGbl_PreprocessOnly || ((AslGbl_ExceptionCount[ASL_ERROR] > 0) && + if (AslGbl_ParserErrorDetected || AslGbl_PreprocessOnly || + ((AslGbl_ExceptionCount[ASL_ERROR] > 0) && (!AslGbl_IgnoreErrors) && AslGbl_Files[ASL_FILE_AML_OUTPUT].Handle)) { DeleteAmlFile = TRUE; + Status = -1; } /* Close all open files */ while (CurrentFileNode) { + /* + * Set the program return status based on file errors. If there are any + * errors and during compilation, the command is not considered + * successful. + */ + if (Status != -1 && !AslGbl_IgnoreErrors && + CurrentFileNode->ParserErrorDetected) + { + Status = -1; + } + switch (FlSwitchFileSet (CurrentFileNode->Files[ASL_FILE_INPUT].Filename)) { case SWITCH_TO_SAME_FILE: @@ -952,6 +971,8 @@ CmCleanupAndExit ( { UtDeleteLocalCaches (); } + + return (Status); } Modified: vendor-sys/acpica/dist/source/compiler/aslcompiler.h ============================================================================== --- vendor-sys/acpica/dist/source/compiler/aslcompiler.h Fri Aug 16 15:31:46 2019 (r351126) +++ vendor-sys/acpica/dist/source/compiler/aslcompiler.h Fri Aug 16 17:03:01 2019 (r351127) @@ -266,7 +266,7 @@ void CmDoOutputFiles ( void); -void +int CmCleanupAndExit ( void); @@ -1052,6 +1052,7 @@ void FlCloseFile ( UINT32 FileId); +ACPI_PRINTF_LIKE (2) void FlPrintFile ( UINT32 FileId, @@ -1195,6 +1196,7 @@ OtXrefWalkPart1 ( /* * aslutils - common compiler utilities */ +ACPI_PRINTF_LIKE(2) void DbgPrint ( UINT32 Type, @@ -1288,6 +1290,10 @@ UtInternalizeName ( char *ExternalName, char **ConvertedName); +BOOLEAN +UtNameContainsAllPrefix ( + ACPI_PARSE_OBJECT *Op); + void UtAttachNamepathToOwner ( ACPI_PARSE_OBJECT *Op, @@ -1303,7 +1309,16 @@ UINT64 UtDoConstant ( char *String); +char * +AcpiUtStrdup ( + char *String); +char * +AcpiUtStrcat ( + char *String1, + char *String2); + + /* * asluuid - UUID support */ @@ -1604,6 +1619,7 @@ DtCreateTemplates ( /* * ASL/ASL+ converter debug */ +ACPI_PRINTF_LIKE (1) void CvDbgPrint ( char *Fmt, Added: vendor-sys/acpica/dist/source/compiler/aslcompiler.y ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor-sys/acpica/dist/source/compiler/aslcompiler.y Fri Aug 16 17:03:01 2019 (r351127) @@ -0,0 +1,4921 @@ +%{ +/****************************************************************************** + * + * Module Name: aslparser.y - Master Bison/Yacc input file for iASL + * + *****************************************************************************/ + +/****************************************************************************** + * + * 1. Copyright Notice + * + * Some or all of this work - Copyright (c) 1999 - 2019, Intel Corp. + * All rights reserved. + * + * 2. License + * + * 2.1. This is your license from Intel Corp. under its intellectual property + * rights. You may have additional license terms from the party that provided + * you this software, covering your right to use that party's intellectual + * property rights. + * + * 2.2. Intel grants, free of charge, to any person ("Licensee") obtaining a + * copy of the source code appearing in this file ("Covered Code") an + * irrevocable, perpetual, worldwide license under Intel's copyrights in the + * base code distributed originally by Intel ("Original Intel Code") to copy, + * make derivatives, distribute, use and display any portion of the Covered + * Code in any form, with the right to sublicense such rights; and + * + * 2.3. Intel grants Licensee a non-exclusive and non-transferable patent + * license (with the right to sublicense), under only those claims of Intel + * patents that are infringed by the Original Intel Code, to make, use, sell, + * offer to sell, and import the Covered Code and derivative works thereof + * solely to the minimum extent necessary to exercise the above copyright + * license, and in no event shall the patent license extend to any additions + * to or modifications of the Original Intel Code. No other license or right + * is granted directly or by implication, estoppel or otherwise; + * + * The above copyright and patent license is granted only if the following + * conditions are met: + * + * 3. Conditions + * + * 3.1. Redistribution of Source with Rights to Further Distribute Source. + * Redistribution of source code of any substantial portion of the Covered + * Code or modification with rights to further distribute source must include + * the above Copyright Notice, the above License, this list of Conditions, + * and the following Disclaimer and Export Compliance provision. In addition, + * Licensee must cause all Covered Code to which Licensee contributes to + * contain a file documenting the changes Licensee made to create that Covered + * Code and the date of any change. Licensee must include in that file the + * documentation of any changes made by any predecessor Licensee. Licensee + * must include a prominent statement that the modification is derived, + * directly or indirectly, from Original Intel Code. + * + * 3.2. Redistribution of Source with no Rights to Further Distribute Source. + * Redistribution of source code of any substantial portion of the Covered + * Code or modification without rights to further distribute source must + * include the following Disclaimer and Export Compliance provision in the + * documentation and/or other materials provided with distribution. In + * addition, Licensee may not authorize further sublicense of source of any + * portion of the Covered Code, and must include terms to the effect that the + * license from Licensee to its licensee is limited to the intellectual + * property embodied in the software Licensee provides to its licensee, and + * not to intellectual property embodied in modifications its licensee may + * make. + * + * 3.3. Redistribution of Executable. Redistribution in executable form of any + * substantial portion of the Covered Code or modification must reproduce the + * above Copyright Notice, and the following Disclaimer and Export Compliance + * provision in the documentation and/or other materials provided with the + * distribution. + * + * 3.4. Intel retains all right, title, and interest in and to the Original + * Intel Code. + * + * 3.5. Neither the name Intel nor any other trademark owned or controlled by + * Intel shall be used in advertising or otherwise to promote the sale, use or + * other dealings in products derived from or relating to the Covered Code + * without prior written authorization from Intel. + * + * 4. Disclaimer and Export Compliance + * + * 4.1. INTEL MAKES NO WARRANTY OF ANY KIND REGARDING ANY SOFTWARE PROVIDED + * HERE. ANY SOFTWARE ORIGINATING FROM INTEL OR DERIVED FROM INTEL SOFTWARE + * IS PROVIDED "AS IS," AND INTEL WILL NOT PROVIDE ANY SUPPORT, ASSISTANCE, + * INSTALLATION, TRAINING OR OTHER SERVICES. INTEL WILL NOT PROVIDE ANY + * UPDATES, ENHANCEMENTS OR EXTENSIONS. INTEL SPECIFICALLY DISCLAIMS ANY + * IMPLIED WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT AND FITNESS FOR A + * PARTICULAR PURPOSE. + * + * 4.2. IN NO EVENT SHALL INTEL HAVE ANY LIABILITY TO LICENSEE, ITS LICENSEES + * OR ANY OTHER THIRD PARTY, FOR ANY LOST PROFITS, LOST DATA, LOSS OF USE OR + * COSTS OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR FOR ANY INDIRECT, + * SPECIAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THIS AGREEMENT, UNDER ANY + * CAUSE OF ACTION OR THEORY OF LIABILITY, AND IRRESPECTIVE OF WHETHER INTEL + * HAS ADVANCE NOTICE OF THE POSSIBILITY OF SUCH DAMAGES. THESE LIMITATIONS + * SHALL APPLY NOTWITHSTANDING THE FAILURE OF THE ESSENTIAL PURPOSE OF ANY + * LIMITED REMEDY. + * + * 4.3. Licensee shall not export, either directly or indirectly, any of this + * software or system incorporating such software without first obtaining any + * required license or other approval from the U. S. Department of Commerce or + * any other agency or department of the United States Government. In the + * event Licensee exports any such software from the United States or + * re-exports any such software from a foreign destination, Licensee shall + * ensure that the distribution and export/re-export of the software is in + * compliance with all laws, regulations, orders, or other restrictions of the + * U.S. Export Administration Regulations. Licensee agrees that neither it nor + * any of its subsidiaries will export/re-export any technical data, process, + * software, or service, directly or indirectly, to any country for which the + * United States government or any agency thereof requires an export license, + * other governmental approval, or letter of assurance, without first obtaining + * such license, approval or letter. + * + ***************************************************************************** + * + * Alternatively, you may choose to be licensed under the terms of the + * following license: + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions, and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * substantially similar to the "NO WARRANTY" disclaimer below + * ("Disclaimer") and any redistribution must be conditioned upon + * including a substantially similar Disclaimer requirement for further + * binary redistribution. + * 3. Neither the names of the above-listed copyright holders nor the names + * of any contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + * Alternatively, you may choose to be licensed under the terms of the + * GNU General Public License ("GPL") version 2 as published by the Free + * Software Foundation. + * + *****************************************************************************/ + +#include "aslcompiler.h" +#include "acpi.h" +#include "accommon.h" + +#define _COMPONENT ACPI_COMPILER + ACPI_MODULE_NAME ("aslparse") + +/* + * Global Notes: + * + * October 2005: The following list terms have been optimized (from the + * original ASL grammar in the ACPI specification) to force the immediate + * reduction of each list item so that the parse stack use doesn't increase on + * each list element and possibly overflow on very large lists (>4000 items). + * This dramatically reduces use of the parse stack overall. + * + * ArgList, TermList, ByteList, DWordList, PackageList, + * ResourceMacroList, and FieldUnitList + */ + +void * +AslLocalAllocate ( + unsigned int Size); + + +/* Bison/yacc configuration */ + +#define static +#undef malloc +#define malloc AslLocalAllocate +#undef alloca +#define alloca AslLocalAllocate +#define yytname AslCompilername + +#define YYINITDEPTH 600 /* State stack depth */ +#define YYDEBUG 1 /* Enable debug output */ +#define YYERROR_VERBOSE 1 /* Verbose error messages */ +#define YYFLAG -32768 + +/* Define YYMALLOC/YYFREE to prevent redefinition errors */ + +#define YYMALLOC AslLocalAllocate +#define YYFREE ACPI_FREE +%} + +/* + * Declare the type of values in the grammar + */ +%union { + UINT64 i; + char *s; + ACPI_PARSE_OBJECT *n; +} + +/* + * These shift/reduce conflicts are expected. There should be zero + * reduce/reduce conflicts. + */ +%expect 124 + +/*! [Begin] no source code translation */ + +/* + * The M4 macro processor is used to bring in the parser items, + * in order to keep this master file smaller, and to break up + * the various parser items. + */ + + +/* Token types */ + + + +/****************************************************************************** + * + * Token types: These are returned by the lexer + * + * NOTE: This list MUST match the AslKeywordMapping table found + * in aslmap.c EXACTLY! Double check any changes! + * + *****************************************************************************/ + +/* + * Most tokens are defined to return , which is a UINT64. + * + * These tokens return , a pointer to the associated lexed string: + * + * PARSEOP_NAMESEG + * PARSEOP_NAMESTRING + * PARSEOP_STRING_LITERAL + * PARSEOP_STRUCTURE_NAMESTRING + */ +%token PARSEOP_ACCESSAS +%token PARSEOP_ACCESSATTRIB_BLOCK +%token PARSEOP_ACCESSATTRIB_BLOCK_CALL +%token PARSEOP_ACCESSATTRIB_BYTE +%token PARSEOP_ACCESSATTRIB_BYTES +%token PARSEOP_ACCESSATTRIB_QUICK +%token PARSEOP_ACCESSATTRIB_RAW_BYTES +%token PARSEOP_ACCESSATTRIB_RAW_PROCESS +%token PARSEOP_ACCESSATTRIB_SND_RCV +%token PARSEOP_ACCESSATTRIB_WORD +%token PARSEOP_ACCESSATTRIB_WORD_CALL +%token PARSEOP_ACCESSTYPE_ANY +%token PARSEOP_ACCESSTYPE_BUF +%token PARSEOP_ACCESSTYPE_BYTE +%token PARSEOP_ACCESSTYPE_DWORD +%token PARSEOP_ACCESSTYPE_QWORD +%token PARSEOP_ACCESSTYPE_WORD +%token PARSEOP_ACQUIRE +%token PARSEOP_ADD +%token PARSEOP_ADDRESSINGMODE_7BIT +%token PARSEOP_ADDRESSINGMODE_10BIT +%token PARSEOP_ADDRESSTYPE_ACPI +%token PARSEOP_ADDRESSTYPE_MEMORY +%token PARSEOP_ADDRESSTYPE_NVS +%token PARSEOP_ADDRESSTYPE_RESERVED +%token PARSEOP_ALIAS +%token PARSEOP_AND +%token PARSEOP_ARG0 +%token PARSEOP_ARG1 +%token PARSEOP_ARG2 +%token PARSEOP_ARG3 +%token PARSEOP_ARG4 +%token PARSEOP_ARG5 +%token PARSEOP_ARG6 +%token PARSEOP_BANKFIELD +%token PARSEOP_BITSPERBYTE_EIGHT +%token PARSEOP_BITSPERBYTE_FIVE +%token PARSEOP_BITSPERBYTE_NINE +%token PARSEOP_BITSPERBYTE_SEVEN +%token PARSEOP_BITSPERBYTE_SIX +%token PARSEOP_BREAK +%token PARSEOP_BREAKPOINT +%token PARSEOP_BUFFER +%token PARSEOP_BUSMASTERTYPE_MASTER +%token PARSEOP_BUSMASTERTYPE_NOTMASTER +%token PARSEOP_BYTECONST +%token PARSEOP_CASE +%token PARSEOP_CLOCKPHASE_FIRST +%token PARSEOP_CLOCKPHASE_SECOND +%token PARSEOP_CLOCKPOLARITY_HIGH +%token PARSEOP_CLOCKPOLARITY_LOW +%token PARSEOP_CONCATENATE +%token PARSEOP_CONCATENATERESTEMPLATE +%token PARSEOP_CONDREFOF +%token PARSEOP_CONNECTION +%token PARSEOP_CONTINUE +%token PARSEOP_COPYOBJECT +%token PARSEOP_CREATEBITFIELD +%token PARSEOP_CREATEBYTEFIELD +%token PARSEOP_CREATEDWORDFIELD +%token PARSEOP_CREATEFIELD +%token PARSEOP_CREATEQWORDFIELD +%token PARSEOP_CREATEWORDFIELD +%token PARSEOP_DATABUFFER +%token PARSEOP_DATATABLEREGION +%token PARSEOP_DEBUG +%token PARSEOP_DECODETYPE_POS +%token PARSEOP_DECODETYPE_SUB +%token PARSEOP_DECREMENT +%token PARSEOP_DEFAULT +%token PARSEOP_DEFAULT_ARG +%token PARSEOP_DEFINITION_BLOCK +%token PARSEOP_DEREFOF +%token PARSEOP_DEVICE +%token PARSEOP_DEVICEPOLARITY_HIGH +%token PARSEOP_DEVICEPOLARITY_LOW +%token PARSEOP_DIVIDE +%token PARSEOP_DMA +%token PARSEOP_DMATYPE_A +%token PARSEOP_DMATYPE_COMPATIBILITY +%token PARSEOP_DMATYPE_B +%token PARSEOP_DMATYPE_F +%token PARSEOP_DWORDCONST +%token PARSEOP_DWORDIO +%token PARSEOP_DWORDMEMORY +%token PARSEOP_DWORDSPACE +%token PARSEOP_EISAID +%token PARSEOP_ELSE +%token PARSEOP_ELSEIF +%token PARSEOP_ENDDEPENDENTFN +%token PARSEOP_ENDIAN_BIG +%token PARSEOP_ENDIAN_LITTLE +%token PARSEOP_ENDTAG +%token PARSEOP_ERRORNODE +%token PARSEOP_EVENT +%token PARSEOP_EXTENDEDIO +%token PARSEOP_EXTENDEDMEMORY +%token PARSEOP_EXTENDEDSPACE +%token PARSEOP_EXTERNAL +%token PARSEOP_FATAL +%token PARSEOP_FIELD +%token PARSEOP_FINDSETLEFTBIT +%token PARSEOP_FINDSETRIGHTBIT +%token PARSEOP_FIXEDDMA +%token PARSEOP_FIXEDIO +%token PARSEOP_FLOWCONTROL_HW +%token PARSEOP_FLOWCONTROL_NONE +%token PARSEOP_FLOWCONTROL_SW +%token PARSEOP_FROMBCD +%token PARSEOP_FUNCTION +%token PARSEOP_GPIO_INT +%token PARSEOP_GPIO_IO +%token PARSEOP_I2C_SERIALBUS +%token PARSEOP_I2C_SERIALBUS_V2 +%token PARSEOP_IF +%token PARSEOP_INCLUDE +%token PARSEOP_INCLUDE_END +%token PARSEOP_INCREMENT +%token PARSEOP_INDEX +%token PARSEOP_INDEXFIELD +%token PARSEOP_INTEGER +%token PARSEOP_INTERRUPT +%token PARSEOP_INTLEVEL_ACTIVEBOTH +%token PARSEOP_INTLEVEL_ACTIVEHIGH +%token PARSEOP_INTLEVEL_ACTIVELOW +%token PARSEOP_INTTYPE_EDGE +%token PARSEOP_INTTYPE_LEVEL +%token PARSEOP_IO +%token PARSEOP_IODECODETYPE_10 +%token PARSEOP_IODECODETYPE_16 +%token PARSEOP_IORESTRICT_IN +%token PARSEOP_IORESTRICT_NONE +%token PARSEOP_IORESTRICT_OUT +%token PARSEOP_IORESTRICT_PRESERVE +%token PARSEOP_IRQ +%token PARSEOP_IRQNOFLAGS +%token PARSEOP_LAND +%token PARSEOP_LEQUAL +%token PARSEOP_LGREATER +%token PARSEOP_LGREATEREQUAL +%token PARSEOP_LLESS +%token PARSEOP_LLESSEQUAL +%token PARSEOP_LNOT +%token PARSEOP_LNOTEQUAL +%token PARSEOP_LOAD +%token PARSEOP_LOADTABLE +%token PARSEOP_LOCAL0 +%token PARSEOP_LOCAL1 +%token PARSEOP_LOCAL2 +%token PARSEOP_LOCAL3 +%token PARSEOP_LOCAL4 +%token PARSEOP_LOCAL5 +%token PARSEOP_LOCAL6 +%token PARSEOP_LOCAL7 +%token PARSEOP_LOCKRULE_LOCK +%token PARSEOP_LOCKRULE_NOLOCK +%token PARSEOP_LOR +%token PARSEOP_MATCH +%token PARSEOP_MATCHTYPE_MEQ +%token PARSEOP_MATCHTYPE_MGE +%token PARSEOP_MATCHTYPE_MGT +%token PARSEOP_MATCHTYPE_MLE +%token PARSEOP_MATCHTYPE_MLT +%token PARSEOP_MATCHTYPE_MTR +%token PARSEOP_MAXTYPE_FIXED +%token PARSEOP_MAXTYPE_NOTFIXED +%token PARSEOP_MEMORY24 +%token PARSEOP_MEMORY32 +%token PARSEOP_MEMORY32FIXED +%token PARSEOP_MEMTYPE_CACHEABLE +%token PARSEOP_MEMTYPE_NONCACHEABLE +%token PARSEOP_MEMTYPE_PREFETCHABLE +%token PARSEOP_MEMTYPE_WRITECOMBINING +%token PARSEOP_METHOD +%token PARSEOP_METHODCALL +%token PARSEOP_MID +%token PARSEOP_MINTYPE_FIXED +%token PARSEOP_MINTYPE_NOTFIXED +%token PARSEOP_MOD +%token PARSEOP_MULTIPLY +%token PARSEOP_MUTEX +%token PARSEOP_NAME +%token PARSEOP_NAMESEG +%token PARSEOP_NAMESTRING +%token PARSEOP_NAND +%token PARSEOP_NOOP +%token PARSEOP_NOR +%token PARSEOP_NOT +%token PARSEOP_NOTIFY +%token PARSEOP_OBJECTTYPE +%token PARSEOP_OBJECTTYPE_BFF +%token PARSEOP_OBJECTTYPE_BUF +%token PARSEOP_OBJECTTYPE_DDB +%token PARSEOP_OBJECTTYPE_DEV +%token PARSEOP_OBJECTTYPE_EVT +%token PARSEOP_OBJECTTYPE_FLD +%token PARSEOP_OBJECTTYPE_INT +%token PARSEOP_OBJECTTYPE_MTH +%token PARSEOP_OBJECTTYPE_MTX +%token PARSEOP_OBJECTTYPE_OPR +%token PARSEOP_OBJECTTYPE_PKG +%token PARSEOP_OBJECTTYPE_POW +%token PARSEOP_OBJECTTYPE_PRO +%token PARSEOP_OBJECTTYPE_STR +%token PARSEOP_OBJECTTYPE_THZ +%token PARSEOP_OBJECTTYPE_UNK +%token PARSEOP_OFFSET +%token PARSEOP_ONE +%token PARSEOP_ONES +%token PARSEOP_OPERATIONREGION +%token PARSEOP_OR +%token PARSEOP_PACKAGE +%token PARSEOP_PACKAGE_LENGTH +%token PARSEOP_PARITYTYPE_EVEN +%token PARSEOP_PARITYTYPE_MARK +%token PARSEOP_PARITYTYPE_NONE +%token PARSEOP_PARITYTYPE_ODD +%token PARSEOP_PARITYTYPE_SPACE +%token PARSEOP_PINCONFIG +%token PARSEOP_PINFUNCTION +%token PARSEOP_PINGROUP +%token PARSEOP_PINGROUPCONFIG +%token PARSEOP_PINGROUPFUNCTION +%token PARSEOP_PIN_NOPULL +%token PARSEOP_PIN_PULLDEFAULT +%token PARSEOP_PIN_PULLDOWN +%token PARSEOP_PIN_PULLUP +%token PARSEOP_POWERRESOURCE +%token PARSEOP_PROCESSOR +%token PARSEOP_QWORDCONST +%token PARSEOP_QWORDIO +%token PARSEOP_QWORDMEMORY +%token PARSEOP_QWORDSPACE +%token PARSEOP_RANGETYPE_ENTIRE +%token PARSEOP_RANGETYPE_ISAONLY +%token PARSEOP_RANGETYPE_NONISAONLY +%token PARSEOP_RAW_DATA +%token PARSEOP_READWRITETYPE_BOTH +%token PARSEOP_READWRITETYPE_READONLY +%token PARSEOP_REFOF +%token PARSEOP_REGIONSPACE_CMOS +%token PARSEOP_REGIONSPACE_EC +%token PARSEOP_REGIONSPACE_FFIXEDHW +%token PARSEOP_REGIONSPACE_GPIO +%token PARSEOP_REGIONSPACE_GSBUS +%token PARSEOP_REGIONSPACE_IO +%token PARSEOP_REGIONSPACE_IPMI +%token PARSEOP_REGIONSPACE_MEM +%token PARSEOP_REGIONSPACE_PCC +%token PARSEOP_REGIONSPACE_PCI +%token PARSEOP_REGIONSPACE_PCIBAR +%token PARSEOP_REGIONSPACE_SMBUS +%token PARSEOP_REGISTER +%token PARSEOP_RELEASE +%token PARSEOP_RESERVED_BYTES +%token PARSEOP_RESET +%token PARSEOP_RESOURCETEMPLATE +%token PARSEOP_RESOURCETYPE_CONSUMER +%token PARSEOP_RESOURCETYPE_PRODUCER +%token PARSEOP_RETURN +%token PARSEOP_REVISION +%token PARSEOP_SCOPE +%token PARSEOP_SERIALIZERULE_NOTSERIAL +%token PARSEOP_SERIALIZERULE_SERIAL +%token PARSEOP_SHARETYPE_EXCLUSIVE +%token PARSEOP_SHARETYPE_EXCLUSIVEWAKE +%token PARSEOP_SHARETYPE_SHARED +%token PARSEOP_SHARETYPE_SHAREDWAKE +%token PARSEOP_SHIFTLEFT +%token PARSEOP_SHIFTRIGHT +%token PARSEOP_SIGNAL +%token PARSEOP_SIZEOF +%token PARSEOP_SLAVEMODE_CONTROLLERINIT +%token PARSEOP_SLAVEMODE_DEVICEINIT +%token PARSEOP_SLEEP +%token PARSEOP_SPI_SERIALBUS +%token PARSEOP_SPI_SERIALBUS_V2 +%token PARSEOP_STALL +%token PARSEOP_STARTDEPENDENTFN +%token PARSEOP_STARTDEPENDENTFN_NOPRI +%token PARSEOP_STOPBITS_ONE +%token PARSEOP_STOPBITS_ONEPLUSHALF +%token PARSEOP_STOPBITS_TWO +%token PARSEOP_STOPBITS_ZERO +%token PARSEOP_STORE +%token PARSEOP_STRING_LITERAL +%token PARSEOP_SUBTRACT +%token PARSEOP_SWITCH +%token PARSEOP_THERMALZONE +%token PARSEOP_TIMER +%token PARSEOP_TOBCD +%token PARSEOP_TOBUFFER +%token PARSEOP_TODECIMALSTRING +%token PARSEOP_TOHEXSTRING +%token PARSEOP_TOINTEGER +%token PARSEOP_TOSTRING +%token PARSEOP_TOUUID +%token PARSEOP_TRANSLATIONTYPE_DENSE +%token PARSEOP_TRANSLATIONTYPE_SPARSE +%token PARSEOP_TYPE_STATIC +%token PARSEOP_TYPE_TRANSLATION +%token PARSEOP_UART_SERIALBUS +%token PARSEOP_UART_SERIALBUS_V2 +%token PARSEOP_UNICODE +%token PARSEOP_UNLOAD +%token PARSEOP_UPDATERULE_ONES +%token PARSEOP_UPDATERULE_PRESERVE +%token PARSEOP_UPDATERULE_ZEROS +%token PARSEOP_VAR_PACKAGE +%token PARSEOP_VENDORLONG +%token PARSEOP_VENDORSHORT +%token PARSEOP_WAIT +%token PARSEOP_WHILE +%token PARSEOP_WIREMODE_FOUR +%token PARSEOP_WIREMODE_THREE +%token PARSEOP_WORDBUSNUMBER +%token PARSEOP_WORDCONST +%token PARSEOP_WORDIO +%token PARSEOP_WORDSPACE +%token PARSEOP_XFERSIZE_8 +%token PARSEOP_XFERSIZE_16 +%token PARSEOP_XFERSIZE_32 +%token PARSEOP_XFERSIZE_64 +%token PARSEOP_XFERSIZE_128 +%token PARSEOP_XFERSIZE_256 +%token PARSEOP_XFERTYPE_8 +%token PARSEOP_XFERTYPE_8_16 +%token PARSEOP_XFERTYPE_16 +%token PARSEOP_XOR +%token PARSEOP_ZERO + +/* ToPld macro */ + +%token PARSEOP_TOPLD +%token PARSEOP_PLD_REVISION *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Fri Aug 16 17:03:37 2019 Return-Path: Delivered-To: svn-src-vendor@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BE25FA8A53; Fri, 16 Aug 2019 17:03:37 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4698m94f5qz4StV; Fri, 16 Aug 2019 17:03:37 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6735544FC; Fri, 16 Aug 2019 17:03:37 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7GH3bVn088613; Fri, 16 Aug 2019 17:03:37 GMT (envelope-from jkim@FreeBSD.org) Received: (from jkim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7GH3bxR088612; Fri, 16 Aug 2019 17:03:37 GMT (envelope-from jkim@FreeBSD.org) Message-Id: <201908161703.x7GH3bxR088612@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jkim set sender to jkim@FreeBSD.org using -f From: Jung-uk Kim Date: Fri, 16 Aug 2019 17:03:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r351128 - vendor-sys/acpica/20190816 X-SVN-Group: vendor-sys X-SVN-Commit-Author: jkim X-SVN-Commit-Paths: vendor-sys/acpica/20190816 X-SVN-Commit-Revision: 351128 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2019 17:03:37 -0000 Author: jkim Date: Fri Aug 16 17:03:37 2019 New Revision: 351128 URL: https://svnweb.freebsd.org/changeset/base/351128 Log: Tag ACPICA 20190816. Added: vendor-sys/acpica/20190816/ - copied from r351127, vendor-sys/acpica/dist/