From owner-svn-src-vendor@freebsd.org Wed Dec 26 07:57:22 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAB9A134629B; Wed, 26 Dec 2018 07:57:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E1826B054; Wed, 26 Dec 2018 07:57:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4050420AAC; Wed, 26 Dec 2018 07:57:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBQ7vM01046663; Wed, 26 Dec 2018 07:57:22 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBQ7vMVt046662; Wed, 26 Dec 2018 07:57:22 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201812260757.wBQ7vMVt046662@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 26 Dec 2018 07:57:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342469 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib/libzfs_core/common X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib/libzfs_core/common X-SVN-Commit-Revision: 342469 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4E1826B054 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.964,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2018 07:57:22 -0000 Author: avg Date: Wed Dec 26 07:57:21 2018 New Revision: 342469 URL: https://svnweb.freebsd.org/changeset/base/342469 Log: 9630 add lzc_rename and lzc_destroy to libzfs_core illumos/illumos-gate@049ba636fa37a2892809192fc671bff9158a01cd https://github.com/illumos/illumos-gate/commit/049ba636fa37a2892809192fc671bff9158a01cd https://www.illumos.org/issues/9630 Rename and destroy are very useful operations that deserve to be in libzfs_core. And they are not hard to implement too. Reviewed by: Andy Stormont Reviewed by: Matt Ahrens Reviewed by: Serapheim Dimitropoulos Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Wed Dec 26 04:06:16 2018 (r342468) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Wed Dec 26 07:57:21 2018 (r342469) @@ -3686,7 +3686,6 @@ zfs_ioc_pool_discard_checkpoint(const char *poolname, /* * inputs: * zc_name name of dataset to destroy - * zc_objset_type type of objset * zc_defer_destroy mark for deferred destroy * * outputs: none @@ -3694,16 +3693,24 @@ zfs_ioc_pool_discard_checkpoint(const char *poolname, static int zfs_ioc_destroy(zfs_cmd_t *zc) { + objset_t *os; + dmu_objset_type_t ost; int err; - if (zc->zc_objset_type == DMU_OST_ZFS) + err = dmu_objset_hold(zc->zc_name, FTAG, &os); + if (err != 0) + return (err); + ost = dmu_objset_type(os); + dmu_objset_rele(os, FTAG); + + if (ost == DMU_OST_ZFS) zfs_unmount_snap(zc->zc_name); if (strchr(zc->zc_name, '@')) err = dsl_destroy_snapshot(zc->zc_name, zc->zc_defer_destroy); else err = dsl_destroy_head(zc->zc_name); - if (zc->zc_objset_type == DMU_OST_ZVOL && err == 0) + if (ost == DMU_OST_ZVOL && err == 0) (void) zvol_remove_minor(zc->zc_name); return (err); } @@ -3854,8 +3861,11 @@ recursive_unmount(const char *fsname, void *arg) static int zfs_ioc_rename(zfs_cmd_t *zc) { + objset_t *os; + dmu_objset_type_t ost; boolean_t recursive = zc->zc_cookie & 1; char *at; + int err; /* "zfs rename" from and to ...%recv datasets should both fail */ zc->zc_name[sizeof (zc->zc_name) - 1] = '\0'; @@ -3865,6 +3875,12 @@ zfs_ioc_rename(zfs_cmd_t *zc) strchr(zc->zc_name, '%') || strchr(zc->zc_value, '%')) return (SET_ERROR(EINVAL)); + err = dmu_objset_hold(zc->zc_name, FTAG, &os); + if (err != 0) + return (err); + ost = dmu_objset_type(os); + dmu_objset_rele(os, FTAG); + at = strchr(zc->zc_name, '@'); if (at != NULL) { /* snaps must be in same fs */ @@ -3873,7 +3889,7 @@ zfs_ioc_rename(zfs_cmd_t *zc) if (strncmp(zc->zc_name, zc->zc_value, at - zc->zc_name + 1)) return (SET_ERROR(EXDEV)); *at = '\0'; - if (zc->zc_objset_type == DMU_OST_ZFS) { + if (ost == DMU_OST_ZFS) { error = dmu_objset_find(zc->zc_name, recursive_unmount, at + 1, recursive ? DS_FIND_CHILDREN : 0); @@ -3888,7 +3904,7 @@ zfs_ioc_rename(zfs_cmd_t *zc) return (error); } else { - if (zc->zc_objset_type == DMU_OST_ZVOL) + if (ost == DMU_OST_ZVOL) (void) zvol_remove_minor(zc->zc_name); return (dsl_dir_rename(zc->zc_name, zc->zc_value)); } From owner-svn-src-vendor@freebsd.org Wed Dec 26 07:57:23 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F49C13462A0; Wed, 26 Dec 2018 07:57:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 338856B055; Wed, 26 Dec 2018 07:57:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 27AD420AAD; Wed, 26 Dec 2018 07:57:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBQ7vNuE046672; Wed, 26 Dec 2018 07:57:23 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBQ7vMS2046669; Wed, 26 Dec 2018 07:57:22 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201812260757.wBQ7vMS2046669@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 26 Dec 2018 07:57:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342469 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib/libzfs_core/common X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib/libzfs_core/common X-SVN-Commit-Revision: 342469 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 338856B055 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.964,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2018 07:57:23 -0000 Author: avg Date: Wed Dec 26 07:57:21 2018 New Revision: 342469 URL: https://svnweb.freebsd.org/changeset/base/342469 Log: 9630 add lzc_rename and lzc_destroy to libzfs_core illumos/illumos-gate@049ba636fa37a2892809192fc671bff9158a01cd https://github.com/illumos/illumos-gate/commit/049ba636fa37a2892809192fc671bff9158a01cd https://www.illumos.org/issues/9630 Rename and destroy are very useful operations that deserve to be in libzfs_core. And they are not hard to implement too. Reviewed by: Andy Stormont Reviewed by: Matt Ahrens Reviewed by: Serapheim Dimitropoulos Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Dec 26 04:06:16 2018 (r342468) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Dec 26 07:57:21 2018 (r342469) @@ -3609,32 +3609,34 @@ zfs_create(libzfs_handle_t *hdl, const char *path, zfs int zfs_destroy(zfs_handle_t *zhp, boolean_t defer) { - zfs_cmd_t zc = { 0 }; + int error; + if (zhp->zfs_type != ZFS_TYPE_SNAPSHOT && defer) + return (EINVAL); + if (zhp->zfs_type == ZFS_TYPE_BOOKMARK) { nvlist_t *nv = fnvlist_alloc(); fnvlist_add_boolean(nv, zhp->zfs_name); - int error = lzc_destroy_bookmarks(nv, NULL); + error = lzc_destroy_bookmarks(nv, NULL); fnvlist_free(nv); if (error != 0) { - return (zfs_standard_error_fmt(zhp->zfs_hdl, errno, + return (zfs_standard_error_fmt(zhp->zfs_hdl, error, dgettext(TEXT_DOMAIN, "cannot destroy '%s'"), zhp->zfs_name)); } return (0); } - (void) strlcpy(zc.zc_name, zhp->zfs_name, sizeof (zc.zc_name)); - - if (ZFS_IS_VOLUME(zhp)) { - zc.zc_objset_type = DMU_OST_ZVOL; + if (zhp->zfs_type == ZFS_TYPE_SNAPSHOT) { + nvlist_t *nv = fnvlist_alloc(); + fnvlist_add_boolean(nv, zhp->zfs_name); + error = lzc_destroy_snaps(nv, defer, NULL); + fnvlist_free(nv); } else { - zc.zc_objset_type = DMU_OST_ZFS; + error = lzc_destroy(zhp->zfs_name); } - zc.zc_defer_destroy = defer; - if (zfs_ioctl(zhp->zfs_hdl, ZFS_IOC_DESTROY, &zc) != 0 && - errno != ENOENT) { + if (error != 0 && error != ENOENT) { return (zfs_standard_error_fmt(zhp->zfs_hdl, errno, dgettext(TEXT_DOMAIN, "cannot destroy '%s'"), zhp->zfs_name)); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Wed Dec 26 04:06:16 2018 (r342468) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Wed Dec 26 07:57:21 2018 (r342469) @@ -2133,7 +2133,6 @@ recv_rename(libzfs_handle_t *hdl, const char *name, co int baselen, char *newname, recvflags_t *flags) { static int seq; - zfs_cmd_t zc = { 0 }; int err; prop_changelist_t *clp; zfs_handle_t *zhp; @@ -2150,19 +2149,13 @@ recv_rename(libzfs_handle_t *hdl, const char *name, co if (err) return (err); - zc.zc_objset_type = DMU_OST_ZFS; - (void) strlcpy(zc.zc_name, name, sizeof (zc.zc_name)); - if (tryname) { (void) strcpy(newname, tryname); - - (void) strlcpy(zc.zc_value, tryname, sizeof (zc.zc_value)); - if (flags->verbose) { (void) printf("attempting rename %s to %s\n", - zc.zc_name, zc.zc_value); + name, newname); } - err = ioctl(hdl->libzfs_fd, ZFS_IOC_RENAME, &zc); + err = lzc_rename(name, newname); if (err == 0) changelist_rename(clp, name, tryname); } else { @@ -2174,13 +2167,11 @@ recv_rename(libzfs_handle_t *hdl, const char *name, co (void) snprintf(newname, ZFS_MAX_DATASET_NAME_LEN, "%.*srecv-%u-%u", baselen, name, getpid(), seq); - (void) strlcpy(zc.zc_value, newname, sizeof (zc.zc_value)); - if (flags->verbose) { (void) printf("failed - trying rename %s to %s\n", - zc.zc_name, zc.zc_value); + name, newname); } - err = ioctl(hdl->libzfs_fd, ZFS_IOC_RENAME, &zc); + err = lzc_rename(name, newname); if (err == 0) changelist_rename(clp, name, newname); if (err && flags->verbose) { @@ -2205,7 +2196,6 @@ static int recv_destroy(libzfs_handle_t *hdl, const char *name, int baselen, char *newname, recvflags_t *flags) { - zfs_cmd_t zc = { 0 }; int err = 0; prop_changelist_t *clp; zfs_handle_t *zhp; @@ -2228,17 +2218,20 @@ recv_destroy(libzfs_handle_t *hdl, const char *name, i if (err) return (err); - zc.zc_objset_type = DMU_OST_ZFS; - zc.zc_defer_destroy = defer; - (void) strlcpy(zc.zc_name, name, sizeof (zc.zc_name)); - if (flags->verbose) - (void) printf("attempting destroy %s\n", zc.zc_name); - err = ioctl(hdl->libzfs_fd, ZFS_IOC_DESTROY, &zc); + (void) printf("attempting destroy %s\n", name); + if (zhp->zfs_type == ZFS_TYPE_SNAPSHOT) { + nvlist_t *nv = fnvlist_alloc(); + fnvlist_add_boolean(nv, name); + err = lzc_destroy_snaps(nv, defer, NULL); + fnvlist_free(nv); + } else { + err = lzc_destroy(name); + } if (err == 0) { if (flags->verbose) (void) printf("success\n"); - changelist_remove(clp, zc.zc_name); + changelist_remove(clp, name); } (void) changelist_postfix(clp); Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Wed Dec 26 04:06:16 2018 (r342468) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Wed Dec 26 07:57:21 2018 (r342469) @@ -249,6 +249,34 @@ lzc_remap(const char *fsname) return (error); } +int +lzc_rename(const char *source, const char *target) +{ + zfs_cmd_t zc = { 0 }; + int error; + + ASSERT3S(g_refcount, >, 0); + VERIFY3S(g_fd, !=, -1); + + (void) strlcpy(zc.zc_name, source, sizeof (zc.zc_name)); + (void) strlcpy(zc.zc_value, target, sizeof (zc.zc_value)); + error = ioctl(g_fd, ZFS_IOC_RENAME, &zc); + if (error != 0) + error = errno; + return (error); +} + +int +lzc_destroy(const char *fsname) +{ + int error; + + nvlist_t *args = fnvlist_alloc(); + error = lzc_ioctl(ZFS_IOC_DESTROY, fsname, args, NULL); + nvlist_free(args); + return (error); +} + /* * Creates snapshots. * Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Wed Dec 26 04:06:16 2018 (r342468) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Wed Dec 26 07:57:21 2018 (r342469) @@ -91,6 +91,9 @@ boolean_t lzc_exists(const char *); int lzc_rollback(const char *, char *, int); int lzc_rollback_to(const char *, const char *); +int lzc_rename(const char *, const char *); +int lzc_destroy(const char *); + int lzc_channel_program(const char *, const char *, uint64_t, uint64_t, nvlist_t *, nvlist_t **); int lzc_channel_program_nosync(const char *, const char *, uint64_t, From owner-svn-src-vendor@freebsd.org Wed Dec 26 10:41:19 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D7DC134C6AF; Wed, 26 Dec 2018 10:41:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C897C74817; Wed, 26 Dec 2018 10:41:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BB6E822618; Wed, 26 Dec 2018 10:41:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBQAfIr4036736; Wed, 26 Dec 2018 10:41:18 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBQAfHcG036526; Wed, 26 Dec 2018 10:41:17 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201812261041.wBQAfHcG036526@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 26 Dec 2018 10:41:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342532 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendor/illumos/dist/lib/li... X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendor/illumos/dist/lib/libzfs/common vendor/illumo... X-SVN-Commit-Revision: 342532 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: C897C74817 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2018 10:41:19 -0000 Author: avg Date: Wed Dec 26 10:41:17 2018 New Revision: 342532 URL: https://svnweb.freebsd.org/changeset/base/342532 Log: 5882 Temporary pool names illumos/illumos-gate@04e56356520b98d5a93c496b10f02530bb6647e0 https://github.com/illumos/illumos-gate/commit/04e56356520b98d5a93c496b10f02530bb6647e0 https://www.illumos.org/issues/5882 This is an import of the temporary pool names functionality from ZoL: https://github.com/zfsonlinux/zfs/commit/ e2282ef57edc79cdce2a4b9b7e3333c56494a807 https://github.com/zfsonlinux/zfs/commit/ 26b42f3f9d03f85cc7966dc2fe4dfe9216601b0e https://github.com/zfsonlinux/zfs/commit/ 2f3ec9006146844af6763d1fa4e823fd9047fd54 https://github.com/zfsonlinux/zfs/commit/ 00d2a8c92f614f49d23dea5d73f7ea7eb489ccf1 https://github.com/zfsonlinux/zfs/commit/ 83e9986f6eefdf0afc387f06407087bba3ead4e9 https://github.com/zfsonlinux/zfs/commit/ 023bbe6f017380f4a04c5060feb24dd8cdda9fce It is intended to assist the creation and management of virtual machines that have their rootfs on ZFS on hosts that also have their rootfs on ZFS. These situations cause SPA namespace collisions when the standard name rpool is used in both cases. The solution is either to give each guest pool a name unique to the host, which is not always desireable, or boot a VM environment containing an ISO image to install it, which is cumbersome. As a side note, this commit includes the removal of `zpool import -r`, which previously did nothing. patch [Magnifier] (14.3 KB) Richard Yao, 2015-04-30 04:33 PM Reviewed by: Matt Ahrens Reviewed by: Igor Kozhukhov Reviewed by: John Kennedy Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zpool_prop.c Wed Dec 26 10:41:14 2018 (r342531) +++ vendor-sys/illumos/dist/common/zfs/zpool_prop.c Wed Dec 26 10:41:17 2018 (r342532) @@ -136,6 +136,8 @@ zpool_prop_init(void) PROP_READONLY, ZFS_TYPE_POOL, "NAME"); zprop_register_hidden(ZPOOL_PROP_MAXBLOCKSIZE, "maxblocksize", PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_POOL, "MAXBLOCKSIZE"); + zprop_register_hidden(ZPOOL_PROP_TNAME, "tname", PROP_TYPE_STRING, + PROP_ONETIME, ZFS_TYPE_POOL, "TNAME"); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Dec 26 10:41:14 2018 (r342531) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Dec 26 10:41:17 2018 (r342532) @@ -4578,12 +4578,18 @@ spa_create(const char *pool, nvlist_t *nvroot, nvlist_ uint_t nspares, nl2cache; uint64_t version, obj; boolean_t has_features; + char *poolname; + nvlist_t *nvl; + if (nvlist_lookup_string(props, + zpool_prop_to_name(ZPOOL_PROP_TNAME), &poolname) != 0) + poolname = (char *)pool; + /* * If this pool already exists, return failure. */ mutex_enter(&spa_namespace_lock); - if (spa_lookup(pool) != NULL) { + if (spa_lookup(poolname) != NULL) { mutex_exit(&spa_namespace_lock); return (SET_ERROR(EEXIST)); } @@ -4591,9 +4597,12 @@ spa_create(const char *pool, nvlist_t *nvroot, nvlist_ /* * Allocate a new spa_t structure. */ + nvl = fnvlist_alloc(); + fnvlist_add_string(nvl, ZPOOL_CONFIG_POOL_NAME, pool); (void) nvlist_lookup_string(props, zpool_prop_to_name(ZPOOL_PROP_ALTROOT), &altroot); - spa = spa_add(pool, NULL, altroot); + spa = spa_add(poolname, nvl, altroot); + fnvlist_free(nvl); spa_activate(spa, spa_mode_global); if (props && (error = spa_prop_validate(spa, props))) { @@ -4602,6 +4611,12 @@ spa_create(const char *pool, nvlist_t *nvroot, nvlist_ mutex_exit(&spa_namespace_lock); return (error); } + + /* + * Temporary pool names should never be written to disk. + */ + if (poolname != pool) + spa->spa_import_flags |= ZFS_IMPORT_TEMP_NAME; has_features = B_FALSE; for (nvpair_t *elem = nvlist_next_nvpair(props, NULL); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c Wed Dec 26 10:41:14 2018 (r342531) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c Wed Dec 26 10:41:17 2018 (r342532) @@ -208,6 +208,7 @@ spa_write_cachefile(spa_t *target, boolean_t removing, nvlist_t *nvl; boolean_t ccw_failure; int error; + char *pool_name; ASSERT(MUTEX_HELD(&spa_namespace_lock)); @@ -254,7 +255,14 @@ spa_write_cachefile(spa_t *target, boolean_t removing, if (nvl == NULL) nvl = fnvlist_alloc(); - fnvlist_add_nvlist(nvl, spa->spa_name, + if (spa->spa_import_flags & ZFS_IMPORT_TEMP_NAME) { + pool_name = fnvlist_lookup_string( + spa->spa_config, ZPOOL_CONFIG_POOL_NAME); + } else { + pool_name = spa_name(spa); + } + + fnvlist_add_nvlist(nvl, pool_name, spa->spa_config); mutex_exit(&spa->spa_props_lock); } @@ -359,6 +367,7 @@ spa_config_generate(spa_t *spa, vdev_t *vd, uint64_t t unsigned long hostid = 0; boolean_t locked = B_FALSE; uint64_t split_guid; + char *pool_name; if (vd == NULL) { vd = rvd; @@ -375,10 +384,27 @@ spa_config_generate(spa_t *spa, vdev_t *vd, uint64_t t if (txg == -1ULL) txg = spa->spa_config_txg; + /* + * Originally, users had to handle spa namespace collisions by either + * exporting the already imported pool or by specifying a new name for + * the pool with a conflicting name. In the case of root pools from + * virtual guests, neither approach to collision resolution is + * reasonable. This is addressed by extending the new name syntax with + * an option to specify that the new name is temporary. When specified, + * ZFS_IMPORT_TEMP_NAME will be set in spa->spa_import_flags to tell us + * to use the previous name, which we do below. + */ + if (spa->spa_import_flags & ZFS_IMPORT_TEMP_NAME) { + pool_name = fnvlist_lookup_string(spa->spa_config, + ZPOOL_CONFIG_POOL_NAME); + } else { + pool_name = spa_name(spa); + } + config = fnvlist_alloc(); fnvlist_add_uint64(config, ZPOOL_CONFIG_VERSION, spa_version(spa)); - fnvlist_add_string(config, ZPOOL_CONFIG_POOL_NAME, spa_name(spa)); + fnvlist_add_string(config, ZPOOL_CONFIG_POOL_NAME, pool_name); fnvlist_add_uint64(config, ZPOOL_CONFIG_POOL_STATE, spa_state(spa)); fnvlist_add_uint64(config, ZPOOL_CONFIG_POOL_TXG, txg); fnvlist_add_uint64(config, ZPOOL_CONFIG_POOL_GUID, spa_guid(spa)); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Wed Dec 26 10:41:14 2018 (r342531) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Wed Dec 26 10:41:17 2018 (r342532) @@ -1492,6 +1492,7 @@ zfs_ioc_pool_create(zfs_cmd_t *zc) nvlist_t *config, *props = NULL; nvlist_t *rootprops = NULL; nvlist_t *zplprops = NULL; + char *spa_name = zc->zc_name; if (error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size, zc->zc_iflags, &config)) @@ -1507,6 +1508,7 @@ zfs_ioc_pool_create(zfs_cmd_t *zc) if (props) { nvlist_t *nvl = NULL; uint64_t version = SPA_VERSION; + char *tname; (void) nvlist_lookup_uint64(props, zpool_prop_to_name(ZPOOL_PROP_VERSION), &version); @@ -1529,6 +1531,10 @@ zfs_ioc_pool_create(zfs_cmd_t *zc) zplprops, NULL); if (error != 0) goto pool_props_bad; + + if (nvlist_lookup_string(props, + zpool_prop_to_name(ZPOOL_PROP_TNAME), &tname) == 0) + spa_name = tname; } error = spa_create(zc->zc_name, config, props, zplprops); @@ -1536,9 +1542,9 @@ zfs_ioc_pool_create(zfs_cmd_t *zc) /* * Set the remaining root properties */ - if (!error && (error = zfs_set_prop_nvlist(zc->zc_name, + if (!error && (error = zfs_set_prop_nvlist(spa_name, ZPROP_SRC_LOCAL, rootprops, NULL)) != 0) - (void) spa_destroy(zc->zc_name); + (void) spa_destroy(spa_name); pool_props_bad: nvlist_free(rootprops); Modified: vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Wed Dec 26 10:41:14 2018 (r342531) +++ vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Wed Dec 26 10:41:17 2018 (r342532) @@ -210,6 +210,7 @@ typedef enum { ZPOOL_PROP_MAXBLOCKSIZE, ZPOOL_PROP_BOOTSIZE, ZPOOL_PROP_CHECKPOINT, + ZPOOL_PROP_TNAME, ZPOOL_NUM_PROPS } zpool_prop_t; @@ -855,7 +856,7 @@ typedef struct vdev_stat { * is passed between kernel and userland as an nvlist uint64 array. */ typedef struct ddt_object { - uint64_t ddo_count; /* number of elments in ddt */ + uint64_t ddo_count; /* number of elments in ddt */ uint64_t ddo_dspace; /* size of ddt on disk */ uint64_t ddo_mspace; /* size of ddt in-core */ } ddt_object_t; @@ -1061,6 +1062,7 @@ typedef enum { #define ZFS_IMPORT_MISSING_LOG 0x4 #define ZFS_IMPORT_ONLY 0x8 #define ZFS_IMPORT_CHECKPOINT 0x10 +#define ZFS_IMPORT_TEMP_NAME 0x20 /* * Channel program argument/return nvlist keys and defaults. From owner-svn-src-vendor@freebsd.org Wed Dec 26 10:41:20 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5C49134C6BA; Wed, 26 Dec 2018 10:41:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CC5674824; Wed, 26 Dec 2018 10:41:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 56A8A2261A; Wed, 26 Dec 2018 10:41:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBQAfJ82036744; Wed, 26 Dec 2018 10:41:19 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBQAfIPf036741; Wed, 26 Dec 2018 10:41:18 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201812261041.wBQAfIPf036741@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 26 Dec 2018 10:41:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342532 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendor/illumos/dist/lib/li... X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendor/illumos/dist/lib/libzfs/common vendor/illumo... X-SVN-Commit-Revision: 342532 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 8CC5674824 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2018 10:41:20 -0000 Author: avg Date: Wed Dec 26 10:41:17 2018 New Revision: 342532 URL: https://svnweb.freebsd.org/changeset/base/342532 Log: 5882 Temporary pool names illumos/illumos-gate@04e56356520b98d5a93c496b10f02530bb6647e0 https://github.com/illumos/illumos-gate/commit/04e56356520b98d5a93c496b10f02530bb6647e0 https://www.illumos.org/issues/5882 This is an import of the temporary pool names functionality from ZoL: https://github.com/zfsonlinux/zfs/commit/ e2282ef57edc79cdce2a4b9b7e3333c56494a807 https://github.com/zfsonlinux/zfs/commit/ 26b42f3f9d03f85cc7966dc2fe4dfe9216601b0e https://github.com/zfsonlinux/zfs/commit/ 2f3ec9006146844af6763d1fa4e823fd9047fd54 https://github.com/zfsonlinux/zfs/commit/ 00d2a8c92f614f49d23dea5d73f7ea7eb489ccf1 https://github.com/zfsonlinux/zfs/commit/ 83e9986f6eefdf0afc387f06407087bba3ead4e9 https://github.com/zfsonlinux/zfs/commit/ 023bbe6f017380f4a04c5060feb24dd8cdda9fce It is intended to assist the creation and management of virtual machines that have their rootfs on ZFS on hosts that also have their rootfs on ZFS. These situations cause SPA namespace collisions when the standard name rpool is used in both cases. The solution is either to give each guest pool a name unique to the host, which is not always desireable, or boot a VM environment containing an ISO image to install it, which is cumbersome. As a side note, this commit includes the removal of `zpool import -r`, which previously did nothing. patch [Magnifier] (14.3 KB) Richard Yao, 2015-04-30 04:33 PM Reviewed by: Matt Ahrens Reviewed by: Igor Kozhukhov Reviewed by: John Kennedy Approved by: Dan McDonald Author: Andriy Gapon Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/man/man1m/zpool.1m Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Wed Dec 26 10:41:14 2018 (r342531) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Wed Dec 26 10:41:17 2018 (r342532) @@ -220,8 +220,9 @@ get_usage(zpool_help_t idx) case HELP_CREATE: return (gettext("\tcreate [-fnd] [-B] " "[-o property=value] ... \n" - "\t [-O file-system-property=value] ... \n" - "\t [-m mountpoint] [-R root] ...\n")); + "\t [-O file-system-property=value] ...\n" + "\t [-m mountpoint] [-R root] [-t tempname] " + " ...\n")); case HELP_CHECKPOINT: return (gettext("\tcheckpoint [--discard] ...\n")); case HELP_DESTROY: @@ -239,7 +240,7 @@ get_usage(zpool_help_t idx) "[-R root] [-F [-n]] -a\n" "\timport [-o mntopts] [-o property=value] ... \n" "\t [-d dir | -c cachefile] [-D] [-f] [-m] [-N] " - "[-R root] [-F [-n]]\n" + "[-R root] [-F [-n]] [-t]\n" "\t [--rewind-to-checkpoint] [newpool]\n")); case HELP_IOSTAT: return (gettext("\tiostat [-v] [-T d|u] [pool] ... [interval " @@ -492,6 +493,21 @@ add_prop_list(const char *propname, char *propval, nvl } /* + * Set a default property pair (name, string-value) in a property nvlist + */ +static int +add_prop_list_default(const char *propname, char *propval, nvlist_t **props, + boolean_t poolprop) +{ + char *pval; + + if (nvlist_lookup_string(*props, propname, &pval) == 0) + return (0); + + return (add_prop_list(propname, propval, props, poolprop)); +} + +/* * zpool add [-fn] ... * * -f Force addition of devices, even if they appear in use @@ -849,15 +865,16 @@ errout: /* * zpool create [-fnd] [-B] [-o property=value] ... * [-O file-system-property=value] ... - * [-R root] [-m mountpoint] ... + * [-R root] [-m mountpoint] [-t tempname] ... * * -B Create boot partition. * -f Force creation, even if devices appear in use * -n Do not create the pool, but display the resulting layout if it * were to be created. - * -R Create a pool under an alternate root - * -m Set default mountpoint for the root dataset. By default it's + * -R Create a pool under an alternate root + * -m Set default mountpoint for the root dataset. By default it's * '/' + * -t Use the temporary name until the pool is exported. * -o Set property=value. * -d Don't automatically enable all supported pool features * (individual features can be enabled with -o). @@ -881,6 +898,7 @@ zpool_do_create(int argc, char **argv) int c; nvlist_t *nvroot = NULL; char *poolname; + char *tname = NULL; int ret = 1; char *altroot = NULL; char *mountpoint = NULL; @@ -889,7 +907,7 @@ zpool_do_create(int argc, char **argv) char *propval; /* check options */ - while ((c = getopt(argc, argv, ":fndBR:m:o:O:")) != -1) { + while ((c = getopt(argc, argv, ":fndBR:m:o:O:t:")) != -1) { switch (c) { case 'f': force = B_TRUE; @@ -914,11 +932,7 @@ zpool_do_create(int argc, char **argv) if (add_prop_list(zpool_prop_to_name( ZPOOL_PROP_ALTROOT), optarg, &props, B_TRUE)) goto errout; - if (nvlist_lookup_string(props, - zpool_prop_to_name(ZPOOL_PROP_CACHEFILE), - &propval) == 0) - break; - if (add_prop_list(zpool_prop_to_name( + if (add_prop_list_default(zpool_prop_to_name( ZPOOL_PROP_CACHEFILE), "none", &props, B_TRUE)) goto errout; break; @@ -991,6 +1005,27 @@ zpool_do_create(int argc, char **argv) goto errout; } break; + case 't': + /* + * Sanity check temporary pool name. + */ + if (strchr(optarg, '/') != NULL) { + (void) fprintf(stderr, gettext("cannot create " + "'%s': invalid character '/' in temporary " + "name\n"), optarg); + (void) fprintf(stderr, gettext("use 'zfs " + "create' to create a dataset\n")); + goto errout; + } + + if (add_prop_list(zpool_prop_to_name( + ZPOOL_PROP_TNAME), optarg, &props, B_TRUE)) + goto errout; + if (add_prop_list_default(zpool_prop_to_name( + ZPOOL_PROP_CACHEFILE), "none", &props, B_TRUE)) + goto errout; + tname = optarg; + break; case ':': (void) fprintf(stderr, gettext("missing argument for " "'%c' option\n"), optopt); @@ -1196,8 +1231,8 @@ zpool_do_create(int argc, char **argv) ret = 1; if (zpool_create(g_zfs, poolname, nvroot, props, fsprops) == 0) { - zfs_handle_t *pool = zfs_open(g_zfs, poolname, - ZFS_TYPE_FILESYSTEM); + zfs_handle_t *pool = zfs_open(g_zfs, + tname ? tname : poolname, ZFS_TYPE_FILESYSTEM); if (pool != NULL) { if (zfs_mount(pool, NULL, 0) == 0) ret = zfs_shareall(pool); @@ -1224,7 +1259,7 @@ badusage: /* * zpool destroy * - * -f Forcefully unmount any datasets + * -f Forcefully unmount any datasets * * Destroy the given pool. Automatically unmounts any datasets in the pool. */ @@ -2098,8 +2133,8 @@ do_import(nvlist_t *config, const char *newname, const * zpool checkpoint * checkpoint --discard * - * -d Discard the checkpoint from a checkpointed - * --discard pool. + * -d Discard the checkpoint from a checkpointed + * --discard pool. * * Checkpoints the specified pool, by taking a "snapshot" of its * current state. A pool can only have one checkpoint at a time. @@ -2172,45 +2207,49 @@ zpool_do_checkpoint(int argc, char **argv) * import [-o mntopts] [-o prop=value] ... [-R root] [-D] * [-d dir | -c cachefile] [-f] -a * import [-o mntopts] [-o prop=value] ... [-R root] [-D] - * [-d dir | -c cachefile] [-f] [-n] [-F] [newpool] + * [-d dir | -c cachefile] [-f] [-n] [-F] [-t] + * [newpool] * - * -c Read pool information from a cachefile instead of searching + * -c Read pool information from a cachefile instead of searching * devices. * - * -d Scan in a specific directory, other than /dev/dsk. More than + * -d Scan in a specific directory, other than /dev/dsk. More than * one directory can be specified using multiple '-d' options. * - * -D Scan for previously destroyed pools or import all or only - * specified destroyed pools. + * -D Scan for previously destroyed pools or import all or only + * specified destroyed pools. * - * -R Temporarily import the pool, with all mountpoints relative to + * -R Temporarily import the pool, with all mountpoints relative to * the given root. The pool will remain exported when the machine * is rebooted. * - * -V Import even in the presence of faulted vdevs. This is an - * intentionally undocumented option for testing purposes, and - * treats the pool configuration as complete, leaving any bad + * -V Import even in the presence of faulted vdevs. This is an + * intentionally undocumented option for testing purposes, and + * treats the pool configuration as complete, leaving any bad * vdevs in the FAULTED state. In other words, it does verbatim * import. * - * -f Force import, even if it appears that the pool is active. + * -f Force import, even if it appears that the pool is active. * - * -F Attempt rewind if necessary. + * -F Attempt rewind if necessary. * - * -n See if rewind would work, but don't actually rewind. + * -n See if rewind would work, but don't actually rewind. * - * -N Import the pool but don't mount datasets. + * -N Import the pool but don't mount datasets. * - * -T Specify a starting txg to use for import. This option is - * intentionally undocumented option for testing purposes. + * -t Use newpool as a temporary pool name instead of renaming + * the pool. * - * -a Import all pools found. + * -T Specify a starting txg to use for import. This option is + * intentionally undocumented option for testing purposes. * - * -o Set property=value and/or temporary mount options (without '='). + * -a Import all pools found. * - * --rewind-to-checkpoint - * Import the pool and revert back to the checkpoint. + * -o Set property=value and/or temporary mount options (without '='). * + * --rewind-to-checkpoint + * Import the pool and revert back to the checkpoint. + * * The import command scans for pools to import, and import pools based on pool * name and GUID. The pool can also be renamed as part of the import process. */ @@ -2251,7 +2290,7 @@ zpool_do_import(int argc, char **argv) }; /* check options */ - while ((c = getopt_long(argc, argv, ":aCc:d:DEfFmnNo:rR:T:VX", + while ((c = getopt_long(argc, argv, ":aCc:d:DEfFmnNo:rR:tT:VX", long_options, NULL)) != -1) { switch (c) { case 'a': @@ -2306,14 +2345,16 @@ zpool_do_import(int argc, char **argv) if (add_prop_list(zpool_prop_to_name( ZPOOL_PROP_ALTROOT), optarg, &props, B_TRUE)) goto error; - if (nvlist_lookup_string(props, - zpool_prop_to_name(ZPOOL_PROP_CACHEFILE), - &propval) == 0) - break; - if (add_prop_list(zpool_prop_to_name( + if (add_prop_list_default(zpool_prop_to_name( ZPOOL_PROP_CACHEFILE), "none", &props, B_TRUE)) goto error; break; + case 't': + flags |= ZFS_IMPORT_TEMP_NAME; + if (add_prop_list_default(zpool_prop_to_name( + ZPOOL_PROP_CACHEFILE), "none", &props, B_TRUE)) + goto error; + break; case 'T': errno = 0; txg = strtoull(optarg, &endptr, 0); @@ -2449,9 +2490,9 @@ zpool_do_import(int argc, char **argv) (void) fprintf(stderr, gettext("cannot import '%s': " "a pool with that name already exists\n"), argv[0]); - (void) fprintf(stderr, gettext("use the form '%s " - " ' to give it a new name\n"), - "zpool import"); + (void) fprintf(stderr, gettext("use the form 'zpool import " + "[-t] ' to give it a new temporary " + "or permanent name\n")); err = 1; } else if (pools == NULL && idata.exists) { (void) fprintf(stderr, gettext("cannot import '%s': " @@ -3367,7 +3408,7 @@ list_callback(zpool_handle_t *zhp, void *data) * -o List of properties to display. Defaults to * "name,size,allocated,free,expandsize,fragmentation,capacity," * "dedupratio,health,altroot" - * -p Diplay values in parsable (exact) format. + * -p Diplay values in parsable (exact) format. * -T Display a timestamp in date(1) or Unix format * * List all pools in the system, whether or not they're healthy. Output space @@ -5839,7 +5880,7 @@ get_callback(zpool_handle_t *zhp, void *data) * by a single tab. * -o List of columns to display. Defaults to * "name,property,value,source". - * -p Diplay values in parsable (exact) format. + * -p Diplay values in parsable (exact) format. * * Get properties of pools in the system. Output space statistics * for each one as well as other attributes. Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Wed Dec 26 10:41:14 2018 (r342531) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Wed Dec 26 10:41:17 2018 (r342532) @@ -643,11 +643,22 @@ zpool_valid_proplist(libzfs_handle_t *hdl, const char goto error; } break; + case ZPOOL_PROP_READONLY: if (!flags.import) { zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "property '%s' can only be set at " "import time"), propname); + (void) zfs_error(hdl, EZFS_BADPROP, errbuf); + goto error; + } + break; + + case ZPOOL_PROP_TNAME: + if (!flags.create) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "property '%s' can only be set at " + "creation time"), propname); (void) zfs_error(hdl, EZFS_BADPROP, errbuf); goto error; } Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Wed Dec 26 10:41:14 2018 (r342531) +++ vendor/illumos/dist/man/man1m/zpool.1m Wed Dec 26 10:41:17 2018 (r342532) @@ -58,6 +58,7 @@ .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ... .Op Fl R Ar root +.Op Fl t Ar tempname .Ar pool vdev Ns ... .Nm .Cm destroy @@ -95,7 +96,7 @@ .Op Fl R Ar root .Nm .Cm import -.Op Fl Dfm +.Op Fl Dfmt .Op Fl F Op Fl n .Op Fl -rewind-to-checkpoint .Op Fl c Ar cachefile Ns | Ns Fl d Ar dir @@ -871,6 +872,7 @@ specified device or devices are cleared. .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ... .Op Fl R Ar root +.Op Fl t Ar tempname .Ar pool vdev Ns ... .Xc Creates a new storage pool containing the virtual devices specified on the @@ -992,6 +994,16 @@ for a list of valid properties that can be set. .It Fl R Ar root Equivalent to .Fl o Sy cachefile Ns = Ns Sy none Fl o Sy altroot Ns = Ns Ar root +.It Fl t Ar tempname +Sets the in-core pool name to +.Pa tempname +while the on-disk name will be the name specified as the pool name +.Pa pool . +This will set the default cachefile property to +.Sy none. +This is intended to handle name space collisions when creating pools +for other systems, such as virtual machines or physical machines +whose pools live on network block devices. .El .It Xo .Nm @@ -1237,7 +1249,7 @@ property to .It Xo .Nm .Cm import -.Op Fl Dfm +.Op Fl Dfmt .Op Fl F Op Fl n .Op Fl -rewind-to-checkpoint .Op Fl c Ar cachefile Ns | Ns Fl d Ar dir @@ -1328,6 +1340,20 @@ and the .Sy altroot property to .Ar root . +.It Fl t +Used with +.Ar newpool . +Specifies that +.Ar newpool +is temporary. +Temporary pool names last until export. +Ensures that the original pool name will be used in all label updates and +therefore is retained upon export. +Will also set +.Sy cachefile +property to +.Sy none +when not explicitly specified. .It Fl -rewind-to-checkpoint Rewinds pool to the checkpointed state. Once the pool is imported with this flag there is no way to undo the rewind. From owner-svn-src-vendor@freebsd.org Sat Dec 29 06:52:55 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC89614352EB; Sat, 29 Dec 2018 06:52:54 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66EC689589; Sat, 29 Dec 2018 06:52:54 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 53C06742D; Sat, 29 Dec 2018 06:52:54 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBT6qsTh010403; Sat, 29 Dec 2018 06:52:54 GMT (envelope-from cem@FreeBSD.org) Received: (from cem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBT6qr9N010397; Sat, 29 Dec 2018 06:52:53 GMT (envelope-from cem@FreeBSD.org) Message-Id: <201812290652.wBT6qr9N010397@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: cem set sender to cem@FreeBSD.org using -f From: Conrad Meyer Date: Sat, 29 Dec 2018 06:52:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342590 - in vendor/zstd/1.3.8: . contrib/adaptive-compression contrib/meson contrib/pzstd contrib/seekable_format contrib/seekable_format/examples doc lib lib/common lib/compress lib/d... X-SVN-Group: vendor X-SVN-Commit-Author: cem X-SVN-Commit-Paths: in vendor/zstd/1.3.8: . contrib/adaptive-compression contrib/meson contrib/pzstd contrib/seekable_format contrib/seekable_format/examples doc lib lib/common lib/compress lib/decompress lib/dictBuilder... X-SVN-Commit-Revision: 342590 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 66EC689589 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.998,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2018 06:52:55 -0000 Author: cem Date: Sat Dec 29 06:52:52 2018 New Revision: 342590 URL: https://svnweb.freebsd.org/changeset/base/342590 Log: tag import of zstd 1.3.8 Added: vendor/zstd/1.3.8/ - copied from r342588, vendor/zstd/dist/ vendor/zstd/1.3.8/lib/decompress/zstd_ddict.c - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/zstd_ddict.c vendor/zstd/1.3.8/lib/decompress/zstd_ddict.h - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/zstd_ddict.h vendor/zstd/1.3.8/lib/decompress/zstd_decompress_block.c - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/zstd_decompress_block.c vendor/zstd/1.3.8/lib/decompress/zstd_decompress_block.h - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/zstd_decompress_block.h vendor/zstd/1.3.8/lib/decompress/zstd_decompress_internal.h - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/zstd_decompress_internal.h vendor/zstd/1.3.8/programs/benchfn.c - copied unchanged from r342589, vendor/zstd/dist/programs/benchfn.c vendor/zstd/1.3.8/programs/benchfn.h - copied unchanged from r342589, vendor/zstd/dist/programs/benchfn.h vendor/zstd/1.3.8/programs/benchzstd.c - copied unchanged from r342589, vendor/zstd/dist/programs/benchzstd.c vendor/zstd/1.3.8/programs/benchzstd.h - copied unchanged from r342589, vendor/zstd/dist/programs/benchzstd.h vendor/zstd/1.3.8/programs/util.c - copied unchanged from r342589, vendor/zstd/dist/programs/util.c vendor/zstd/1.3.8/tests/regression/ - copied from r342589, vendor/zstd/dist/tests/regression/ Replaced: vendor/zstd/1.3.8/Makefile - copied unchanged from r342589, vendor/zstd/dist/Makefile vendor/zstd/1.3.8/README.md - copied unchanged from r342589, vendor/zstd/dist/README.md vendor/zstd/1.3.8/appveyor.yml - copied unchanged from r342589, vendor/zstd/dist/appveyor.yml vendor/zstd/1.3.8/contrib/adaptive-compression/Makefile - copied unchanged from r342589, vendor/zstd/dist/contrib/adaptive-compression/Makefile vendor/zstd/1.3.8/contrib/adaptive-compression/adapt.c - copied unchanged from r342589, vendor/zstd/dist/contrib/adaptive-compression/adapt.c vendor/zstd/1.3.8/contrib/adaptive-compression/datagencli.c - copied unchanged from r342589, vendor/zstd/dist/contrib/adaptive-compression/datagencli.c vendor/zstd/1.3.8/contrib/pzstd/Makefile - copied unchanged from r342589, vendor/zstd/dist/contrib/pzstd/Makefile vendor/zstd/1.3.8/contrib/seekable_format/examples/parallel_processing.c - copied unchanged from r342589, vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c vendor/zstd/1.3.8/contrib/seekable_format/zstdseek_compress.c - copied unchanged from r342589, vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c vendor/zstd/1.3.8/contrib/seekable_format/zstdseek_decompress.c - copied unchanged from r342589, vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c vendor/zstd/1.3.8/doc/zstd_compression_format.md - copied unchanged from r342589, vendor/zstd/dist/doc/zstd_compression_format.md vendor/zstd/1.3.8/doc/zstd_manual.html - copied unchanged from r342589, vendor/zstd/dist/doc/zstd_manual.html vendor/zstd/1.3.8/lib/BUCK - copied unchanged from r342589, vendor/zstd/dist/lib/BUCK vendor/zstd/1.3.8/lib/Makefile - copied unchanged from r342589, vendor/zstd/dist/lib/Makefile vendor/zstd/1.3.8/lib/README.md - copied unchanged from r342589, vendor/zstd/dist/lib/README.md vendor/zstd/1.3.8/lib/common/bitstream.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/bitstream.h vendor/zstd/1.3.8/lib/common/compiler.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/compiler.h vendor/zstd/1.3.8/lib/common/cpu.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/cpu.h vendor/zstd/1.3.8/lib/common/debug.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/debug.h vendor/zstd/1.3.8/lib/common/error_private.c - copied unchanged from r342589, vendor/zstd/dist/lib/common/error_private.c vendor/zstd/1.3.8/lib/common/fse.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/fse.h vendor/zstd/1.3.8/lib/common/huf.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/huf.h vendor/zstd/1.3.8/lib/common/pool.c - copied unchanged from r342589, vendor/zstd/dist/lib/common/pool.c vendor/zstd/1.3.8/lib/common/zstd_common.c - copied unchanged from r342589, vendor/zstd/dist/lib/common/zstd_common.c vendor/zstd/1.3.8/lib/common/zstd_errors.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/zstd_errors.h vendor/zstd/1.3.8/lib/common/zstd_internal.h - copied unchanged from r342589, vendor/zstd/dist/lib/common/zstd_internal.h vendor/zstd/1.3.8/lib/compress/fse_compress.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/fse_compress.c vendor/zstd/1.3.8/lib/compress/hist.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/hist.c vendor/zstd/1.3.8/lib/compress/hist.h - copied unchanged from r342589, vendor/zstd/dist/lib/compress/hist.h vendor/zstd/1.3.8/lib/compress/huf_compress.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/huf_compress.c vendor/zstd/1.3.8/lib/compress/zstd_compress.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_compress.c vendor/zstd/1.3.8/lib/compress/zstd_compress_internal.h - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_compress_internal.h vendor/zstd/1.3.8/lib/compress/zstd_double_fast.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_double_fast.c vendor/zstd/1.3.8/lib/compress/zstd_fast.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_fast.c vendor/zstd/1.3.8/lib/compress/zstd_lazy.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_lazy.c vendor/zstd/1.3.8/lib/compress/zstd_ldm.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_ldm.c vendor/zstd/1.3.8/lib/compress/zstd_ldm.h - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_ldm.h vendor/zstd/1.3.8/lib/compress/zstd_opt.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_opt.c vendor/zstd/1.3.8/lib/compress/zstd_opt.h - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstd_opt.h vendor/zstd/1.3.8/lib/compress/zstdmt_compress.c - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstdmt_compress.c vendor/zstd/1.3.8/lib/compress/zstdmt_compress.h - copied unchanged from r342589, vendor/zstd/dist/lib/compress/zstdmt_compress.h vendor/zstd/1.3.8/lib/decompress/huf_decompress.c - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/huf_decompress.c vendor/zstd/1.3.8/lib/decompress/zstd_decompress.c - copied unchanged from r342589, vendor/zstd/dist/lib/decompress/zstd_decompress.c vendor/zstd/1.3.8/lib/dictBuilder/cover.c - copied unchanged from r342589, vendor/zstd/dist/lib/dictBuilder/cover.c vendor/zstd/1.3.8/lib/dictBuilder/fastcover.c - copied unchanged from r342589, vendor/zstd/dist/lib/dictBuilder/fastcover.c vendor/zstd/1.3.8/lib/dictBuilder/zdict.c - copied unchanged from r342589, vendor/zstd/dist/lib/dictBuilder/zdict.c vendor/zstd/1.3.8/lib/legacy/zstd_v04.c - copied unchanged from r342589, vendor/zstd/dist/lib/legacy/zstd_v04.c vendor/zstd/1.3.8/lib/legacy/zstd_v05.c - copied unchanged from r342589, vendor/zstd/dist/lib/legacy/zstd_v05.c vendor/zstd/1.3.8/lib/zstd.h - copied unchanged from r342589, vendor/zstd/dist/lib/zstd.h vendor/zstd/1.3.8/programs/Makefile - copied unchanged from r342589, vendor/zstd/dist/programs/Makefile vendor/zstd/1.3.8/programs/README.md - copied unchanged from r342589, vendor/zstd/dist/programs/README.md vendor/zstd/1.3.8/programs/datagen.c - copied unchanged from r342589, vendor/zstd/dist/programs/datagen.c vendor/zstd/1.3.8/programs/dibio.c - copied unchanged from r342589, vendor/zstd/dist/programs/dibio.c vendor/zstd/1.3.8/programs/fileio.c - copied unchanged from r342589, vendor/zstd/dist/programs/fileio.c vendor/zstd/1.3.8/programs/fileio.h - copied unchanged from r342589, vendor/zstd/dist/programs/fileio.h vendor/zstd/1.3.8/programs/platform.h - copied unchanged from r342589, vendor/zstd/dist/programs/platform.h vendor/zstd/1.3.8/programs/util.h - copied unchanged from r342589, vendor/zstd/dist/programs/util.h vendor/zstd/1.3.8/programs/zstd.1 - copied unchanged from r342589, vendor/zstd/dist/programs/zstd.1 vendor/zstd/1.3.8/programs/zstd.1.md - copied unchanged from r342589, vendor/zstd/dist/programs/zstd.1.md vendor/zstd/1.3.8/programs/zstdcli.c - copied unchanged from r342589, vendor/zstd/dist/programs/zstdcli.c vendor/zstd/1.3.8/programs/zstdgrep - copied unchanged from r342589, vendor/zstd/dist/programs/zstdgrep vendor/zstd/1.3.8/programs/zstdgrep.1 - copied unchanged from r342589, vendor/zstd/dist/programs/zstdgrep.1 vendor/zstd/1.3.8/programs/zstdless.1 - copied unchanged from r342589, vendor/zstd/dist/programs/zstdless.1 vendor/zstd/1.3.8/tests/Makefile - copied unchanged from r342589, vendor/zstd/dist/tests/Makefile vendor/zstd/1.3.8/tests/README.md - copied unchanged from r342589, vendor/zstd/dist/tests/README.md vendor/zstd/1.3.8/tests/datagencli.c - copied unchanged from r342589, vendor/zstd/dist/tests/datagencli.c vendor/zstd/1.3.8/tests/decodecorpus.c - copied unchanged from r342589, vendor/zstd/dist/tests/decodecorpus.c vendor/zstd/1.3.8/tests/fullbench.c - copied unchanged from r342589, vendor/zstd/dist/tests/fullbench.c vendor/zstd/1.3.8/tests/fuzz/Makefile - copied unchanged from r342589, vendor/zstd/dist/tests/fuzz/Makefile vendor/zstd/1.3.8/tests/fuzz/simple_round_trip.c - copied unchanged from r342589, vendor/zstd/dist/tests/fuzz/simple_round_trip.c vendor/zstd/1.3.8/tests/fuzz/stream_round_trip.c - copied unchanged from r342589, vendor/zstd/dist/tests/fuzz/stream_round_trip.c vendor/zstd/1.3.8/tests/fuzz/zstd_helpers.c - copied unchanged from r342589, vendor/zstd/dist/tests/fuzz/zstd_helpers.c vendor/zstd/1.3.8/tests/fuzzer.c - copied unchanged from r342589, vendor/zstd/dist/tests/fuzzer.c vendor/zstd/1.3.8/tests/libzstd_partial_builds.sh - copied unchanged from r342589, vendor/zstd/dist/tests/libzstd_partial_builds.sh vendor/zstd/1.3.8/tests/longmatch.c - copied unchanged from r342589, vendor/zstd/dist/tests/longmatch.c vendor/zstd/1.3.8/tests/paramgrill.c - copied unchanged from r342589, vendor/zstd/dist/tests/paramgrill.c vendor/zstd/1.3.8/tests/playTests.sh - copied unchanged from r342589, vendor/zstd/dist/tests/playTests.sh vendor/zstd/1.3.8/tests/poolTests.c - copied unchanged from r342589, vendor/zstd/dist/tests/poolTests.c vendor/zstd/1.3.8/tests/roundTripCrash.c - copied unchanged from r342589, vendor/zstd/dist/tests/roundTripCrash.c vendor/zstd/1.3.8/tests/seqgen.c - copied unchanged from r342589, vendor/zstd/dist/tests/seqgen.c vendor/zstd/1.3.8/tests/symbols.c - copied unchanged from r342589, vendor/zstd/dist/tests/symbols.c vendor/zstd/1.3.8/tests/zstreamtest.c - copied unchanged from r342589, vendor/zstd/dist/tests/zstreamtest.c vendor/zstd/1.3.8/zlibWrapper/Makefile - copied unchanged from r342589, vendor/zstd/dist/zlibWrapper/Makefile vendor/zstd/1.3.8/zlibWrapper/examples/zwrapbench.c - copied unchanged from r342589, vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c vendor/zstd/1.3.8/zlibWrapper/zstd_zlibwrapper.c - copied unchanged from r342589, vendor/zstd/dist/zlibWrapper/zstd_zlibwrapper.c Deleted: vendor/zstd/1.3.8/NEWS vendor/zstd/1.3.8/contrib/meson/ vendor/zstd/1.3.8/programs/bench.c vendor/zstd/1.3.8/programs/bench.h Copied: vendor/zstd/1.3.8/Makefile (from r342589, vendor/zstd/dist/Makefile) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.3.8/Makefile Sat Dec 29 06:52:52 2018 (r342590, copy of r342589, vendor/zstd/dist/Makefile) @@ -0,0 +1,396 @@ +# ################################################################ +# Copyright (c) 2015-present, Yann Collet, Facebook, Inc. +# All rights reserved. +# +# This source code is licensed under both the BSD-style license (found in the +# LICENSE file in the root directory of this source tree) and the GPLv2 (found +# in the COPYING file in the root directory of this source tree). +# ################################################################ + +PRGDIR = programs +ZSTDDIR = lib +BUILDIR = build +ZWRAPDIR = zlibWrapper +TESTDIR = tests +FUZZDIR = $(TESTDIR)/fuzz + +# Define nul output +VOID = /dev/null + +ifneq (,$(filter Windows%,$(OS))) +EXT =.exe +else +EXT = +endif + +## default: Build lib-release and zstd-release +.PHONY: default +default: lib-release zstd-release + +.PHONY: all +all: allmost examples manual contrib + +.PHONY: allmost +allmost: allzstd zlibwrapper + +# skip zwrapper, can't build that on alternate architectures without the proper zlib installed +.PHONY: allzstd +allzstd: lib + $(MAKE) -C $(PRGDIR) all + $(MAKE) -C $(TESTDIR) all + +.PHONY: all32 +all32: + $(MAKE) -C $(PRGDIR) zstd32 + $(MAKE) -C $(TESTDIR) all32 + +.PHONY: lib lib-release libzstd.a +lib lib-release : + @$(MAKE) -C $(ZSTDDIR) $@ + +.PHONY: zstd zstd-release +zstd zstd-release: + @$(MAKE) -C $(PRGDIR) $@ + cp $(PRGDIR)/zstd$(EXT) . + +.PHONY: zstdmt +zstdmt: + @$(MAKE) -C $(PRGDIR) $@ + cp $(PRGDIR)/zstd$(EXT) ./zstdmt$(EXT) + +.PHONY: zlibwrapper +zlibwrapper: lib + $(MAKE) -C $(ZWRAPDIR) all + +## test: run long-duration tests +.PHONY: test +DEBUGLEVEL ?= 1 +test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror +test: + MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants + $(MAKE) -C $(TESTDIR) $@ + +## shortest: same as `make check` +.PHONY: shortest +shortest: + $(MAKE) -C $(TESTDIR) $@ + +## check: run basic tests for `zstd` cli +.PHONY: check +check: shortest + +## examples: build all examples in `/examples` directory +.PHONY: examples +examples: lib + CPPFLAGS=-I../lib LDFLAGS=-L../lib $(MAKE) -C examples/ all + +## manual: generate API documentation in html format +.PHONY: manual +manual: + $(MAKE) -C contrib/gen_html $@ + +## man: generate man page +.PHONY: man +man: + $(MAKE) -C programs $@ + +## contrib: build all supported projects in `/contrib` directory +.PHONY: contrib +contrib: lib + $(MAKE) -C contrib/pzstd all + $(MAKE) -C contrib/seekable_format/examples all + $(MAKE) -C contrib/adaptive-compression all + $(MAKE) -C contrib/largeNbDicts all + +.PHONY: cleanTabs +cleanTabs: + cd contrib; ./cleanTabs + +.PHONY: clean +clean: + @$(MAKE) -C $(ZSTDDIR) $@ > $(VOID) + @$(MAKE) -C $(PRGDIR) $@ > $(VOID) + @$(MAKE) -C $(TESTDIR) $@ > $(VOID) + @$(MAKE) -C $(ZWRAPDIR) $@ > $(VOID) + @$(MAKE) -C examples/ $@ > $(VOID) + @$(MAKE) -C contrib/gen_html $@ > $(VOID) + @$(MAKE) -C contrib/pzstd $@ > $(VOID) + @$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID) + @$(MAKE) -C contrib/adaptive-compression $@ > $(VOID) + @$(MAKE) -C contrib/largeNbDicts $@ > $(VOID) + @$(RM) zstd$(EXT) zstdmt$(EXT) tmp* + @$(RM) -r lz4 + @echo Cleaning completed + +#------------------------------------------------------------------------------ +# make install is validated only for Linux, macOS, Hurd and some BSD targets +#------------------------------------------------------------------------------ +ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD DragonFly NetBSD MSYS_NT Haiku)) + +HOST_OS = POSIX +CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release + +HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0) +EGREP_OPTIONS ?= +ifeq ($HAVE_COLORNEVER, 1) +EGREP_OPTIONS += --color=never +endif +EGREP = egrep $(EGREP_OPTIONS) + +# Print a two column output of targets and their description. To add a target description, put a +# comment in the Makefile with the format "## : ". For example: +# +## list: Print all targets and their descriptions (if provided) +.PHONY: list +list: + @TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \ + | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \ + | $(EGREP) -v -e '^[^[:alnum:]]' | sort); \ + { \ + printf "Target Name\tDescription\n"; \ + printf "%0.s-" {1..16}; printf "\t"; printf "%0.s-" {1..40}; printf "\n"; \ + for target in $$TARGETS; do \ + line=$$($(EGREP) "^##[[:space:]]+$$target:" $(lastword $(MAKEFILE_LIST))); \ + description=$$(echo $$line | awk '{i=index($$0,":"); print substr($$0,i+1)}' | xargs); \ + printf "$$target\t$$description\n"; \ + done \ + } | column -t -s $$'\t' + +.PHONY: install clangtest armtest usan asan uasan +install: + @$(MAKE) -C $(ZSTDDIR) $@ + @$(MAKE) -C $(PRGDIR) $@ + +.PHONY: uninstall +uninstall: + @$(MAKE) -C $(ZSTDDIR) $@ + @$(MAKE) -C $(PRGDIR) $@ + +.PHONY: travis-install +travis-install: + $(MAKE) install PREFIX=~/install_test_dir + +.PHONY: gcc5build +gcc5build: clean + gcc-5 -v + CC=gcc-5 $(MAKE) all MOREFLAGS="-Werror" + +.PHONY: gcc6build +gcc6build: clean + gcc-6 -v + CC=gcc-6 $(MAKE) all MOREFLAGS="-Werror" + +.PHONY: gcc7build +gcc7build: clean + gcc-7 -v + CC=gcc-7 $(MAKE) all MOREFLAGS="-Werror" + +.PHONY: clangbuild +clangbuild: clean + clang -v + CXX=clang++ CC=clang $(MAKE) all MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" + +m32build: clean + gcc -v + $(MAKE) all32 + +armbuild: clean + CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allzstd + +aarch64build: clean + CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allzstd + +ppcbuild: clean + CC=powerpc-linux-gnu-gcc CFLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd + +ppc64build: clean + CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allzstd + +armfuzz: clean + CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +aarch64fuzz: clean + ld -v + CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +ppcfuzz: clean + CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +ppc64fuzz: clean + CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +.PHONY: cxxtest +cxxtest: CXXFLAGS += -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror +cxxtest: clean + $(MAKE) -C $(PRGDIR) all CC="$(CXX) -Wno-deprecated" CFLAGS="$(CXXFLAGS)" # adding -Wno-deprecated to avoid clang++ warning on dealing with C files directly + +gcc5test: clean + gcc-5 -v + $(MAKE) all CC=gcc-5 MOREFLAGS="-Werror" + +gcc6test: clean + gcc-6 -v + $(MAKE) all CC=gcc-6 MOREFLAGS="-Werror" + +clangtest: clean + clang -v + $(MAKE) all CXX=clang++ CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" + +armtest: clean + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static ZSTDRTTEST= MOREFLAGS="-Werror -static" FUZZER_FLAGS=--no-big-tests + +aarch64test: + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static ZSTDRTTEST= MOREFLAGS="-Werror -static" FUZZER_FLAGS=--no-big-tests + +ppctest: clean + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static" FUZZER_FLAGS=--no-big-tests + +ppc64test: clean + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests + +arm-ppc-compilation: + $(MAKE) -C $(PRGDIR) clean zstd CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static ZSTDRTTEST= MOREFLAGS="-Werror -static" + $(MAKE) -C $(PRGDIR) clean zstd CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static ZSTDRTTEST= MOREFLAGS="-Werror -static" + $(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static" + $(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static" + +regressiontest: + $(MAKE) -C $(FUZZDIR) regressiontest + +uasanregressiontest: + $(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=address,undefined" CXXFLAGS="-O3 -fsanitize=address,undefined" + +msanregressiontest: + $(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=memory" CXXFLAGS="-O3 -fsanitize=memory" + +# run UBsan with -fsanitize-recover=signed-integer-overflow +# due to a bug in UBsan when doing pointer subtraction +# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63303 + +usan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=undefined -Werror" + +asan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=address -Werror" + +asan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=address -Werror" $(MAKE) -C $(TESTDIR) $* + +msan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=memory -fno-omit-frame-pointer -Werror" HAVE_LZMA=0 # datagen.c fails this test for no obvious reason + +msan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=memory -fno-omit-frame-pointer -Werror" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) HAVE_LZMA=0 $* + +asan32: clean + $(MAKE) -C $(TESTDIR) test32 CC=clang MOREFLAGS="-g -fsanitize=address" + +uasan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=address,undefined -Werror" + +uasan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=address,undefined -Werror" $(MAKE) -C $(TESTDIR) $* + +tsan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=thread -Werror" $(MAKE) -C $(TESTDIR) $* FUZZER_FLAGS=--no-big-tests + +apt-install: + sudo apt-get -yq --no-install-suggests --no-install-recommends --force-yes install $(APT_PACKAGES) + +apt-add-repo: + sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test + sudo apt-get update -y -qq + +ppcinstall: + APT_PACKAGES="qemu-system-ppc qemu-user-static gcc-powerpc-linux-gnu" $(MAKE) apt-install + +arminstall: + APT_PACKAGES="qemu-system-arm qemu-user-static gcc-arm-linux-gnueabi libc6-dev-armel-cross gcc-aarch64-linux-gnu libc6-dev-arm64-cross" $(MAKE) apt-install + +valgrindinstall: + APT_PACKAGES="valgrind" $(MAKE) apt-install + +libc6install: + APT_PACKAGES="libc6-dev-i386 gcc-multilib" $(MAKE) apt-install + +gcc6install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-6 gcc-6-multilib" $(MAKE) apt-install + +gcc7install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-7 gcc-7-multilib" $(MAKE) apt-install + +gcc8install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-8 gcc-8-multilib" $(MAKE) apt-install + +gpp6install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 g++-multilib gcc-6 g++-6 g++-6-multilib" $(MAKE) apt-install + +clang38install: + APT_PACKAGES="clang-3.8" $(MAKE) apt-install + +# Ubuntu 14.04 ships a too-old lz4 +lz4install: + [ -e lz4 ] || git clone https://github.com/lz4/lz4 && sudo $(MAKE) -C lz4 install + +endif + + +ifneq (,$(filter MSYS%,$(shell uname))) +HOST_OS = MSYS +CMAKE_PARAMS = -G"MSYS Makefiles" -DZSTD_MULTITHREAD_SUPPORT:BOOL=OFF -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON +endif + + +#------------------------------------------------------------------------ +# target specific tests +#------------------------------------------------------------------------ +ifneq (,$(filter $(HOST_OS),MSYS POSIX)) +cmakebuild: + cmake --version + $(RM) -r $(BUILDIR)/cmake/build + mkdir $(BUILDIR)/cmake/build + cd $(BUILDIR)/cmake/build ; cmake -DCMAKE_INSTALL_PREFIX:PATH=~/install_test_dir $(CMAKE_PARAMS) .. ; $(MAKE) install ; $(MAKE) uninstall + +c90build: clean + $(CC) -v + CFLAGS="-std=c90 -Werror" $(MAKE) allmost # will fail, due to missing support for `long long` + +gnu90build: clean + $(CC) -v + CFLAGS="-std=gnu90 -Werror" $(MAKE) allmost + +c99build: clean + $(CC) -v + CFLAGS="-std=c99 -Werror" $(MAKE) allmost + +gnu99build: clean + $(CC) -v + CFLAGS="-std=gnu99 -Werror" $(MAKE) allmost + +c11build: clean + $(CC) -v + CFLAGS="-std=c11 -Werror" $(MAKE) allmost + +bmix64build: clean + $(CC) -v + CFLAGS="-O3 -mbmi -Werror" $(MAKE) -C $(TESTDIR) test + +bmix32build: clean + $(CC) -v + CFLAGS="-O3 -mbmi -mx32 -Werror" $(MAKE) -C $(TESTDIR) test + +bmi32build: clean + $(CC) -v + CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(TESTDIR) test + +# static analyzer test uses clang's scan-build +# does not analyze zlibWrapper, due to detected issues in zlib source code +staticAnalyze: SCANBUILD ?= scan-build +staticAnalyze: + $(CC) -v + CC=$(CC) CPPFLAGS=-g $(SCANBUILD) --status-bugs -v $(MAKE) allzstd examples contrib +endif Copied: vendor/zstd/1.3.8/README.md (from r342589, vendor/zstd/dist/README.md) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.3.8/README.md Sat Dec 29 06:52:52 2018 (r342590, copy of r342589, vendor/zstd/dist/README.md) @@ -0,0 +1,167 @@ +

Zstandard

+ +__Zstandard__, or `zstd` as short version, is a fast lossless compression algorithm, +targeting real-time compression scenarios at zlib-level and better compression ratios. +It's backed by a very fast entropy stage, provided by [Huff0 and FSE library](https://github.com/Cyan4973/FiniteStateEntropy). + +The project is provided as an open-source dual [BSD](LICENSE) and [GPLv2](COPYING) licensed **C** library, +and a command line utility producing and decoding `.zst`, `.gz`, `.xz` and `.lz4` files. +Should your project require another programming language, +a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages). + +**Development branch status:** + +[![Build Status][travisDevBadge]][travisLink] +[![Build status][AppveyorDevBadge]][AppveyorLink] +[![Build status][CircleDevBadge]][CircleLink] + +[travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" +[travisLink]: https://travis-ci.org/facebook/zstd +[AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true "Windows test suite" +[AppveyorLink]: https://ci.appveyor.com/project/YannCollet/zstd-p0yf0 +[CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite" +[CircleLink]: https://circleci.com/gh/facebook/zstd + +## Benchmarks + +For reference, several fast compression algorithms were tested and compared +on a server running Linux Debian (`Linux version 4.14.0-3-amd64`), +with a Core i7-6700K CPU @ 4.0GHz, +using [lzbench], an open-source in-memory benchmark by @inikep +compiled with [gcc] 7.3.0, +on the [Silesia compression corpus]. + +[lzbench]: https://github.com/inikep/lzbench +[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia +[gcc]: https://gcc.gnu.org/ + +| Compressor name | Ratio | Compression| Decompress.| +| --------------- | ------| -----------| ---------- | +| **zstd 1.3.4 -1** | 2.877 | 470 MB/s | 1380 MB/s | +| zlib 1.2.11 -1 | 2.743 | 110 MB/s | 400 MB/s | +| brotli 1.0.2 -0 | 2.701 | 410 MB/s | 430 MB/s | +| quicklz 1.5.0 -1 | 2.238 | 550 MB/s | 710 MB/s | +| lzo1x 2.09 -1 | 2.108 | 650 MB/s | 830 MB/s | +| lz4 1.8.1 | 2.101 | 750 MB/s | 3700 MB/s | +| snappy 1.1.4 | 2.091 | 530 MB/s | 1800 MB/s | +| lzf 3.6 -1 | 2.077 | 400 MB/s | 860 MB/s | + +[zlib]: http://www.zlib.net/ +[LZ4]: http://www.lz4.org/ + +Zstd can also offer stronger compression ratios at the cost of compression speed. +Speed vs Compression trade-off is configurable by small increments. +Decompression speed is preserved and remains roughly the same at all settings, +a property shared by most LZ compression algorithms, such as [zlib] or lzma. + +The following tests were run +on a server running Linux Debian (`Linux version 4.14.0-3-amd64`) +with a Core i7-6700K CPU @ 4.0GHz, +using [lzbench], an open-source in-memory benchmark by @inikep +compiled with [gcc] 7.3.0, +on the [Silesia compression corpus]. + +Compression Speed vs Ratio | Decompression Speed +---------------------------|-------------------- +![Compression Speed vs Ratio](doc/images/CSpeed2.png "Compression Speed vs Ratio") | ![Decompression Speed](doc/images/DSpeed3.png "Decompression Speed") + +A few other algorithms can produce higher compression ratios at slower speeds, falling outside of the graph. +For a larger picture including slow modes, [click on this link](doc/images/DCspeed5.png). + + +## The case for Small Data compression + +Previous charts provide results applicable to typical file and stream scenarios (several MB). Small data comes with different perspectives. + +The smaller the amount of data to compress, the more difficult it is to compress. This problem is common to all compression algorithms, and reason is, compression algorithms learn from past data how to compress future data. But at the beginning of a new data set, there is no "past" to build upon. + +To solve this situation, Zstd offers a __training mode__, which can be used to tune the algorithm for a selected type of data. +Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. +Using this dictionary, the compression ratio achievable on small data improves dramatically. + +The following example uses the `github-users` [sample set](https://github.com/facebook/zstd/releases/tag/v1.1.3), created from [github public API](https://developer.github.com/v3/users/#get-all-users). +It consists of roughly 10K records weighing about 1KB each. + +Compression Ratio | Compression Speed | Decompression Speed +------------------|-------------------|-------------------- +![Compression Ratio](doc/images/dict-cr.png "Compression Ratio") | ![Compression Speed](doc/images/dict-cs.png "Compression Speed") | ![Decompression Speed](doc/images/dict-ds.png "Decompression Speed") + + +These compression gains are achieved while simultaneously providing _faster_ compression and decompression speeds. + +Training works if there is some correlation in a family of small data samples. The more data-specific a dictionary is, the more efficient it is (there is no _universal dictionary_). +Hence, deploying one dictionary per type of data will provide the greatest benefits. +Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file. + +### Dictionary compression How To: + +1. Create the dictionary + + `zstd --train FullPathToTrainingSet/* -o dictionaryName` + +2. Compress with dictionary + + `zstd -D dictionaryName FILE` + +3. Decompress with dictionary + + `zstd -D dictionaryName --decompress FILE.zst` + + +## Build instructions + +### Makefile + +If your system is compatible with standard `make` (or `gmake`), +invoking `make` in root directory will generate `zstd` cli in root directory. + +Other available options include: +- `make install` : create and install zstd cli, library and man pages +- `make check` : create and run `zstd`, tests its behavior on local platform + +### cmake + +A `cmake` project generator is provided within `build/cmake`. +It can generate Makefiles or other build scripts +to create `zstd` binary, and `libzstd` dynamic and static libraries. + +By default, `CMAKE_BUILD_TYPE` is set to `Release`. + +### Meson + +A Meson project is provided within [`build/meson`](build/meson). Follow +build instructions in that directory. + +You can also take a look at [`.travis.yml`](.travis.yml) file for an +example about how Meson is used to build this project. + +Note that default build type is **release**. + +### Visual Studio (Windows) + +Going into `build` directory, you will find additional possibilities: +- Projects for Visual Studio 2005, 2008 and 2010. + + VS2010 project is compatible with VS2012, VS2013, VS2015 and VS2017. +- Automated build scripts for Visual compiler by [@KrzysFR](https://github.com/KrzysFR), in `build/VS_scripts`, + which will build `zstd` cli and `libzstd` library without any need to open Visual Studio solution. + +### Buck + +You can build the zstd binary via buck by executing: `buck build programs:zstd` from the root of the repo. +The output binary will be in `buck-out/gen/programs/`. + +## Status + +Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases. +Zstandard is considered safe for production environments. + +## License + +Zstandard is dual-licensed under [BSD](LICENSE) and [GPLv2](COPYING). + +## Contributing + +The "dev" branch is the one where all contributions are merged before reaching "master". +If you plan to propose a patch, please commit into the "dev" branch, or its own feature branch. +Direct commit to "master" are not permitted. +For more information, please read [CONTRIBUTING](CONTRIBUTING.md). Copied: vendor/zstd/1.3.8/appveyor.yml (from r342589, vendor/zstd/dist/appveyor.yml) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.3.8/appveyor.yml Sat Dec 29 06:52:52 2018 (r342590, copy of r342589, vendor/zstd/dist/appveyor.yml) @@ -0,0 +1,251 @@ +- + version: 1.0.{build} + branches: + only: + - master + - appveyorTest + - /visual*/ + environment: + matrix: + - COMPILER: "gcc" + HOST: "mingw" + PLATFORM: "x64" + SCRIPT: "make allzstd MOREFLAGS=-static && make -C tests test-symbols fullbench-lib" + ARTIFACT: "true" + BUILD: "true" + - COMPILER: "gcc" + HOST: "mingw" + PLATFORM: "x86" + SCRIPT: "make allzstd MOREFLAGS=-static" + ARTIFACT: "true" + BUILD: "true" + - COMPILER: "clang" + HOST: "mingw" + PLATFORM: "x64" + SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allzstd" + BUILD: "true" + + - COMPILER: "gcc" + HOST: "mingw" + PLATFORM: "x64" + SCRIPT: "" + TEST: "cmake" + + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "x64" + CONFIGURATION: "Debug" + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "Win32" + CONFIGURATION: "Debug" + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "x64" + CONFIGURATION: "Release" + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "Win32" + CONFIGURATION: "Release" + + install: + - ECHO Installing %COMPILER% %PLATFORM% %CONFIGURATION% + - SET PATH_ORIGINAL=%PATH% + - if [%HOST%]==[mingw] ( + SET "PATH_MINGW32=C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin" && + SET "PATH_MINGW64=C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin" && + COPY C:\msys64\usr\bin\make.exe C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin\make.exe && + COPY C:\msys64\usr\bin\make.exe C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin\make.exe + ) + - IF [%HOST%]==[visual] IF [%PLATFORM%]==[x64] ( + SET ADDITIONALPARAM=/p:LibraryPath="C:\Program Files\Microsoft SDKs\Windows\v7.1\lib\x64;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib\amd64;C:\Program Files (x86)\Microsoft Visual Studio 10.0\;C:\Program Files (x86)\Microsoft Visual Studio 10.0\lib\amd64;" + ) + + build_script: + - if [%HOST%]==[mingw] ( + ( if [%PLATFORM%]==[x64] ( + SET "PATH=%PATH_MINGW64%;%PATH_ORIGINAL%" + ) else if [%PLATFORM%]==[x86] ( + SET "PATH=%PATH_MINGW32%;%PATH_ORIGINAL%" + ) ) + ) + - if [%HOST%]==[mingw] if [%BUILD%]==[true] ( + make -v && + sh -c "%COMPILER% -v" && + ECHO Building zlib to static link && + SET "CC=%COMPILER%" && + sh -c "cd .. && git clone --depth 1 --branch v1.2.11 https://github.com/madler/zlib" && + sh -c "cd ../zlib && make -f win32/Makefile.gcc libz.a" + ECHO Building zstd && + SET "CPPFLAGS=-I../../zlib" && + SET "LDFLAGS=../../zlib/libz.a" && + sh -c "%SCRIPT%" && + ( if [%COMPILER%]==[gcc] if [%ARTIFACT%]==[true] + ECHO Creating artifacts && + ECHO %cd% && + lib\dll\example\build_package.bat && + make -C programs DEBUGFLAGS= clean zstd && + cd programs\ && 7z a -tzip -mx9 zstd-win-binary-%PLATFORM%.zip zstd.exe && + appveyor PushArtifact zstd-win-binary-%PLATFORM%.zip && + cp zstd.exe ..\bin\zstd.exe && + git clone --depth 1 --branch master https://github.com/facebook/zstd && + cd zstd && + git archive --format=tar master -o zstd-src.tar && + ..\zstd -19 zstd-src.tar && + appveyor PushArtifact zstd-src.tar.zst && + certUtil -hashfile zstd-src.tar.zst SHA256 > zstd-src.tar.zst.sha256.sig && + appveyor PushArtifact zstd-src.tar.zst.sha256.sig && + cd ..\..\bin\ && + 7z a -tzip -mx9 zstd-win-release-%PLATFORM%.zip * && + appveyor PushArtifact zstd-win-release-%PLATFORM%.zip + ) + ) + - if [%HOST%]==[visual] ( + ECHO *** && + ECHO *** Building Visual Studio 2008 %PLATFORM%\%CONFIGURATION% in %APPVEYOR_BUILD_FOLDER% && + ECHO *** && + msbuild "build\VS2008\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v90 /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2008\bin\%PLATFORM%\%CONFIGURATION%\*.exe && + MD5sum build/VS2008/bin/%PLATFORM%/%CONFIGURATION%/*.exe && + COPY build\VS2008\bin\%PLATFORM%\%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2008_%PLATFORM%_%CONFIGURATION%.exe && + ECHO *** && + ECHO *** Building Visual Studio 2010 %PLATFORM%\%CONFIGURATION% && + ECHO *** && + msbuild "build\VS2010\zstd.sln" %ADDITIONALPARAM% /m /verbosity:minimal /property:PlatformToolset=v100 /p:ForceImportBeforeCppTargets=%APPVEYOR_BUILD_FOLDER%\build\VS2010\CompileAsCpp.props /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + msbuild "build\VS2010\zstd.sln" %ADDITIONALPARAM% /m /verbosity:minimal /property:PlatformToolset=v100 /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2010_%PLATFORM%_%CONFIGURATION%.exe && + ECHO *** && + ECHO *** Building Visual Studio 2012 %PLATFORM%\%CONFIGURATION% && + ECHO *** && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v110 /p:ForceImportBeforeCppTargets=%APPVEYOR_BUILD_FOLDER%\build\VS2010\CompileAsCpp.props /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v110 /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2012_%PLATFORM%_%CONFIGURATION%.exe && + ECHO *** && + ECHO *** Building Visual Studio 2013 %PLATFORM%\%CONFIGURATION% && + ECHO *** && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v120 /p:ForceImportBeforeCppTargets=%APPVEYOR_BUILD_FOLDER%\build\VS2010\CompileAsCpp.props /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v120 /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2013_%PLATFORM%_%CONFIGURATION%.exe && + ECHO *** && + ECHO *** Building Visual Studio 2015 %PLATFORM%\%CONFIGURATION% && + ECHO *** && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v140 /p:ForceImportBeforeCppTargets=%APPVEYOR_BUILD_FOLDER%\build\VS2010\CompileAsCpp.props /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v140 /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2015_%PLATFORM%_%CONFIGURATION%.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe tests\ + ) + + test_script: + - ECHO Testing %COMPILER% %PLATFORM% %CONFIGURATION% + - SET "CC=gcc" + - SET "CXX=g++" + - if [%TEST%]==[cmake] ( + mkdir build\cmake\build && + cd build\cmake\build && + cmake -G "Visual Studio 14 2015 Win64" .. && + cd ..\..\.. && + make clean + ) + - SET "FUZZERTEST=-T30s" + - if [%HOST%]==[visual] if [%CONFIGURATION%]==[Release] ( + CD tests && + SET ZSTD=./zstd.exe && + sh -e playTests.sh --test-large-data && + fullbench.exe -i1 && + fullbench.exe -i1 -P0 && + fuzzer_VS2008_%PLATFORM%_Release.exe %FUZZERTEST% && + fuzzer_VS2010_%PLATFORM%_Release.exe %FUZZERTEST% && + fuzzer_VS2012_%PLATFORM%_Release.exe %FUZZERTEST% && + fuzzer_VS2013_%PLATFORM%_Release.exe %FUZZERTEST% && + fuzzer_VS2015_%PLATFORM%_Release.exe %FUZZERTEST% + ) + +- + version: 1.0.{build} + environment: + matrix: + - COMPILER: "gcc" + HOST: "mingw" + PLATFORM: "x64" + SCRIPT: "CPPFLAGS=-DDEBUGLEVEL=2 CFLAGS=-Werror make -j allzstd DEBUGLEVEL=2" + - COMPILER: "gcc" + HOST: "mingw" + PLATFORM: "x86" + SCRIPT: "CFLAGS=-Werror make -j allzstd" + - COMPILER: "clang" + HOST: "mingw" + PLATFORM: "x64" + SCRIPT: "CFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make -j allzstd" + + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "x64" + CONFIGURATION: "Debug" + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "Win32" + CONFIGURATION: "Debug" + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "x64" + CONFIGURATION: "Release" + - COMPILER: "visual" + HOST: "visual" + PLATFORM: "Win32" + CONFIGURATION: "Release" + + install: + - ECHO Installing %COMPILER% %PLATFORM% %CONFIGURATION% + - SET PATH_ORIGINAL=%PATH% + - if [%HOST%]==[mingw] ( + SET "PATH_MINGW32=C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin" && + SET "PATH_MINGW64=C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin" && + COPY C:\msys64\usr\bin\make.exe C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin\make.exe && + COPY C:\msys64\usr\bin\make.exe C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin\make.exe + ) + - IF [%HOST%]==[visual] IF [%PLATFORM%]==[x64] ( + SET ADDITIONALPARAM=/p:LibraryPath="C:\Program Files\Microsoft SDKs\Windows\v7.1\lib\x64;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib\amd64;C:\Program Files (x86)\Microsoft Visual Studio 10.0\;C:\Program Files (x86)\Microsoft Visual Studio 10.0\lib\amd64;" + ) + + build_script: + - ECHO Building %COMPILER% %PLATFORM% %CONFIGURATION% + - if [%HOST%]==[mingw] ( + ( if [%PLATFORM%]==[x64] ( + SET "PATH=%PATH_MINGW64%;%PATH_ORIGINAL%" + ) else if [%PLATFORM%]==[x86] ( + SET "PATH=%PATH_MINGW32%;%PATH_ORIGINAL%" + ) ) && + make -v && + sh -c "%COMPILER% -v" && + set "CC=%COMPILER%" && + sh -c "%SCRIPT%" + ) + - if [%HOST%]==[visual] ( + ECHO *** && + ECHO *** Building Visual Studio 2015 %PLATFORM%\%CONFIGURATION% && + ECHO *** && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v140 /p:ForceImportBeforeCppTargets=%APPVEYOR_BUILD_FOLDER%\build\VS2010\CompileAsCpp.props /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + msbuild "build\VS2010\zstd.sln" /m /verbosity:minimal /property:PlatformToolset=v140 /t:Clean,Build /p:Platform=%PLATFORM% /p:Configuration=%CONFIGURATION% /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" && + DIR build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe && + MD5sum build/VS2010/bin/%PLATFORM%_%CONFIGURATION%/*.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\fuzzer.exe tests\fuzzer_VS2015_%PLATFORM%_%CONFIGURATION%.exe && + COPY build\VS2010\bin\%PLATFORM%_%CONFIGURATION%\*.exe tests\ + ) Copied: vendor/zstd/1.3.8/contrib/adaptive-compression/Makefile (from r342589, vendor/zstd/dist/contrib/adaptive-compression/Makefile) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.3.8/contrib/adaptive-compression/Makefile Sat Dec 29 06:52:52 2018 (r342590, copy of r342589, vendor/zstd/dist/contrib/adaptive-compression/Makefile) @@ -0,0 +1,76 @@ + +ZSTDDIR = ../../lib +PRGDIR = ../../programs +ZSTDCOMMON_FILES := $(ZSTDDIR)/common/*.c +ZSTDCOMP_FILES := $(ZSTDDIR)/compress/*.c +ZSTDDECOMP_FILES := $(ZSTDDIR)/decompress/*.c +ZSTD_FILES := $(ZSTDDECOMP_FILES) $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES) + +MULTITHREAD_LDFLAGS = -pthread +DEBUGFLAGS= -g -DZSTD_DEBUG=1 +CPPFLAGS += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \ + -I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(PRGDIR) +CFLAGS ?= -O3 +CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \ + -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \ + -Wstrict-prototypes -Wundef -Wformat-security \ + -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \ + -Wredundant-decls +CFLAGS += $(DEBUGFLAGS) +CFLAGS += $(MOREFLAGS) +FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITHREAD_LDFLAGS) + +all: adapt datagen + +adapt: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c + $(CC) $(FLAGS) $^ -o $@ + +adapt-debug: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c + $(CC) $(FLAGS) -DDEBUG_MODE=2 $^ -o adapt + +datagen : $(PRGDIR)/datagen.c datagencli.c + $(CC) $(FLAGS) $^ -o $@ + +test-adapt-correctness: datagen adapt + @./test-correctness.sh + @echo "test correctness complete" + +test-adapt-performance: datagen adapt + @./test-performance.sh + @echo "test performance complete" + +clean: + @$(RM) -f adapt datagen + @$(RM) -rf *.dSYM + @$(RM) -f tmp* + @$(RM) -f tests/*.zst + @$(RM) -f tests/tmp* + @echo "finished cleaning" + +#----------------------------------------------------------------------------- +# make install is validated only for Linux, macOS, BSD, Hurd and Solaris targets +#----------------------------------------------------------------------------- +ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS)) + +ifneq (,$(filter $(shell uname),SunOS)) +INSTALL ?= ginstall +else +INSTALL ?= install +endif + +PREFIX ?= /usr/local +DESTDIR ?= +BINDIR ?= $(PREFIX)/bin + +INSTALL_PROGRAM ?= $(INSTALL) -m 755 + +install: adapt + @echo Installing binaries + @$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)/ + @$(INSTALL_PROGRAM) adapt $(DESTDIR)$(BINDIR)/zstd-adaptive + @echo zstd-adaptive installation completed + +uninstall: + @$(RM) $(DESTDIR)$(BINDIR)/zstd-adaptive + @echo zstd-adaptive programs successfully uninstalled +endif Copied: vendor/zstd/1.3.8/contrib/adaptive-compression/adapt.c (from r342589, vendor/zstd/dist/contrib/adaptive-compression/adapt.c) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.3.8/contrib/adaptive-compression/adapt.c Sat Dec 29 06:52:52 2018 (r342590, copy of r342589, vendor/zstd/dist/contrib/adaptive-compression/adapt.c) @@ -0,0 +1,1128 @@ +/* + * Copyright (c) 2017-present, Facebook, Inc. + * All rights reserved. + * + * This source code is licensed under both the BSD-style license (found in the + * LICENSE file in the root directory of this source tree) and the GPLv2 (found + * in the COPYING file in the root directory of this source tree). + */ + +#include /* fprintf */ +#include /* malloc, free */ +#include /* pthread functions */ +#include /* memset */ +#include "zstd_internal.h" +#include "util.h" + +#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) +#define PRINT(...) fprintf(stdout, __VA_ARGS__) +#define DEBUG(l, ...) { if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } } +#define FILE_CHUNK_SIZE 4 << 20 +#define MAX_NUM_JOBS 2 +#define stdinmark "/*stdin*\\" +#define stdoutmark "/*stdout*\\" +#define MAX_PATH 256 +#define DEFAULT_DISPLAY_LEVEL 1 +#define DEFAULT_COMPRESSION_LEVEL 6 +#define MAX_COMPRESSION_LEVEL_CHANGE 2 +#define CONVERGENCE_LOWER_BOUND 5 +#define CLEVEL_DECREASE_COOLDOWN 5 +#define CHANGE_BY_TWO_THRESHOLD 0.1 +#define CHANGE_BY_ONE_THRESHOLD 0.65 + +#ifndef DEBUG_MODE +static int g_displayLevel = DEFAULT_DISPLAY_LEVEL; +#else +static int g_displayLevel = DEBUG_MODE; +#endif + +static unsigned g_compressionLevel = DEFAULT_COMPRESSION_LEVEL; +static UTIL_time_t g_startTime; +static size_t g_streamedSize = 0; +static unsigned g_useProgressBar = 1; +static unsigned g_forceCompressionLevel = 0; +static unsigned g_minCLevel = 1; +static unsigned g_maxCLevel; + +typedef struct { + void* start; + size_t size; + size_t capacity; +} buffer_t; + +typedef struct { + size_t filled; + buffer_t buffer; +} inBuff_t; + +typedef struct { + buffer_t src; + buffer_t dst; + unsigned jobID; + unsigned lastJobPlusOne; + size_t compressedSize; + size_t dictSize; +} jobDescription; + +typedef struct { + pthread_mutex_t pMutex; + int noError; +} mutex_t; + +typedef struct { + pthread_cond_t pCond; + int noError; +} cond_t; + +typedef struct { + unsigned compressionLevel; + unsigned numJobs; + unsigned nextJobID; + unsigned threadError; + + /* + * JobIDs for the next jobs to be created, compressed, and written + */ + unsigned jobReadyID; + unsigned jobCompressedID; + unsigned jobWriteID; + unsigned allJobsCompleted; + + /* + * counter for how many jobs in a row the compression level has not changed + * if the counter becomes >= CONVERGENCE_LOWER_BOUND, the next time the + * compression level tries to change (by non-zero amount) resets the counter + * to 1 and does not apply the change *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sat Dec 29 06:51:14 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD7011435177; Sat, 29 Dec 2018 06:51:13 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59CC58927E; Sat, 29 Dec 2018 06:51:13 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 35F5973AB; Sat, 29 Dec 2018 06:51:13 +0000 (UTC) (envelope-from cem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBT6pDux006153; Sat, 29 Dec 2018 06:51:13 GMT (envelope-from cem@FreeBSD.org) Received: (from cem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBT6pAxw006140; Sat, 29 Dec 2018 06:51:11 GMT (envelope-from cem@FreeBSD.org) Message-Id: <201812290651.wBT6pAxw006140@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: cem set sender to cem@FreeBSD.org using -f From: Conrad Meyer Date: Sat, 29 Dec 2018 06:51:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r342589 - in vendor/zstd/dist: . contrib/adaptive-compression contrib/meson contrib/pzstd contrib/seekable_format contrib/seekable_format/examples doc lib lib/common lib/compress lib/de... X-SVN-Group: vendor X-SVN-Commit-Author: cem X-SVN-Commit-Paths: in vendor/zstd/dist: . contrib/adaptive-compression contrib/meson contrib/pzstd contrib/seekable_format contrib/seekable_format/examples doc lib lib/common lib/compress lib/decompress lib/dictBuilder ... X-SVN-Commit-Revision: 342589 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 59CC58927E X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.998,0] X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2018 06:51:14 -0000 Author: cem Date: Sat Dec 29 06:51:10 2018 New Revision: 342589 URL: https://svnweb.freebsd.org/changeset/base/342589 Log: import zstd 1.3.8 Added: vendor/zstd/dist/lib/decompress/zstd_ddict.c (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_ddict.h (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_decompress_block.c (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_decompress_block.h (contents, props changed) vendor/zstd/dist/lib/decompress/zstd_decompress_internal.h (contents, props changed) vendor/zstd/dist/programs/benchfn.c (contents, props changed) vendor/zstd/dist/programs/benchfn.h (contents, props changed) vendor/zstd/dist/programs/benchzstd.c (contents, props changed) vendor/zstd/dist/programs/benchzstd.h (contents, props changed) vendor/zstd/dist/programs/util.c (contents, props changed) vendor/zstd/dist/tests/regression/ vendor/zstd/dist/tests/regression/Makefile (contents, props changed) vendor/zstd/dist/tests/regression/config.c (contents, props changed) vendor/zstd/dist/tests/regression/config.h (contents, props changed) vendor/zstd/dist/tests/regression/data.c (contents, props changed) vendor/zstd/dist/tests/regression/data.h (contents, props changed) vendor/zstd/dist/tests/regression/levels.h (contents, props changed) vendor/zstd/dist/tests/regression/method.c (contents, props changed) vendor/zstd/dist/tests/regression/method.h (contents, props changed) vendor/zstd/dist/tests/regression/result.c (contents, props changed) vendor/zstd/dist/tests/regression/result.h (contents, props changed) vendor/zstd/dist/tests/regression/results.csv vendor/zstd/dist/tests/regression/test.c (contents, props changed) Deleted: vendor/zstd/dist/NEWS vendor/zstd/dist/contrib/meson/ vendor/zstd/dist/programs/bench.c vendor/zstd/dist/programs/bench.h Modified: vendor/zstd/dist/Makefile vendor/zstd/dist/README.md vendor/zstd/dist/appveyor.yml vendor/zstd/dist/contrib/adaptive-compression/Makefile vendor/zstd/dist/contrib/adaptive-compression/adapt.c vendor/zstd/dist/contrib/adaptive-compression/datagencli.c vendor/zstd/dist/contrib/pzstd/Makefile vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c vendor/zstd/dist/doc/zstd_compression_format.md vendor/zstd/dist/doc/zstd_manual.html vendor/zstd/dist/lib/BUCK vendor/zstd/dist/lib/Makefile vendor/zstd/dist/lib/README.md vendor/zstd/dist/lib/common/bitstream.h vendor/zstd/dist/lib/common/compiler.h vendor/zstd/dist/lib/common/cpu.h vendor/zstd/dist/lib/common/debug.h vendor/zstd/dist/lib/common/error_private.c vendor/zstd/dist/lib/common/fse.h vendor/zstd/dist/lib/common/huf.h vendor/zstd/dist/lib/common/pool.c vendor/zstd/dist/lib/common/zstd_common.c vendor/zstd/dist/lib/common/zstd_errors.h vendor/zstd/dist/lib/common/zstd_internal.h vendor/zstd/dist/lib/compress/fse_compress.c vendor/zstd/dist/lib/compress/hist.c vendor/zstd/dist/lib/compress/hist.h vendor/zstd/dist/lib/compress/huf_compress.c vendor/zstd/dist/lib/compress/zstd_compress.c vendor/zstd/dist/lib/compress/zstd_compress_internal.h vendor/zstd/dist/lib/compress/zstd_double_fast.c vendor/zstd/dist/lib/compress/zstd_fast.c vendor/zstd/dist/lib/compress/zstd_lazy.c vendor/zstd/dist/lib/compress/zstd_ldm.c vendor/zstd/dist/lib/compress/zstd_ldm.h vendor/zstd/dist/lib/compress/zstd_opt.c vendor/zstd/dist/lib/compress/zstd_opt.h vendor/zstd/dist/lib/compress/zstdmt_compress.c vendor/zstd/dist/lib/compress/zstdmt_compress.h vendor/zstd/dist/lib/decompress/huf_decompress.c vendor/zstd/dist/lib/decompress/zstd_decompress.c vendor/zstd/dist/lib/dictBuilder/cover.c vendor/zstd/dist/lib/dictBuilder/fastcover.c vendor/zstd/dist/lib/dictBuilder/zdict.c vendor/zstd/dist/lib/legacy/zstd_v04.c vendor/zstd/dist/lib/legacy/zstd_v05.c vendor/zstd/dist/lib/zstd.h vendor/zstd/dist/programs/Makefile vendor/zstd/dist/programs/README.md vendor/zstd/dist/programs/datagen.c vendor/zstd/dist/programs/dibio.c vendor/zstd/dist/programs/fileio.c vendor/zstd/dist/programs/fileio.h vendor/zstd/dist/programs/platform.h vendor/zstd/dist/programs/util.h vendor/zstd/dist/programs/zstd.1 vendor/zstd/dist/programs/zstd.1.md vendor/zstd/dist/programs/zstdcli.c vendor/zstd/dist/programs/zstdgrep vendor/zstd/dist/programs/zstdgrep.1 vendor/zstd/dist/programs/zstdless.1 vendor/zstd/dist/tests/Makefile vendor/zstd/dist/tests/README.md vendor/zstd/dist/tests/datagencli.c vendor/zstd/dist/tests/decodecorpus.c vendor/zstd/dist/tests/fullbench.c vendor/zstd/dist/tests/fuzz/Makefile vendor/zstd/dist/tests/fuzz/simple_round_trip.c vendor/zstd/dist/tests/fuzz/stream_round_trip.c vendor/zstd/dist/tests/fuzz/zstd_helpers.c vendor/zstd/dist/tests/fuzzer.c vendor/zstd/dist/tests/libzstd_partial_builds.sh vendor/zstd/dist/tests/longmatch.c vendor/zstd/dist/tests/paramgrill.c vendor/zstd/dist/tests/playTests.sh vendor/zstd/dist/tests/poolTests.c vendor/zstd/dist/tests/roundTripCrash.c vendor/zstd/dist/tests/seqgen.c vendor/zstd/dist/tests/symbols.c vendor/zstd/dist/tests/zstreamtest.c vendor/zstd/dist/zlibWrapper/Makefile vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c vendor/zstd/dist/zlibWrapper/zstd_zlibwrapper.c Modified: vendor/zstd/dist/Makefile ============================================================================== --- vendor/zstd/dist/Makefile Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/Makefile Sat Dec 29 06:51:10 2018 (r342589) @@ -64,7 +64,8 @@ zlibwrapper: lib ## test: run long-duration tests .PHONY: test -test: MOREFLAGS += -g -DDEBUGLEVEL=1 -Werror +DEBUGLEVEL ?= 1 +test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror test: MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants $(MAKE) -C $(TESTDIR) $@ @@ -129,7 +130,12 @@ ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFree HOST_OS = POSIX CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release -EGREP = egrep --color=never +HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0) +EGREP_OPTIONS ?= +ifeq ($HAVE_COLORNEVER, 1) +EGREP_OPTIONS += --color=never +endif +EGREP = egrep $(EGREP_OPTIONS) # Print a two column output of targets and their description. To add a target description, put a # comment in the Makefile with the format "## : ". For example: Modified: vendor/zstd/dist/README.md ============================================================================== --- vendor/zstd/dist/README.md Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/README.md Sat Dec 29 06:51:10 2018 (r342589) @@ -9,8 +9,12 @@ and a command line utility producing and decoding `.zs Should your project require another programming language, a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages). -Development branch status : [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] [![Build status][CircleDevBadge]][CircleLink] +**Development branch status:** +[![Build Status][travisDevBadge]][travisLink] +[![Build status][AppveyorDevBadge]][AppveyorLink] +[![Build status][CircleDevBadge]][CircleLink] + [travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" [travisLink]: https://travis-ci.org/facebook/zstd [AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true "Windows test suite" @@ -18,7 +22,7 @@ Development branch status : [![Build Status][travisDev [CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite" [CircleLink]: https://circleci.com/gh/facebook/zstd -### Benchmarks +## Benchmarks For reference, several fast compression algorithms were tested and compared on a server running Linux Debian (`Linux version 4.14.0-3-amd64`), @@ -42,7 +46,7 @@ on the [Silesia compression corpus]. | snappy 1.1.4 | 2.091 | 530 MB/s | 1800 MB/s | | lzf 3.6 -1 | 2.077 | 400 MB/s | 860 MB/s | -[zlib]:http://www.zlib.net/ +[zlib]: http://www.zlib.net/ [LZ4]: http://www.lz4.org/ Zstd can also offer stronger compression ratios at the cost of compression speed. @@ -65,7 +69,7 @@ A few other algorithms can produce higher compression For a larger picture including slow modes, [click on this link](doc/images/DCspeed5.png). -### The case for Small Data compression +## The case for Small Data compression Previous charts provide results applicable to typical file and stream scenarios (several MB). Small data comes with different perspectives. @@ -89,24 +93,24 @@ Training works if there is some correlation in a famil Hence, deploying one dictionary per type of data will provide the greatest benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file. -#### Dictionary compression How To: +### Dictionary compression How To: -1) Create the dictionary +1. Create the dictionary -`zstd --train FullPathToTrainingSet/* -o dictionaryName` + `zstd --train FullPathToTrainingSet/* -o dictionaryName` -2) Compress with dictionary +2. Compress with dictionary -`zstd -D dictionaryName FILE` + `zstd -D dictionaryName FILE` -3) Decompress with dictionary +3. Decompress with dictionary -`zstd -D dictionaryName --decompress FILE.zst` + `zstd -D dictionaryName --decompress FILE.zst` -### Build instructions +## Build instructions -#### Makefile +### Makefile If your system is compatible with standard `make` (or `gmake`), invoking `make` in root directory will generate `zstd` cli in root directory. @@ -115,7 +119,7 @@ Other available options include: - `make install` : create and install zstd cli, library and man pages - `make check` : create and run `zstd`, tests its behavior on local platform -#### cmake +### cmake A `cmake` project generator is provided within `build/cmake`. It can generate Makefiles or other build scripts @@ -123,29 +127,39 @@ to create `zstd` binary, and `libzstd` dynamic and sta By default, `CMAKE_BUILD_TYPE` is set to `Release`. -#### Meson +### Meson -A Meson project is provided within `contrib/meson`. +A Meson project is provided within [`build/meson`](build/meson). Follow +build instructions in that directory. -#### Visual Studio (Windows) +You can also take a look at [`.travis.yml`](.travis.yml) file for an +example about how Meson is used to build this project. +Note that default build type is **release**. + +### Visual Studio (Windows) + Going into `build` directory, you will find additional possibilities: - Projects for Visual Studio 2005, 2008 and 2010. + VS2010 project is compatible with VS2012, VS2013, VS2015 and VS2017. - Automated build scripts for Visual compiler by [@KrzysFR](https://github.com/KrzysFR), in `build/VS_scripts`, which will build `zstd` cli and `libzstd` library without any need to open Visual Studio solution. +### Buck -### Status +You can build the zstd binary via buck by executing: `buck build programs:zstd` from the root of the repo. +The output binary will be in `buck-out/gen/programs/`. +## Status + Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases. Zstandard is considered safe for production environments. -### License +## License Zstandard is dual-licensed under [BSD](LICENSE) and [GPLv2](COPYING). -### Contributing +## Contributing The "dev" branch is the one where all contributions are merged before reaching "master". If you plan to propose a patch, please commit into the "dev" branch, or its own feature branch. Modified: vendor/zstd/dist/appveyor.yml ============================================================================== --- vendor/zstd/dist/appveyor.yml Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/appveyor.yml Sat Dec 29 06:51:10 2018 (r342589) @@ -3,6 +3,8 @@ branches: only: - master + - appveyorTest + - /visual*/ environment: matrix: - COMPILER: "gcc" Modified: vendor/zstd/dist/contrib/adaptive-compression/Makefile ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/Makefile Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/adaptive-compression/Makefile Sat Dec 29 06:51:10 2018 (r342589) @@ -22,10 +22,10 @@ FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITH all: adapt datagen -adapt: $(ZSTD_FILES) adapt.c +adapt: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c $(CC) $(FLAGS) $^ -o $@ -adapt-debug: $(ZSTD_FILES) adapt.c +adapt-debug: $(ZSTD_FILES) $(PRGDIR)/util.c adapt.c $(CC) $(FLAGS) -DDEBUG_MODE=2 $^ -o adapt datagen : $(PRGDIR)/datagen.c datagencli.c Modified: vendor/zstd/dist/contrib/adaptive-compression/adapt.c ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/adapt.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/adaptive-compression/adapt.c Sat Dec 29 06:51:10 2018 (r342589) @@ -579,7 +579,7 @@ static void* compressionThread(void* arg) params.cParams.windowLog = 23; { size_t const initError = ZSTD_compressBegin_advanced(ctx->cctx, job->src.start + job->dictSize - useDictSize, useDictSize, params, 0); - size_t const windowSizeError = ZSTD_CCtx_setParameter(ctx->cctx, ZSTD_p_forceMaxWindow, 1); + size_t const windowSizeError = ZSTD_CCtx_setParameter(ctx->cctx, ZSTD_c_forceMaxWindow, 1); if (ZSTD_isError(initError) || ZSTD_isError(windowSizeError)) { DISPLAY("Error: something went wrong while starting compression\n"); signalErrorToThreads(ctx); Modified: vendor/zstd/dist/contrib/adaptive-compression/datagencli.c ============================================================================== --- vendor/zstd/dist/contrib/adaptive-compression/datagencli.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/adaptive-compression/datagencli.c Sat Dec 29 06:51:10 2018 (r342589) @@ -120,7 +120,7 @@ int main(int argc, const char** argv) DISPLAYLEVEL(4, "Compressible data Generator \n"); if (probaU32!=COMPRESSIBILITY_DEFAULT) DISPLAYLEVEL(3, "Compressibility : %i%%\n", probaU32); - DISPLAYLEVEL(3, "Seed = %u \n", seed); + DISPLAYLEVEL(3, "Seed = %u \n", (unsigned)seed); RDG_genStdout(size, (double)probaU32/100, litProba, seed); DISPLAYLEVEL(1, "\n"); Modified: vendor/zstd/dist/contrib/pzstd/Makefile ============================================================================== --- vendor/zstd/dist/contrib/pzstd/Makefile Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/pzstd/Makefile Sat Dec 29 06:51:10 2018 (r342589) @@ -171,7 +171,7 @@ roundtripcheck: roundtrip check $(TESTPROG) ./test/RoundTripTest$(EXT) $(TESTFLAGS) # Build the main binary -pzstd$(EXT): main.o Options.o Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a +pzstd$(EXT): main.o $(PROGDIR)/util.o Options.o Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a $(LD_COMMAND) # Target that depends on all the tests Modified: vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c ============================================================================== --- vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/seekable_format/examples/parallel_processing.c Sat Dec 29 06:51:10 2018 (r342589) @@ -148,20 +148,20 @@ static void sumFile_orDie(const char* fname, int nbThr size_t const initResult = ZSTD_seekable_initFile(seekable, fin); if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); } - size_t const numFrames = ZSTD_seekable_getNumFrames(seekable); + unsigned const numFrames = ZSTD_seekable_getNumFrames(seekable); struct sum_job* jobs = (struct sum_job*)malloc(numFrames * sizeof(struct sum_job)); - size_t i; - for (i = 0; i < numFrames; i++) { - jobs[i] = (struct sum_job){ fname, 0, i, 0 }; - POOL_add(pool, sumFrame, &jobs[i]); + unsigned fnb; + for (fnb = 0; fnb < numFrames; fnb++) { + jobs[fnb] = (struct sum_job){ fname, 0, fnb, 0 }; + POOL_add(pool, sumFrame, &jobs[fnb]); } unsigned long long total = 0; - for (i = 0; i < numFrames; i++) { - while (!jobs[i].done) SLEEP(5); /* wake up every 5 milliseconds to check */ - total += jobs[i].sum; + for (fnb = 0; fnb < numFrames; fnb++) { + while (!jobs[fnb].done) SLEEP(5); /* wake up every 5 milliseconds to check */ + total += jobs[fnb].sum; } printf("Sum: %llu\n", total); Modified: vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c ============================================================================== --- vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/seekable_format/zstdseek_compress.c Sat Dec 29 06:51:10 2018 (r342589) @@ -8,6 +8,8 @@ */ #include /* malloc, free */ +#include /* UINT_MAX */ +#include #define XXH_STATIC_LINKING_ONLY #define XXH_NAMESPACE ZSTD_ @@ -139,7 +141,7 @@ size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs, int compressionLevel, int checksumFlag, - U32 maxFrameSize) + unsigned maxFrameSize) { zcs->framelog.size = 0; zcs->frameCSize = 0; @@ -167,9 +169,9 @@ size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream } size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl, - unsigned compressedSize, - unsigned decompressedSize, - unsigned checksum) + unsigned compressedSize, + unsigned decompressedSize, + unsigned checksum) { if (fl->size == ZSTD_SEEKABLE_MAXFRAMES) return ERROR(frameIndex_tooLarge); @@ -184,7 +186,8 @@ size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl, if (newEntries == NULL) return ERROR(memory_allocation); fl->entries = newEntries; - fl->capacity = newCapacity; + assert(newCapacity <= UINT_MAX); + fl->capacity = (U32)newCapacity; } fl->entries[fl->size] = (framelogEntry_t){ @@ -268,7 +271,7 @@ size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStr static inline size_t ZSTD_seekable_seekTableSize(const ZSTD_frameLog* fl) { size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0); - size_t const seekTableLen = ZSTD_skippableHeaderSize + + size_t const seekTableLen = ZSTD_SKIPPABLEHEADERSIZE + sizePerFrame * fl->size + ZSTD_seekTableFooterSize; @@ -307,32 +310,32 @@ size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, size_t const seekTableLen = ZSTD_seekable_seekTableSize(fl); CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_MAGIC_SKIPPABLE_START | 0xE, 0)); - CHECK_Z(ZSTD_stwrite32(fl, output, seekTableLen - ZSTD_skippableHeaderSize, - 4)); + assert(seekTableLen <= (size_t)UINT_MAX); + CHECK_Z(ZSTD_stwrite32(fl, output, (U32)seekTableLen - ZSTD_SKIPPABLEHEADERSIZE, 4)); while (fl->seekTableIndex < fl->size) { + unsigned long long const start = ZSTD_SKIPPABLEHEADERSIZE + sizePerFrame * fl->seekTableIndex; + assert(start + 8 <= UINT_MAX); CHECK_Z(ZSTD_stwrite32(fl, output, fl->entries[fl->seekTableIndex].cSize, - ZSTD_skippableHeaderSize + - sizePerFrame * fl->seekTableIndex + 0)); + (U32)start + 0)); CHECK_Z(ZSTD_stwrite32(fl, output, fl->entries[fl->seekTableIndex].dSize, - ZSTD_skippableHeaderSize + - sizePerFrame * fl->seekTableIndex + 4)); + (U32)start + 4)); if (fl->checksumFlag) { CHECK_Z(ZSTD_stwrite32( fl, output, fl->entries[fl->seekTableIndex].checksum, - ZSTD_skippableHeaderSize + - sizePerFrame * fl->seekTableIndex + 8)); + (U32)start + 8)); } fl->seekTableIndex++; } + assert(seekTableLen <= UINT_MAX); CHECK_Z(ZSTD_stwrite32(fl, output, fl->size, - seekTableLen - ZSTD_seekTableFooterSize)); + (U32)seekTableLen - ZSTD_seekTableFooterSize)); if (output->size - output->pos < 1) return seekTableLen - fl->seekTablePos; if (fl->seekTablePos < seekTableLen - 4) { @@ -345,7 +348,7 @@ size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, } CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_SEEKABLE_MAGICNUMBER, - seekTableLen - 4)); + (U32)seekTableLen - 4)); if (fl->seekTablePos != seekTableLen) return ERROR(GENERIC); return 0; Modified: vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c ============================================================================== --- vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/contrib/seekable_format/zstdseek_decompress.c Sat Dec 29 06:51:10 2018 (r342589) @@ -54,8 +54,9 @@ # define LONG_SEEK fseek #endif -#include /* malloc, free */ -#include /* FILE* */ +#include /* malloc, free */ +#include /* FILE* */ +#include /* UNIT_MAX */ #include #define XXH_STATIC_LINKING_ONLY @@ -200,13 +201,14 @@ size_t ZSTD_seekable_free(ZSTD_seekable* zs) * Performs a binary search to find the last frame with a decompressed offset * <= pos * @return : the frame's index */ -U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long pos) +unsigned ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long pos) { U32 lo = 0; - U32 hi = zs->seekTable.tableLen; + U32 hi = (U32)zs->seekTable.tableLen; + assert(zs->seekTable.tableLen <= UINT_MAX); if (pos >= zs->seekTable.entries[zs->seekTable.tableLen].dOffset) { - return zs->seekTable.tableLen; + return (U32)zs->seekTable.tableLen; } while (lo + 1 < hi) { @@ -220,31 +222,32 @@ U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* co return lo; } -U32 ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs) +unsigned ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs) { - return zs->seekTable.tableLen; + assert(zs->seekTable.tableLen <= UINT_MAX); + return (unsigned)zs->seekTable.tableLen; } -unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, U32 frameIndex) +unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE; return zs->seekTable.entries[frameIndex].cOffset; } -unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, U32 frameIndex) +unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE; return zs->seekTable.entries[frameIndex].dOffset; } -size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, U32 frameIndex) +size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge); return zs->seekTable.entries[frameIndex + 1].cOffset - zs->seekTable.entries[frameIndex].cOffset; } -size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, U32 frameIndex) +size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, unsigned frameIndex) { if (frameIndex > zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge); return zs->seekTable.entries[frameIndex + 1].dOffset - @@ -275,7 +278,7 @@ static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekabl { U32 const numFrames = MEM_readLE32(zs->inBuff); U32 const sizePerEntry = 8 + (checksumFlag?4:0); U32 const tableSize = sizePerEntry * numFrames; - U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + ZSTD_skippableHeaderSize; + U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + ZSTD_SKIPPABLEHEADERSIZE; U32 remaining = frameSize - ZSTD_seekTableFooterSize; /* don't need to re-read footer */ { @@ -290,7 +293,7 @@ static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekabl if (MEM_readLE32(zs->inBuff) != (ZSTD_MAGIC_SKIPPABLE_START | 0xE)) { return ERROR(prefix_unknown); } - if (MEM_readLE32(zs->inBuff+4) + ZSTD_skippableHeaderSize != frameSize) { + if (MEM_readLE32(zs->inBuff+4) + ZSTD_SKIPPABLEHEADERSIZE != frameSize) { return ERROR(prefix_unknown); } @@ -444,7 +447,7 @@ size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, voi return len; } -size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, U32 frameIndex) +size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned frameIndex) { if (frameIndex >= zs->seekTable.tableLen) { return ERROR(frameIndex_tooLarge); Modified: vendor/zstd/dist/doc/zstd_compression_format.md ============================================================================== --- vendor/zstd/dist/doc/zstd_compression_format.md Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/doc/zstd_compression_format.md Sat Dec 29 06:51:10 2018 (r342589) @@ -16,7 +16,7 @@ Distribution of this document is unlimited. ### Version -0.3.0 (25/09/18) +0.3.1 (25/10/18) Introduction @@ -913,13 +913,39 @@ Note that blocks which are not `Compressed_Block` are ###### Offset updates rules The newest offset takes the lead in offset history, -shifting others back (up to its previous place if it was already present). +shifting others back by one rank, +up to the previous rank of the new offset _if it was present in history_. -This means that when `Repeated_Offset1` (most recent) is used, history is unmodified. -When `Repeated_Offset2` is used, it's swapped with `Repeated_Offset1`. -If any other offset is used, it becomes `Repeated_Offset1` and the rest are shift back by one. +__Examples__ : +In the common case, when new offset is not part of history : +`Repeated_Offset3` = `Repeated_Offset2` +`Repeated_Offset2` = `Repeated_Offset1` +`Repeated_Offset1` = `NewOffset` +When the new offset _is_ part of history, there may be specific adjustments. + +When `NewOffset` == `Repeated_Offset1`, offset history remains actually unmodified. + +When `NewOffset` == `Repeated_Offset2`, +`Repeated_Offset1` and `Repeated_Offset2` ranks are swapped. +`Repeated_Offset3` is unmodified. + +When `NewOffset` == `Repeated_Offset3`, +there is actually no difference with the common case : +all offsets are shifted by one rank, +`NewOffset` (== `Repeated_Offset3`) becomes the new `Repeated_Offset1`. + +Also worth mentioning, the specific corner case when `offset_value` == 3, +and the literal length of the current sequence is zero. +In which case , `NewOffset` = `Repeated_Offset1` - 1_byte. +Here also, from an offset history update perspective, it's just a common case : +`Repeated_Offset3` = `Repeated_Offset2` +`Repeated_Offset2` = `Repeated_Offset1` +`Repeated_Offset1` = `NewOffset` ( == `Repeated_Offset1` - 1_byte ) + + + Skippable Frames ---------------- @@ -1629,6 +1655,7 @@ or at least provide a meaningful error code explaining Version changes --------------- +- 0.3.1 : minor clarification regarding offset history update rules - 0.3.0 : minor edits to match RFC8478 - 0.2.9 : clarifications for huffman weights direct representation, by Ulrich Kunitz - 0.2.8 : clarifications for IETF RFC discuss Modified: vendor/zstd/dist/doc/zstd_manual.html ============================================================================== --- vendor/zstd/dist/doc/zstd_manual.html Sat Dec 29 03:18:16 2018 (r342588) +++ vendor/zstd/dist/doc/zstd_manual.html Sat Dec 29 06:51:10 2018 (r342589) @@ -1,10 +1,10 @@ -zstd 1.3.7 Manual +zstd 1.3.8 Manual -

zstd 1.3.7 Manual

+

zstd 1.3.8 Manual

Contents

    @@ -19,16 +19,19 @@
  1. Streaming compression - HowTo
  2. Streaming decompression - HowTo
  3. ADVANCED AND EXPERIMENTAL FUNCTIONS
    4. -
  4. Frame size functions
    5. -
  5. Memory management
    6. -
  6. Advanced compression functions
    7. -
  7. Advanced decompression functions
    8. -
  8. Advanced streaming functions
    9. -
  9. Buffer-less and synchronous inner streaming functions
    10. -
  10. Buffer-less streaming compression (synchronous mode)
    11. -
  11. Buffer-less streaming decompression (synchronous mode)
    12. -
  12. New advanced API (experimental)
    13. -
  13. Block level API
    14. +
  14. Candidate API for promotion to stable status
    15. +
  15. Advanced compression API
    16. +
  16. experimental API (static linking only)
    17. +
  17. Frame size functions
    18. +
  18. Memory management
    19. +
  19. Advanced compression functions
    20. +
  20. Advanced decompression functions
    21. +
  21. Advanced streaming functions
    22. +
  22. Buffer-less and synchronous inner streaming functions
    23. +
  23. Buffer-less streaming compression (synchronous mode)
    24. +
  24. Buffer-less streaming decompression (synchronous mode)
    25. +
  25. ZSTD_getFrameHeader() :
    26. +
  26. Block level API

Introduction

@@ -64,7 +67,7 @@
 
 
Version

 
-
unsigned ZSTD_versionNumber(void);   /**< useful to check dll version */
+
unsigned ZSTD_versionNumber(void);   /**< to check runtime library version */
 


 
Default constant

 
@@ -139,11 +142,13 @@ int         ZSTD_maxCLevel(void);               /*
 ZSTD_CCtx* ZSTD_createCCtx(void);
 size_t     ZSTD_freeCCtx(ZSTD_CCtx* cctx);
 


-
size_t ZSTD_compressCCtx(ZSTD_CCtx* ctx,
+
size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
                          void* dst, size_t dstCapacity,
                    const void* src, size_t srcSize,
                          int compressionLevel);
-
  Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see ZSTD_createCCtx()). 
+
  Same as ZSTD_compress(), using an explicit ZSTD_CCtx
+  The function will compress at requested compression level,
+  ignoring any other parameter 
 


 
 
Decompression context
  When decompressing many times,
@@ -155,10 +160,13 @@ size_t     ZSTD_freeCCtx(ZSTD_CCtx* cctx);
 ZSTD_DCtx* ZSTD_createDCtx(void);
 size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
 


-
size_t ZSTD_decompressDCtx(ZSTD_DCtx* ctx,
+
size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
                            void* dst, size_t dstCapacity,
                      const void* src, size_t srcSize);
-
  Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see ZSTD_createDCtx()) 
+
  Same as ZSTD_decompress(),
+  requires an allocated ZSTD_DCtx.
+  Compatible with sticky parameters.
+ 
 


 
 
Simple dictionary API

@@ -168,18 +176,22 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                          const void* src, size_t srcSize,
                          const void* dict,size_t dictSize,
                                int compressionLevel);
-
  Compression using a predefined Dictionary (see dictBuilder/zdict.h).
+
  Compression at an explicit compression level using a Dictionary.
+  A dictionary can be any arbitrary data segment (also called a prefix),
+  or a buffer with specified information (see dictBuilder/zdict.h).
   Note : This function loads the dictionary, resulting in significant startup delay.
-  Note : When `dict == NULL || dictSize < 8` no dictionary is used. 
+         It's intended for a dictionary used only once.
+  Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. 
 


 
 
size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
                                  void* dst, size_t dstCapacity,
                            const void* src, size_t srcSize,
                            const void* dict,size_t dictSize);
-
  Decompression using a predefined Dictionary (see dictBuilder/zdict.h).
+
  Decompression using a known Dictionary.
   Dictionary must be identical to the one used during compression.
   Note : This function loads the dictionary, resulting in significant startup delay.
+         It's intended for a dictionary used only once.
   Note : When `dict == NULL || dictSize < 8` no dictionary is used. 
 


 
@@ -187,11 +199,12 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
 
 
ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
                              int compressionLevel);
-
  When compressing multiple messages / blocks with the same dictionary, it's recommended to load it just once.
-  ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup delay.
+
  When compressing multiple messages / blocks using the same dictionary, it's recommended to load it only once.
+  ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup cost.
   ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.
-  `dictBuffer` can be released after ZSTD_CDict creation, since its content is copied within CDict
-  Note : A ZSTD_CDict can be created with an empty dictionary, but it is inefficient for small data. 
+ `dictBuffer` can be released after ZSTD_CDict creation, because its content is copied within CDict.
+  Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate `dictBuffer` content.
+  Note : A ZSTD_CDict can be created from an empty dictBuffer, but it is inefficient when used to compress small data. 
 


 
 
size_t      ZSTD_freeCDict(ZSTD_CDict* CDict);
@@ -203,16 +216,14 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                           const void* src, size_t srcSize,
                           const ZSTD_CDict* cdict);
 
  Compression using a digested Dictionary.
-  Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times.
-  Note that compression level is decided during dictionary creation.
-  Frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no)
-  Note : ZSTD_compress_usingCDict() can be used with a ZSTD_CDict created from an empty dictionary.
-         But it is inefficient for small data, and it is recommended to use ZSTD_compressCCtx(). 
+  Recommended when same dictionary is used multiple times.
+  Note : compression level is _decided at dictionary creation time_,
+     and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) 
 


 
 
ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize);
 
  Create a digested dictionary, ready to start decompression operation without startup delay.
-  dictBuffer can be released after DDict creation, as its content is copied inside DDict 
+  dictBuffer can be released after DDict creation, as its content is copied inside DDict. 
 


 
 
size_t      ZSTD_freeDDict(ZSTD_DDict* ddict);
@@ -224,7 +235,7 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                             const void* src, size_t srcSize,
                             const ZSTD_DDict* ddict);
 
  Decompression using a digested Dictionary.
-  Faster startup than ZSTD_decompress_usingDict(), recommended when same dictionary is used multiple times. 
+  Recommended when same dictionary is used multiple times. 
 


 
 
Streaming

@@ -245,14 +256,18 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
   A ZSTD_CStream object is required to track streaming operation.
   Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
   ZSTD_CStream objects can be reused multiple times on consecutive compression operations.
-  It is recommended to re-use ZSTD_CStream in situations where many streaming operations will be achieved consecutively,
-  since it will play nicer with system's memory, by re-using already allocated memory.
-  Use one separate ZSTD_CStream per thread for parallel execution.
+  It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory.
 
-  Start a new compression by initializing ZSTD_CStream context.
-  Use ZSTD_initCStream() to start a new compression operation.
-  Use variants ZSTD_initCStream_usingDict() or ZSTD_initCStream_usingCDict() for streaming with dictionary (experimental section)
+  For parallel execution, use one separate ZSTD_CStream per thread.
 
+  note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
+
+  Parameters are sticky : when starting a new compression on the same context,
+  it will re-use the same sticky parameters as previous compression session.
+  When in doubt, it's recommended to fully initialize the context before usage.
+  Use ZSTD_initCStream() to set the parameter to a selected compression level.
+  Use advanced API (ZSTD_CCtx_setParameter(), etc.) to set more specific parameters.
+
   Use ZSTD_compressStream() as many times as necessary to consume input stream.
   The function will automatically update both `pos` fields within `input` and `output`.
   Note that the function may not consume the entire input,
@@ -260,12 +275,11 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
   in which case `input.pos < input.size`.
   The caller must check if input has been entirely consumed.
   If not, the caller must make some room to receive more compressed data,
-  typically by emptying output buffer, or allocating a new output buffer,
   and then present again remaining input data.
-  @return : a size hint, preferred nb of bytes to use as input for next function call
-            or an error code, which can be tested using ZSTD_isError().
-            Note 1 : it's just a hint, to help latency a little, any other value will work fine.
-            Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
+ @return : a size hint, preferred nb of bytes to use as input for next function call
+           or an error code, which can be tested using ZSTD_isError().
+           Note 1 : it's just a hint, to help latency a little, any value will work fine.
+           Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
 
   At any moment, it's possible to flush whatever data might remain stuck within internal buffer,
   using ZSTD_flushStream(). `output->pos` will be updated.
@@ -305,25 +319,24 @@ size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffe
   Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
   ZSTD_DStream objects can be re-used multiple times.
 
-  Use ZSTD_initDStream() to start a new decompression operation,
-   or ZSTD_initDStream_usingDict() if decompression requires a dictionary.
-   @return : recommended first input size
+  Use ZSTD_initDStream() to start a new decompression operation.
+ @return : recommended first input size
+  Alternatively, use advanced API to set specific properties.
 
   Use ZSTD_decompressStream() repetitively to consume your input.
   The function will update both `pos` fields.
   If `input.pos < input.size`, some input has not been consumed.
   It's up to the caller to present again remaining data.
-  The function tries to flush all data decoded immediately, repecting buffer sizes.
+  The function tries to flush all data decoded immediately, respecting output buffer size.
   If `output.pos < output.size`, decoder has flushed everything it could.
-  But if `output.pos == output.size`, there is no such guarantee,
-  it's likely that some decoded data was not flushed and still remains within internal buffers.
+  But if `output.pos == output.size`, there might be some data left within internal buffers.,
   In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
-  When no additional input is provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
+  Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
  @return : 0 when a frame is completely decoded and fully flushed,
         or an error code, which can be tested using ZSTD_isError(),
         or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
-                                the return value is a suggested next input size (a hint for better latency)
-                                that will never load more than the current frame.
+                                the return value is a suggested next input size (just a hint for better latency)
+                                that will never request more than the remaining frame size.
  
 

 
@@ -340,32 +353,477 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_o
 
size_t ZSTD_DStreamOutSize(void);   /*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */
 


 
ADVANCED AND EXPERIMENTAL FUNCTIONS
- The definitions in this section are considered experimental.
- They should never be used with a dynamic library, as prototypes may change in the future.
+ The definitions in the following section are considered experimental.
  They are provided for advanced scenarios.
+ They should never be used with a dynamic library, as prototypes may change in the future.
  Use them only in association with static linking.
  
 

 
+
Candidate API for promotion to stable status
+ The following symbols and constants form the "staging area" :
+ they are considered to join "stable API" by v1.4.0.
+ The proposal is written so that it can be made stable "as is",
+ though it's still possible to suggest improvements.
+ Staging is in fact last chance for changes,
+ the API is locked once reaching "stable" status.
+ 
+

+
 
int ZSTD_minCLevel(void);  /*!< minimum negative compression level allowed */
 


-
typedef enum { ZSTD_fast=1, ZSTD_dfast, ZSTD_greedy, ZSTD_lazy, ZSTD_lazy2,
-               ZSTD_btlazy2, ZSTD_btopt, ZSTD_btultra } ZSTD_strategy;   /* from faster to stronger */
+
size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);
+
 `src` should point to the start of a ZSTD frame or skippable frame.
+ `srcSize` must be >= first frame size
+ @return : the compressed size of the first frame starting at `src`,
+           suitable to pass as `srcSize` to `ZSTD_decompress` or similar,
+        or an error code if input is invalid 
+


+
+
size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
+size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
+size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs);
+size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
+size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
+size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
+
  These functions give the _current_ memory usage of selected object.
+  Note that object memory usage can evolve (increase or decrease) over time. 
+


+
+
Advanced compression API

+
+
typedef enum { ZSTD_fast=1,
+               ZSTD_dfast=2,
+               ZSTD_greedy=3,
+               ZSTD_lazy=4,
+               ZSTD_lazy2=5,
+               ZSTD_btlazy2=6,
+               ZSTD_btopt=7,
+               ZSTD_btultra=8,
+               ZSTD_btultra2=9
+               /* note : new strategies _might_ be added in the future.
+                         Only the order (from fast to strong) is guaranteed */
+} ZSTD_strategy;
 


+
typedef enum {
+
+    /* compression parameters */
+    ZSTD_c_compressionLevel=100, /* Update all compression parameters according to pre-defined cLevel table
+                              * Default level is ZSTD_CLEVEL_DEFAULT==3.
+                              * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
+                              * Note 1 : it's possible to pass a negative compression level.
+                              * Note 2 : setting a level sets all default values of other compression parameters */
+    ZSTD_c_windowLog=101,    /* Maximum allowed back-reference distance, expressed as power of 2.
+                              * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
+                              * Special: value 0 means "use default windowLog".
+                              * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT
+                              *       requires explicitly allowing such window size at decompression stage if using streaming. */
+    ZSTD_c_hashLog=102,      /* Size of the initial probe table, as a power of 2.
+                              * Resulting memory usage is (1 << (hashLog+2)).
+                              * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
+                              * Larger tables improve compression ratio of strategies <= dFast,
+                              * and improve speed of strategies > dFast.
+                              * Special: value 0 means "use default hashLog". */
+    ZSTD_c_chainLog=103,     /* Size of the multi-probe search table, as a power of 2.
+                              * Resulting memory usage is (1 << (chainLog+2)).
+                              * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX.
+                              * Larger tables result in better and slower compression.
+                              * This parameter is useless when using "fast" strategy.
+                              * It's still useful when using "dfast" strategy,
+                              * in which case it defines a secondary probe table.
+                              * Special: value 0 means "use default chainLog". */
+    ZSTD_c_searchLog=104,    /* Number of search attempts, as a power of 2.
+                              * More attempts result in better and slower compression.
+                              * This parameter is useless when using "fast" and "dFast" strategies.
+                              * Special: value 0 means "use default searchLog". */
+    ZSTD_c_minMatch=105,     /* Minimum size of searched matches.
+                              * Note that Zstandard can still find matches of smaller size,
+                              * it just tweaks its search algorithm to look for this size and larger.
+                              * Larger values increase compression and decompression speed, but decrease ratio.
+                              * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX.
+                              * Note that currently, for all strategies < btopt, effective minimum is 4.
+                              *                    , for all strategies > fast, effective maximum is 6.
+                              * Special: value 0 means "use default minMatchLength". */
+    ZSTD_c_targetLength=106, /* Impact of this field depends on strategy.
+                              * For strategies btopt, btultra & btultra2:
+                              *     Length of Match considered "good enough" to stop search.
+                              *     Larger values make compression stronger, and slower.
+                              * For strategy fast:
+                              *     Distance between match sampling.
+                              *     Larger values make compression faster, and weaker.
+                              * Special: value 0 means "use default targetLength". */
+    ZSTD_c_strategy=107,     /* See ZSTD_strategy enum definition.
+                              * The higher the value of selected strategy, the more complex it is,
+                              * resulting in stronger and slower compression.
+                              * Special: value 0 means "use default strategy". */
+
+    /* LDM mode parameters */
+    ZSTD_c_enableLongDistanceMatching=160, /* Enable long distance matching.
+                                     * This parameter is designed to improve compression ratio
+                                     * for large inputs, by finding large matches at long distance.
+                                     * It increases memory usage and window size.
+                                     * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB
+                                     * except when expressly set to a different value. */
+    ZSTD_c_ldmHashLog=161,   /* Size of the table for long distance matching, as a power of 2.
+                              * Larger values increase memory usage and compression ratio,
+                              * but decrease compression speed.
+                              * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX
+                              * default: windowlog - 7.
+                              * Special: value 0 means "automatically determine hashlog". */
+    ZSTD_c_ldmMinMatch=162,  /* Minimum match size for long distance matcher.
+                              * Larger/too small values usually decrease compression ratio.
+                              * Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX.
+                              * Special: value 0 means "use default value" (default: 64). */
+    ZSTD_c_ldmBucketSizeLog=163, /* Log size of each bucket in the LDM hash table for collision resolution.
+                              * Larger values improve collision resolution but decrease compression speed.
+                              * The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX.
+                              * Special: value 0 means "use default value" (default: 3). */
+    ZSTD_c_ldmHashRateLog=164, /* Frequency of inserting/looking up entries into the LDM hash table.
+                              * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
+                              * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
+                              * Larger values improve compression speed.
+                              * Deviating far from default value will likely result in a compression ratio decrease.
+                              * Special: value 0 means "automatically determine hashRateLog". */
+
+    /* frame parameters */
+    ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1)
+                              * Content size must be known at the beginning of compression.
+                              * This is automatically the case when using ZSTD_compress2(),
+                              * For streaming variants, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
+    ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
+    ZSTD_c_dictIDFlag=202,   /* When applicable, dictionary's ID is written into frame header (default:1) */
+
+    /* multi-threading parameters */
+    /* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
+     * They return an error otherwise. */
+    ZSTD_c_nbWorkers=400,    /* Select how many threads will be spawned to compress in parallel.
+                              * When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() :
+                              * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
+                              * while compression work is performed in parallel, within worker threads.
+                              * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
+                              *  in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
+                              * More workers improve speed, but also increase memory usage.
+                              * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */
+    ZSTD_c_jobSize=401,      /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
+                              * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
+                              * 0 means default, which is dynamically determined based on compression parameters.
+                              * Job size must be a minimum of overlap size, or 1 MB, whichever is largest.
+                              * The minimum size is automatically and transparently enforced */
+    ZSTD_c_overlapLog=402,   /* Control the overlap size, as a fraction of window size.
+                              * The overlap size is an amount of data reloaded from previous job at the beginning of a new job.
+                              * It helps preserve compression ratio, while each job is compressed in parallel.
+                              * This value is enforced only when nbWorkers >= 1.
+                              * Larger values increase compression ratio, but decrease speed.
+                              * Possible values range from 0 to 9 :
+                              * - 0 means "default" : value will be determined by the library, depending on strategy
+                              * - 1 means "no overlap"
+                              * - 9 means "full overlap", using a full window size.
+                              * Each intermediate rank increases/decreases load size by a factor 2 :
+                              * 9: full window;  8: w/2;  7: w/4;  6: w/8;  5:w/16;  4: w/32;  3:w/64;  2:w/128;  1:no overlap;  0:default
+                              * default value varies between 6 and 9, depending on strategy */
+
+    /* note : additional experimental parameters are also available
+     * within the experimental section of the API.
+     * At the time of this writing, they include :
+     * ZSTD_c_rsyncable
+     * ZSTD_c_format
+     * ZSTD_c_forceMaxWindow
+     * ZSTD_c_forceAttachDict
+     * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
+     * note : never ever use experimentalParam? names directly;
+     *        also, the enums values themselves are unstable and can still change.
+     */
+     ZSTD_c_experimentalParam1=500,
+     ZSTD_c_experimentalParam2=10,
+     ZSTD_c_experimentalParam3=1000,
+     ZSTD_c_experimentalParam4=1001
+} ZSTD_cParameter;
+


 
typedef struct {
-    unsigned windowLog;      /**< largest match distance : larger == more compression, more memory needed during decompression */
-    unsigned chainLog;       /**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */
-    unsigned hashLog;        /**< dispatch table : larger == faster, more memory */
-    unsigned searchLog;      /**< nb of searches : larger == more compression, slower */
-    unsigned searchLength;   /**< match length searched : larger == faster decompression, sometimes less compression */
-    unsigned targetLength;   /**< acceptable match size for optimal parser (only) : larger == more compression, slower */
-    ZSTD_strategy strategy;
+    size_t error;
+    int lowerBound;
+    int upperBound;
+} ZSTD_bounds;
+


+
ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam);
+
  All parameters must belong to an interval with lower and upper bounds,
+  otherwise they will either trigger an error or be automatically clamped.
+ @return : a structure, ZSTD_bounds, which contains
+         - an error status field, which must be tested using ZSTD_isError()
+         - lower and upper bounds, both inclusive
+ 
+


+
+
size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value);
+
  Set one compression parameter, selected by enum ZSTD_cParameter.
+  All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds().
+  Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
+  Setting a parameter is generally only possible during frame initialization (before starting compression).
+  Exception : when using multi-threading mode (nbWorkers >= 1),
+              the following parameters can be updated _during_ compression (within same frame):
+              => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy.
+              new parameters will be active for next job only (after a flush()).
+ @return : an error code (which can be tested using ZSTD_isError()).
+ 
+


+
+
size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize);
+
  Total input data size to be compressed as a single frame.
+  Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag.
+  This value will also be controlled at end of frame, and trigger an error if not respected.
+ @result : 0, or an error code (which can be tested with ZSTD_isError()).
+  Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
+           In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN.
+           ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
+  Note 2 : pledgedSrcSize is only valid once, for the next frame.
+           It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
+  Note 3 : Whenever all input data is provided and consumed in a single round,
+           for example with ZSTD_compress2(),
+           or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end),
+           this value is automatically overriden by srcSize instead.
+ 
+


+
+
size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
+
  Create an internal CDict from `dict` buffer.
+  Decompression will have to use same dictionary.
+ @result : 0, or an error code (which can be tested with ZSTD_isError()).
+  Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
+           meaning "return to no-dictionary mode".
+  Note 1 : Dictionary is sticky, it will be used for all future compressed frames.

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***