From owner-svn-src-vendor@freebsd.org Sun Sep 10 00:56:30 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DAADE15AEA; Sun, 10 Sep 2017 00:56:30 +0000 (UTC) (envelope-from marius@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DC30083E25; Sun, 10 Sep 2017 00:56:29 +0000 (UTC) (envelope-from marius@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8A0uTsO013584; Sun, 10 Sep 2017 00:56:29 GMT (envelope-from marius@FreeBSD.org) Received: (from marius@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8A0uShH013582; Sun, 10 Sep 2017 00:56:28 GMT (envelope-from marius@FreeBSD.org) Message-Id: <201709100056.v8A0uShH013582@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: marius set sender to marius@FreeBSD.org using -f From: Marius Strobl Date: Sun, 10 Sep 2017 00:56:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323381 - vendor/zlib/dist X-SVN-Group: vendor X-SVN-Commit-Author: marius X-SVN-Commit-Paths: vendor/zlib/dist X-SVN-Commit-Revision: 323381 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Sep 2017 00:56:30 -0000 Author: marius Date: Sun Sep 10 00:56:28 2017 New Revision: 323381 URL: https://svnweb.freebsd.org/changeset/base/323381 Log: Permit a deflateParams() parameter change as soon as possible. This commit allows a parameter change even if the input data has not all been compressed and copied to the application output buffer, so long as all of the input data has been compressed to the internal pending output buffer. This also allows an immediate deflateParams change so long as there have been no deflate calls since initialization or reset. Obtained from: upstream (f9694097dd69354b03cb8af959094c7f260db0a1) Modified: vendor/zlib/dist/deflate.c vendor/zlib/dist/zlib.h Modified: vendor/zlib/dist/deflate.c ============================================================================== --- vendor/zlib/dist/deflate.c Sat Sep 9 22:02:36 2017 (r323380) +++ vendor/zlib/dist/deflate.c Sun Sep 10 00:56:28 2017 (r323381) @@ -494,7 +494,7 @@ int ZEXPORT deflateResetKeep (strm) s->wrap == 2 ? crc32(0L, Z_NULL, 0) : #endif adler32(0L, Z_NULL, 0); - s->last_flush = Z_NO_FLUSH; + s->last_flush = -2; _tr_init(s); @@ -587,12 +587,12 @@ int ZEXPORT deflateParams(strm, level, strategy) func = configuration_table[s->level].func; if ((strategy != s->strategy || func != configuration_table[level].func) && - s->high_water) { + s->last_flush != -2) { /* Flush the last buffer: */ int err = deflate(strm, Z_BLOCK); if (err == Z_STREAM_ERROR) return err; - if (strm->avail_out == 0) + if (strm->avail_in || (s->strstart - s->block_start) + s->lookahead) return Z_BUF_ERROR; } if (s->level != level) { Modified: vendor/zlib/dist/zlib.h ============================================================================== --- vendor/zlib/dist/zlib.h Sat Sep 9 22:02:36 2017 (r323380) +++ vendor/zlib/dist/zlib.h Sun Sep 10 00:56:28 2017 (r323381) @@ -712,11 +712,12 @@ ZEXTERN int ZEXPORT deflateParams OF((z_streamp strm, used to switch between compression and straight copy of the input data, or to switch to a different kind of input data requiring a different strategy. If the compression approach (which is a function of the level) or the - strategy is changed, and if any input has been consumed in a previous - deflate() call, then the input available so far is compressed with the old - level and strategy using deflate(strm, Z_BLOCK). There are three approaches - for the compression levels 0, 1..3, and 4..9 respectively. The new level - and strategy will take effect at the next call of deflate(). + strategy is changed, and if there have been any deflate() calls since the + state was initialized or reset, then the input available so far is + compressed with the old level and strategy using deflate(strm, Z_BLOCK). + There are three approaches for the compression levels 0, 1..3, and 4..9 + respectively. The new level and strategy will take effect at the next call + of deflate(). If a deflate(strm, Z_BLOCK) is performed by deflateParams(), and it does not have enough output space to complete, then the parameter change will not From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:17:15 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55F01E18707; Wed, 13 Sep 2017 10:17:15 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2B36D80D34; Wed, 13 Sep 2017 10:17:15 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAHERl051640; Wed, 13 Sep 2017 10:17:14 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAHE3V051639; Wed, 13 Sep 2017 10:17:14 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131017.v8DAHE3V051639@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:17:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323523 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 323523 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:17:15 -0000 Author: avg Date: Wed Sep 13 10:17:14 2017 New Revision: 323523 URL: https://svnweb.freebsd.org/changeset/base/323523 Log: 8331 zfs_unshare returns wrong error code for smb unshare failure illumos/illumos-gate@4f4378cc54b7deec3a35c529dc397dbdc325b4bb https://github.com/illumos/illumos-gate/commit/4f4378cc54b7deec3a35c529dc397dbdc325b4bb https://www.illumos.org/issues/8331 zfs_unshare returns EZFS_UNSHARENFSFAILED on error for all share types. Reviewed by: Marcel Telka Reviewed by: Toomas Soome Approved by: Dan McDonald Author: Andrew Stormont Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Wed Sep 13 07:09:58 2017 (r323522) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Wed Sep 13 10:17:14 2017 (r323523) @@ -24,6 +24,7 @@ * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2014, 2016 by Delphix. All rights reserved. * Copyright 2016 Igor Kozhukhov + * Copyright 2017 RackTop Systems. */ /* @@ -91,7 +92,7 @@ zfs_share_type_t zfs_is_shared_proto(zfs_handle_t *, c zfs_share_proto_t); /* - * The share protocols table must be in the same order as the zfs_share_prot_t + * The share protocols table must be in the same order as the zfs_share_proto_t * enum in libzfs_impl.h */ typedef struct { @@ -916,7 +917,7 @@ unshare_one(libzfs_handle_t *hdl, const char *name, co if ((err = zfs_init_libshare_arg(hdl, SA_INIT_ONE_SHARE_FROM_NAME, (void *)name)) != SA_OK) { free(mntpt); /* don't need the copy anymore */ - return (zfs_error_fmt(hdl, EZFS_UNSHARENFSFAILED, + return (zfs_error_fmt(hdl, proto_table[proto].p_unshare_err, dgettext(TEXT_DOMAIN, "cannot unshare '%s': %s"), name, _sa_errorstr(err))); } @@ -927,12 +928,13 @@ unshare_one(libzfs_handle_t *hdl, const char *name, co if (share != NULL) { err = zfs_sa_disable_share(share, proto_table[proto].p_name); if (err != SA_OK) { - return (zfs_error_fmt(hdl, EZFS_UNSHARENFSFAILED, + return (zfs_error_fmt(hdl, + proto_table[proto].p_unshare_err, dgettext(TEXT_DOMAIN, "cannot unshare '%s': %s"), name, _sa_errorstr(err))); } } else { - return (zfs_error_fmt(hdl, EZFS_UNSHARENFSFAILED, + return (zfs_error_fmt(hdl, proto_table[proto].p_unshare_err, dgettext(TEXT_DOMAIN, "cannot unshare '%s': not found"), name)); } From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:25:45 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7200E18E61; Wed, 13 Sep 2017 10:25:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 751E781453; Wed, 13 Sep 2017 10:25:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAPi0E055620; Wed, 13 Sep 2017 10:25:44 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAPieT055617; Wed, 13 Sep 2017 10:25:44 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131025.v8DAPieT055617@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:25:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323526 - in vendor-sys/illumos/dist/uts/common: fs fs/zfs sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common: fs fs/zfs sys X-SVN-Commit-Revision: 323526 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:25:45 -0000 Author: avg Date: Wed Sep 13 10:25:44 2017 New Revision: 323526 URL: https://svnweb.freebsd.org/changeset/base/323526 Log: 8376 cached v_path should be kept fresh illumos/illumos-gate@e2fc3408efa6cdfc5e33c73c3567efc8c7592707 https://github.com/illumos/illumos-gate/commit/e2fc3408efa6cdfc5e33c73c3567efc8c7592707 https://www.illumos.org/issues/8376 The logic for generating and maintaining the cached v_path value on vnodes could stand to be improved. If vnodes were purely ephemeral, then freshly calculating v_path at the time of lookup() would result in correct values (at a performance cost). When they persist, either as referenced by other structures (such as open files, process cwd, dnlc entries, etc), the opportunity for the v_path to become stale arises. This is exacerbated by the current behavior that, when v_path is found to be invalid (during a vnodetopath operation) will strive to recalculate it, but not preserve the result. The overall situation leads to both performance and correctness (due to lack of results) problems relating to v_path. This has been addressed in SmartOS through a series of changes. Firstly, to do proper invalidation of v_path when it's found to be stale: - OS-3891 stale v_path slows vfs lookups OS-3891 revealed that some logic made assumptions about v_path never transitioning from non-NULL to NULL. It was addressed here: - OS-4317 v_path accesses can race While the pathological stale v_path behavior had been addressed, there are still cases where the absence of valid v_path information was causing problems. The largest patch in this series addressed it by performing v_path checking and updates during vnode lookups/updates, when it is most convenient: - OS-5167 cached v_path should be kept fresh Two smaller updates are included too, to prevent erroneous behavior introduced by the prior changes: - OS-5846 procfs should follow VFS rules - OS-6134 vn_reinit balks on zeroed vnodes Reviewed by: Jerry Jelinek Reviewed by: Robert Mustacchi Approved by: Gordon Ross Author: Patrick Mooney Modified: vendor-sys/illumos/dist/uts/common/fs/vnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_dir.c vendor-sys/illumos/dist/uts/common/sys/vnode.h Modified: vendor-sys/illumos/dist/uts/common/fs/vnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/vnode.c Wed Sep 13 10:23:55 2017 (r323525) +++ vendor-sys/illumos/dist/uts/common/fs/vnode.c Wed Sep 13 10:25:44 2017 (r323526) @@ -21,7 +21,7 @@ /* * Copyright (c) 1988, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright 2017, Joyent, Inc. * Copyright 2016 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2011, 2017 by Delphix. All rights reserved. */ @@ -68,6 +68,8 @@ #include #include #include +#include +#include /* Determine if this vnode is a file that is read-only */ #define ISROFILE(vp) \ @@ -104,6 +106,9 @@ kmutex_t vskstat_tree_lock; /* Global variable which enables/disables the vopstats collection */ int vopstats_enabled = 1; +/* Global used for empty/invalid v_path */ +char *vn_vpath_empty = ""; + /* * forward declarations for internal vnode specific data (vsd) */ @@ -2286,7 +2291,8 @@ vn_cache_constructor(void *buf, void *cdrarg, int kmfl cv_init(&vp->v_cv, NULL, CV_DEFAULT, NULL); rw_init(&vp->v_nbllock, NULL, RW_DEFAULT, NULL); vp->v_femhead = NULL; /* Must be done before vn_reinit() */ - vp->v_path = NULL; + vp->v_path = vn_vpath_empty; + vp->v_path_stamp = 0; vp->v_mpssdata = NULL; vp->v_vsd = NULL; vp->v_fopdata = NULL; @@ -2333,6 +2339,7 @@ void vn_recycle(vnode_t *vp) { ASSERT(vp->v_pages == NULL); + VERIFY(vp->v_path != NULL); /* * XXX - This really belongs in vn_reinit(), but we have some issues @@ -2355,10 +2362,11 @@ vn_recycle(vnode_t *vp) kmem_free(vp->v_femhead, sizeof (*(vp->v_femhead))); vp->v_femhead = NULL; } - if (vp->v_path) { + if (vp->v_path != vn_vpath_empty) { kmem_free(vp->v_path, strlen(vp->v_path) + 1); - vp->v_path = NULL; + vp->v_path = vn_vpath_empty; } + vp->v_path_stamp = 0; if (vp->v_fopdata != NULL) { free_fopdata(vp); @@ -2396,6 +2404,15 @@ vn_reinit(vnode_t *vp) vp->v_locality = NULL; vp->v_xattrdir = NULL; + /* + * In a few specific instances, vn_reinit() is used to initialize + * locally defined vnode_t instances. Lacking the construction offered + * by vn_alloc(), these vnodes require v_path initialization. + */ + if (vp->v_path == NULL) { + vp->v_path = vn_vpath_empty; + } + /* Handles v_femhead, v_path, and the r/w/map counts */ vn_recycle(vp); } @@ -2429,9 +2446,10 @@ vn_free(vnode_t *vp) */ ASSERT((vp->v_count == 0) || (vp->v_count == 1)); ASSERT(vp->v_count_dnlc == 0); - if (vp->v_path != NULL) { + VERIFY(vp->v_path != NULL); + if (vp->v_path != vn_vpath_empty) { kmem_free(vp->v_path, strlen(vp->v_path) + 1); - vp->v_path = NULL; + vp->v_path = vn_vpath_empty; } /* If FEM was in use, make sure everything gets cleaned up */ @@ -2954,125 +2972,256 @@ fs_new_caller_id() } /* - * Given a starting vnode and a path, updates the path in the target vnode in - * a safe manner. If the vnode already has path information embedded, then the - * cached path is left untouched. + * The value stored in v_path is relative to rootdir, located in the global + * zone. Zones or chroot environments which reside deeper inside the VFS + * hierarchy will have a relative view of MAXPATHLEN since they are unaware of + * what lies below their perceived root. In order to keep v_path usable for + * these child environments, its allocations are allowed to exceed MAXPATHLEN. + * + * An upper bound of max_vnode_path is placed upon v_path allocations to + * prevent the system from going too wild at the behest of pathological + * behavior from the operator. */ - size_t max_vnode_path = 4 * MAXPATHLEN; + void -vn_setpath(vnode_t *rootvp, struct vnode *startvp, struct vnode *vp, - const char *path, size_t plen) +vn_clearpath(vnode_t *vp, hrtime_t compare_stamp) { - char *rpath; - vnode_t *base; - size_t rpathlen, rpathalloc; - int doslash = 1; + char *buf; - if (*path == '/') { - base = rootvp; - path++; - plen--; - } else { - base = startvp; - } - + mutex_enter(&vp->v_lock); /* - * We cannot grab base->v_lock while we hold vp->v_lock because of - * the potential for deadlock. + * If the snapshot of v_path_stamp passed in via compare_stamp does not + * match the present value on the vnode, it indicates that subsequent + * changes have occurred. The v_path value is not cleared in this case + * since the new value may be valid. */ - mutex_enter(&base->v_lock); - if (base->v_path == NULL) { - mutex_exit(&base->v_lock); + if (compare_stamp != 0 && vp->v_path_stamp != compare_stamp) { + mutex_exit(&vp->v_lock); return; } + buf = vp->v_path; + vp->v_path = vn_vpath_empty; + vp->v_path_stamp = 0; + mutex_exit(&vp->v_lock); + if (buf != vn_vpath_empty) { + kmem_free(buf, strlen(buf) + 1); + } +} - rpathlen = strlen(base->v_path); - rpathalloc = rpathlen + plen + 1; - /* Avoid adding a slash if there's already one there */ - if (base->v_path[rpathlen-1] == '/') - doslash = 0; - else - rpathalloc++; +static void +vn_setpath_common(vnode_t *pvp, vnode_t *vp, const char *name, size_t len, + boolean_t is_rename) +{ + char *buf, *oldbuf; + hrtime_t pstamp; + size_t baselen, buflen = 0; - /* - * We don't want to call kmem_alloc(KM_SLEEP) with kernel locks held, - * so we must do this dance. If, by chance, something changes the path, - * just give up since there is no real harm. - */ - mutex_exit(&base->v_lock); + /* Handle the vn_setpath_str case. */ + if (pvp == NULL) { + if (len + 1 > max_vnode_path) { + DTRACE_PROBE4(vn__setpath__too__long, vnode_t *, pvp, + vnode_t *, vp, char *, name, size_t, len + 1); + return; + } + buf = kmem_alloc(len + 1, KM_SLEEP); + bcopy(name, buf, len); + buf[len] = '\0'; - /* Paths should stay within reason */ - if (rpathalloc > max_vnode_path) + mutex_enter(&vp->v_lock); + oldbuf = vp->v_path; + vp->v_path = buf; + vp->v_path_stamp = gethrtime(); + mutex_exit(&vp->v_lock); + if (oldbuf != vn_vpath_empty) { + kmem_free(oldbuf, strlen(oldbuf) + 1); + } return; + } - rpath = kmem_alloc(rpathalloc, KM_SLEEP); + /* Take snapshot of parent dir */ + mutex_enter(&pvp->v_lock); - mutex_enter(&base->v_lock); - if (base->v_path == NULL || strlen(base->v_path) != rpathlen) { - mutex_exit(&base->v_lock); - kmem_free(rpath, rpathalloc); + if ((pvp->v_flag & VTRAVERSE) != 0) { + /* + * When the parent vnode has VTRAVERSE set in its flags, normal + * assumptions about v_path calculation no longer apply. The + * primary situation where this occurs is via the VFS tricks + * which procfs plays in order to allow /proc/PID/(root|cwd) to + * yield meaningful results. + * + * When this flag is set, v_path on the child must not be + * updated since the calculated value is likely to be + * incorrect, given the current context. + */ + mutex_exit(&pvp->v_lock); return; } - bcopy(base->v_path, rpath, rpathlen); - mutex_exit(&base->v_lock); - if (doslash) - rpath[rpathlen++] = '/'; - bcopy(path, rpath + rpathlen, plen); - rpath[rpathlen + plen] = '\0'; +retrybuf: + if (pvp->v_path == vn_vpath_empty) { + /* + * Without v_path from the parent directory, generating a child + * path from the name is impossible. + */ + if (len > 0) { + pstamp = pvp->v_path_stamp; + mutex_exit(&pvp->v_lock); + vn_clearpath(vp, pstamp); + return; + } + /* + * The only feasible case here is where a NUL lookup is being + * performed on rootdir prior to its v_path being populated. + */ + ASSERT(pvp->v_path_stamp == 0); + baselen = 0; + pstamp = 0; + } else { + pstamp = pvp->v_path_stamp; + baselen = strlen(pvp->v_path); + /* ignore a trailing slash if present */ + if (pvp->v_path[baselen - 1] == '/') { + /* This should only the be case for rootdir */ + ASSERT(baselen == 1 && pvp == rootdir); + baselen--; + } + } + mutex_exit(&pvp->v_lock); + + if (buflen != 0) { + /* Free the existing (mis-sized) buffer in case of retry */ + kmem_free(buf, buflen); + } + /* base, '/', name and trailing NUL */ + buflen = baselen + len + 2; + if (buflen > max_vnode_path) { + DTRACE_PROBE4(vn__setpath_too__long, vnode_t *, pvp, + vnode_t *, vp, char *, name, size_t, buflen); + return; + } + buf = kmem_alloc(buflen, KM_SLEEP); + + mutex_enter(&pvp->v_lock); + if (pvp->v_path_stamp != pstamp) { + size_t vlen; + + /* + * Since v_path_stamp changed on the parent, it is likely that + * v_path has been altered as well. If the length does not + * exactly match what was previously measured, the buffer + * allocation must be repeated for proper sizing. + */ + if (pvp->v_path == vn_vpath_empty) { + /* Give up if parent lack v_path */ + mutex_exit(&pvp->v_lock); + kmem_free(buf, buflen); + return; + } + vlen = strlen(pvp->v_path); + if (pvp->v_path[vlen - 1] == '/') { + vlen--; + } + if (vlen != baselen) { + goto retrybuf; + } + } + bcopy(pvp->v_path, buf, baselen); + mutex_exit(&pvp->v_lock); + + buf[baselen] = '/'; + baselen++; + bcopy(name, &buf[baselen], len + 1); + mutex_enter(&vp->v_lock); - if (vp->v_path != NULL) { + if (vp->v_path_stamp == 0) { + /* never-visited vnode can inherit stamp from parent */ + ASSERT(vp->v_path == vn_vpath_empty); + vp->v_path_stamp = pstamp; + vp->v_path = buf; mutex_exit(&vp->v_lock); - kmem_free(rpath, rpathalloc); + } else if (vp->v_path_stamp < pstamp || is_rename) { + /* + * Install the updated path and stamp, ensuring that the v_path + * pointer is valid at all times for dtrace. + */ + oldbuf = vp->v_path; + vp->v_path = buf; + vp->v_path_stamp = gethrtime(); + mutex_exit(&vp->v_lock); + kmem_free(oldbuf, strlen(oldbuf) + 1); } else { - vp->v_path = rpath; + /* + * If the timestamp matches or is greater, it means another + * thread performed the update first while locks were dropped + * here to make the allocation. We defer to the newer value. + */ mutex_exit(&vp->v_lock); + kmem_free(buf, buflen); } + ASSERT(MUTEX_NOT_HELD(&vp->v_lock)); } -/* - * Sets the path to the vnode to be the given string, regardless of current - * context. The string must be a complete path from rootdir. This is only used - * by fsop_root() for setting the path based on the mountpoint. - */ void -vn_setpath_str(struct vnode *vp, const char *str, size_t len) +vn_updatepath(vnode_t *pvp, vnode_t *vp, const char *name) { - char *buf = kmem_alloc(len + 1, KM_SLEEP); + size_t len; - mutex_enter(&vp->v_lock); - if (vp->v_path != NULL) { - mutex_exit(&vp->v_lock); - kmem_free(buf, len + 1); + /* + * If the parent is older or empty, there's nothing further to do. + */ + if (pvp->v_path == vn_vpath_empty || + pvp->v_path_stamp <= vp->v_path_stamp) { return; } - vp->v_path = buf; - bcopy(str, vp->v_path, len); - vp->v_path[len] = '\0'; + /* + * Given the lack of appropriate context, meaningful updates to v_path + * cannot be made for during lookups for the '.' or '..' entries. + */ + len = strlen(name); + if (len == 0 || (len == 1 && name[0] == '.') || + (len == 2 && name[0] == '.' && name[1] == '.')) { + return; + } - mutex_exit(&vp->v_lock); + vn_setpath_common(pvp, vp, name, len, B_FALSE); } /* + * Given a starting vnode and a path, updates the path in the target vnode in + * a safe manner. If the vnode already has path information embedded, then the + * cached path is left untouched. + */ +/* ARGSUSED */ +void +vn_setpath(vnode_t *rootvp, vnode_t *pvp, vnode_t *vp, const char *name, + size_t len) +{ + vn_setpath_common(pvp, vp, name, len, B_FALSE); +} + +/* + * Sets the path to the vnode to be the given string, regardless of current + * context. The string must be a complete path from rootdir. This is only used + * by fsop_root() for setting the path based on the mountpoint. + */ +void +vn_setpath_str(vnode_t *vp, const char *str, size_t len) +{ + vn_setpath_common(NULL, vp, str, len, B_FALSE); +} + +/* * Called from within filesystem's vop_rename() to handle renames once the * target vnode is available. */ void -vn_renamepath(vnode_t *dvp, vnode_t *vp, const char *nm, size_t len) +vn_renamepath(vnode_t *pvp, vnode_t *vp, const char *name, size_t len) { - char *tmp; - - mutex_enter(&vp->v_lock); - tmp = vp->v_path; - vp->v_path = NULL; - mutex_exit(&vp->v_lock); - vn_setpath(rootdir, dvp, vp, nm, len); - if (tmp != NULL) - kmem_free(tmp, strlen(tmp) + 1); + vn_setpath_common(pvp, vp, name, len, B_TRUE); } /* @@ -3083,37 +3232,42 @@ void vn_copypath(struct vnode *src, struct vnode *dst) { char *buf; - int alloc; + hrtime_t stamp; + size_t buflen; mutex_enter(&src->v_lock); - if (src->v_path == NULL) { + if (src->v_path == vn_vpath_empty) { mutex_exit(&src->v_lock); return; } - alloc = strlen(src->v_path) + 1; - - /* avoid kmem_alloc() with lock held */ + buflen = strlen(src->v_path) + 1; mutex_exit(&src->v_lock); - buf = kmem_alloc(alloc, KM_SLEEP); + + buf = kmem_alloc(buflen, KM_SLEEP); + mutex_enter(&src->v_lock); - if (src->v_path == NULL || strlen(src->v_path) + 1 != alloc) { + if (src->v_path == vn_vpath_empty || + strlen(src->v_path) + 1 != buflen) { mutex_exit(&src->v_lock); - kmem_free(buf, alloc); + kmem_free(buf, buflen); return; } - bcopy(src->v_path, buf, alloc); + bcopy(src->v_path, buf, buflen); + stamp = src->v_path_stamp; mutex_exit(&src->v_lock); mutex_enter(&dst->v_lock); - if (dst->v_path != NULL) { + if (dst->v_path != vn_vpath_empty) { mutex_exit(&dst->v_lock); - kmem_free(buf, alloc); + kmem_free(buf, buflen); return; } dst->v_path = buf; + dst->v_path_stamp = stamp; mutex_exit(&dst->v_lock); } + /* * XXX Private interface for segvn routines that handle vnode * large page segments. @@ -3453,9 +3607,7 @@ fop_lookup( } if (ret == 0 && *vpp) { VOPSTATS_UPDATE(*vpp, lookup); - if ((*vpp)->v_path == NULL) { - vn_setpath(rootdir, dvp, *vpp, nm, strlen(nm)); - } + vn_updatepath(dvp, *vpp, nm); } return (ret); @@ -3495,9 +3647,7 @@ fop_create( (dvp, name, vap, excl, mode, vpp, cr, flags, ct, vsecp); if (ret == 0 && *vpp) { VOPSTATS_UPDATE(*vpp, create); - if ((*vpp)->v_path == NULL) { - vn_setpath(rootdir, dvp, *vpp, name, strlen(name)); - } + vn_updatepath(dvp, *vpp, name); } return (ret); @@ -3617,10 +3767,7 @@ fop_mkdir( (dvp, dirname, vap, vpp, cr, ct, flags, vsecp); if (ret == 0 && *vpp) { VOPSTATS_UPDATE(*vpp, mkdir); - if ((*vpp)->v_path == NULL) { - vn_setpath(rootdir, dvp, *vpp, dirname, - strlen(dirname)); - } + vn_updatepath(dvp, *vpp, dirname); } return (ret); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_dir.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_dir.c Wed Sep 13 10:23:55 2017 (r323525) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_dir.c Wed Sep 13 10:25:44 2017 (r323526) @@ -23,6 +23,7 @@ * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2013, 2016 by Delphix. All rights reserved. * Copyright 2017 Nexenta Systems, Inc. + * Copyright (c) 2015, Joyent, Inc. */ #include @@ -893,9 +894,9 @@ zfs_link_destroy(zfs_dirlock_t *dl, znode_t *zp, dmu_t if (zp->z_links <= zp_is_dir) { zfs_panic_recover("zfs: link count on %s is %u, " "should be at least %u", - zp->z_vnode->v_path ? zp->z_vnode->v_path : - "", (int)zp->z_links, - zp_is_dir + 1); + zp->z_vnode->v_path != vn_vpath_empty ? + zp->z_vnode->v_path : "", + (int)zp->z_links, zp_is_dir + 1); zp->z_links = zp_is_dir + 1; } if (--zp->z_links == zp_is_dir) { Modified: vendor-sys/illumos/dist/uts/common/sys/vnode.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/vnode.h Wed Sep 13 10:23:55 2017 (r323525) +++ vendor-sys/illumos/dist/uts/common/sys/vnode.h Wed Sep 13 10:25:44 2017 (r323526) @@ -21,7 +21,7 @@ /* * Copyright (c) 1988, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright (c) 2017, Joyent, Inc. * Copyright (c) 2011, 2017 by Delphix. All rights reserved. */ @@ -223,6 +223,59 @@ struct vsd_node { * In particular, file systems should not access other fields; they may * change or even be removed. The functionality which was once provided * by these fields is available through vn_* functions. + * + * VNODE PATH THEORY: + * In each vnode, the v_path field holds a cached version of the canonical + * filesystem path which that node represents. Because vnodes lack contextual + * information about their own name or position in the VFS hierarchy, this path + * must be calculated when the vnode is instantiated by operations such as + * fop_create, fop_lookup, or fop_mkdir. During said operations, both the + * parent vnode (and its cached v_path) and future name are known, so the + * v_path of the resulting object can easily be set. + * + * The caching nature of v_path is complicated in the face of directory + * renames. Filesystem drivers are responsible for calling vn_renamepath when + * a fop_rename operation succeeds. While the v_path on the renamed vnode will + * be updated, existing children of the directory (direct, or at deeper levels) + * will now possess v_path caches which are stale. + * + * It is expensive (and for non-directories, impossible) to recalculate stale + * v_path entries during operations such as vnodetopath. The best time during + * which to correct such wrongs is the same as when v_path is first + * initialized: during fop_create/fop_lookup/fop_mkdir/etc, where adequate + * context is available to generate the current path. + * + * In order to quickly detect stale v_path entries (without full lookup + * verification) to trigger a v_path update, the v_path_stamp field has been + * added to vnode_t. As part of successful fop_create/fop_lookup/fop_mkdir + * operations, where the name and parent vnode are available, the following + * rules are used to determine updates to the child: + * + * 1. If the parent lacks a v_path, clear any existing v_path and v_path_stamp + * on the child. Until the parent v_path is refreshed to a valid state, the + * child v_path must be considered invalid too. + * + * 2. If the child lacks a v_path (implying v_path_stamp == 0), it inherits the + * v_path_stamp value from its parent and its v_path is updated. + * + * 3. If the child v_path_stamp is less than v_path_stamp in the parent, it is + * an indication that the child v_path is stale. The v_path is updated and + * v_path_stamp in the child is set to the current hrtime(). + * + * It does _not_ inherit the parent v_path_stamp in order to propagate the + * the time of v_path invalidation through the directory structure. This + * prevents concurrent invalidations (operating with a now-incorrect v_path) + * at deeper levels in the tree from persisting. + * + * 4. If the child v_path_stamp is greater or equal to the parent, no action + * needs to be taken. + * + * Note that fop_rename operations do not follow this ruleset. They perform an + * explicit update of v_path and v_path_stamp (setting it to the current time) + * + * With these constraints in place, v_path invalidations and updates should + * proceed in a timely manner as vnodes are accessed. While there still are + * limited cases where vnodetopath operations will fail, the risk is minimized. */ struct fem_head; /* from fem.h */ @@ -249,6 +302,7 @@ typedef struct vnode { void *v_locality; /* hook for locality info */ struct fem_head *v_femhead; /* fs monitoring */ char *v_path; /* cached path */ + hrtime_t v_path_stamp; /* timestamp for cached path */ uint_t v_rdcnt; /* open for read count (VREG only) */ uint_t v_wrcnt; /* open for write count (VREG only) */ u_longlong_t v_mmap_read; /* mmap read count */ @@ -350,6 +404,14 @@ typedef struct vn_vfslocks_entry { #define V_SYSATTR 0x40000 /* vnode is a GFS system attribute */ /* + * Indication that VOP_LOOKUP operations on this vnode may yield results from a + * different VFS instance. The main use of this is to suppress v_path + * calculation logic when filesystems such as procfs emit results which defy + * expectations about normal VFS behavior. + */ +#define VTRAVERSE 0x80000 + +/* * Vnode attributes. A bit-mask is supplied as part of the * structure to indicate the attributes the caller wants to * set (setattr) or extract (getattr). @@ -1293,6 +1355,11 @@ void vn_setpath(vnode_t *rootvp, struct vnode *startvp const char *path, size_t plen); void vn_renamepath(vnode_t *dvp, vnode_t *vp, const char *nm, size_t len); +/* Private vnode manipulation functions */ +void vn_clearpath(vnode_t *, hrtime_t); +void vn_updatepath(vnode_t *, vnode_t *, const char *); + + /* Vnode event notification */ void vnevent_rename_src(vnode_t *, vnode_t *, char *, caller_context_t *); void vnevent_rename_dest(vnode_t *, vnode_t *, char *, caller_context_t *); @@ -1338,6 +1405,9 @@ void reparse_point_init(void); u_longlong_t fs_new_caller_id(); int vn_vmpss_usepageio(vnode_t *); + +/* Empty v_path placeholder */ +extern char *vn_vpath_empty; /* * Needed for use of IS_VMODSORT() in kernel. From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:33:11 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 242D5E19382; Wed, 13 Sep 2017 10:33:11 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1C0081902; Wed, 13 Sep 2017 10:33:10 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAX9bd059753; Wed, 13 Sep 2017 10:33:09 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAX9vp059752; Wed, 13 Sep 2017 10:33:09 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131033.v8DAX9vp059752@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:33:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323527 - vendor/illumos/dist/lib/libzpool/common X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzpool/common X-SVN-Commit-Revision: 323527 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:33:11 -0000 Author: avg Date: Wed Sep 13 10:33:09 2017 New Revision: 323527 URL: https://svnweb.freebsd.org/changeset/base/323527 Log: 5815 libzpool's panic function doesn't set global panicstr, ::status not as useful illumos/illumos-gate@fae6347731c9d3f46b26338313b0422927f29cf6 https://github.com/illumos/illumos-gate/commit/fae6347731c9d3f46b26338313b0422927f29cf6 https://www.illumos.org/issues/5815 When panic() is called from within ztest, the mdb ::status command isn't as useful as it could be since the global panicstr variable isn't updated. We should modify the function to make sure panicstr is set, so ::status can present the error message just like it does on a failed assertion. Reviewed by: Matthew Ahrens Reviewed by: Sebastien Roy Reviewed by: Gordon Ross Reviewed by: Rich Lowe Approved by: Dan McDonald Author: Prakash Surya Modified: vendor/illumos/dist/lib/libzpool/common/kernel.c Modified: vendor/illumos/dist/lib/libzpool/common/kernel.c ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Sep 13 10:25:44 2017 (r323526) +++ vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Sep 13 10:33:09 2017 (r323527) @@ -699,11 +699,9 @@ static char ce_suffix[CE_IGNORE][2] = { "", "\n", "\n" void vpanic(const char *fmt, va_list adx) { - (void) fprintf(stderr, "error: "); - (void) vfprintf(stderr, fmt, adx); - (void) fprintf(stderr, "\n"); - - abort(); /* think of it as a "user-level crash dump" */ + char buf[512]; + (void) vsnprintf(buf, 512, fmt, adx); + assfail(buf, NULL, 0); } void From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:41:49 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55DC2E19A35; Wed, 13 Sep 2017 10:41:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2509381F30; Wed, 13 Sep 2017 10:41:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAfmSF061532; Wed, 13 Sep 2017 10:41:48 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAfmPx061531; Wed, 13 Sep 2017 10:41:48 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131041.v8DAfmPx061531@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:41:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323529 - vendor-sys/illumos/dist/uts/common X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common X-SVN-Commit-Revision: 323529 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:41:49 -0000 Author: avg Date: Wed Sep 13 10:41:47 2017 New Revision: 323529 URL: https://svnweb.freebsd.org/changeset/base/323529 Log: 5115 Want Intel 40GbE NIC driver for illumos illumos/illumos-gate@9d26e4fc021e249c93c2861629cc665e4f5bd4d6 https://github.com/illumos/illumos-gate/commit/9d26e4fc021e249c93c2861629cc665e4f5bd4d6 https://www.illumos.org/issues/5115 Intel's NICs based on the XL710 chipset exist 1 . There exist drivers for Linux and FreeBSD 2 . It does not appear to be derived from the ixgbe driver source, so it would probably require porting the i40e source from FBSD to Illumos, unless a driver exists for a GLDv3-like platform under CDDL or similar license (none are known to currently be available or being worked on). Reviewed by: Dan McDonald Reviewed by: Hans Rosenfeld Approved by: Richard Lowe Author: Robert Mustacchi Note: this change has nothing to do with FreeBSD. Modified: vendor-sys/illumos/dist/uts/common/Makefile.files Modified: vendor-sys/illumos/dist/uts/common/Makefile.files ============================================================================== --- vendor-sys/illumos/dist/uts/common/Makefile.files Wed Sep 13 10:34:31 2017 (r323528) +++ vendor-sys/illumos/dist/uts/common/Makefile.files Wed Sep 13 10:41:47 2017 (r323529) @@ -21,11 +21,11 @@ # # Copyright (c) 1991, 2010, Oracle and/or its affiliates. All rights reserved. -# Copyright (c) 2012 Nexenta Systems, Inc. All rights reserved. -# Copyright (c) 2012 Joyent, Inc. All rights reserved. # Copyright (c) 2011, 2014 by Delphix. All rights reserved. # Copyright (c) 2013 by Saso Kiselkov. All rights reserved. -# Copyright 2014 Nexenta Systems, Inc. All rights reserved. +# Copyright 2015 Nexenta Systems, Inc. All rights reserved. +# Copyright 2016 Garrett D'Amore +# Copyright 2016 Joyent, Inc. # # @@ -1951,6 +1951,17 @@ IXGBE_OBJS = ixgbe_82598.o ixgbe_82599.o ixgbe_api. ixgbe_log.o ixgbe_main.o \ ixgbe_osdep.o ixgbe_rx.o ixgbe_stat.o \ ixgbe_tx.o ixgbe_x540.o ixgbe_mbx.o + +# +# Intel 40GbE PCIe NIC driver module +# + +# illumos-written ones. +I40E_OBJS = i40e_main.o i40e_osdep.o i40e_intr.o i40e_transceiver.o \ + i40e_stats.o i40e_gld.o +# Intel-written ones. +I40E_INTC_OBJS = i40e_adminq.o i40e_common.o i40e_hmc.o i40e_lan_hmc.o \ + i40e_nvm.o # # NIU 10G/1G driver module From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:45:50 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8C15E19CAD; Wed, 13 Sep 2017 10:45:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 825FC821A6; Wed, 13 Sep 2017 10:45:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAjn3l063850; Wed, 13 Sep 2017 10:45:49 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAjntu063845; Wed, 13 Sep 2017 10:45:49 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131045.v8DAjntu063845@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:45:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323530 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua vendor-sys/illumo... X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua vendor-sys/illumos/dist/uts/common/fs/zfs/... X-SVN-Commit-Revision: 323530 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:45:50 -0000 Author: avg Date: Wed Sep 13 10:45:49 2017 New Revision: 323530 URL: https://svnweb.freebsd.org/changeset/base/323530 Log: 7431 ZFS Channel Programs illumos/illumos-gate@dfc115332c94a2f62058ac7f2bce7631fbd20b3d https://github.com/illumos/illumos-gate/commit/dfc115332c94a2f62058ac7f2bce7631fbd20b3d https://www.illumos.org/issues/7431 ZFS channel programs (ZCP) adds support for performing compound ZFS administrative actions via Lua scripts in a sandboxed environment (with time and memory limits). This initial commit includes both base support for running ZCP scripts, and a small initial library of API calls which support getting properties and listing, destroying, and promoting datasets. Testing: in addition to the included unit tests, channel programs have been in use at Delphix for several months for batch destroying filesystems. The dsl_destroy_snaps_nvl() call has also been replaced with For reference, the new zfs-program manpage is included below. ZFS-PROGRAM(1M) 1M ZFS-PROGRAM(1M) NAME zfs program – executes ZFS channel programs SYNOPSIS zfs program [-t timeout] [-m memory-limit] pool script DESCRIPTION The ZFS channel program interface allows ZFS administrative operations to be run programmatically as a Lua script. The entire script is executed atomically, with no other administrative operations taking effect concurrently. A library of ZFS calls is made available to channel program scripts. Channel programs may only be run with root privileges. A modified version of the Lua 5.2 interpreter is used to run channel program scripts. The Lua 5.2 manual can be found at: http://www.lua.org/manual/5.2/ ... Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: John Kennedy Reviewed by: Dan Kimmel Approved by: Garrett D'Amore Author: Chris Williamson Added: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/README.zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lapi.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lapi.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lauxlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lauxlib.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lbaselib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lbitlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcode.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcode.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcompat.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcorolib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lctype.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lctype.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldebug.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldebug.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldo.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldo.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldump.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lfunc.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lfunc.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llimits.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lmem.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lmem.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lobject.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lobject.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lopcodes.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lopcodes.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lparser.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lparser.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstate.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstate.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstring.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstring.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltablib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltm.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltm.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lua.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/luaconf.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lualib.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lundump.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lundump.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lvm.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lvm.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lzio.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lzio.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_global.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_iter.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_prop.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_global.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_iter.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_synctask.c (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_destroy.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dir.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_vfsops.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Added: vendor/illumos/dist/man/man1m/zfs-program.1m Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/lib/libzpool/common/kernel.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h vendor/illumos/dist/man/man1m/zfs.1m Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_prop.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor-sys/illumos/dist/common/zfs/zfs_prop.c Wed Sep 13 10:45:49 2017 (r323530) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. * Copyright (c) 2014 Integros [integros.com] @@ -381,7 +381,8 @@ zfs_prop_init(void) zprop_register_number(ZFS_PROP_WRITTEN, "written", 0, PROP_READONLY, ZFS_TYPE_DATASET, "", "WRITTEN"); zprop_register_number(ZFS_PROP_LOGICALUSED, "logicalused", 0, - PROP_READONLY, ZFS_TYPE_DATASET, "", "LUSED"); + PROP_READONLY, ZFS_TYPE_FILESYSTEM | ZFS_TYPE_VOLUME, "", + "LUSED"); zprop_register_number(ZFS_PROP_LOGICALREFERENCED, "logicalreferenced", 0, PROP_READONLY, ZFS_TYPE_DATASET, "", "LREFER"); @@ -582,6 +583,15 @@ zfs_prop_readonly(zfs_prop_t prop) { return (zfs_prop_table[prop].pd_attr == PROP_READONLY || zfs_prop_table[prop].pd_attr == PROP_ONETIME); +} + +/* + * Returns TRUE if the property is visible (not hidden). + */ +boolean_t +zfs_prop_visible(zfs_prop_t prop) +{ + return (zfs_prop_table[prop].pd_visible); } /* Modified: vendor-sys/illumos/dist/uts/common/Makefile.files ============================================================================== --- vendor-sys/illumos/dist/uts/common/Makefile.files Wed Sep 13 10:41:47 2017 (r323529) +++ vendor-sys/illumos/dist/uts/common/Makefile.files Wed Sep 13 10:45:49 2017 (r323530) @@ -1308,6 +1308,35 @@ SMBFS_OBJS += smbfs_vfsops.o smbfs_vnops.o smbfs_node. $(SMBFS_COMMON_OBJS) +LUA_OBJS += \ + ldo.o \ + lvm.o \ + lbitlib.o \ + lopcodes.o \ + lstring.o \ + ltable.o \ + ltm.o \ + lcorolib.o \ + lauxlib.o \ + ldebug.o \ + lstate.o \ + lgc.o \ + lmem.o \ + lctype.o \ + lfunc.o \ + ldump.o \ + lundump.o \ + lstrlib.o \ + ltablib.o \ + lapi.o \ + lobject.o \ + lbaselib.o \ + lcompat.o \ + lzio.o \ + lcode.o \ + llex.o \ + lparser.o + ZFS_COMMON_OBJS += \ abd.o \ arc.o \ @@ -1375,6 +1404,11 @@ ZFS_COMMON_OBJS += \ zap.o \ zap_leaf.o \ zap_micro.o \ + zcp.o \ + zcp_get.o \ + zcp_global.o \ + zcp_iter.o \ + zcp_synctask.o \ zfs_byteswap.o \ zfs_debug.o \ zfs_fm.o \ @@ -1958,7 +1992,7 @@ IXGBE_OBJS = ixgbe_82598.o ixgbe_82599.o ixgbe_api. # illumos-written ones. I40E_OBJS = i40e_main.o i40e_osdep.o i40e_intr.o i40e_transceiver.o \ - i40e_stats.o i40e_gld.o + i40e_stats.o i40e_gld.o # Intel-written ones. I40E_INTC_OBJS = i40e_adminq.o i40e_common.o i40e_hmc.o i40e_lan_hmc.o \ i40e_nvm.o Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Wed Sep 13 10:45:49 2017 (r323530) @@ -1616,7 +1616,6 @@ dsl_dataset_snapshot_tmp(const char *fsname, const cha return (error); } - void dsl_dataset_sync(dsl_dataset_t *ds, zio_t *zio, dmu_tx_t *tx) { @@ -1684,30 +1683,17 @@ dsl_dataset_sync_done(dsl_dataset_t *ds, dmu_tx_t *tx) dmu_buf_rele(ds->ds_dbuf, ds); } -static void -get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) +int +get_clones_stat_impl(dsl_dataset_t *ds, nvlist_t *val) { uint64_t count = 0; objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; zap_cursor_t zc; zap_attribute_t za; - nvlist_t *propval = fnvlist_alloc(); - nvlist_t *val; ASSERT(dsl_pool_config_held(ds->ds_dir->dd_pool)); /* - * We use nvlist_alloc() instead of fnvlist_alloc() because the - * latter would allocate the list with NV_UNIQUE_NAME flag. - * As a result, every time a clone name is appended to the list - * it would be (linearly) searched for for a duplicate name. - * We already know that all clone names must be unique and we - * want avoid the quadratic complexity of double-checking that - * because we can have a large number of clones. - */ - VERIFY0(nvlist_alloc(&val, 0, KM_SLEEP)); - - /* * There may be missing entries in ds_next_clones_obj * due to a bug in a previous version of the code. * Only trust it if it has the right number of entries. @@ -1716,8 +1702,9 @@ get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) VERIFY0(zap_count(mos, dsl_dataset_phys(ds)->ds_next_clones_obj, &count)); } - if (count != dsl_dataset_phys(ds)->ds_num_children - 1) - goto fail; + if (count != dsl_dataset_phys(ds)->ds_num_children - 1) { + return (ENOENT); + } for (zap_cursor_init(&zc, mos, dsl_dataset_phys(ds)->ds_next_clones_obj); zap_cursor_retrieve(&zc, &za) == 0; @@ -1731,16 +1718,43 @@ get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) dsl_dataset_rele(clone, FTAG); } zap_cursor_fini(&zc); - fnvlist_add_nvlist(propval, ZPROP_VALUE, val); - fnvlist_add_nvlist(nv, zfs_prop_to_name(ZFS_PROP_CLONES), propval); -fail: - nvlist_free(val); - nvlist_free(propval); + return (0); } -static void -get_receive_resume_stats(dsl_dataset_t *ds, nvlist_t *nv) +void +get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) { + nvlist_t *propval = fnvlist_alloc(); + nvlist_t *val; + + /* + * We use nvlist_alloc() instead of fnvlist_alloc() because the + * latter would allocate the list with NV_UNIQUE_NAME flag. + * As a result, every time a clone name is appended to the list + * it would be (linearly) searched for for a duplicate name. + * We already know that all clone names must be unique and we + * want avoid the quadratic complexity of double-checking that + * because we can have a large number of clones. + */ + VERIFY0(nvlist_alloc(&val, 0, KM_SLEEP)); + + if (get_clones_stat_impl(ds, val) == 0) { + fnvlist_add_nvlist(propval, ZPROP_VALUE, val); + fnvlist_add_nvlist(nv, zfs_prop_to_name(ZFS_PROP_CLONES), + propval); + } else { + nvlist_free(val); + nvlist_free(propval); + } +} + +/* + * Returns a string that represents the receive resume stats token. It should + * be freed with strfree(). + */ +char * +get_receive_resume_stats_impl(dsl_dataset_t *ds) +{ dsl_pool_t *dp = ds->ds_dir->dd_pool; if (dsl_dataset_has_resume_receive_state(ds)) { @@ -1807,84 +1821,359 @@ get_receive_resume_stats(dsl_dataset_t *ds, nvlist_t * ZFS_SEND_RESUME_TOKEN_VERSION, (longlong_t)cksum.zc_word[0], (longlong_t)packed_size, str); - dsl_prop_nvlist_add_string(nv, - ZFS_PROP_RECEIVE_RESUME_TOKEN, propval); kmem_free(packed, packed_size); kmem_free(str, compressed_size * 2 + 1); kmem_free(compressed, packed_size); - strfree(propval); + return (propval); } + return (strdup("")); } +/* + * Returns a string that represents the receive resume stats token of the + * dataset's child. It should be freed with strfree(). + */ +char * +get_child_receive_stats(dsl_dataset_t *ds) +{ + char recvname[ZFS_MAX_DATASET_NAME_LEN + 6]; + dsl_dataset_t *recv_ds; + dsl_dataset_name(ds, recvname); + if (strlcat(recvname, "/", sizeof (recvname)) < + sizeof (recvname) && + strlcat(recvname, recv_clone_name, sizeof (recvname)) < + sizeof (recvname) && + dsl_dataset_hold(ds->ds_dir->dd_pool, recvname, FTAG, + &recv_ds) == 0) { + char *propval = get_receive_resume_stats_impl(recv_ds); + dsl_dataset_rele(recv_ds, FTAG); + return (propval); + } + return (strdup("")); +} + +static void +get_receive_resume_stats(dsl_dataset_t *ds, nvlist_t *nv) +{ + char *propval = get_receive_resume_stats_impl(ds); + if (strcmp(propval, "") != 0) { + dsl_prop_nvlist_add_string(nv, + ZFS_PROP_RECEIVE_RESUME_TOKEN, propval); + } else { + char *childval = get_child_receive_stats(ds); + if (strcmp(childval, "") != 0) { + dsl_prop_nvlist_add_string(nv, + ZFS_PROP_RECEIVE_RESUME_TOKEN, childval); + } + strfree(childval); + } + strfree(propval); +} + +uint64_t +dsl_get_refratio(dsl_dataset_t *ds) +{ + uint64_t ratio = dsl_dataset_phys(ds)->ds_compressed_bytes == 0 ? 100 : + (dsl_dataset_phys(ds)->ds_uncompressed_bytes * 100 / + dsl_dataset_phys(ds)->ds_compressed_bytes); + return (ratio); +} + +uint64_t +dsl_get_logicalreferenced(dsl_dataset_t *ds) +{ + return (dsl_dataset_phys(ds)->ds_uncompressed_bytes); +} + +uint64_t +dsl_get_compressratio(dsl_dataset_t *ds) +{ + if (ds->ds_is_snapshot) { + return (dsl_get_refratio(ds)); + } else { + dsl_dir_t *dd = ds->ds_dir; + mutex_enter(&dd->dd_lock); + uint64_t val = dsl_dir_get_compressratio(dd); + mutex_exit(&dd->dd_lock); + return (val); + } +} + +uint64_t +dsl_get_used(dsl_dataset_t *ds) +{ + if (ds->ds_is_snapshot) { + return (dsl_dataset_phys(ds)->ds_unique_bytes); + } else { + dsl_dir_t *dd = ds->ds_dir; + mutex_enter(&dd->dd_lock); + uint64_t val = dsl_dir_get_used(dd); + mutex_exit(&dd->dd_lock); + return (val); + } +} + +uint64_t +dsl_get_creation(dsl_dataset_t *ds) +{ + return (dsl_dataset_phys(ds)->ds_creation_time); +} + +uint64_t +dsl_get_creationtxg(dsl_dataset_t *ds) +{ + return (dsl_dataset_phys(ds)->ds_creation_txg); +} + +uint64_t +dsl_get_refquota(dsl_dataset_t *ds) +{ + return (ds->ds_quota); +} + +uint64_t +dsl_get_refreservation(dsl_dataset_t *ds) +{ + return (ds->ds_reserved); +} + +uint64_t +dsl_get_guid(dsl_dataset_t *ds) +{ + return (dsl_dataset_phys(ds)->ds_guid); +} + +uint64_t +dsl_get_unique(dsl_dataset_t *ds) +{ + return (dsl_dataset_phys(ds)->ds_unique_bytes); +} + +uint64_t +dsl_get_objsetid(dsl_dataset_t *ds) +{ + return (ds->ds_object); +} + +uint64_t +dsl_get_userrefs(dsl_dataset_t *ds) +{ + return (ds->ds_userrefs); +} + +uint64_t +dsl_get_defer_destroy(dsl_dataset_t *ds) +{ + return (DS_IS_DEFER_DESTROY(ds) ? 1 : 0); +} + +uint64_t +dsl_get_referenced(dsl_dataset_t *ds) +{ + return (dsl_dataset_phys(ds)->ds_referenced_bytes); +} + +uint64_t +dsl_get_numclones(dsl_dataset_t *ds) +{ + ASSERT(ds->ds_is_snapshot); + return (dsl_dataset_phys(ds)->ds_num_children - 1); +} + +uint64_t +dsl_get_inconsistent(dsl_dataset_t *ds) +{ + return ((dsl_dataset_phys(ds)->ds_flags & DS_FLAG_INCONSISTENT) ? + 1 : 0); +} + +uint64_t +dsl_get_available(dsl_dataset_t *ds) +{ + uint64_t refdbytes = dsl_get_referenced(ds); + uint64_t availbytes = dsl_dir_space_available(ds->ds_dir, + NULL, 0, TRUE); + if (ds->ds_reserved > dsl_dataset_phys(ds)->ds_unique_bytes) { + availbytes += + ds->ds_reserved - dsl_dataset_phys(ds)->ds_unique_bytes; + } + if (ds->ds_quota != 0) { + /* + * Adjust available bytes according to refquota + */ + if (refdbytes < ds->ds_quota) { + availbytes = MIN(availbytes, + ds->ds_quota - refdbytes); + } else { + availbytes = 0; + } + } + return (availbytes); +} + +int +dsl_get_written(dsl_dataset_t *ds, uint64_t *written) +{ + dsl_pool_t *dp = ds->ds_dir->dd_pool; + dsl_dataset_t *prev; + int err = dsl_dataset_hold_obj(dp, + dsl_dataset_phys(ds)->ds_prev_snap_obj, FTAG, &prev); + if (err == 0) { + uint64_t comp, uncomp; + err = dsl_dataset_space_written(prev, ds, written, + &comp, &uncomp); + dsl_dataset_rele(prev, FTAG); + } + return (err); +} + +/* + * 'snap' should be a buffer of size ZFS_MAX_DATASET_NAME_LEN. + */ +int +dsl_get_prev_snap(dsl_dataset_t *ds, char *snap) +{ + dsl_pool_t *dp = ds->ds_dir->dd_pool; + if (ds->ds_prev != NULL && ds->ds_prev != dp->dp_origin_snap) { + dsl_dataset_name(ds->ds_prev, snap); + return (0); + } else { + return (ENOENT); + } +} + +/* + * Returns the mountpoint property and source for the given dataset in the value + * and source buffers. The value buffer must be at least as large as MAXPATHLEN + * and the source buffer as least as large a ZFS_MAX_DATASET_NAME_LEN. + * Returns 0 on success and an error on failure. + */ +int +dsl_get_mountpoint(dsl_dataset_t *ds, const char *dsname, char *value, + char *source) +{ + int error; + dsl_pool_t *dp = ds->ds_dir->dd_pool; + + /* Retrieve the mountpoint value stored in the zap opbject */ + error = dsl_prop_get_ds(ds, zfs_prop_to_name(ZFS_PROP_MOUNTPOINT), 1, + ZAP_MAXVALUELEN, value, source); + if (error != 0) { + return (error); + } + + /* Process the dsname and source to find the full mountpoint string */ + if (value[0] == '/') { + char *buf = kmem_alloc(ZAP_MAXVALUELEN, KM_SLEEP); + char *root = buf; + const char *relpath; + + /* + * If we inherit the mountpoint, even from a dataset + * with a received value, the source will be the path of + * the dataset we inherit from. If source is + * ZPROP_SOURCE_VAL_RECVD, the received value is not + * inherited. + */ + if (strcmp(source, ZPROP_SOURCE_VAL_RECVD) == 0) { + relpath = ""; + } else { + ASSERT0(strncmp(dsname, source, strlen(source))); + relpath = dsname + strlen(source); + if (relpath[0] == '/') + relpath++; + } + + spa_altroot(dp->dp_spa, root, ZAP_MAXVALUELEN); + + /* + * Special case an alternate root of '/'. This will + * avoid having multiple leading slashes in the + * mountpoint path. + */ + if (strcmp(root, "/") == 0) + root++; + + /* + * If the mountpoint is '/' then skip over this + * if we are obtaining either an alternate root or + * an inherited mountpoint. + */ + char *mnt = value; + if (value[1] == '\0' && (root[0] != '\0' || + relpath[0] != '\0')) + mnt = value + 1; + + if (relpath[0] == '\0') { + (void) snprintf(value, ZAP_MAXVALUELEN, "%s%s", + root, mnt); + } else { + (void) snprintf(value, ZAP_MAXVALUELEN, "%s%s%s%s", + root, mnt, relpath[0] == '@' ? "" : "/", + relpath); + } + kmem_free(buf, ZAP_MAXVALUELEN); + } else { + /* 'legacy' or 'none' */ + (void) snprintf(value, ZAP_MAXVALUELEN, "%s", value); + } + return (0); +} + void dsl_dataset_stats(dsl_dataset_t *ds, nvlist_t *nv) { dsl_pool_t *dp = ds->ds_dir->dd_pool; - uint64_t refd, avail, uobjs, aobjs, ratio; ASSERT(dsl_pool_config_held(dp)); - ratio = dsl_dataset_phys(ds)->ds_compressed_bytes == 0 ? 100 : - (dsl_dataset_phys(ds)->ds_uncompressed_bytes * 100 / - dsl_dataset_phys(ds)->ds_compressed_bytes); - - dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFRATIO, ratio); + dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFRATIO, + dsl_get_refratio(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_LOGICALREFERENCED, - dsl_dataset_phys(ds)->ds_uncompressed_bytes); + dsl_get_logicalreferenced(ds)); + dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_COMPRESSRATIO, + dsl_get_compressratio(ds)); + dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USED, + dsl_get_used(ds)); if (ds->ds_is_snapshot) { - dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_COMPRESSRATIO, ratio); - dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USED, - dsl_dataset_phys(ds)->ds_unique_bytes); get_clones_stat(ds, nv); } else { - if (ds->ds_prev != NULL && ds->ds_prev != dp->dp_origin_snap) { - char buf[ZFS_MAX_DATASET_NAME_LEN]; - dsl_dataset_name(ds->ds_prev, buf); - dsl_prop_nvlist_add_string(nv, ZFS_PROP_PREV_SNAP, buf); - } - + char buf[ZFS_MAX_DATASET_NAME_LEN]; + if (dsl_get_prev_snap(ds, buf) == 0) + dsl_prop_nvlist_add_string(nv, ZFS_PROP_PREV_SNAP, + buf); dsl_dir_stats(ds->ds_dir, nv); } - dsl_dataset_space(ds, &refd, &avail, &uobjs, &aobjs); - dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_AVAILABLE, avail); - dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFERENCED, refd); - + dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_AVAILABLE, + dsl_get_available(ds)); + dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFERENCED, + dsl_get_referenced(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_CREATION, - dsl_dataset_phys(ds)->ds_creation_time); + dsl_get_creation(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_CREATETXG, - dsl_dataset_phys(ds)->ds_creation_txg); + dsl_get_creationtxg(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFQUOTA, - ds->ds_quota); + dsl_get_refquota(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFRESERVATION, - ds->ds_reserved); + dsl_get_refreservation(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_GUID, - dsl_dataset_phys(ds)->ds_guid); + dsl_get_guid(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_UNIQUE, - dsl_dataset_phys(ds)->ds_unique_bytes); + dsl_get_unique(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_OBJSETID, - ds->ds_object); + dsl_get_objsetid(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USERREFS, - ds->ds_userrefs); + dsl_get_userrefs(ds)); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_DEFER_DESTROY, - DS_IS_DEFER_DESTROY(ds) ? 1 : 0); + dsl_get_defer_destroy(ds)); if (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) { - uint64_t written, comp, uncomp; - dsl_pool_t *dp = ds->ds_dir->dd_pool; - dsl_dataset_t *prev; - - int err = dsl_dataset_hold_obj(dp, - dsl_dataset_phys(ds)->ds_prev_snap_obj, FTAG, &prev); - if (err == 0) { - err = dsl_dataset_space_written(prev, ds, &written, - &comp, &uncomp); - dsl_dataset_rele(prev, FTAG); - if (err == 0) { - dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_WRITTEN, - written); - } + uint64_t written; + if (dsl_get_written(ds, &written) == 0) { + dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_WRITTEN, + written); } } @@ -1921,27 +2210,19 @@ dsl_dataset_fast_stat(dsl_dataset_t *ds, dmu_objset_st dsl_pool_t *dp = ds->ds_dir->dd_pool; ASSERT(dsl_pool_config_held(dp)); - stat->dds_creation_txg = dsl_dataset_phys(ds)->ds_creation_txg; - stat->dds_inconsistent = - dsl_dataset_phys(ds)->ds_flags & DS_FLAG_INCONSISTENT; - stat->dds_guid = dsl_dataset_phys(ds)->ds_guid; + stat->dds_creation_txg = dsl_get_creationtxg(ds); + stat->dds_inconsistent = dsl_get_inconsistent(ds); + stat->dds_guid = dsl_get_guid(ds); stat->dds_origin[0] = '\0'; if (ds->ds_is_snapshot) { stat->dds_is_snapshot = B_TRUE; - stat->dds_num_clones = - dsl_dataset_phys(ds)->ds_num_children - 1; + stat->dds_num_clones = dsl_get_numclones(ds); } else { stat->dds_is_snapshot = B_FALSE; stat->dds_num_clones = 0; if (dsl_dir_is_clone(ds->ds_dir)) { - dsl_dataset_t *ods; - - VERIFY0(dsl_dataset_hold_obj(dp, - dsl_dir_phys(ds->ds_dir)->dd_origin_obj, - FTAG, &ods)); - dsl_dataset_name(ods, stat->dds_origin); - dsl_dataset_rele(ods, FTAG); + dsl_dir_get_origin(ds->ds_dir, stat->dds_origin); } } } @@ -2348,22 +2629,12 @@ struct promotenode { dsl_dataset_t *ds; }; -typedef struct dsl_dataset_promote_arg { - const char *ddpa_clonename; - dsl_dataset_t *ddpa_clone; - list_t shared_snaps, origin_snaps, clone_snaps; - dsl_dataset_t *origin_origin; /* origin of the origin */ - uint64_t used, comp, uncomp, unique, cloneusedsnap, originusedsnap; - char *err_ds; - cred_t *cr; -} dsl_dataset_promote_arg_t; - static int snaplist_space(list_t *l, uint64_t mintxg, uint64_t *spacep); static int promote_hold(dsl_dataset_promote_arg_t *ddpa, dsl_pool_t *dp, void *tag); static void promote_rele(dsl_dataset_promote_arg_t *ddpa, void *tag); -static int +int dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) { dsl_dataset_promote_arg_t *ddpa = arg; @@ -2375,14 +2646,19 @@ dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) uint64_t unused; uint64_t ss_mv_cnt; size_t max_snap_len; + boolean_t conflicting_snaps; err = promote_hold(ddpa, dp, FTAG); if (err != 0) return (err); hds = ddpa->ddpa_clone; + snap = list_head(&ddpa->shared_snaps); + origin_ds = snap->ds; max_snap_len = MAXNAMELEN - strlen(ddpa->ddpa_clonename) - 1; + snap = list_head(&ddpa->origin_snaps); + if (dsl_dataset_phys(hds)->ds_flags & DS_FLAG_NOPROMOTE) { promote_rele(ddpa, FTAG); return (SET_ERROR(EXDEV)); @@ -2397,9 +2673,6 @@ dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) return (0); } - snap = list_head(&ddpa->shared_snaps); - origin_ds = snap->ds; - /* compute origin's new unique space */ snap = list_tail(&ddpa->clone_snaps); ASSERT3U(dsl_dataset_phys(snap->ds)->ds_prev_snap_obj, ==, @@ -2423,6 +2696,7 @@ dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) * Note however, if we stop before we reach the ORIGIN we get: * uN + kN + kN-1 + ... + kM - uM-1 */ + conflicting_snaps = B_FALSE; ss_mv_cnt = 0; ddpa->used = dsl_dataset_phys(origin_ds)->ds_referenced_bytes; ddpa->comp = dsl_dataset_phys(origin_ds)->ds_compressed_bytes; @@ -2451,12 +2725,12 @@ dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) } err = dsl_dataset_snap_lookup(hds, ds->ds_snapname, &val); if (err == 0) { - (void) strcpy(ddpa->err_ds, snap->ds->ds_snapname); - err = SET_ERROR(EEXIST); + fnvlist_add_boolean(ddpa->err_ds, + snap->ds->ds_snapname); + conflicting_snaps = B_TRUE; + } else if (err != ENOENT) { goto out; } - if (err != ENOENT) - goto out; /* The very first snapshot does not have a deadlist */ if (dsl_dataset_phys(ds)->ds_prev_snap_obj == 0) @@ -2470,6 +2744,15 @@ dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) } /* + * In order to return the full list of conflicting snapshots, we check + * whether there was a conflict after traversing all of them. + */ + if (conflicting_snaps) { + err = SET_ERROR(EEXIST); + goto out; + } + + /* * If we are a clone of a clone then we never reached ORIGIN, * so we need to subtract out the clone origin's used space. */ @@ -2531,7 +2814,7 @@ out: return (err); } -static void +void dsl_dataset_promote_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_promote_arg_t *ddpa = arg; @@ -2856,6 +3139,7 @@ dsl_dataset_promote(const char *name, char *conflsnap) dsl_dataset_promote_arg_t ddpa = { 0 }; uint64_t numsnaps; int error; + nvpair_t *snap_pair; objset_t *os; /* @@ -2873,12 +3157,22 @@ dsl_dataset_promote(const char *name, char *conflsnap) return (error); ddpa.ddpa_clonename = name; - ddpa.err_ds = conflsnap; + ddpa.err_ds = fnvlist_alloc(); ddpa.cr = CRED(); - return (dsl_sync_task(name, dsl_dataset_promote_check, + error = dsl_sync_task(name, dsl_dataset_promote_check, dsl_dataset_promote_sync, &ddpa, - 2 + numsnaps, ZFS_SPACE_CHECK_RESERVED)); + 2 + numsnaps, ZFS_SPACE_CHECK_RESERVED); + + /* + * Return the first conflicting snapshot found. + */ + snap_pair = nvlist_next_nvpair(ddpa.err_ds, NULL); + if (snap_pair != NULL && conflsnap != NULL) + (void) strcpy(conflsnap, nvpair_name(snap_pair)); + + fnvlist_free(ddpa.err_ds); + return (error); } int Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Wed Sep 13 10:45:49 2017 (r323530) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2013 by Joyent, Inc. All rights reserved. * Copyright (c) 2014 Integros [integros.com] @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -41,14 +42,8 @@ #include #include #include +#include -typedef struct dmu_snapshots_destroy_arg { - nvlist_t *dsda_snaps; - nvlist_t *dsda_successful_snaps; - boolean_t dsda_defer; - nvlist_t *dsda_errlist; -} dmu_snapshots_destroy_arg_t; - int dsl_destroy_snapshot_check_impl(dsl_dataset_t *ds, boolean_t defer) { @@ -85,51 +80,33 @@ dsl_destroy_snapshot_check_impl(dsl_dataset_t *ds, boo return (0); } -static int +int dsl_destroy_snapshot_check(void *arg, dmu_tx_t *tx) { - dmu_snapshots_destroy_arg_t *dsda = arg; + dsl_destroy_snapshot_arg_t *ddsa = arg; + const char *dsname = ddsa->ddsa_name; + boolean_t defer = ddsa->ddsa_defer; + dsl_pool_t *dp = dmu_tx_pool(tx); - nvpair_t *pair; int error = 0; + dsl_dataset_t *ds; - if (!dmu_tx_is_syncing(tx)) + error = dsl_dataset_hold(dp, dsname, FTAG, &ds); + + /* + * If the snapshot does not exist, silently ignore it, and + * dsl_destroy_snapshot_sync() will be a no-op + * (it's "already destroyed"). + */ + if (error == ENOENT) return (0); - for (pair = nvlist_next_nvpair(dsda->dsda_snaps, NULL); - pair != NULL; pair = nvlist_next_nvpair(dsda->dsda_snaps, pair)) { - dsl_dataset_t *ds; - - error = dsl_dataset_hold(dp, nvpair_name(pair), - FTAG, &ds); - - /* - * If the snapshot does not exist, silently ignore it - * (it's "already destroyed"). - */ - if (error == ENOENT) - continue; - - if (error == 0) { - error = dsl_destroy_snapshot_check_impl(ds, - dsda->dsda_defer); - dsl_dataset_rele(ds, FTAG); - } - - if (error == 0) { - fnvlist_add_boolean(dsda->dsda_successful_snaps, - nvpair_name(pair)); - } else { - fnvlist_add_int32(dsda->dsda_errlist, - nvpair_name(pair), error); - } + if (error == 0) { + error = dsl_destroy_snapshot_check_impl(ds, defer); + dsl_dataset_rele(ds, FTAG); } - pair = nvlist_next_nvpair(dsda->dsda_errlist, NULL); - if (pair != NULL) - return (fnvpair_value_int32(pair)); - - return (0); + return (error); } struct process_old_arg { @@ -473,23 +450,22 @@ dsl_destroy_snapshot_sync_impl(dsl_dataset_t *ds, bool dmu_object_free_zapified(mos, obj, tx); } -static void +void dsl_destroy_snapshot_sync(void *arg, dmu_tx_t *tx) { - dmu_snapshots_destroy_arg_t *dsda = arg; + dsl_destroy_snapshot_arg_t *ddsa = arg; + const char *dsname = ddsa->ddsa_name; + boolean_t defer = ddsa->ddsa_defer; + dsl_pool_t *dp = dmu_tx_pool(tx); - nvpair_t *pair; + dsl_dataset_t *ds; - for (pair = nvlist_next_nvpair(dsda->dsda_successful_snaps, NULL); - pair != NULL; - pair = nvlist_next_nvpair(dsda->dsda_successful_snaps, pair)) { - dsl_dataset_t *ds; - - VERIFY0(dsl_dataset_hold(dp, nvpair_name(pair), FTAG, &ds)); - - dsl_destroy_snapshot_sync_impl(ds, dsda->dsda_defer, tx); - dsl_dataset_rele(ds, FTAG); - } + int error = dsl_dataset_hold(dp, dsname, FTAG, &ds); + if (error == ENOENT) + return; + ASSERT0(error); + dsl_destroy_snapshot_sync_impl(ds, defer, tx); + dsl_dataset_rele(ds, FTAG); } /* @@ -509,25 +485,85 @@ int dsl_destroy_snapshots_nvl(nvlist_t *snaps, boolean_t defer, nvlist_t *errlist) { - dmu_snapshots_destroy_arg_t dsda; - int error; - nvpair_t *pair; - - pair = nvlist_next_nvpair(snaps, NULL); - if (pair == NULL) + if (nvlist_next_nvpair(snaps, NULL) == NULL) return (0); - dsda.dsda_snaps = snaps; - dsda.dsda_successful_snaps = fnvlist_alloc(); - dsda.dsda_defer = defer; - dsda.dsda_errlist = errlist; + nvlist_t *arg = fnvlist_alloc(); + nvlist_t *snaps_normalized = fnvlist_alloc(); + /* + * lzc_destroy_snaps() is documented to take an nvlist whose + * values "don't matter". We need to convert that nvlist to one + * that we know can be converted to LUA. + */ + for (nvpair_t *pair = nvlist_next_nvpair(snaps, NULL); + pair != NULL; pair = nvlist_next_nvpair(snaps, pair)) { + fnvlist_add_boolean_value(snaps_normalized, + nvpair_name(pair), B_TRUE); + } + fnvlist_add_nvlist(arg, "snaps", snaps_normalized); + fnvlist_add_boolean_value(arg, "defer", defer); - error = dsl_sync_task(nvpair_name(pair), - dsl_destroy_snapshot_check, dsl_destroy_snapshot_sync, - &dsda, 0, ZFS_SPACE_CHECK_NONE); - fnvlist_free(dsda.dsda_successful_snaps); + nvlist_t *wrapper = fnvlist_alloc(); + fnvlist_add_nvlist(wrapper, ZCP_ARG_ARGLIST, arg); + fnvlist_free(arg); - return (error); + const char *program = + "arg = ...\n" + "snaps = arg['snaps']\n" + "defer = arg['defer']\n" + "errors = { }\n" + "has_errors = false\n" + "for snap, v in pairs(snaps) do\n" + " errno = zfs.check.destroy{snap, defer=defer}\n" + " zfs.debug('snap: ' .. snap .. ' errno: ' .. errno)\n" + " if errno == ENOENT then\n" + " snaps[snap] = nil\n" + " elseif errno ~= 0 then\n" + " errors[snap] = errno\n" + " has_errors = true\n" + " end\n" + "end\n" + "if has_errors then\n" + " return errors\n" + "end\n" + "for snap, v in pairs(snaps) do\n" + " errno = zfs.sync.destroy{snap, defer=defer}\n" + " assert(errno == 0)\n" + "end\n" + "return { }\n"; + + nvlist_t *result = fnvlist_alloc(); + int error = zcp_eval(nvpair_name(nvlist_next_nvpair(snaps, NULL)), + program, + 0, + zfs_lua_max_memlimit, + fnvlist_lookup_nvpair(wrapper, ZCP_ARG_ARGLIST), result); + if (error != 0) { + char *errorstr = NULL; + (void) nvlist_lookup_string(result, ZCP_RET_ERROR, &errorstr); + if (errorstr != NULL) { + zfs_dbgmsg(errorstr); + } + return (error); + } *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:45:51 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3544E19CB3; Wed, 13 Sep 2017 10:45:51 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 88A2D821A7; Wed, 13 Sep 2017 10:45:51 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAjosq063866; Wed, 13 Sep 2017 10:45:50 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAjnv0063855; Wed, 13 Sep 2017 10:45:49 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131045.v8DAjnv0063855@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:45:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323530 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua vendor-sys/illumo... X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua vendor-sys/illumos/dist/uts/common/fs/zfs/... X-SVN-Commit-Revision: 323530 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:45:52 -0000 Author: avg Date: Wed Sep 13 10:45:49 2017 New Revision: 323530 URL: https://svnweb.freebsd.org/changeset/base/323530 Log: 7431 ZFS Channel Programs illumos/illumos-gate@dfc115332c94a2f62058ac7f2bce7631fbd20b3d https://github.com/illumos/illumos-gate/commit/dfc115332c94a2f62058ac7f2bce7631fbd20b3d https://www.illumos.org/issues/7431 ZFS channel programs (ZCP) adds support for performing compound ZFS administrative actions via Lua scripts in a sandboxed environment (with time and memory limits). This initial commit includes both base support for running ZCP scripts, and a small initial library of API calls which support getting properties and listing, destroying, and promoting datasets. Testing: in addition to the included unit tests, channel programs have been in use at Delphix for several months for batch destroying filesystems. The dsl_destroy_snaps_nvl() call has also been replaced with For reference, the new zfs-program manpage is included below. ZFS-PROGRAM(1M) 1M ZFS-PROGRAM(1M) NAME zfs program – executes ZFS channel programs SYNOPSIS zfs program [-t timeout] [-m memory-limit] pool script DESCRIPTION The ZFS channel program interface allows ZFS administrative operations to be run programmatically as a Lua script. The entire script is executed atomically, with no other administrative operations taking effect concurrently. A library of ZFS calls is made available to channel program scripts. Channel programs may only be run with root privileges. A modified version of the Lua 5.2 interpreter is used to run channel program scripts. The Lua 5.2 manual can be found at: http://www.lua.org/manual/5.2/ ... Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: John Kennedy Reviewed by: Dan Kimmel Approved by: Garrett D'Amore Author: Chris Williamson Added: vendor/illumos/dist/man/man1m/zfs-program.1m Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/lib/libzpool/common/kernel.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h vendor/illumos/dist/man/man1m/zfs.1m Changes in other areas also in this revision: Added: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/README.zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lapi.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lapi.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lauxlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lauxlib.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lbaselib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lbitlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcode.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcode.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcompat.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcorolib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lctype.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lctype.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldebug.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldebug.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldo.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldo.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldump.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lfunc.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lfunc.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llimits.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lmem.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lmem.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lobject.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lobject.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lopcodes.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lopcodes.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lparser.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lparser.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstate.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstate.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstring.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstring.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltablib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltm.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltm.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lua.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/luaconf.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lualib.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lundump.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lundump.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lvm.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lvm.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lzio.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lzio.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_global.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_iter.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_prop.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_global.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_iter.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_synctask.c (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_destroy.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dir.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_vfsops.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c ============================================================================== --- vendor/illumos/dist/cmd/zfs/zfs_main.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/cmd/zfs/zfs_main.c Wed Sep 13 10:45:49 2017 (r323530) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. * Copyright 2012 Milan Jurik. All rights reserved. * Copyright (c) 2012, Joyent, Inc. All rights reserved. * Copyright (c) 2011-2012 Pawel Jakub Dawidek. All rights reserved. @@ -50,6 +50,7 @@ #include #include #include +#include #include #include #include @@ -106,6 +107,7 @@ static int zfs_do_holds(int argc, char **argv); static int zfs_do_release(int argc, char **argv); static int zfs_do_diff(int argc, char **argv); static int zfs_do_bookmark(int argc, char **argv); +static int zfs_do_channel_program(int argc, char **argv); /* * Enable a reasonable set of defaults for libumem debugging on DEBUG builds. @@ -153,6 +155,7 @@ typedef enum { HELP_RELEASE, HELP_DIFF, HELP_BOOKMARK, + HELP_CHANNEL_PROGRAM, } zfs_help_t; typedef struct zfs_command { @@ -180,6 +183,7 @@ static zfs_command_t command_table[] = { { "promote", zfs_do_promote, HELP_PROMOTE }, { "rename", zfs_do_rename, HELP_RENAME }, { "bookmark", zfs_do_bookmark, HELP_BOOKMARK }, + { "program", zfs_do_channel_program, HELP_CHANNEL_PROGRAM }, { NULL }, { "list", zfs_do_list, HELP_LIST }, { NULL }, @@ -324,6 +328,10 @@ get_usage(zfs_help_t idx) "[snapshot|filesystem]\n")); case HELP_BOOKMARK: return (gettext("\tbookmark \n")); + case HELP_CHANNEL_PROGRAM: + return (gettext("\tprogram [-t ] " + "[-m ] " + "[lua args...]\n")); } abort(); @@ -352,6 +360,18 @@ safe_malloc(size_t size) return (data); } +void * +safe_realloc(void *data, size_t size) +{ + void *newp; + if ((newp = realloc(data, size)) == NULL) { + free(data); + nomem(); + } + + return (newp); +} + static char * safe_strdup(char *str) { @@ -6945,6 +6965,190 @@ zfs_do_bookmark(int argc, char **argv) dgettext(TEXT_DOMAIN, err_msg)); } + return (ret != 0); + +usage: + usage(B_FALSE); + return (-1); +} + +static int +zfs_do_channel_program(int argc, char **argv) +{ + int ret, fd; + char c; + char *progbuf, *filename, *poolname; + size_t progsize, progread; + nvlist_t *outnvl; + uint64_t instrlimit = ZCP_DEFAULT_INSTRLIMIT; + uint64_t memlimit = ZCP_DEFAULT_MEMLIMIT; + zpool_handle_t *zhp; + + /* check options */ + while (-1 != + (c = getopt(argc, argv, "t:(instr-limit)m:(memory-limit)"))) { + switch (c) { + case 't': + case 'm': { + uint64_t arg; + char *endp; + + errno = 0; + arg = strtoull(optarg, &endp, 0); + if (errno != 0 || *endp != '\0') { + (void) fprintf(stderr, gettext( + "invalid argument " + "'%s': expected integer\n"), optarg); + goto usage; + } + + if (c == 't') { + if (arg > ZCP_MAX_INSTRLIMIT || arg == 0) { + (void) fprintf(stderr, gettext( + "Invalid instruction limit: " + "%s\n"), optarg); + return (1); + } else { + instrlimit = arg; + } + } else { + ASSERT3U(c, ==, 'm'); + if (arg > ZCP_MAX_MEMLIMIT || arg == 0) { + (void) fprintf(stderr, gettext( + "Invalid memory limit: " + "%s\n"), optarg); + return (1); + } else { + memlimit = arg; + } + } + break; + } + case '?': + (void) fprintf(stderr, gettext("invalid option '%c'\n"), + optopt); + goto usage; + } + } + + argc -= optind; + argv += optind; + + if (argc < 2) { + (void) fprintf(stderr, + gettext("invalid number of arguments\n")); + goto usage; + } + + poolname = argv[0]; + filename = argv[1]; + if (strcmp(filename, "-") == 0) { + fd = 0; + filename = "standard input"; + } else if ((fd = open(filename, O_RDONLY)) < 0) { + (void) fprintf(stderr, gettext("cannot open '%s': %s\n"), + filename, strerror(errno)); + return (1); + } + + if ((zhp = zpool_open(g_zfs, poolname)) == NULL) { + (void) fprintf(stderr, gettext("cannot open pool '%s'"), + poolname); + return (1); + } + zpool_close(zhp); + + /* + * Read in the channel program, expanding the program buffer as + * necessary. + */ + progread = 0; + progsize = 1024; + progbuf = safe_malloc(progsize); + do { + ret = read(fd, progbuf + progread, progsize - progread); + progread += ret; + if (progread == progsize && ret > 0) { + progsize *= 2; + progbuf = safe_realloc(progbuf, progsize); + } + } while (ret > 0); + + if (fd != 0) + (void) close(fd); + if (ret < 0) { + free(progbuf); + (void) fprintf(stderr, + gettext("cannot read '%s': %s\n"), + filename, strerror(errno)); + return (1); + } + progbuf[progread] = '\0'; + + /* + * Any remaining arguments are passed as arguments to the lua script as + * a string array: + * { + * "argv" -> [ "arg 1", ... "arg n" ], + * } + */ + nvlist_t *argnvl = fnvlist_alloc(); + fnvlist_add_string_array(argnvl, ZCP_ARG_CLIARGV, argv + 2, argc - 2); + + ret = lzc_channel_program(poolname, progbuf, instrlimit, memlimit, + argnvl, &outnvl); + + if (ret != 0) { + /* + * On error, report the error message handed back by lua if one + * exists. Otherwise, generate an appropriate error message, + * falling back on strerror() for an unexpected return code. + */ + char *errstring = NULL; + if (nvlist_exists(outnvl, ZCP_RET_ERROR)) { + (void) nvlist_lookup_string(outnvl, + ZCP_RET_ERROR, &errstring); + if (errstring == NULL) + errstring = strerror(ret); + } else { + switch (ret) { + case EINVAL: + errstring = + "Invalid instruction or memory limit."; + break; + case ENOMEM: + errstring = "Return value too large."; + break; + case ENOSPC: + errstring = "Memory limit exhausted."; + break; + case ETIME: + errstring = "Timed out."; + break; + case EPERM: + errstring = "Permission denied. Channel " + "programs must be run as root."; + break; + default: + errstring = strerror(ret); + } + } + (void) fprintf(stderr, + gettext("Channel program execution failed:\n%s\n"), + errstring); + } else { + (void) printf("Channel program fully executed "); + if (nvlist_empty(outnvl)) { + (void) printf("with no return value.\n"); + } else { + (void) printf("with return value:\n"); + dump_nvlist(outnvl, 4); + } + } + + free(progbuf); + fnvlist_free(outnvl); + fnvlist_free(argnvl); return (ret != 0); usage: Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Wed Sep 13 10:45:49 2017 (r323530) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. * Copyright (c) 2012 by Frederik Wessels. All rights reserved. * Copyright (c) 2013 by Prasad Joshi (sTec). All rights reserved. * Copyright 2016 Igor Kozhukhov . @@ -5159,6 +5159,11 @@ get_history_one(zpool_handle_t *zhp, void *data) (void) printf(" output:\n"); dump_nvlist(fnvlist_lookup_nvlist(rec, ZPOOL_HIST_OUTPUT_NVL), 8); + } + if (nvlist_exists(rec, ZPOOL_HIST_ERRNO)) { + (void) printf(" errno: %lld\n", + fnvlist_lookup_int64(rec, + ZPOOL_HIST_ERRNO)); } } else { if (!cb->internal) Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Sep 13 10:45:49 2017 (r323530) @@ -2323,6 +2323,74 @@ zfs_get_clones_nvl(zfs_handle_t *zhp) } /* + * Accepts a property and value and checks that the value + * matches the one found by the channel program. If they are + * not equal, print both of them. + */ +void +zcp_check(zfs_handle_t *zhp, zfs_prop_t prop, uint64_t intval, + const char *strval) +{ + if (!zhp->zfs_hdl->libzfs_prop_debug) + return; + int error; + char *poolname = zhp->zpool_hdl->zpool_name; + const char *program = + "args = ...\n" + "ds = args['dataset']\n" + "prop = args['property']\n" + "value, setpoint = zfs.get_prop(ds, prop)\n" + "return {value=value, setpoint=setpoint}\n"; + nvlist_t *outnvl; + nvlist_t *retnvl; + nvlist_t *argnvl = fnvlist_alloc(); + + fnvlist_add_string(argnvl, "dataset", zhp->zfs_name); + fnvlist_add_string(argnvl, "property", zfs_prop_to_name(prop)); + + error = lzc_channel_program(poolname, program, + 10 * 1000 * 1000, 10 * 1024 * 1024, argnvl, &outnvl); + + if (error == 0) { + retnvl = fnvlist_lookup_nvlist(outnvl, "return"); + if (zfs_prop_get_type(prop) == PROP_TYPE_NUMBER) { + int64_t ans; + error = nvlist_lookup_int64(retnvl, "value", &ans); + if (error != 0) { + (void) fprintf(stderr, "zcp check error: %u\n", + error); + return; + } + if (ans != intval) { + (void) fprintf(stderr, + "%s: zfs found %lld, but zcp found %lld\n", + zfs_prop_to_name(prop), + (longlong_t)intval, (longlong_t)ans); + } + } else { + char *str_ans; + error = nvlist_lookup_string(retnvl, "value", &str_ans); + if (error != 0) { + (void) fprintf(stderr, "zcp check error: %u\n", + error); + return; + } + if (strcmp(strval, str_ans) != 0) { + (void) fprintf(stderr, + "%s: zfs found %s, but zcp found %s\n", + zfs_prop_to_name(prop), + strval, str_ans); + } + } + } else { + (void) fprintf(stderr, + "zcp check failed, channel program error: %u\n", error); + } + nvlist_free(argnvl); + nvlist_free(outnvl); +} + +/* * Retrieve a property from the given object. If 'literal' is specified, then * numbers are left as exact values. Otherwise, numbers are converted to a * human-readable form. @@ -2368,6 +2436,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char &t) == 0) (void) snprintf(propbuf, proplen, "%llu", val); } + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_MOUNTPOINT: @@ -2436,7 +2505,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char /* 'legacy' or 'none' */ (void) strlcpy(propbuf, str, proplen); } - + zcp_check(zhp, prop, NULL, propbuf); break; case ZFS_PROP_ORIGIN: @@ -2444,6 +2513,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char if (str == NULL) return (-1); (void) strlcpy(propbuf, str, proplen); + zcp_check(zhp, prop, NULL, str); break; case ZFS_PROP_CLONES: @@ -2458,7 +2528,6 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char if (get_numeric_property(zhp, prop, src, &source, &val) != 0) return (-1); - /* * If quota or reservation is 0, we translate this into 'none' * (unless literal is set), and indicate that it's the default @@ -2477,6 +2546,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char else zfs_nicenum(val, propbuf, proplen); } + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_FILESYSTEM_LIMIT: @@ -2501,6 +2571,8 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char } else { zfs_nicenum(val, propbuf, proplen); } + + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_REFRATIO: @@ -2510,6 +2582,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char (void) snprintf(propbuf, proplen, "%llu.%02llux", (u_longlong_t)(val / 100), (u_longlong_t)(val % 100)); + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_TYPE: @@ -2530,6 +2603,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char abort(); } (void) snprintf(propbuf, proplen, "%s", str); + zcp_check(zhp, prop, NULL, propbuf); break; case ZFS_PROP_MOUNTED: @@ -2555,6 +2629,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char * consumers. */ (void) strlcpy(propbuf, zhp->zfs_name, proplen); + zcp_check(zhp, prop, NULL, propbuf); break; case ZFS_PROP_MLSLABEL: @@ -2604,26 +2679,33 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char if (get_numeric_property(zhp, prop, src, &source, &val) != 0) return (-1); (void) snprintf(propbuf, proplen, "%llu", (u_longlong_t)val); + zcp_check(zhp, prop, val, NULL); break; default: switch (zfs_prop_get_type(prop)) { case PROP_TYPE_NUMBER: if (get_numeric_property(zhp, prop, src, - &source, &val) != 0) + &source, &val) != 0) { return (-1); - if (literal) + } + + if (literal) { (void) snprintf(propbuf, proplen, "%llu", (u_longlong_t)val); - else + } else { zfs_nicenum(val, propbuf, proplen); + } + zcp_check(zhp, prop, val, NULL); break; case PROP_TYPE_STRING: str = getprop_string(zhp, prop, &source); if (str == NULL) return (-1); + (void) strlcpy(propbuf, str, proplen); + zcp_check(zhp, prop, NULL, str); break; case PROP_TYPE_INDEX: @@ -2632,7 +2714,9 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char return (-1); if (zfs_prop_index_to_string(prop, val, &strval) != 0) return (-1); + (void) strlcpy(propbuf, strval, proplen); + zcp_check(zhp, prop, NULL, strval); break; default: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h Wed Sep 13 10:45:49 2017 (r323530) @@ -79,6 +79,7 @@ struct libzfs_handle { libzfs_fru_t **libzfs_fru_hash; libzfs_fru_t *libzfs_fru_list; char libzfs_chassis_id[256]; + boolean_t libzfs_prop_debug; }; struct zfs_handle { Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_util.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Wed Sep 13 10:45:49 2017 (r323530) @@ -651,6 +651,10 @@ libzfs_init(void) zpool_feature_init(); libzfs_mnttab_init(hdl); + if (getenv("ZFS_PROP_DEBUG") != NULL) { + hdl->libzfs_prop_debug = B_TRUE; + } + return (hdl); } Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Wed Sep 13 10:45:49 2017 (r323530) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 RackTop Systems. @@ -152,7 +152,15 @@ lzc_ioctl(zfs_ioc_t ioc, const char *name, } while (ioctl(g_fd, ioc, &zc) != 0) { - if (errno == ENOMEM && resultp != NULL) { + /* + * If ioctl exited with ENOMEM, we retry the ioctl after + * increasing the size of the destination nvlist. + * + * Channel programs that exit with ENOMEM probably ran over the + * lua memory sandbox; they should not be retried. + */ + if (errno == ENOMEM && resultp != NULL && + ioc != ZFS_IOC_CHANNEL_PROGRAM) { free((void *)(uintptr_t)zc.zc_nvlist_dst); zc.zc_nvlist_dst_size *= 2; zc.zc_nvlist_dst = (uint64_t)(uintptr_t) @@ -864,6 +872,57 @@ lzc_destroy_bookmarks(nvlist_t *bmarks, nvlist_t **err pool[strcspn(pool, "/#")] = '\0'; error = lzc_ioctl(ZFS_IOC_DESTROY_BOOKMARKS, pool, bmarks, errlist); + + return (error); +} + +/* + * Executes a channel program. + * + * If this function returns 0 the channel program was successfully loaded and + * ran without failing. Note that individual commands the channel program ran + * may have failed and the channel program is responsible for reporting such + * errors through outnvl if they are important. + * + * This method may also return: + * + * EINVAL The program contains syntax errors, or an invalid memory or time + * limit was given. No part of the channel program was executed. + * If caused by syntax errors, 'outnvl' contains information about the + * errors. + * + * ECHRNG The program was executed, but encountered a runtime error, such as + * calling a function with incorrect arguments, invoking the error() + * function directly, failing an assert() command, etc. Some portion + * of the channel program may have executed and committed changes. + * Information about the failure can be found in 'outnvl'. + * + * ENOMEM The program fully executed, but the output buffer was not large + * enough to store the returned value. No output is returned through + * 'outnvl'. + * + * ENOSPC The program was terminated because it exceeded its memory usage + * limit. Some portion of the channel program may have executed and + * committed changes to disk. No output is returned through 'outnvl'. + * + * ETIME The program was terminated because it exceeded its Lua instruction + * limit. Some portion of the channel program may have executed and + * committed changes to disk. No output is returned through 'outnvl'. + */ +int +lzc_channel_program(const char *pool, const char *program, uint64_t instrlimit, + uint64_t memlimit, nvlist_t *argnvl, nvlist_t **outnvl) +{ + int error; + nvlist_t *args; + + args = fnvlist_alloc(); + fnvlist_add_string(args, ZCP_ARG_PROGRAM, program); + fnvlist_add_nvlist(args, ZCP_ARG_ARGLIST, argnvl); + fnvlist_add_uint64(args, ZCP_ARG_INSTRLIMIT, instrlimit); + fnvlist_add_uint64(args, ZCP_ARG_MEMLIMIT, memlimit); + error = lzc_ioctl(ZFS_IOC_CHANNEL_PROGRAM, pool, args, outnvl); + fnvlist_free(args); return (error); } Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Wed Sep 13 10:45:49 2017 (r323530) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 RackTop Systems. */ @@ -85,6 +85,9 @@ boolean_t lzc_exists(const char *); int lzc_rollback(const char *, char *, int); int lzc_rollback_to(const char *, const char *); + +int lzc_channel_program(const char *, const char *, uint64_t, uint64_t, + nvlist_t *, nvlist_t **); #ifdef __cplusplus } Modified: vendor/illumos/dist/lib/libzpool/common/kernel.c ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Sep 13 10:45:49 2017 (r323530) @@ -702,6 +702,7 @@ vpanic(const char *fmt, va_list adx) char buf[512]; (void) vsnprintf(buf, 512, fmt, adx); assfail(buf, NULL, 0); + abort(); /* necessary to make vpanic meet noreturn requirements */ } void Modified: vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Wed Sep 13 10:45:49 2017 (r323530) @@ -62,6 +62,7 @@ extern "C" { #include #include #include +#include #include #include #include @@ -107,8 +108,8 @@ extern void dprintf_setup(int *argc, char **argv); extern void cmn_err(int, const char *, ...); extern void vcmn_err(int, const char *, __va_list); -extern void panic(const char *, ...); -extern void vpanic(const char *, __va_list); +extern void panic(const char *, ...) __NORETURN; +extern void vpanic(const char *, __va_list) __NORETURN; #define fm_panic panic @@ -317,6 +318,7 @@ extern void kstat_runq_back_to_waitq(kstat_io_t *); #define KM_SLEEP UMEM_NOFAIL #define KM_PUSHPAGE KM_SLEEP #define KM_NOSLEEP UMEM_DEFAULT +#define KM_NORMALPRI 0 /* not needed with UMEM_DEFAULT */ #define KMC_NODEBUG UMC_NODEBUG #define KMC_NOTOUCH 0 /* not needed for userland caches */ #define kmem_alloc(_s, _f) umem_alloc(_s, _f) Added: vendor/illumos/dist/man/man1m/zfs-program.1m ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/illumos/dist/man/man1m/zfs-program.1m Wed Sep 13 10:45:49 2017 (r323530) @@ -0,0 +1,499 @@ +.\" This file and its contents are supplied under the terms of the +.\" Common Development and Distribution License ("CDDL"), version 1.0. +.\" You may only use this file in accordance with the terms of version +.\" 1.0 of the CDDL. +.\" +.\" A full copy of the text of the CDDL should have accompanied this +.\" source. A copy of the CDDL is also available via the Internet at +.\" http://www.illumos.org/license/CDDL. +.\" +.\" +.\" Copyright (c) 2016 by Delphix. All Rights Reserved. +.\" +.Dd January 21, 2016 +.Dt ZFS-PROGRAM 1M +.Os +.Sh NAME +.Nm zfs program +.Nd executes ZFS channel programs +.Sh SYNOPSIS +.Cm zfs program +.Op Fl t Ar instruction-limit +.Op Fl m Ar memory-limit +.Ar pool +.Ar script +.\".Op Ar optional arguments to channel program +.Sh DESCRIPTION +The ZFS channel program interface allows ZFS administrative operations to be +run programmatically as a Lua script. +The entire script is executed atomically, with no other administrative +operations taking effect concurrently. +A library of ZFS calls is made available to channel program scripts. +Channel programs may only be run with root privileges. +.Pp +A modified version of the Lua 5.2 interpreter is used to run channel program +scripts. +The Lua 5.2 manual can be found at: +.Bd -centered -offset indent +.Lk http://www.lua.org/manual/5.2/ +.Ed +.Pp +The channel program given by +.Ar script +will be run on +.Ar pool , +and any attempts to access or modify other pools will cause an error. +.Sh OPTIONS +.Bl -tag -width "-t" +.It Fl t Ar instruction-limit +Execution time limit, in number of Lua instructions to execute. +If a channel program executes more than the specified number of instructions, +it will be stopped and an error will be returned. +The default limit is 10 million instructions, and it can be set to a maximum of +100 million instructions. +.It Fl m Ar memory-limit +Memory limit, in bytes. +If a channel program attempts to allocate more memory than the given limit, it +will be stopped and an error returned. +The default memory limit is 10 MB, and can be set to a maximum of 100 MB. +.El +.Pp +All remaining argument strings will be passed directly to the Lua script as +described in the +.Sx LUA INTERFACE +section below. +.Sh LUA INTERFACE +A channel program can be invoked either from the command line, or via a library +call to +.Fn lzc_channel_program . +.Ss Arguments +Arguments passed to the channel program are converted to a Lua table. +If invoked from the command line, extra arguments to the Lua script will be +accessible as an array stored in the argument table with the key 'argv': +.Bd -literal -offset indent +args = ... +argv = args["argv"] +-- argv == {1="arg1", 2="arg2", ...} +.Ed +.Pp +If invoked from the libZFS interface, an arbitrary argument list can be +passed to the channel program, which is accessible via the same +"..." syntax in Lua: +.Bd -literal -offset indent +args = ... +-- args == {"foo"="bar", "baz"={...}, ...} +.Ed +.Pp +Note that because Lua arrays are 1-indexed, arrays passed to Lua from the +libZFS interface will have their indices incremented by 1. +That is, the element +in +.Va arr[0] +in a C array passed to a channel program will be stored in +.Va arr[1] +when accessed from Lua. +.Ss Return Values +Lua return statements take the form: +.Bd -literal -offset indent +return ret0, ret1, ret2, ... +.Ed +.Pp +Return statements returning multiple values are permitted internally in a +channel program script, but attempting to return more than one value from the +top level of the channel program is not permitted and will throw an error. +However, tables containing multiple values can still be returned. +If invoked from the command line, a return statement: +.Bd -literal -offset indent +a = {foo="bar", baz=2} +return a +.Ed +.Pp +Will be output formatted as: +.Bd -literal -offset indent +Channel program fully executed with return value: + return: + baz: 2 + foo: 'bar' +.Ed +.Ss Fatal Errors +If the channel program encounters a fatal error while running, a non-zero exit +status will be returned. +If more information about the error is available, a singleton list will be +returned detailing the error: +.Bd -literal -offset indent +error: "error string, including Lua stack trace" +.Ed +.Pp +If a fatal error is returned, the channel program may have not executed at all, +may have partially executed, or may have fully executed but failed to pass a +return value back to userland. +.Pp +If the channel program exhausts an instruction or memory limit, a fatal error +will be generated and the program will be stopped, leaving the program partially +executed. +No attempt is made to reverse or undo any operations already performed. +Note that because both the instruction count and amount of memory used by a +channel program are deterministic when run against the same inputs and +filesystem state, as long as a channel program has run successfully once, you +can guarantee that it will finish successfully against a similar size system. +.Pp +If a channel program attempts to return too large a value, the program will +fully execute but exit with a nonzero status code and no return value. +.Pp +.Em Note: +ZFS API functions do not generate Fatal Errors when correctly invoked, they +return an error code and the channel program continues executing. +See the +.Sx ZFS API +section below for function-specific details on error return codes. +.Ss Lua to C Value Conversion +When invoking a channel program via the libZFS interface, it is necessary to +translate arguments and return values from Lua values to their C equivalents, +and vice-versa. +.Pp +There is a correspondence between nvlist values in C and Lua tables. +A Lua table which is returned from the channel program will be recursively +converted to an nvlist, with table values converted to their natural +equivalents: +.Bd -literal -offset indent +string -> string +number -> int64 +boolean -> boolean_value +nil -> boolean (no value) +table -> nvlist +.Ed +.Pp +Likewise, table keys are replaced by string equivalents as follows: +.Bd -literal -offset indent +string -> no change +number -> signed decimal string ("%lld") +boolean -> "true" | "false" +.Ed +.Pp +Any collision of table key strings (for example, the string "true" and a +true boolean value) will cause a fatal error. +.Pp +Lua numbers are represented internally as signed 64-bit integers. +.Sh LUA STANDARD LIBRARY +The following Lua built-in base library functions are available: +.Bd -literal -offset indent +assert rawlen +collectgarbage rawget +error rawset +getmetatable select +ipairs setmetatable +next tonumber +pairs tostring +rawequal type +.Ed +.Pp +All functions in the +.Em coroutine , +.Em string , +and +.Em table +built-in submodules are also available. +A complete list and documentation of these modules is available in the Lua +manual. +.Pp +The following functions base library functions have been disabled and are +not available for use in channel programs: +.Bd -literal -offset indent +dofile +loadfile +load +pcall +print +xpcall +.Ed +.Sh ZFS API +.Ss Function Arguments +Each API function takes a fixed set of required positional arguments and +optional keyword arguments. +For example, the destroy function takes a single positional string argument +(the name of the dataset to destroy) and an optional "defer" keyword boolean +argument. +When using parentheses to specify the arguments to a Lua function, only +positional arguments can be used: +.Bd -literal -offset indent +zfs.sync.destroy("rpool@snap") +.Ed +.Pp +To use keyword arguments, functions must be called with a single argument that +is a Lua table containing entries mapping integers to positional arguments and +strings to keyword arguments: +.Bd -literal -offset indent +zfs.sync.destroy({1="rpool@snap", defer=true}) +.Ed +.Pp +The Lua language allows curly braces to be used in place of parenthesis as +syntactic sugar for this calling convention: +.Bd -literal -offset indent +zfs.sync.snapshot{"rpool@snap", defer=true} +.Ed +.Ss Function Return Values +If an API function succeeds, it returns 0. +If it fails, it returns an error code and the channel program continues +executing. +API functions do not generate Fatal Errors except in the case of an +unrecoverable internal file system error. +.Pp +In addition to returning an error code, some functions also return extra +details describing what caused the error. +This extra description is given as a second return value, and will always be a +Lua table, or Nil if no error details were returned. +Different keys will exist in the error details table depending on the function +and error case. +Any such function may be called expecting a single return value: +.Bd -literal -offset indent +errno = zfs.sync.promote(dataset) +.Ed +.Pp +Or, the error details can be retrieved: +.Bd -literal -offset indent +errno, details = zfs.sync.promote(dataset) +if (errno == EEXIST) then + assert(details ~= Nil) + list_of_conflicting_snapshots = details +end +.Ed +.Pp +The following global aliases for API function error return codes are defined +for use in channel programs: +.Bd -literal -offset indent +EPERM ECHILD ENODEV ENOSPC +ENOENT EAGAIN ENOTDIR ESPIPE +ESRCH ENOMEM EISDIR EROFS +EINTR EACCES EINVAL EMLINK +EIO EFAULT ENFILE EPIPE +ENXIO ENOTBLK EMFILE EDOM +E2BIG EBUSY ENOTTY ERANGE +ENOEXEC EEXIST ETXTBSY EDQUOT +EBADF EXDEV EFBIG +.Ed +.Ss API Functions +For detailed descriptions of the exact behavior of any zfs administrative +operations, see the main +.Xr zfs 1 +manual page. +.Bl -tag -width "xx" +.It Em zfs.debug(msg) +Record a debug message in the zfs_dbgmsg log. +A log of these messages can be printed via mdb's "::zfs_dbgmsg" command, or +can be monitored live by running: +.Bd -literal -offset indent + dtrace -n 'zfs-dbgmsg{trace(stringof(arg0))}' +.Ed +.Pp +msg (string) +.Bd -ragged -compact -offset "xxxx" +Debug message to be printed. +.Ed +.It Em zfs.get_prop(dataset, property) +Returns two values. +First, a string, number or table containing the property value for the given +dataset. +Second, a string containing the source of the property (i.e. the name of the +dataset in which it was set or nil if it is readonly). +Throws a Lua error if the dataset is invalid or the property doesn't exist. +Note that Lua only supports int64 number types whereas ZFS number properties +are uint64. +This means very large values (like guid) may wrap around and appear negative. +.Pp +dataset (string) +.Bd -ragged -compact -offset "xxxx" +Filesystem or snapshot path to retrieve properties from. +.Ed +.Pp +property (string) +.Bd -ragged -compact -offset "xxxx" +Name of property to retrieve. +All filesystem, snapshot and volume properties are supported except +for 'mounted' and 'iscsioptions.' +Also supports the 'written@snap' and 'written#bookmark' properties and +the '@id' properties, though the id must be in numeric +form. +.Ed +.El +.Bl -tag -width "xx" +.It Sy zfs.sync submodule +The sync submodule contains functions that modify the on-disk state. +They are executed in "syncing context". +.Pp +The available sync submodule functions are as follows: +.Bl -tag -width "xx" +.It Em zfs.sync.destroy(dataset, [defer=true|false]) +Destroy the given dataset. +Returns 0 on successful destroy, or a nonzero error code if the dataset could +not be destroyed (for example, if the dataset has any active children or +clones). +.Pp +dataset (string) +.Bd -ragged -compact -offset "xxxx" +Filesystem or snapshot to be destroyed. +.Ed +.Pp +[optional] defer (boolean) +.Bd -ragged -compact -offset "xxxx" +Valid only for destroying snapshots. +If set to true, and the snapshot has holds or clones, allows the snapshot to be +marked for deferred deletion rather than failing. +.Ed +.It Em zfs.sync.promote(dataset) +Promote the given clone to a filesystem. *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:54:58 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BC47E1A26B; Wed, 13 Sep 2017 10:54:58 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB878827A9; Wed, 13 Sep 2017 10:54:57 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAsvxT067962; Wed, 13 Sep 2017 10:54:57 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAsv08067960; Wed, 13 Sep 2017 10:54:57 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131054.v8DAsv08067960@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:54:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323531 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 323531 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:54:58 -0000 Author: avg Date: Wed Sep 13 10:54:56 2017 New Revision: 323531 URL: https://svnweb.freebsd.org/changeset/base/323531 Log: 8521 nvlist memory leak in get_clones_stat() and spa_load_best() illumos/illumos-gate@7d3000f774e20097a1ee45cbd06d0e38065ddd5a https://github.com/illumos/illumos-gate/commit/7d3000f774e20097a1ee45cbd06d0e38065ddd5a https://www.illumos.org/issues/8521 Yuri reported this to the mailing list: doing a `reboot -d` on current illumos-gate HEAD gives the following ":: findleaks -dv" output: findleaks: maximum buffers => 301061 findleaks: actual buffers => 297587 findleaks: findleaks: potential pointers => 29289774 findleaks: dismissals => 26242305 (89.5%) findleaks: misses => 331153 ( 1.1%) findleaks: dups => 2419681 ( 8.2%) findleaks: follows => 296635 ( 1.0%) findleaks: findleaks: peak memory usage => 7353 kB findleaks: elapsed CPU time => 1.5 seconds findleaks: elapsed wall time => 2.0 seconds findleaks: CACHE LEAKED BUFCTL CALLER ffffff03d222b008 120 ffffff03ef7ceb78 nv_alloc_sys+0x1f ffffff03d222a448 123 ffffff03f4150cc8 nv_alloc_sys+0x1f ffffff03d222b448 5 ffffff03f28bd598 nv_alloc_sys+0x1f ffffff03d222b888 87 ffffff03f28c10f0 nv_alloc_sys+0x1f ffffff03d222c008 21 ffffff03f4139310 nv_alloc_sys+0x1f ffffff03d222b888 43 ffffff040ef3f3e8 nv_alloc_sys+0x1f ffffff03d222c008 120 ffffff03f4591e58 nv_alloc_sys+0x1f ffffff03d222b008 121 ffffff03f352c068 nv_alloc_sys+0x1f ffffff03d222a448 112 ffffff03f414e5f8 nv_alloc_sys+0x1f ffffff03d222b008 119 ffffff03ee92fdc0 nv_alloc_sys+0x1f ffffff03d222b888 46 ffffff03f28c1378 nv_alloc_sys+0x1f ffffff03d222b448 4 ffffff03f28c7708 nv_alloc_sys+0x1f ffffff03d222c008 20 ffffff03f2a6e7e8 nv_alloc_sys+0x1f Reviewed by: Steve Gonczi Reviewed by: George Wilson Reviewed by: Yuri Pankov Reviewed by: Matt Ahrens Approved by: Dan McDonald Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Wed Sep 13 10:45:49 2017 (r323530) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c Wed Sep 13 10:54:56 2017 (r323531) @@ -1742,10 +1742,10 @@ get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) fnvlist_add_nvlist(propval, ZPROP_VALUE, val); fnvlist_add_nvlist(nv, zfs_prop_to_name(ZFS_PROP_CLONES), propval); - } else { - nvlist_free(val); - nvlist_free(propval); } + + nvlist_free(val); + nvlist_free(propval); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Sep 13 10:45:49 2017 (r323530) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Sep 13 10:54:56 2017 (r323531) @@ -3025,6 +3025,8 @@ spa_load_best(spa_t *spa, spa_load_state_t state, int if (config && (rewind_error || state != SPA_LOAD_RECOVER)) spa_config_set(spa, config); + else + nvlist_free(config); if (state == SPA_LOAD_RECOVER) { ASSERT3P(loadinfo, ==, NULL); From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:56:20 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6A9AE1A358; Wed, 13 Sep 2017 10:56:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A3E33829EC; Wed, 13 Sep 2017 10:56:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAuJdK068136; Wed, 13 Sep 2017 10:56:19 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAuJKs068135; Wed, 13 Sep 2017 10:56:19 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131056.v8DAuJKs068135@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:56:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323533 - vendor-sys/illumos/dist/uts/common/fs/zfs/lua X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs/lua X-SVN-Commit-Revision: 323533 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:56:21 -0000 Author: avg Date: Wed Sep 13 10:56:19 2017 New Revision: 323533 URL: https://svnweb.freebsd.org/changeset/base/323533 Log: 8552 ZFS LUA code uses floating point math illumos/illumos-gate@916c8d881190bd2c3ca20d9fca919aecff504435 https://github.com/illumos/illumos-gate/commit/916c8d881190bd2c3ca20d9fca919aecff504435 https://www.illumos.org/issues/8552 In the LUA interpreter used by "zfs program", the lua format() function accidentally includes support for '%f' and friends, which can cause compilation problems when building on platforms that don't support floating-point math in the kernel (e.g. sparc). Support for '%f' friends (%f %e %E %g %G) should be removed, since there's no way to supply a floating-point value anyway (all numbers in ZFS LUA are int64_t's). Reviewed by: Yuri Pankov Reviewed by: Igor Kozhukhov Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Wed Sep 13 10:56:02 2017 (r323532) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c Wed Sep 13 10:56:19 2017 (r323533) @@ -958,6 +958,7 @@ static int str_format (lua_State *L) { nb = str_sprintf(buff, form, ni); break; } +#if defined(LUA_USE_FLOAT_FORMATS) case 'e': case 'E': case 'f': #if defined(LUA_USE_AFORMAT) case 'a': case 'A': @@ -967,6 +968,7 @@ static int str_format (lua_State *L) { nb = str_sprintf(buff, form, (LUA_FLTFRM_T)luaL_checknumber(L, arg)); break; } +#endif case 'q': { addquoted(L, &b, arg); break; From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:57:53 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E8D3E1A4B0; Wed, 13 Sep 2017 10:57:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1B60F82BAF; Wed, 13 Sep 2017 10:57:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAvqC2068231; Wed, 13 Sep 2017 10:57:52 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAvq3K068230; Wed, 13 Sep 2017 10:57:52 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131057.v8DAvq3K068230@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:57:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323534 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 323534 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:57:53 -0000 Author: avg Date: Wed Sep 13 10:57:52 2017 New Revision: 323534 URL: https://svnweb.freebsd.org/changeset/base/323534 Log: 8590 memory leak in dsl_destroy_snapshots_nvl() illumos/illumos-gate@e6ab4525d156c82445c116ecf6b2b874d5e9009d https://github.com/illumos/illumos-gate/commit/e6ab4525d156c82445c116ecf6b2b874d5e9009d https://www.illumos.org/issues/8590 In dsl_destroy_snapshots_nvl(), "snaps_normalized" is not freed after it is added to "arg". Reviewed by: Pavel Zakharov Reviewed by: Steve Gonczi Reviewed by: George Wilson Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Wed Sep 13 10:56:19 2017 (r323533) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c Wed Sep 13 10:57:52 2017 (r323534) @@ -501,6 +501,7 @@ dsl_destroy_snapshots_nvl(nvlist_t *snaps, boolean_t d nvpair_name(pair), B_TRUE); } fnvlist_add_nvlist(arg, "snaps", snaps_normalized); + fnvlist_free(snaps_normalized); fnvlist_add_boolean_value(arg, "defer", defer); nvlist_t *wrapper = fnvlist_alloc(); From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:59:39 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 141A2E1A5C2; Wed, 13 Sep 2017 10:59:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1EE682CFB; Wed, 13 Sep 2017 10:59:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAxcw0068360; Wed, 13 Sep 2017 10:59:38 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAxcZZ068359; Wed, 13 Sep 2017 10:59:38 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131059.v8DAxcZZ068359@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:59:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323535 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 323535 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:59:39 -0000 Author: avg Date: Wed Sep 13 10:59:36 2017 New Revision: 323535 URL: https://svnweb.freebsd.org/changeset/base/323535 Log: 8585 improve batching done in zil_commit() illumos/illumos-gate@1271e4b10dfaaed576c08a812f466f6e81370e5e https://github.com/illumos/illumos-gate/commit/1271e4b10dfaaed576c08a812f466f6e81370e5e https://www.illumos.org/issues/8585 The current implementation of zil_commit() can introduce significant latency, beyond what is inherent due to the latency of the underlying storage. The additional latency comes from two main problems: 1. When there's outstanding ZIL blocks being written (i.e. there's already a "writer thread" in progress), then any new calls to zil_commit() will block waiting for the currently oustanding ZIL blocks to complete. The blocks written for each "writer thread" is coined a "batch", and there can only ever be a single "batch" being written at a time. When a batch is being written, any new ZIL transactions will have to wait for the next batch to be written, which won't occur until the current batch finishes. As a result, the underlying storage may not be used as efficiently as possible. While "new" threads enter zil_commit() and are blocked waiting for the next batch, it's possible that the underlying storage isn't fully utilized by the current batch of ZIL blocks. In that case, it'd be better to allow these new threads to generate (and issue) a new ZIL block, such that it could be serviced by the underlying storage concurrently with the other ZIL blocks that are being serviced. 2. Any call to zil_commit() must wait for all ZIL blocks in its "batch" to complete, prior to zil_commit() returning. The size of any given batch is proportional to the number of ZIL transaction in the queue at the time that the batch starts processing the queue; which doesn't occur until the previous batch completes. Thus, if there's a lot of transactions in the queue, the batch could be composed of many ZIL blocks, and each call to zil_commit() will have to wait for all of these writes to complete (even if the thread calling zil_commit() only cared about one of the transactions in the batch). Reviewed by: Brad Lewis Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Dan McDonald Author: Prakash Surya Modified: vendor/illumos/dist/cmd/ztest/ztest.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c vendor-sys/illumos/dist/uts/common/sys/debug.h Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Wed Sep 13 10:57:52 2017 (r323534) +++ vendor/illumos/dist/cmd/ztest/ztest.c Wed Sep 13 10:59:36 2017 (r323535) @@ -1829,13 +1829,14 @@ ztest_get_done(zgd_t *zgd, int error) ztest_object_unlock(zd, object); if (error == 0 && zgd->zgd_bp) - zil_add_block(zgd->zgd_zilog, zgd->zgd_bp); + zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); umem_free(zgd, sizeof (*zgd)); } static int -ztest_get_data(void *arg, lr_write_t *lr, char *buf, zio_t *zio) +ztest_get_data(void *arg, lr_write_t *lr, char *buf, struct lwb *lwb, + zio_t *zio) { ztest_ds_t *zd = arg; objset_t *os = zd->zd_os; @@ -1849,6 +1850,10 @@ ztest_get_data(void *arg, lr_write_t *lr, char *buf, z zgd_t *zgd; int error; + ASSERT3P(lwb, !=, NULL); + ASSERT3P(zio, !=, NULL); + ASSERT3U(size, !=, 0); + ztest_object_lock(zd, object, RL_READER); error = dmu_bonus_hold(os, object, FTAG, &db); if (error) { @@ -1869,7 +1874,7 @@ ztest_get_data(void *arg, lr_write_t *lr, char *buf, z db = NULL; zgd = umem_zalloc(sizeof (*zgd), UMEM_NOFAIL); - zgd->zgd_zilog = zd->zd_zilog; + zgd->zgd_lwb = lwb; zgd->zgd_private = zd; if (buf != NULL) { /* immediate write */ From owner-svn-src-vendor@freebsd.org Wed Sep 13 10:59:39 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27D13E1A5C3; Wed, 13 Sep 2017 10:59:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B866882CFA; Wed, 13 Sep 2017 10:59:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DAxbgj068354; Wed, 13 Sep 2017 10:59:37 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DAxbJ1068345; Wed, 13 Sep 2017 10:59:37 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201709131059.v8DAxbJ1068345@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 13 Sep 2017 10:59:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323535 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 323535 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 10:59:39 -0000 Author: avg Date: Wed Sep 13 10:59:36 2017 New Revision: 323535 URL: https://svnweb.freebsd.org/changeset/base/323535 Log: 8585 improve batching done in zil_commit() illumos/illumos-gate@1271e4b10dfaaed576c08a812f466f6e81370e5e https://github.com/illumos/illumos-gate/commit/1271e4b10dfaaed576c08a812f466f6e81370e5e https://www.illumos.org/issues/8585 The current implementation of zil_commit() can introduce significant latency, beyond what is inherent due to the latency of the underlying storage. The additional latency comes from two main problems: 1. When there's outstanding ZIL blocks being written (i.e. there's already a "writer thread" in progress), then any new calls to zil_commit() will block waiting for the currently oustanding ZIL blocks to complete. The blocks written for each "writer thread" is coined a "batch", and there can only ever be a single "batch" being written at a time. When a batch is being written, any new ZIL transactions will have to wait for the next batch to be written, which won't occur until the current batch finishes. As a result, the underlying storage may not be used as efficiently as possible. While "new" threads enter zil_commit() and are blocked waiting for the next batch, it's possible that the underlying storage isn't fully utilized by the current batch of ZIL blocks. In that case, it'd be better to allow these new threads to generate (and issue) a new ZIL block, such that it could be serviced by the underlying storage concurrently with the other ZIL blocks that are being serviced. 2. Any call to zil_commit() must wait for all ZIL blocks in its "batch" to complete, prior to zil_commit() returning. The size of any given batch is proportional to the number of ZIL transaction in the queue at the time that the batch starts processing the queue; which doesn't occur until the previous batch completes. Thus, if there's a lot of transactions in the queue, the batch could be composed of many ZIL blocks, and each call to zil_commit() will have to wait for all of these writes to complete (even if the thread calling zil_commit() only cared about one of the transactions in the batch). Reviewed by: Brad Lewis Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Dan McDonald Author: Prakash Surya Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c vendor-sys/illumos/dist/uts/common/sys/debug.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/ztest/ztest.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Wed Sep 13 10:59:36 2017 (r323535) @@ -1645,6 +1645,13 @@ dmu_sync_late_arrival(zio_t *pio, objset_t *os, dmu_sy return (SET_ERROR(EIO)); } + /* + * In order to prevent the zgd's lwb from being free'd prior to + * dmu_sync_late_arrival_done() being called, we have to ensure + * the lwb's "max txg" takes this tx's txg into account. + */ + zil_lwb_add_txg(zgd->zgd_lwb, dmu_tx_get_txg(tx)); + dsa = kmem_alloc(sizeof (dmu_sync_arg_t), KM_SLEEP); dsa->dsa_dr = NULL; dsa->dsa_done = done; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Wed Sep 13 10:59:36 2017 (r323535) @@ -911,7 +911,7 @@ uint64_t dmu_tx_get_txg(dmu_tx_t *tx); * {zfs,zvol,ztest}_get_done() args */ typedef struct zgd { - struct zilog *zgd_zilog; + struct lwb *zgd_lwb; struct blkptr *zgd_bp; dmu_buf_t *zgd_db; struct rl *zgd_rl; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h Wed Sep 13 10:59:36 2017 (r323535) @@ -40,6 +40,7 @@ extern "C" { struct dsl_pool; struct dsl_dataset; +struct lwb; /* * Intent log format: @@ -140,6 +141,7 @@ typedef enum zil_create { /* * Intent log transaction types and record structures */ +#define TX_COMMIT 0 /* Commit marker (no on-disk state) */ #define TX_CREATE 1 /* Create file */ #define TX_MKDIR 2 /* Make directory */ #define TX_MKXATTR 3 /* Make XATTR directory */ @@ -388,7 +390,8 @@ typedef int zil_parse_blk_func_t(zilog_t *zilog, blkpt typedef int zil_parse_lr_func_t(zilog_t *zilog, lr_t *lr, void *arg, uint64_t txg); typedef int zil_replay_func_t(); -typedef int zil_get_data_t(void *arg, lr_write_t *lr, char *dbuf, zio_t *zio); +typedef int zil_get_data_t(void *arg, lr_write_t *lr, char *dbuf, + struct lwb *lwb, zio_t *zio); extern int zil_parse(zilog_t *zilog, zil_parse_blk_func_t *parse_blk_func, zil_parse_lr_func_t *parse_lr_func, void *arg, uint64_t txg); @@ -426,7 +429,8 @@ extern void zil_clean(zilog_t *zilog, uint64_t synced_ extern int zil_suspend(const char *osname, void **cookiep); extern void zil_resume(void *cookie); -extern void zil_add_block(zilog_t *zilog, const blkptr_t *bp); +extern void zil_lwb_add_block(struct lwb *lwb, const blkptr_t *bp); +extern void zil_lwb_add_txg(struct lwb *lwb, uint64_t txg); extern int zil_bp_tree_add(zilog_t *zilog, const blkptr_t *bp); extern void zil_set_sync(zilog_t *zilog, uint64_t syncval); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h Wed Sep 13 10:59:36 2017 (r323535) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] */ @@ -37,22 +37,76 @@ extern "C" { #endif /* - * Log write buffer. + * Possbile states for a given lwb structure. An lwb will start out in + * the "closed" state, and then transition to the "opened" state via a + * call to zil_lwb_write_open(). After the lwb is "open", it can + * transition into the "issued" state via zil_lwb_write_issue(). After + * the lwb's zio completes, and the vdev's are flushed, the lwb will + * transition into the "done" state via zil_lwb_write_done(), and the + * structure eventually freed. */ +typedef enum { + LWB_STATE_CLOSED, + LWB_STATE_OPENED, + LWB_STATE_ISSUED, + LWB_STATE_DONE, + LWB_NUM_STATES +} lwb_state_t; + +/* + * Log write block (lwb) + * + * Prior to an lwb being issued to disk via zil_lwb_write_issue(), it + * will be protected by the zilog's "zl_writer_lock". Basically, prior + * to it being issued, it will only be accessed by the thread that's + * holding the "zl_writer_lock". After the lwb is issued, the zilog's + * "zl_lock" is used to protect the lwb against concurrent access. + */ typedef struct lwb { zilog_t *lwb_zilog; /* back pointer to log struct */ blkptr_t lwb_blk; /* on disk address of this log blk */ boolean_t lwb_slog; /* lwb_blk is on SLOG device */ int lwb_nused; /* # used bytes in buffer */ int lwb_sz; /* size of block and buffer */ + lwb_state_t lwb_state; /* the state of this lwb */ char *lwb_buf; /* log write buffer */ - zio_t *lwb_zio; /* zio for this buffer */ + zio_t *lwb_write_zio; /* zio for the lwb buffer */ + zio_t *lwb_root_zio; /* root zio for lwb write and flushes */ dmu_tx_t *lwb_tx; /* tx for log block allocation */ uint64_t lwb_max_txg; /* highest txg in this lwb */ list_node_t lwb_node; /* zilog->zl_lwb_list linkage */ + list_t lwb_waiters; /* list of zil_commit_waiter's */ + avl_tree_t lwb_vdev_tree; /* vdevs to flush after lwb write */ + kmutex_t lwb_vdev_lock; /* protects lwb_vdev_tree */ + hrtime_t lwb_issued_timestamp; /* when was the lwb issued? */ } lwb_t; /* + * ZIL commit waiter. + * + * This structure is allocated each time zil_commit() is called, and is + * used by zil_commit() to communicate with other parts of the ZIL, such + * that zil_commit() can know when it safe for it return. For more + * details, see the comment above zil_commit(). + * + * The "zcw_lock" field is used to protect the commit waiter against + * concurrent access. This lock is often acquired while already holding + * the zilog's "zl_writer_lock" or "zl_lock"; see the functions + * zil_process_commit_list() and zil_lwb_flush_vdevs_done() as examples + * of this. Thus, one must be careful not to acquire the + * "zl_writer_lock" or "zl_lock" when already holding the "zcw_lock"; + * e.g. see the zil_commit_waiter_timeout() function. + */ +typedef struct zil_commit_waiter { + kcondvar_t zcw_cv; /* signalled when "done" */ + kmutex_t zcw_lock; /* protects fields of this struct */ + list_node_t zcw_node; /* linkage in lwb_t:lwb_waiter list */ + lwb_t *zcw_lwb; /* back pointer to lwb when linked */ + boolean_t zcw_done; /* B_TRUE when "done", else B_FALSE */ + int zcw_zio_error; /* contains the zio io_error value */ +} zil_commit_waiter_t; + +/* * Intent log transaction lists */ typedef struct itxs { @@ -94,20 +148,20 @@ struct zilog { const zil_header_t *zl_header; /* log header buffer */ objset_t *zl_os; /* object set we're logging */ zil_get_data_t *zl_get_data; /* callback to get object content */ - zio_t *zl_root_zio; /* log writer root zio */ + lwb_t *zl_last_lwb_opened; /* most recent lwb opened */ + hrtime_t zl_last_lwb_latency; /* zio latency of last lwb done */ uint64_t zl_lr_seq; /* on-disk log record sequence number */ uint64_t zl_commit_lr_seq; /* last committed on-disk lr seq */ uint64_t zl_destroy_txg; /* txg of last zil_destroy() */ uint64_t zl_replayed_seq[TXG_SIZE]; /* last replayed rec seq */ uint64_t zl_replaying_seq; /* current replay seq number */ uint32_t zl_suspend; /* log suspend count */ - kcondvar_t zl_cv_writer; /* log writer thread completion */ kcondvar_t zl_cv_suspend; /* log suspend completion */ uint8_t zl_suspending; /* log is currently suspending */ uint8_t zl_keep_first; /* keep first log block in destroy */ uint8_t zl_replay; /* replaying records while set */ uint8_t zl_stop_sync; /* for debugging */ - uint8_t zl_writer; /* boolean: write setup in progress */ + kmutex_t zl_writer_lock; /* single writer, per ZIL, at a time */ uint8_t zl_logbias; /* latency or throughput */ uint8_t zl_sync; /* synchronous or asynchronous */ int zl_parse_error; /* last zil_parse() error */ @@ -115,15 +169,10 @@ struct zilog { uint64_t zl_parse_lr_seq; /* highest lr seq on last parse */ uint64_t zl_parse_blk_count; /* number of blocks parsed */ uint64_t zl_parse_lr_count; /* number of log records parsed */ - uint64_t zl_next_batch; /* next batch number */ - uint64_t zl_com_batch; /* committed batch number */ - kcondvar_t zl_cv_batch[2]; /* batch condition variables */ itxg_t zl_itxg[TXG_SIZE]; /* intent log txg chains */ list_t zl_itx_commit_list; /* itx list to be committed */ uint64_t zl_cur_used; /* current commit log size used */ list_t zl_lwb_list; /* in-flight log write list */ - kmutex_t zl_vdev_lock; /* protects zl_vdev_tree */ - avl_tree_t zl_vdev_tree; /* vdevs to flush in zil_commit() */ avl_tree_t zl_bp_tree; /* track bps during log parse */ clock_t zl_replay_time; /* lbolt of when replay started */ uint64_t zl_replay_blks; /* number of log blocks replayed */ @@ -131,6 +180,7 @@ struct zilog { uint_t zl_prev_blks[ZIL_PREV_BLKS]; /* size - sector rounded */ uint_t zl_prev_rotor; /* rotor for zl_prev[] */ txg_node_t zl_dirty_link; /* protected by dp_dirty_zilogs list */ + uint64_t zl_dirty_max_txg; /* highest txg used to dirty zilog */ }; typedef struct zil_bp_node { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h Wed Sep 13 10:59:36 2017 (r323535) @@ -556,6 +556,7 @@ extern enum zio_checksum zio_checksum_dedup_select(spa extern enum zio_compress zio_compress_select(spa_t *spa, enum zio_compress child, enum zio_compress parent); +extern void zio_cancel(zio_t *zio); extern void zio_suspend(spa_t *spa, zio_t *zio); extern int zio_resume(spa_t *spa); extern void zio_resume_wait(spa_t *spa); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Wed Sep 13 10:59:36 2017 (r323535) @@ -158,7 +158,7 @@ txg_fini(dsl_pool_t *dp) tx_state_t *tx = &dp->dp_tx; int c; - ASSERT(tx->tx_threads == 0); + ASSERT0(tx->tx_threads); mutex_destroy(&tx->tx_sync_lock); @@ -199,7 +199,7 @@ txg_sync_start(dsl_pool_t *dp) dprintf("pool %p\n", dp); - ASSERT(tx->tx_threads == 0); + ASSERT0(tx->tx_threads); tx->tx_threads = 2; @@ -261,7 +261,7 @@ txg_sync_stop(dsl_pool_t *dp) /* * Finish off any work in progress. */ - ASSERT(tx->tx_threads == 2); + ASSERT3U(tx->tx_threads, ==, 2); /* * We need to ensure that we've vacated the deferred space_maps. @@ -273,7 +273,7 @@ txg_sync_stop(dsl_pool_t *dp) */ mutex_enter(&tx->tx_sync_lock); - ASSERT(tx->tx_threads == 2); + ASSERT3U(tx->tx_threads, ==, 2); tx->tx_exiting = 1; @@ -608,7 +608,7 @@ txg_wait_synced(dsl_pool_t *dp, uint64_t txg) ASSERT(!dsl_pool_config_held(dp)); mutex_enter(&tx->tx_sync_lock); - ASSERT(tx->tx_threads == 2); + ASSERT3U(tx->tx_threads, ==, 2); if (txg == 0) txg = tx->tx_open_txg + TXG_DEFER_SIZE; if (tx->tx_sync_txg_waiting < txg) @@ -633,7 +633,7 @@ txg_wait_open(dsl_pool_t *dp, uint64_t txg) ASSERT(!dsl_pool_config_held(dp)); mutex_enter(&tx->tx_sync_lock); - ASSERT(tx->tx_threads == 2); + ASSERT3U(tx->tx_threads, ==, 2); if (txg == 0) txg = tx->tx_open_txg + 1; if (tx->tx_quiesce_txg_waiting < txg) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Wed Sep 13 10:59:36 2017 (r323535) @@ -83,6 +83,7 @@ #include #include #include +#include /* * Programming rules. @@ -1032,7 +1033,7 @@ zfs_get_done(zgd_t *zgd, int error) VN_RELE_ASYNC(ZTOV(zp), dsl_pool_vnrele_taskq(dmu_objset_pool(os))); if (error == 0 && zgd->zgd_bp) - zil_add_block(zgd->zgd_zilog, zgd->zgd_bp); + zil_lwb_add_block(zgd->zgd_lwb, zgd->zgd_bp); kmem_free(zgd, sizeof (zgd_t)); } @@ -1045,7 +1046,7 @@ static int zil_fault_io = 0; * Get data to generate a TX_WRITE intent log record. */ int -zfs_get_data(void *arg, lr_write_t *lr, char *buf, zio_t *zio) +zfs_get_data(void *arg, lr_write_t *lr, char *buf, struct lwb *lwb, zio_t *zio) { zfsvfs_t *zfsvfs = arg; objset_t *os = zfsvfs->z_os; @@ -1057,8 +1058,9 @@ zfs_get_data(void *arg, lr_write_t *lr, char *buf, zio zgd_t *zgd; int error = 0; - ASSERT(zio != NULL); - ASSERT(size != 0); + ASSERT3P(lwb, !=, NULL); + ASSERT3P(zio, !=, NULL); + ASSERT3U(size, !=, 0); /* * Nothing to do if the file has been removed @@ -1076,7 +1078,7 @@ zfs_get_data(void *arg, lr_write_t *lr, char *buf, zio } zgd = (zgd_t *)kmem_zalloc(sizeof (zgd_t), KM_SLEEP); - zgd->zgd_zilog = zfsvfs->z_log; + zgd->zgd_lwb = lwb; zgd->zgd_private = zp; /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Wed Sep 13 10:57:52 2017 (r323534) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Wed Sep 13 10:59:36 2017 (r323535) @@ -42,32 +42,53 @@ #include /* - * The zfs intent log (ZIL) saves transaction records of system calls - * that change the file system in memory with enough information - * to be able to replay them. These are stored in memory until - * either the DMU transaction group (txg) commits them to the stable pool - * and they can be discarded, or they are flushed to the stable log - * (also in the pool) due to a fsync, O_DSYNC or other synchronous - * requirement. In the event of a panic or power fail then those log - * records (transactions) are replayed. + * The ZFS Intent Log (ZIL) saves "transaction records" (itxs) of system + * calls that change the file system. Each itx has enough information to + * be able to replay them after a system crash, power loss, or + * equivalent failure mode. These are stored in memory until either: * - * There is one ZIL per file system. Its on-disk (pool) format consists - * of 3 parts: + * 1. they are committed to the pool by the DMU transaction group + * (txg), at which point they can be discarded; or + * 2. they are committed to the on-disk ZIL for the dataset being + * modified (e.g. due to an fsync, O_DSYNC, or other synchronous + * requirement). * - * - ZIL header - * - ZIL blocks - * - ZIL records + * In the event of a crash or power loss, the itxs contained by each + * dataset's on-disk ZIL will be replayed when that dataset is first + * instantianted (e.g. if the dataset is a normal fileystem, when it is + * first mounted). * - * A log record holds a system call transaction. Log blocks can - * hold many log records and the blocks are chained together. - * Each ZIL block contains a block pointer (blkptr_t) to the next - * ZIL block in the chain. The ZIL header points to the first - * block in the chain. Note there is not a fixed place in the pool - * to hold blocks. They are dynamically allocated and freed as - * needed from the blocks available. Figure X shows the ZIL structure: + * As hinted at above, there is one ZIL per dataset (both the in-memory + * representation, and the on-disk representation). The on-disk format + * consists of 3 parts: + * + * - a single, per-dataset, ZIL header; which points to a chain of + * - zero or more ZIL blocks; each of which contains + * - zero or more ZIL records + * + * A ZIL record holds the information necessary to replay a single + * system call transaction. A ZIL block can hold many ZIL records, and + * the blocks are chained together, similarly to a singly linked list. + * + * Each ZIL block contains a block pointer (blkptr_t) to the next ZIL + * block in the chain, and the ZIL header points to the first block in + * the chain. + * + * Note, there is not a fixed place in the pool to hold these ZIL + * blocks; they are dynamically allocated and freed as needed from the + * blocks available on the pool, though they can be preferentially + * allocated from a dedicated "log" vdev. */ /* + * This controls the amount of time that a ZIL block (lwb) will remain + * "open" when it isn't "full", and it has a thread waiting for it to be + * committed to stable storage. Please refer to the zil_commit_waiter() + * function (and the comments within it) for more details. + */ +int zfs_commit_timeout_pct = 5; + +/* * Disable intent logging replay. This global ZIL switch affects all pools. */ int zil_replay_disable = 0; @@ -87,6 +108,7 @@ boolean_t zfs_nocacheflush = B_FALSE; uint64_t zil_slog_bulk = 768 * 1024; static kmem_cache_t *zil_lwb_cache; +static kmem_cache_t *zil_zcw_cache; static void zil_async_to_sync(zilog_t *zilog, uint64_t foid); @@ -436,6 +458,20 @@ zil_free_log_record(zilog_t *zilog, lr_t *lrc, void *t return (0); } +static int +zil_lwb_vdev_compare(const void *x1, const void *x2) +{ + const uint64_t v1 = ((zil_vdev_node_t *)x1)->zv_vdev; + const uint64_t v2 = ((zil_vdev_node_t *)x2)->zv_vdev; + + if (v1 < v2) + return (-1); + if (v1 > v2) + return (1); + + return (0); +} + static lwb_t * zil_alloc_lwb(zilog_t *zilog, blkptr_t *bp, boolean_t slog, uint64_t txg) { @@ -445,10 +481,13 @@ zil_alloc_lwb(zilog_t *zilog, blkptr_t *bp, boolean_t lwb->lwb_zilog = zilog; lwb->lwb_blk = *bp; lwb->lwb_slog = slog; + lwb->lwb_state = LWB_STATE_CLOSED; lwb->lwb_buf = zio_buf_alloc(BP_GET_LSIZE(bp)); lwb->lwb_max_txg = txg; - lwb->lwb_zio = NULL; + lwb->lwb_write_zio = NULL; + lwb->lwb_root_zio = NULL; lwb->lwb_tx = NULL; + lwb->lwb_issued_timestamp = 0; if (BP_GET_CHECKSUM(bp) == ZIO_CHECKSUM_ZILOG2) { lwb->lwb_nused = sizeof (zil_chain_t); lwb->lwb_sz = BP_GET_LSIZE(bp); @@ -461,9 +500,54 @@ zil_alloc_lwb(zilog_t *zilog, blkptr_t *bp, boolean_t list_insert_tail(&zilog->zl_lwb_list, lwb); mutex_exit(&zilog->zl_lock); + ASSERT(!MUTEX_HELD(&lwb->lwb_vdev_lock)); + ASSERT(avl_is_empty(&lwb->lwb_vdev_tree)); + ASSERT(list_is_empty(&lwb->lwb_waiters)); + return (lwb); } +static void +zil_free_lwb(zilog_t *zilog, lwb_t *lwb) +{ + ASSERT(MUTEX_HELD(&zilog->zl_lock)); + ASSERT(!MUTEX_HELD(&lwb->lwb_vdev_lock)); + ASSERT(list_is_empty(&lwb->lwb_waiters)); + + if (lwb->lwb_state == LWB_STATE_OPENED) { + avl_tree_t *t = &lwb->lwb_vdev_tree; + void *cookie = NULL; + zil_vdev_node_t *zv; + + while ((zv = avl_destroy_nodes(t, &cookie)) != NULL) + kmem_free(zv, sizeof (*zv)); + + ASSERT3P(lwb->lwb_root_zio, !=, NULL); + ASSERT3P(lwb->lwb_write_zio, !=, NULL); + + zio_cancel(lwb->lwb_root_zio); + zio_cancel(lwb->lwb_write_zio); + + lwb->lwb_root_zio = NULL; + lwb->lwb_write_zio = NULL; + } else { + ASSERT3S(lwb->lwb_state, !=, LWB_STATE_ISSUED); + } + + ASSERT(avl_is_empty(&lwb->lwb_vdev_tree)); + ASSERT3P(lwb->lwb_write_zio, ==, NULL); + ASSERT3P(lwb->lwb_root_zio, ==, NULL); + + /* + * Clear the zilog's field to indicate this lwb is no longer + * valid, and prevent use-after-free errors. + */ + if (zilog->zl_last_lwb_opened == lwb) + zilog->zl_last_lwb_opened = NULL; + + kmem_cache_free(zil_lwb_cache, lwb); +} + /* * Called when we create in-memory log transactions so that we know * to cleanup the itxs at the end of spa_sync(). @@ -474,12 +558,16 @@ zilog_dirty(zilog_t *zilog, uint64_t txg) dsl_pool_t *dp = zilog->zl_dmu_pool; dsl_dataset_t *ds = dmu_objset_ds(zilog->zl_os); + ASSERT(spa_writeable(zilog->zl_spa)); + if (ds->ds_is_snapshot) panic("dirtying snapshot!"); if (txg_list_add(&dp->dp_dirty_zilogs, zilog, txg)) { /* up the hold count until we can be written out */ dmu_buf_add_ref(ds->ds_dbuf, zilog); + + zilog->zl_dirty_max_txg = MAX(txg, zilog->zl_dirty_max_txg); } } @@ -547,7 +635,7 @@ zil_create(zilog_t *zilog) */ if (BP_IS_HOLE(&blk) || BP_SHOULD_BYTESWAP(&blk)) { tx = dmu_tx_create(zilog->zl_os); - VERIFY(dmu_tx_assign(tx, TXG_WAIT) == 0); + VERIFY0(dmu_tx_assign(tx, TXG_WAIT)); dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx); txg = dmu_tx_get_txg(tx); @@ -564,7 +652,7 @@ zil_create(zilog_t *zilog) } /* - * Allocate a log write buffer (lwb) for the first log block. + * Allocate a log write block (lwb) for the first log block. */ if (error == 0) lwb = zil_alloc_lwb(zilog, &blk, slog, txg); @@ -585,13 +673,13 @@ zil_create(zilog_t *zilog) } /* - * In one tx, free all log blocks and clear the log header. - * If keep_first is set, then we're replaying a log with no content. - * We want to keep the first block, however, so that the first - * synchronous transaction doesn't require a txg_wait_synced() - * in zil_create(). We don't need to txg_wait_synced() here either - * when keep_first is set, because both zil_create() and zil_destroy() - * will wait for any in-progress destroys to complete. + * In one tx, free all log blocks and clear the log header. If keep_first + * is set, then we're replaying a log with no content. We want to keep the + * first block, however, so that the first synchronous transaction doesn't + * require a txg_wait_synced() in zil_create(). We don't need to + * txg_wait_synced() here either when keep_first is set, because both + * zil_create() and zil_destroy() will wait for any in-progress destroys + * to complete. */ void zil_destroy(zilog_t *zilog, boolean_t keep_first) @@ -612,7 +700,7 @@ zil_destroy(zilog_t *zilog, boolean_t keep_first) return; tx = dmu_tx_create(zilog->zl_os); - VERIFY(dmu_tx_assign(tx, TXG_WAIT) == 0); + VERIFY0(dmu_tx_assign(tx, TXG_WAIT)); dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx); txg = dmu_tx_get_txg(tx); @@ -629,8 +717,8 @@ zil_destroy(zilog_t *zilog, boolean_t keep_first) list_remove(&zilog->zl_lwb_list, lwb); if (lwb->lwb_buf != NULL) zio_buf_free(lwb->lwb_buf, lwb->lwb_sz); - zio_free_zil(zilog->zl_spa, txg, &lwb->lwb_blk); - kmem_cache_free(zil_lwb_cache, lwb); + zio_free(zilog->zl_spa, txg, &lwb->lwb_blk); + zil_free_lwb(zilog, lwb); } } else if (!keep_first) { zil_destroy_sync(zilog, tx); @@ -768,24 +856,64 @@ zil_check_log_chain(dsl_pool_t *dp, dsl_dataset_t *ds, return ((error == ECKSUM || error == ENOENT) ? 0 : error); } -static int -zil_vdev_compare(const void *x1, const void *x2) +/* + * When an itx is "skipped", this function is used to properly mark the + * waiter as "done, and signal any thread(s) waiting on it. An itx can + * be skipped (and not committed to an lwb) for a variety of reasons, + * one of them being that the itx was committed via spa_sync(), prior to + * it being committed to an lwb; this can happen if a thread calling + * zil_commit() is racing with spa_sync(). + */ +static void +zil_commit_waiter_skip(zil_commit_waiter_t *zcw) { - const uint64_t v1 = ((zil_vdev_node_t *)x1)->zv_vdev; - const uint64_t v2 = ((zil_vdev_node_t *)x2)->zv_vdev; + mutex_enter(&zcw->zcw_lock); + ASSERT3B(zcw->zcw_done, ==, B_FALSE); + zcw->zcw_done = B_TRUE; + cv_broadcast(&zcw->zcw_cv); + mutex_exit(&zcw->zcw_lock); +} - if (v1 < v2) - return (-1); - if (v1 > v2) - return (1); +/* + * This function is used when the given waiter is to be linked into an + * lwb's "lwb_waiter" list; i.e. when the itx is committed to the lwb. + * At this point, the waiter will no longer be referenced by the itx, + * and instead, will be referenced by the lwb. + */ +static void +zil_commit_waiter_link_lwb(zil_commit_waiter_t *zcw, lwb_t *lwb) +{ + mutex_enter(&zcw->zcw_lock); + ASSERT(!list_link_active(&zcw->zcw_node)); + ASSERT3P(zcw->zcw_lwb, ==, NULL); + ASSERT3P(lwb, !=, NULL); + ASSERT(lwb->lwb_state == LWB_STATE_OPENED || + lwb->lwb_state == LWB_STATE_ISSUED); - return (0); + list_insert_tail(&lwb->lwb_waiters, zcw); + zcw->zcw_lwb = lwb; + mutex_exit(&zcw->zcw_lock); } +/* + * This function is used when zio_alloc_zil() fails to allocate a ZIL + * block, and the given waiter must be linked to the "nolwb waiters" + * list inside of zil_process_commit_list(). + */ +static void +zil_commit_waiter_link_nolwb(zil_commit_waiter_t *zcw, list_t *nolwb) +{ + mutex_enter(&zcw->zcw_lock); + ASSERT(!list_link_active(&zcw->zcw_node)); + ASSERT3P(zcw->zcw_lwb, ==, NULL); + list_insert_tail(nolwb, zcw); + mutex_exit(&zcw->zcw_lock); +} + void -zil_add_block(zilog_t *zilog, const blkptr_t *bp) +zil_lwb_add_block(lwb_t *lwb, const blkptr_t *bp) { - avl_tree_t *t = &zilog->zl_vdev_tree; + avl_tree_t *t = &lwb->lwb_vdev_tree; avl_index_t where; zil_vdev_node_t *zv, zvsearch; int ndvas = BP_GET_NDVAS(bp); @@ -794,14 +922,7 @@ zil_add_block(zilog_t *zilog, const blkptr_t *bp) if (zfs_nocacheflush) return; - ASSERT(zilog->zl_writer); - - /* - * Even though we're zl_writer, we still need a lock because the - * zl_get_data() callbacks may have dmu_sync() done callbacks - * that will run concurrently. - */ - mutex_enter(&zilog->zl_vdev_lock); + mutex_enter(&lwb->lwb_vdev_lock); for (i = 0; i < ndvas; i++) { zvsearch.zv_vdev = DVA_GET_VDEV(&bp->blk_dva[i]); if (avl_find(t, &zvsearch, &where) == NULL) { @@ -810,57 +931,117 @@ zil_add_block(zilog_t *zilog, const blkptr_t *bp) avl_insert(t, zv, where); } } - mutex_exit(&zilog->zl_vdev_lock); + mutex_exit(&lwb->lwb_vdev_lock); } +void +zil_lwb_add_txg(lwb_t *lwb, uint64_t txg) +{ + lwb->lwb_max_txg = MAX(lwb->lwb_max_txg, txg); +} + +/* + * This function is a called after all VDEVs associated with a given lwb + * write have completed their DKIOCFLUSHWRITECACHE command; or as soon + * as the lwb write completes, if "zfs_nocacheflush" is set. + * + * The intention is for this function to be called as soon as the + * contents of an lwb are considered "stable" on disk, and will survive + * any sudden loss of power. At this point, any threads waiting for the + * lwb to reach this state are signalled, and the "waiter" structures + * are marked "done". + */ static void -zil_flush_vdevs(zilog_t *zilog) +zil_lwb_flush_vdevs_done(zio_t *zio) { - spa_t *spa = zilog->zl_spa; - avl_tree_t *t = &zilog->zl_vdev_tree; - void *cookie = NULL; - zil_vdev_node_t *zv; - zio_t *zio; + lwb_t *lwb = zio->io_private; + zilog_t *zilog = lwb->lwb_zilog; + dmu_tx_t *tx = lwb->lwb_tx; + zil_commit_waiter_t *zcw; - ASSERT(zilog->zl_writer); + spa_config_exit(zilog->zl_spa, SCL_STATE, lwb); + zio_buf_free(lwb->lwb_buf, lwb->lwb_sz); + + mutex_enter(&zilog->zl_lock); + /* - * We don't need zl_vdev_lock here because we're the zl_writer, - * and all zl_get_data() callbacks are done. + * Ensure the lwb buffer pointer is cleared before releasing the + * txg. If we have had an allocation failure and the txg is + * waiting to sync then we want zil_sync() to remove the lwb so + * that it's not picked up as the next new one in + * zil_process_commit_list(). zil_sync() will only remove the + * lwb if lwb_buf is null. */ - if (avl_numnodes(t) == 0) - return; + lwb->lwb_buf = NULL; + lwb->lwb_tx = NULL; - spa_config_enter(spa, SCL_STATE, FTAG, RW_READER); + ASSERT3U(lwb->lwb_issued_timestamp, >, 0); + zilog->zl_last_lwb_latency = gethrtime() - lwb->lwb_issued_timestamp; - zio = zio_root(spa, NULL, NULL, ZIO_FLAG_CANFAIL); + lwb->lwb_root_zio = NULL; + lwb->lwb_state = LWB_STATE_DONE; - while ((zv = avl_destroy_nodes(t, &cookie)) != NULL) { - vdev_t *vd = vdev_lookup_top(spa, zv->zv_vdev); - if (vd != NULL) - zio_flush(zio, vd); - kmem_free(zv, sizeof (*zv)); + if (zilog->zl_last_lwb_opened == lwb) { + /* + * Remember the highest committed log sequence number + * for ztest. We only update this value when all the log + * writes succeeded, because ztest wants to ASSERT that + * it got the whole log chain. + */ + zilog->zl_commit_lr_seq = zilog->zl_lr_seq; } + while ((zcw = list_head(&lwb->lwb_waiters)) != NULL) { + mutex_enter(&zcw->zcw_lock); + + ASSERT(list_link_active(&zcw->zcw_node)); + list_remove(&lwb->lwb_waiters, zcw); + + ASSERT3P(zcw->zcw_lwb, ==, lwb); + zcw->zcw_lwb = NULL; + + zcw->zcw_zio_error = zio->io_error; + + ASSERT3B(zcw->zcw_done, ==, B_FALSE); + zcw->zcw_done = B_TRUE; + cv_broadcast(&zcw->zcw_cv); + + mutex_exit(&zcw->zcw_lock); + } + + mutex_exit(&zilog->zl_lock); + /* - * Wait for all the flushes to complete. Not all devices actually - * support the DKIOCFLUSHWRITECACHE ioctl, so it's OK if it fails. + * Now that we've written this log block, we have a stable pointer + * to the next block in the chain, so it's OK to let the txg in + * which we allocated the next block sync. */ - (void) zio_wait(zio); - - spa_config_exit(spa, SCL_STATE, FTAG); + dmu_tx_commit(tx); } /* - * Function called when a log block write completes + * This is called when an lwb write completes. This means, this specific + * lwb was written to disk, and all dependent lwb have also been + * written to disk. + * + * At this point, a DKIOCFLUSHWRITECACHE command hasn't been issued to + * the VDEVs involved in writing out this specific lwb. The lwb will be + * "done" once zil_lwb_flush_vdevs_done() is called, which occurs in the + * zio completion callback for the lwb's root zio. */ static void zil_lwb_write_done(zio_t *zio) { lwb_t *lwb = zio->io_private; + spa_t *spa = zio->io_spa; zilog_t *zilog = lwb->lwb_zilog; - dmu_tx_t *tx = lwb->lwb_tx; + avl_tree_t *t = &lwb->lwb_vdev_tree; + void *cookie = NULL; + zil_vdev_node_t *zv; + ASSERT3S(spa_config_held(spa, SCL_STATE, RW_READER), !=, 0); + ASSERT(BP_GET_COMPRESS(zio->io_bp) == ZIO_COMPRESS_OFF); ASSERT(BP_GET_TYPE(zio->io_bp) == DMU_OT_INTENT_LOG); ASSERT(BP_GET_LEVEL(zio->io_bp) == 0); @@ -869,58 +1050,115 @@ zil_lwb_write_done(zio_t *zio) ASSERT(!BP_IS_HOLE(zio->io_bp)); ASSERT(BP_GET_FILL(zio->io_bp) == 0); - /* - * Ensure the lwb buffer pointer is cleared before releasing - * the txg. If we have had an allocation failure and - * the txg is waiting to sync then we want want zil_sync() - * to remove the lwb so that it's not picked up as the next new - * one in zil_commit_writer(). zil_sync() will only remove - * the lwb if lwb_buf is null. - */ abd_put(zio->io_abd); - zio_buf_free(lwb->lwb_buf, lwb->lwb_sz); + + ASSERT3S(lwb->lwb_state, ==, LWB_STATE_ISSUED); + mutex_enter(&zilog->zl_lock); - lwb->lwb_buf = NULL; - lwb->lwb_tx = NULL; + lwb->lwb_write_zio = NULL; mutex_exit(&zilog->zl_lock); + if (avl_numnodes(t) == 0) + return; + /* - * Now that we've written this log block, we have a stable pointer - * to the next block in the chain, so it's OK to let the txg in - * which we allocated the next block sync. + * If there was an IO error, we're not going to call zio_flush() + * on these vdevs, so we simply empty the tree and free the + * nodes. We avoid calling zio_flush() since there isn't any + * good reason for doing so, after the lwb block failed to be + * written out. */ - dmu_tx_commit(tx); + if (zio->io_error != 0) { + while ((zv = avl_destroy_nodes(t, &cookie)) != NULL) + kmem_free(zv, sizeof (*zv)); + return; + } + + while ((zv = avl_destroy_nodes(t, &cookie)) != NULL) { + vdev_t *vd = vdev_lookup_top(spa, zv->zv_vdev); + if (vd != NULL) + zio_flush(lwb->lwb_root_zio, vd); + kmem_free(zv, sizeof (*zv)); + } } /* - * Initialize the io for a log block. + * This function's purpose is to "open" an lwb such that it is ready to + * accept new itxs being committed to it. To do this, the lwb's zio + * structures are created, and linked to the lwb. This function is + * idempotent; if the passed in lwb has already been opened, this + * function is essentially a no-op. */ static void -zil_lwb_write_init(zilog_t *zilog, lwb_t *lwb) +zil_lwb_write_open(zilog_t *zilog, lwb_t *lwb) { zbookmark_phys_t zb; zio_priority_t prio; + ASSERT(MUTEX_HELD(&zilog->zl_writer_lock)); + ASSERT3P(lwb, !=, NULL); + EQUIV(lwb->lwb_root_zio == NULL, lwb->lwb_state == LWB_STATE_CLOSED); + EQUIV(lwb->lwb_root_zio != NULL, lwb->lwb_state == LWB_STATE_OPENED); + SET_BOOKMARK(&zb, lwb->lwb_blk.blk_cksum.zc_word[ZIL_ZC_OBJSET], ZB_ZIL_OBJECT, ZB_ZIL_LEVEL, lwb->lwb_blk.blk_cksum.zc_word[ZIL_ZC_SEQ]); - if (zilog->zl_root_zio == NULL) { - zilog->zl_root_zio = zio_root(zilog->zl_spa, NULL, NULL, - ZIO_FLAG_CANFAIL); - } - if (lwb->lwb_zio == NULL) { + if (lwb->lwb_root_zio == NULL) { abd_t *lwb_abd = abd_get_from_buf(lwb->lwb_buf, BP_GET_LSIZE(&lwb->lwb_blk)); + if (!lwb->lwb_slog || zilog->zl_cur_used <= zil_slog_bulk) prio = ZIO_PRIORITY_SYNC_WRITE; else prio = ZIO_PRIORITY_ASYNC_WRITE; - lwb->lwb_zio = zio_rewrite(zilog->zl_root_zio, zilog->zl_spa, - 0, &lwb->lwb_blk, lwb_abd, BP_GET_LSIZE(&lwb->lwb_blk), - zil_lwb_write_done, lwb, prio, - ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE, &zb); + + lwb->lwb_root_zio = zio_root(zilog->zl_spa, + zil_lwb_flush_vdevs_done, lwb, ZIO_FLAG_CANFAIL); + ASSERT3P(lwb->lwb_root_zio, !=, NULL); + + lwb->lwb_write_zio = zio_rewrite(lwb->lwb_root_zio, + zilog->zl_spa, 0, &lwb->lwb_blk, lwb_abd, + BP_GET_LSIZE(&lwb->lwb_blk), zil_lwb_write_done, lwb, + prio, ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE, &zb); + ASSERT3P(lwb->lwb_write_zio, !=, NULL); + + lwb->lwb_state = LWB_STATE_OPENED; + + mutex_enter(&zilog->zl_lock); + + /* + * The zilog's "zl_last_lwb_opened" field is used to + * build the lwb/zio dependency chain, which is used to + * preserve the ordering of lwb completions that is + * required by the semantics of the ZIL. Each new lwb + * zio becomes a parent of the "previous" lwb zio, such + * that the new lwb's zio cannot complete until the + * "previous" lwb's zio completes. + * + * This is required by the semantics of zil_commit(); + * the commit waiters attached to the lwbs will be woken + * in the lwb zio's completion callback, so this zio + * dependency graph ensures the waiters are woken in the + * correct order (the same order the lwbs were created). + */ + lwb_t *last_lwb_opened = zilog->zl_last_lwb_opened; + if (last_lwb_opened != NULL && + last_lwb_opened->lwb_state != LWB_STATE_DONE) { + ASSERT(last_lwb_opened->lwb_state == LWB_STATE_OPENED || + last_lwb_opened->lwb_state == LWB_STATE_ISSUED); + ASSERT3P(last_lwb_opened->lwb_root_zio, !=, NULL); + zio_add_child(lwb->lwb_root_zio, + last_lwb_opened->lwb_root_zio); + } + zilog->zl_last_lwb_opened = lwb; + + mutex_exit(&zilog->zl_lock); } + + ASSERT3P(lwb->lwb_root_zio, !=, NULL); + ASSERT3P(lwb->lwb_write_zio, !=, NULL); + ASSERT3S(lwb->lwb_state, ==, LWB_STATE_OPENED); } /* @@ -942,7 +1180,7 @@ uint64_t zil_block_buckets[] = { * Calls are serialized. */ static lwb_t * -zil_lwb_write_start(zilog_t *zilog, lwb_t *lwb) +zil_lwb_write_issue(zilog_t *zilog, lwb_t *lwb) { lwb_t *nlwb = NULL; zil_chain_t *zilc; @@ -954,6 +1192,11 @@ zil_lwb_write_start(zilog_t *zilog, lwb_t *lwb) int i, error; boolean_t slog; + ASSERT(MUTEX_HELD(&zilog->zl_writer_lock)); + ASSERT3P(lwb->lwb_root_zio, !=, NULL); + ASSERT3P(lwb->lwb_write_zio, !=, NULL); + ASSERT3S(lwb->lwb_state, ==, LWB_STATE_OPENED); + if (BP_GET_CHECKSUM(&lwb->lwb_blk) == ZIO_CHECKSUM_ZILOG2) { zilc = (zil_chain_t *)lwb->lwb_buf; bp = &zilc->zc_next_blk; @@ -973,6 +1216,7 @@ zil_lwb_write_start(zilog_t *zilog, lwb_t *lwb) * We dirty the dataset to ensure that zil_sync() will be called * to clean up in the event of allocation failure or I/O failure. */ + tx = dmu_tx_create(zilog->zl_os); /* @@ -1025,6 +1269,7 @@ zil_lwb_write_start(zilog_t *zilog, lwb_t *lwb) zilog->zl_prev_rotor = (zilog->zl_prev_rotor + 1) & (ZIL_PREV_BLKS - 1); BP_ZERO(bp); + /* pass the old blkptr in order to spread log blocks across devs */ error = zio_alloc_zil(spa, txg, bp, &lwb->lwb_blk, zil_blksz, &slog); if (error == 0) { @@ -1033,19 +1278,16 @@ zil_lwb_write_start(zilog_t *zilog, lwb_t *lwb) bp->blk_cksum.zc_word[ZIL_ZC_SEQ]++; /* - * Allocate a new log write buffer (lwb). + * Allocate a new log write block (lwb). */ nlwb = zil_alloc_lwb(zilog, bp, slog, txg); - - /* Record the block for later vdev flushing */ - zil_add_block(zilog, &lwb->lwb_blk); } if (BP_GET_CHECKSUM(&lwb->lwb_blk) == ZIO_CHECKSUM_ZILOG2) { /* For Slim ZIL only write what is used. */ wsz = P2ROUNDUP_TYPED(lwb->lwb_nused, ZIL_MIN_BLKSZ, uint64_t); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Sep 13 15:58:00 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5542FE02A29; Wed, 13 Sep 2017 15:58:00 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0940B6B095; Wed, 13 Sep 2017 15:57:59 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DFvxGF094069; Wed, 13 Sep 2017 15:57:59 GMT (envelope-from pfg@FreeBSD.org) Received: (from pfg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DFvwXv094063; Wed, 13 Sep 2017 15:57:58 GMT (envelope-from pfg@FreeBSD.org) Message-Id: <201709131557.v8DFvwXv094063@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: pfg set sender to pfg@FreeBSD.org using -f From: "Pedro F. Giffuni" Date: Wed, 13 Sep 2017 15:57:58 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323545 - in vendor/NetBSD/libedit/dist: . TEST readline X-SVN-Group: vendor X-SVN-Commit-Author: pfg X-SVN-Commit-Paths: in vendor/NetBSD/libedit/dist: . TEST readline X-SVN-Commit-Revision: 323545 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 15:58:00 -0000 Author: pfg Date: Wed Sep 13 15:57:58 2017 New Revision: 323545 URL: https://svnweb.freebsd.org/changeset/base/323545 Log: Import libedit 2017-09-05 Obtained from: NetBSD Added: vendor/NetBSD/libedit/dist/editline.7 (contents, props changed) vendor/NetBSD/libedit/dist/historyn.c (contents, props changed) vendor/NetBSD/libedit/dist/literal.c (contents, props changed) vendor/NetBSD/libedit/dist/literal.h (contents, props changed) vendor/NetBSD/libedit/dist/tokenizern.c (contents, props changed) Modified: vendor/NetBSD/libedit/dist/Makefile vendor/NetBSD/libedit/dist/TEST/Makefile vendor/NetBSD/libedit/dist/chared.c vendor/NetBSD/libedit/dist/chared.h vendor/NetBSD/libedit/dist/chartype.c vendor/NetBSD/libedit/dist/chartype.h vendor/NetBSD/libedit/dist/common.c vendor/NetBSD/libedit/dist/config.h vendor/NetBSD/libedit/dist/editline.3 vendor/NetBSD/libedit/dist/editrc.5 vendor/NetBSD/libedit/dist/el.c vendor/NetBSD/libedit/dist/el.h vendor/NetBSD/libedit/dist/eln.c vendor/NetBSD/libedit/dist/emacs.c vendor/NetBSD/libedit/dist/filecomplete.c vendor/NetBSD/libedit/dist/filecomplete.h vendor/NetBSD/libedit/dist/hist.c vendor/NetBSD/libedit/dist/hist.h vendor/NetBSD/libedit/dist/histedit.h vendor/NetBSD/libedit/dist/history.c vendor/NetBSD/libedit/dist/keymacro.c vendor/NetBSD/libedit/dist/keymacro.h vendor/NetBSD/libedit/dist/makelist vendor/NetBSD/libedit/dist/map.c vendor/NetBSD/libedit/dist/map.h vendor/NetBSD/libedit/dist/parse.c vendor/NetBSD/libedit/dist/parse.h vendor/NetBSD/libedit/dist/prompt.c vendor/NetBSD/libedit/dist/prompt.h vendor/NetBSD/libedit/dist/read.c vendor/NetBSD/libedit/dist/read.h vendor/NetBSD/libedit/dist/readline.c vendor/NetBSD/libedit/dist/readline/readline.h vendor/NetBSD/libedit/dist/refresh.c vendor/NetBSD/libedit/dist/refresh.h vendor/NetBSD/libedit/dist/search.c vendor/NetBSD/libedit/dist/search.h vendor/NetBSD/libedit/dist/sig.c vendor/NetBSD/libedit/dist/sig.h vendor/NetBSD/libedit/dist/sys.h vendor/NetBSD/libedit/dist/terminal.c vendor/NetBSD/libedit/dist/terminal.h vendor/NetBSD/libedit/dist/tokenizer.c vendor/NetBSD/libedit/dist/tty.c vendor/NetBSD/libedit/dist/tty.h vendor/NetBSD/libedit/dist/vi.c Modified: vendor/NetBSD/libedit/dist/Makefile ============================================================================== --- vendor/NetBSD/libedit/dist/Makefile Wed Sep 13 15:44:54 2017 (r323544) +++ vendor/NetBSD/libedit/dist/Makefile Wed Sep 13 15:57:58 2017 (r323545) @@ -1,9 +1,8 @@ -# $NetBSD: Makefile,v 1.56 2016/03/02 19:24:20 christos Exp $ +# $NetBSD: Makefile,v 1.65 2017/06/30 20:26:52 kre Exp $ # @(#)Makefile 8.1 (Berkeley) 6/4/93 USE_SHLIBDIR= yes -WIDECHAR ?= yes WARNS?= 5 LIB= edit @@ -15,55 +14,80 @@ COPTS+= -Wunused-parameter CWARNFLAGS.gcc+= -Wconversion CWARNFLAGS.clang+= -Wno-cast-qual -OSRCS= chared.c common.c el.c eln.c emacs.c fcns.c filecomplete.c help.c \ - hist.c keymacro.c map.c chartype.c \ - parse.c prompt.c read.c refresh.c search.c sig.c terminal.c tty.c vi.c +SRCS = chared.c chartype.c common.c el.c eln.c emacs.c filecomplete.c \ + hist.c history.c historyn.c keymacro.c literal.c map.c \ + parse.c prompt.c read.c readline.c refresh.c search.c sig.c \ + terminal.c tokenizer.c tokenizern.c tty.c vi.c -MAN= editline.3 editrc.5 +MAN= editline.3 editrc.5 editline.7 -MLINKS= editline.3 el_init.3 editline.3 el_end.3 editline.3 el_reset.3 \ - editline.3 el_gets.3 editline.3 el_getc.3 editline.3 el_push.3 \ - editline.3 el_parse.3 editline.3 el_set.3 editline.3 el_get.3 \ - editline.3 el_source.3 editline.3 el_resize.3 editline.3 el_line.3 \ - editline.3 el_insertstr.3 editline.3 el_deletestr.3 \ - editline.3 history_init.3 editline.3 history_end.3 \ - editline.3 history.3 \ - editline.3 tok_init.3 editline.3 tok_end.3 editline.3 tok_reset.3 \ - editline.3 tok_line.3 editline.3 tok_str.3 +MLINKS= \ +editline.3 el_deletestr.3 \ +editline.3 el_end.3 \ +editline.3 el_get.3 \ +editline.3 el_getc.3 \ +editline.3 el_gets.3 \ +editline.3 el_init.3 \ +editline.3 el_init_fd.3 \ +editline.3 el_insertstr.3 \ +editline.3 el_line.3 \ +editline.3 el_parse.3 \ +editline.3 el_push.3 \ +editline.3 el_reset.3 \ +editline.3 el_resize.3 \ +editline.3 el_set.3 \ +editline.3 el_source.3 \ +editline.3 history.3 \ +editline.3 history_end.3 \ +editline.3 history_init.3 \ +editline.3 tok_end.3 \ +editline.3 tok_init.3 \ +editline.3 tok_line.3 \ +editline.3 tok_reset.3 \ +editline.3 tok_str.3 -# For speed and debugging -#SRCS= ${OSRCS} readline.c tokenizer.c history.c -# For protection -SRCS= editline.c readline.c tokenizer.c history.c +MLINKS+= \ +editline.3 el_wdeletestr.3 \ +editline.3 el_wget.3 \ +editline.3 el_wgetc.3 \ +editline.3 el_wgets.3 \ +editline.3 el_winsertstr.3 \ +editline.3 el_wline.3 \ +editline.3 el_wparse.3 \ +editline.3 el_wpush.3 \ +editline.3 el_wset.3 \ +editline.3 history_w.3 \ +editline.3 history_wend.3 \ +editline.3 history_winit.3 \ +editline.3 tok_wend.3 \ +editline.3 tok_winit.3 \ +editline.3 tok_wline.3 \ +editline.3 tok_wreset.3 \ +editline.3 tok_wstr.3 -.if ${WIDECHAR} == "yes" -SRCS += tokenizern.c historyn.c -CLEANFILES+=tokenizern.c.tmp tokenizern.c historyn.c.tmp historyn.c -CPPFLAGS+=-DWIDECHAR -.endif - LIBEDITDIR?=${.CURDIR} INCS= histedit.h INCSDIR=/usr/include -CLEANFILES+=editline.c -CLEANFILES+=common.h.tmp editline.c.tmp emacs.h.tmp fcns.c.tmp fcns.h.tmp -CLEANFILES+=help.c.tmp help.h.tmp vi.h.tmp tc1.o tc1 -CLEANFILES+=tokenizern.c.tmp tokenizern.c tokenizerw.c.tmp tokenizerw.c +CLEANFILES+=common.h.tmp emacs.h.tmp fcns.h.tmp func.h.tmp +CLEANFILES+=help.h.tmp vi.h.tmp tc1.o tc1 .depend + CPPFLAGS+=-I. -I${LIBEDITDIR} CPPFLAGS+=-I. -I${.CURDIR} -#CPPFLAGS+=-DDEBUG_TTY -DDEBUG_KEY -DDEBUG_READ -DDEBUG -DDEBUG_REFRESH +#CPPFLAGS+=-DDEBUG_TTY -DDEBUG_KEY -DDEBUG -DDEBUG_REFRESH #CPPFLAGS+=-DDEBUG_PASTE -DDEBUG_EDIT AHDR=vi.h emacs.h common.h ASRC=${LIBEDITDIR}/vi.c ${LIBEDITDIR}/emacs.c ${LIBEDITDIR}/common.c -DPSRCS+= ${AHDR} fcns.h help.h fcns.c help.c -CLEANFILES+= ${AHDR} fcns.h help.h fcns.c help.c +DPSRCS+= ${AHDR} fcns.h func.h help.h +CLEANFILES+= ${AHDR} fcns.h func.h help.h SUBDIR= readline +.depend: ${AHDR} fcns.h func.h help.h + vi.h: vi.c makelist Makefile ${_MKTARGET_CREATE} ${HOST_SH} ${LIBEDITDIR}/makelist -h ${LIBEDITDIR}/vi.c \ @@ -87,36 +111,16 @@ fcns.h: ${AHDR} makelist Makefile ${HOST_SH} ${LIBEDITDIR}/makelist -fh ${AHDR} > ${.TARGET}.tmp && \ mv ${.TARGET}.tmp ${.TARGET} -fcns.c: ${AHDR} fcns.h help.h makelist Makefile +func.h: ${AHDR} makelist Makefile ${_MKTARGET_CREATE} ${HOST_SH} ${LIBEDITDIR}/makelist -fc ${AHDR} > ${.TARGET}.tmp && \ mv ${.TARGET}.tmp ${.TARGET} -help.c: ${ASRC} makelist Makefile - ${_MKTARGET_CREATE} - ${HOST_SH} ${LIBEDITDIR}/makelist -bc ${ASRC} > ${.TARGET}.tmp && \ - mv ${.TARGET}.tmp ${.TARGET} - help.h: ${ASRC} makelist Makefile ${_MKTARGET_CREATE} ${HOST_SH} ${LIBEDITDIR}/makelist -bh ${ASRC} > ${.TARGET}.tmp && \ mv ${.TARGET}.tmp ${.TARGET} -editline.c: ${OSRCS} makelist Makefile - ${_MKTARGET_CREATE} - ${HOST_SH} ${LIBEDITDIR}/makelist -e ${OSRCS:T} > ${.TARGET}.tmp && \ - mv ${.TARGET}.tmp ${.TARGET} - -tokenizern.c: makelist Makefile - ${_MKTARGET_CREATE} - ${HOST_SH} ${LIBEDITDIR}/makelist -n tokenizer.c > ${.TARGET}.tmp && \ - mv ${.TARGET}.tmp ${.TARGET} - -historyn.c: makelist Makefile - ${_MKTARGET_CREATE} - ${HOST_SH} ${LIBEDITDIR}/makelist -n history.c > ${.TARGET}.tmp && \ - mv ${.TARGET}.tmp ${.TARGET} - tc1.o: ${LIBEDITDIR}/TEST/tc1.c tc1: libedit.a tc1.o @@ -129,6 +133,7 @@ tc1: libedit.a tc1.o # XXX .if defined(HAVE_GCC) COPTS.editline.c+= -Wno-cast-qual +COPTS.literal.c+= -Wno-sign-conversion COPTS.tokenizer.c+= -Wno-cast-qual COPTS.tokenizern.c+= -Wno-cast-qual .endif Modified: vendor/NetBSD/libedit/dist/TEST/Makefile ============================================================================== --- vendor/NetBSD/libedit/dist/TEST/Makefile Wed Sep 13 15:44:54 2017 (r323544) +++ vendor/NetBSD/libedit/dist/TEST/Makefile Wed Sep 13 15:57:58 2017 (r323545) @@ -1,16 +1,10 @@ -# $NetBSD: Makefile,v 1.6 2016/02/15 21:38:07 christos Exp $ +# $NetBSD: Makefile,v 1.7 2016/03/23 22:27:48 christos Exp $ -WIDECHAR ?= yes - NOMAN=1 PROG=wtc1 CPPFLAGS=-I${.CURDIR}/.. LDADD+=-ledit -ltermlib DPADD+=${LIBEDIT} ${LIBTERMLIB} - -.if "${WIDECHAR}" == "yes" -CPPFLAGS+=-DWIDECHAR -.endif .ifdef DEBUG CPPFLAGS+=-DDEBUG Modified: vendor/NetBSD/libedit/dist/chared.c ============================================================================== --- vendor/NetBSD/libedit/dist/chared.c Wed Sep 13 15:44:54 2017 (r323544) +++ vendor/NetBSD/libedit/dist/chared.c Wed Sep 13 15:57:58 2017 (r323545) @@ -1,4 +1,4 @@ -/* $NetBSD: chared.c,v 1.49 2016/02/24 14:29:21 christos Exp $ */ +/* $NetBSD: chared.c,v 1.56 2016/05/22 19:44:26 christos Exp $ */ /*- * Copyright (c) 1992, 1993 @@ -37,7 +37,7 @@ #if 0 static char sccsid[] = "@(#)chared.c 8.1 (Berkeley) 6/4/93"; #else -__RCSID("$NetBSD: chared.c,v 1.49 2016/02/24 14:29:21 christos Exp $"); +__RCSID("$NetBSD: chared.c,v 1.56 2016/05/22 19:44:26 christos Exp $"); #endif #endif /* not lint && not SCCSID */ @@ -50,16 +50,15 @@ __RCSID("$NetBSD: chared.c,v 1.49 2016/02/24 14:29:21 #include "el.h" #include "common.h" +#include "fcns.h" -private void ch__clearmacro (EditLine *); - /* value to leave unused in line buffer */ #define EL_LEAVE 2 /* cv_undo(): * Handle state for the vi undo command */ -protected void +libedit_private void cv_undo(EditLine *el) { c_undo_t *vu = &el->el_chared.c_undo; @@ -83,8 +82,8 @@ cv_undo(EditLine *el) /* cv_yank(): * Save yank/delete data for paste */ -protected void -cv_yank(EditLine *el, const Char *ptr, int size) +libedit_private void +cv_yank(EditLine *el, const wchar_t *ptr, int size) { c_kill_t *k = &el->el_chared.c_kill; @@ -96,10 +95,10 @@ cv_yank(EditLine *el, const Char *ptr, int size) /* c_insert(): * Insert num characters */ -protected void +libedit_private void c_insert(EditLine *el, int num) { - Char *cp; + wchar_t *cp; if (el->el_line.lastchar + num >= el->el_line.limit) { if (!ch_enlargebufs(el, (size_t)num)) @@ -118,7 +117,7 @@ c_insert(EditLine *el, int num) /* c_delafter(): * Delete num characters after the cursor */ -protected void +libedit_private void c_delafter(EditLine *el, int num) { @@ -131,7 +130,7 @@ c_delafter(EditLine *el, int num) } if (num > 0) { - Char *cp; + wchar_t *cp; for (cp = el->el_line.cursor; cp <= el->el_line.lastchar; cp++) *cp = cp[num]; @@ -144,10 +143,10 @@ c_delafter(EditLine *el, int num) /* c_delafter1(): * Delete the character after the cursor, do not yank */ -protected void +libedit_private void c_delafter1(EditLine *el) { - Char *cp; + wchar_t *cp; for (cp = el->el_line.cursor; cp <= el->el_line.lastchar; cp++) *cp = cp[1]; @@ -159,7 +158,7 @@ c_delafter1(EditLine *el) /* c_delbefore(): * Delete num characters before the cursor */ -protected void +libedit_private void c_delbefore(EditLine *el, int num) { @@ -172,7 +171,7 @@ c_delbefore(EditLine *el, int num) } if (num > 0) { - Char *cp; + wchar_t *cp; for (cp = el->el_line.cursor - num; cp <= el->el_line.lastchar; @@ -187,10 +186,10 @@ c_delbefore(EditLine *el, int num) /* c_delbefore1(): * Delete the character before the cursor, do not yank */ -protected void +libedit_private void c_delbefore1(EditLine *el) { - Char *cp; + wchar_t *cp; for (cp = el->el_line.cursor - 1; cp <= el->el_line.lastchar; cp++) *cp = cp[1]; @@ -202,22 +201,22 @@ c_delbefore1(EditLine *el) /* ce__isword(): * Return if p is part of a word according to emacs */ -protected int +libedit_private int ce__isword(wint_t p) { - return Isalnum(p) || Strchr(STR("*?_-.[]~="), p) != NULL; + return iswalnum(p) || wcschr(L"*?_-.[]~=", p) != NULL; } /* cv__isword(): * Return if p is part of a word according to vi */ -protected int +libedit_private int cv__isword(wint_t p) { - if (Isalnum(p) || p == '_') + if (iswalnum(p) || p == L'_') return 1; - if (Isgraph(p)) + if (iswgraph(p)) return 2; return 0; } @@ -226,18 +225,18 @@ cv__isword(wint_t p) /* cv__isWord(): * Return if p is part of a big word according to vi */ -protected int +libedit_private int cv__isWord(wint_t p) { - return !Isspace(p); + return !iswspace(p); } /* c__prev_word(): * Find the previous word */ -protected Char * -c__prev_word(Char *p, Char *low, int n, int (*wtest)(wint_t)) +libedit_private wchar_t * +c__prev_word(wchar_t *p, wchar_t *low, int n, int (*wtest)(wint_t)) { p--; @@ -260,8 +259,8 @@ c__prev_word(Char *p, Char *low, int n, int (*wtest)(w /* c__next_word(): * Find the next word */ -protected Char * -c__next_word(Char *p, Char *high, int n, int (*wtest)(wint_t)) +libedit_private wchar_t * +c__next_word(wchar_t *p, wchar_t *high, int n, int (*wtest)(wint_t)) { while (n--) { while ((p < high) && !(*wtest)(*p)) @@ -278,8 +277,9 @@ c__next_word(Char *p, Char *high, int n, int (*wtest)( /* cv_next_word(): * Find the next word vi style */ -protected Char * -cv_next_word(EditLine *el, Char *p, Char *high, int n, int (*wtest)(wint_t)) +libedit_private wchar_t * +cv_next_word(EditLine *el, wchar_t *p, wchar_t *high, int n, + int (*wtest)(wint_t)) { int test; @@ -292,7 +292,7 @@ cv_next_word(EditLine *el, Char *p, Char *high, int n, * trailing whitespace! This is not what 'w' does.. */ if (n || el->el_chared.c_vcmd.action != (DELETE|INSERT)) - while ((p < high) && Isspace(*p)) + while ((p < high) && iswspace(*p)) p++; } @@ -307,14 +307,14 @@ cv_next_word(EditLine *el, Char *p, Char *high, int n, /* cv_prev_word(): * Find the previous word vi style */ -protected Char * -cv_prev_word(Char *p, Char *low, int n, int (*wtest)(wint_t)) +libedit_private wchar_t * +cv_prev_word(wchar_t *p, wchar_t *low, int n, int (*wtest)(wint_t)) { int test; p--; while (n--) { - while ((p > low) && Isspace(*p)) + while ((p > low) && iswspace(*p)) p--; test = (*wtest)(*p); while ((p >= low) && (*wtest)(*p) == test) @@ -333,7 +333,7 @@ cv_prev_word(Char *p, Char *low, int n, int (*wtest)(w /* cv_delfini(): * Finish vi delete action */ -protected void +libedit_private void cv_delfini(EditLine *el) { int size; @@ -371,15 +371,15 @@ cv_delfini(EditLine *el) /* cv__endword(): * Go to the end of this word according to vi */ -protected Char * -cv__endword(Char *p, Char *high, int n, int (*wtest)(wint_t)) +libedit_private wchar_t * +cv__endword(wchar_t *p, wchar_t *high, int n, int (*wtest)(wint_t)) { int test; p++; while (n--) { - while ((p < high) && Isspace(*p)) + while ((p < high) && iswspace(*p)) p++; test = (*wtest)(*p); @@ -393,11 +393,9 @@ cv__endword(Char *p, Char *high, int n, int (*wtest)(w /* ch_init(): * Initialize the character editor */ -protected int +libedit_private int ch_init(EditLine *el) { - c_macro_t *ma = &el->el_chared.c_macro; - el->el_line.buffer = el_malloc(EL_BUFSIZ * sizeof(*el->el_line.buffer)); if (el->el_line.buffer == NULL) @@ -449,19 +447,14 @@ ch_init(EditLine *el) el->el_state.argument = 1; el->el_state.lastcmd = ED_UNASSIGNED; - ma->level = -1; - ma->offset = 0; - ma->macro = el_malloc(EL_MAXMACRO * sizeof(*ma->macro)); - if (ma->macro == NULL) - return -1; return 0; } /* ch_reset(): * Reset the character editor */ -protected void -ch_reset(EditLine *el, int mclear) +libedit_private void +ch_reset(EditLine *el) { el->el_line.cursor = el->el_line.buffer; el->el_line.lastchar = el->el_line.buffer; @@ -483,28 +476,17 @@ ch_reset(EditLine *el, int mclear) el->el_state.lastcmd = ED_UNASSIGNED; el->el_history.eventno = 0; - - if (mclear) - ch__clearmacro(el); } -private void -ch__clearmacro(EditLine *el) -{ - c_macro_t *ma = &el->el_chared.c_macro; - while (ma->level >= 0) - el_free(ma->macro[ma->level--]); -} - /* ch_enlargebufs(): * Enlarge line buffer to be able to hold twice as much characters. * Returns 1 if successful, 0 if not. */ -protected int +libedit_private int ch_enlargebufs(EditLine *el, size_t addlen) { size_t sz, newsz; - Char *newbuffer, *oldbuf, *oldkbuf; + wchar_t *newbuffer, *oldbuf, *oldkbuf; sz = (size_t)(el->el_line.limit - el->el_line.buffer + EL_LEAVE); newsz = sz * 2; @@ -589,7 +571,7 @@ ch_enlargebufs(EditLine *el, size_t addlen) /* ch_end(): * Free the data structures used by the editor */ -protected void +libedit_private void ch_end(EditLine *el) { el_free(el->el_line.buffer); @@ -604,21 +586,19 @@ ch_end(EditLine *el) el->el_chared.c_redo.cmd = ED_UNASSIGNED; el_free(el->el_chared.c_kill.buf); el->el_chared.c_kill.buf = NULL; - ch_reset(el, 1); - el_free(el->el_chared.c_macro.macro); - el->el_chared.c_macro.macro = NULL; + ch_reset(el); } /* el_insertstr(): * Insert string at cursorI */ -public int -FUN(el,insertstr)(EditLine *el, const Char *s) +int +el_winsertstr(EditLine *el, const wchar_t *s) { size_t len; - if (s == NULL || (len = Strlen(s)) == 0) + if (s == NULL || (len = wcslen(s)) == 0) return -1; if (el->el_line.lastchar + len >= el->el_line.limit) { if (!ch_enlargebufs(el, len)) @@ -635,7 +615,7 @@ FUN(el,insertstr)(EditLine *el, const Char *s) /* el_deletestr(): * Delete num characters before the cursor */ -public void +void el_deletestr(EditLine *el, int n) { if (n <= 0) @@ -653,7 +633,7 @@ el_deletestr(EditLine *el, int n) /* el_cursor(): * Move the cursor to the left or the right of the current position */ -public int +int el_cursor(EditLine *el, int n) { if (n == 0) @@ -672,15 +652,14 @@ out: /* c_gets(): * Get a string */ -protected int -c_gets(EditLine *el, Char *buf, const Char *prompt) +libedit_private int +c_gets(EditLine *el, wchar_t *buf, const wchar_t *prompt) { - wchar_t wch; ssize_t len; - Char *cp = el->el_line.buffer, ch; + wchar_t *cp = el->el_line.buffer, ch; if (prompt) { - len = (ssize_t)Strlen(prompt); + len = (ssize_t)wcslen(prompt); (void)memcpy(cp, prompt, (size_t)len * sizeof(*cp)); cp += len; } @@ -692,12 +671,11 @@ c_gets(EditLine *el, Char *buf, const Char *prompt) el->el_line.lastchar = cp + 1; re_refresh(el); - if (el_wgetc(el, &wch) != 1) { + if (el_wgetc(el, &ch) != 1) { ed_end_of_file(el, 0); len = -1; break; } - ch = (Char)wch; switch (ch) { @@ -739,10 +717,10 @@ c_gets(EditLine *el, Char *buf, const Char *prompt) /* c_hpos(): * Return the current horizontal position of the cursor */ -protected int +libedit_private int c_hpos(EditLine *el) { - Char *ptr; + wchar_t *ptr; /* * Find how many characters till the beginning of this line. @@ -758,7 +736,7 @@ c_hpos(EditLine *el) } } -protected int +libedit_private int ch_resizefun(EditLine *el, el_zfunc_t f, void *a) { el->el_chared.c_resizefun = f; @@ -766,7 +744,7 @@ ch_resizefun(EditLine *el, el_zfunc_t f, void *a) return 0; } -protected int +libedit_private int ch_aliasfun(EditLine *el, el_afunc_t f, void *a) { el->el_chared.c_aliasfun = f; Modified: vendor/NetBSD/libedit/dist/chared.h ============================================================================== --- vendor/NetBSD/libedit/dist/chared.h Wed Sep 13 15:44:54 2017 (r323544) +++ vendor/NetBSD/libedit/dist/chared.h Wed Sep 13 15:57:58 2017 (r323545) @@ -1,4 +1,4 @@ -/* $NetBSD: chared.h,v 1.27 2016/02/16 22:53:14 christos Exp $ */ +/* $NetBSD: chared.h,v 1.30 2016/05/22 19:44:26 christos Exp $ */ /*- * Copyright (c) 1992, 1993 @@ -40,8 +40,6 @@ #ifndef _h_el_chared #define _h_el_chared -#define EL_MAXMACRO 10 - /* * This is an issue of basic "vi" look-and-feel. Defining VI_MOVE works * like real vi: i.e. the transition from command<->insert modes moves @@ -54,29 +52,22 @@ */ #define VI_MOVE - -typedef struct c_macro_t { - int level; - int offset; - Char **macro; -} c_macro_t; - /* * Undo information for vi - no undo in emacs (yet) */ typedef struct c_undo_t { ssize_t len; /* length of saved line */ int cursor; /* position of saved cursor */ - Char *buf; /* full saved text */ + wchar_t *buf; /* full saved text */ } c_undo_t; /* redo for vi */ typedef struct c_redo_t { - Char *buf; /* redo insert key sequence */ - Char *pos; - Char *lim; + wchar_t *buf; /* redo insert key sequence */ + wchar_t *pos; + wchar_t *lim; el_action_t cmd; /* command to redo */ - Char ch; /* char that invoked it */ + wchar_t ch; /* char that invoked it */ int count; int action; /* from cv_action() */ } c_redo_t; @@ -86,16 +77,16 @@ typedef struct c_redo_t { */ typedef struct c_vcmd_t { int action; - Char *pos; + wchar_t *pos; } c_vcmd_t; /* * Kill buffer for emacs */ typedef struct c_kill_t { - Char *buf; - Char *last; - Char *mark; + wchar_t *buf; + wchar_t *last; + wchar_t *mark; } c_kill_t; typedef void (*el_zfunc_t)(EditLine *, void *); @@ -110,7 +101,6 @@ typedef struct el_chared_t { c_kill_t c_kill; c_redo_t c_redo; c_vcmd_t c_vcmd; - c_macro_t c_macro; el_zfunc_t c_resizefun; el_afunc_t c_aliasfun; void * c_resizearg; @@ -135,30 +125,31 @@ typedef struct el_chared_t { #define MODE_REPLACE_1 2 -protected int cv__isword(wint_t); -protected int cv__isWord(wint_t); -protected void cv_delfini(EditLine *); -protected Char *cv__endword(Char *, Char *, int, int (*)(wint_t)); -protected int ce__isword(wint_t); -protected void cv_undo(EditLine *); -protected void cv_yank(EditLine *, const Char *, int); -protected Char *cv_next_word(EditLine*, Char *, Char *, int, int (*)(wint_t)); -protected Char *cv_prev_word(Char *, Char *, int, int (*)(wint_t)); -protected Char *c__next_word(Char *, Char *, int, int (*)(wint_t)); -protected Char *c__prev_word(Char *, Char *, int, int (*)(wint_t)); -protected void c_insert(EditLine *, int); -protected void c_delbefore(EditLine *, int); -protected void c_delbefore1(EditLine *); -protected void c_delafter(EditLine *, int); -protected void c_delafter1(EditLine *); -protected int c_gets(EditLine *, Char *, const Char *); -protected int c_hpos(EditLine *); +libedit_private int cv__isword(wint_t); +libedit_private int cv__isWord(wint_t); +libedit_private void cv_delfini(EditLine *); +libedit_private wchar_t *cv__endword(wchar_t *, wchar_t *, int, int (*)(wint_t)); +libedit_private int ce__isword(wint_t); +libedit_private void cv_undo(EditLine *); +libedit_private void cv_yank(EditLine *, const wchar_t *, int); +libedit_private wchar_t *cv_next_word(EditLine*, wchar_t *, wchar_t *, int, + int (*)(wint_t)); +libedit_private wchar_t *cv_prev_word(wchar_t *, wchar_t *, int, int (*)(wint_t)); +libedit_private wchar_t *c__next_word(wchar_t *, wchar_t *, int, int (*)(wint_t)); +libedit_private wchar_t *c__prev_word(wchar_t *, wchar_t *, int, int (*)(wint_t)); +libedit_private void c_insert(EditLine *, int); +libedit_private void c_delbefore(EditLine *, int); +libedit_private void c_delbefore1(EditLine *); +libedit_private void c_delafter(EditLine *, int); +libedit_private void c_delafter1(EditLine *); +libedit_private int c_gets(EditLine *, wchar_t *, const wchar_t *); +libedit_private int c_hpos(EditLine *); -protected int ch_init(EditLine *); -protected void ch_reset(EditLine *, int); -protected int ch_resizefun(EditLine *, el_zfunc_t, void *); -protected int ch_aliasfun(EditLine *, el_afunc_t, void *); -protected int ch_enlargebufs(EditLine *, size_t); -protected void ch_end(EditLine *); +libedit_private int ch_init(EditLine *); +libedit_private void ch_reset(EditLine *); +libedit_private int ch_resizefun(EditLine *, el_zfunc_t, void *); +libedit_private int ch_aliasfun(EditLine *, el_afunc_t, void *); +libedit_private int ch_enlargebufs(EditLine *, size_t); +libedit_private void ch_end(EditLine *); #endif /* _h_el_chared */ Modified: vendor/NetBSD/libedit/dist/chartype.c ============================================================================== --- vendor/NetBSD/libedit/dist/chartype.c Wed Sep 13 15:44:54 2017 (r323544) +++ vendor/NetBSD/libedit/dist/chartype.c Wed Sep 13 15:57:58 2017 (r323545) @@ -1,4 +1,4 @@ -/* $NetBSD: chartype.c,v 1.23 2016/02/28 23:02:24 christos Exp $ */ +/* $NetBSD: chartype.c,v 1.31 2017/01/09 02:54:18 christos Exp $ */ /*- * Copyright (c) 2009 The NetBSD Foundation, Inc. @@ -31,7 +31,7 @@ */ #include "config.h" #if !defined(lint) && !defined(SCCSID) -__RCSID("$NetBSD: chartype.c,v 1.23 2016/02/28 23:02:24 christos Exp $"); +__RCSID("$NetBSD: chartype.c,v 1.31 2017/01/09 02:54:18 christos Exp $"); #endif /* not lint && not SCCSID */ #include @@ -42,8 +42,10 @@ __RCSID("$NetBSD: chartype.c,v 1.23 2016/02/28 23:02:2 #define CT_BUFSIZ ((size_t)1024) -#ifdef WIDECHAR -protected int +static int ct_conv_cbuff_resize(ct_buffer_t *, size_t); +static int ct_conv_wbuff_resize(ct_buffer_t *, size_t); + +static int ct_conv_cbuff_resize(ct_buffer_t *conv, size_t csize) { void *p; @@ -64,7 +66,7 @@ ct_conv_cbuff_resize(ct_buffer_t *conv, size_t csize) return 0; } -protected int +static int ct_conv_wbuff_resize(ct_buffer_t *conv, size_t wsize) { void *p; @@ -86,8 +88,8 @@ ct_conv_wbuff_resize(ct_buffer_t *conv, size_t wsize) } -public char * -ct_encode_string(const Char *s, ct_buffer_t *conv) +char * +ct_encode_string(const wchar_t *s, ct_buffer_t *conv) { char *dst; ssize_t used; @@ -116,7 +118,7 @@ ct_encode_string(const Char *s, ct_buffer_t *conv) return conv->cbuff; } -public Char * +wchar_t * ct_decode_string(const char *s, ct_buffer_t *conv) { size_t len; @@ -124,7 +126,7 @@ ct_decode_string(const char *s, ct_buffer_t *conv) if (!s) return NULL; - len = ct_mbstowcs(NULL, s, (size_t)0); + len = mbstowcs(NULL, s, (size_t)0); if (len == (size_t)-1) return NULL; @@ -132,18 +134,18 @@ ct_decode_string(const char *s, ct_buffer_t *conv) if (ct_conv_wbuff_resize(conv, len + CT_BUFSIZ) == -1) return NULL; - ct_mbstowcs(conv->wbuff, s, conv->wsize); + mbstowcs(conv->wbuff, s, conv->wsize); return conv->wbuff; } -protected Char ** +libedit_private wchar_t ** ct_decode_argv(int argc, const char *argv[], ct_buffer_t *conv) { size_t bufspace; int i; - Char *p; - Char **wargv; + wchar_t *p; + wchar_t **wargv; ssize_t bytes; /* Make sure we have enough space in the conversion buffer to store all @@ -154,7 +156,7 @@ ct_decode_argv(int argc, const char *argv[], ct_buffer if (ct_conv_wbuff_resize(conv, bufspace + CT_BUFSIZ) == -1) return NULL; - wargv = el_malloc((size_t)argc * sizeof(*wargv)); + wargv = el_malloc((size_t)(argc + 1) * sizeof(*wargv)); for (i = 0, p = conv->wbuff; i < argc; ++i) { if (!argv[i]) { /* don't pass null pointers to mbstowcs */ @@ -172,13 +174,14 @@ ct_decode_argv(int argc, const char *argv[], ct_buffer bufspace -= (size_t)bytes; p += bytes; } + wargv[i] = NULL; return wargv; } -protected size_t -ct_enc_width(Char c) +libedit_private size_t +ct_enc_width(wchar_t c) { /* UTF-8 encoding specific values */ if (c < 0x80) @@ -193,96 +196,66 @@ ct_enc_width(Char c) return 0; /* not a valid codepoint */ } -protected ssize_t -ct_encode_char(char *dst, size_t len, Char c) +libedit_private ssize_t +ct_encode_char(char *dst, size_t len, wchar_t c) { ssize_t l = 0; if (len < ct_enc_width(c)) return -1; - l = ct_wctomb(dst, c); + l = wctomb(dst, c); if (l < 0) { - ct_wctomb_reset; + wctomb(NULL, L'\0'); l = 0; } return l; } -size_t -ct_mbrtowc(wchar_t *wc, const char *s, size_t n) +libedit_private const wchar_t * +ct_visual_string(const wchar_t *s, ct_buffer_t *conv) { - mbstate_t mbs; - /* This only works because UTF-8 is stateless */ - memset(&mbs, 0, sizeof(mbs)); - return mbrtowc(wc, s, n, &mbs); -} + wchar_t *dst; + ssize_t used; -#else - -size_t -ct_mbrtowc(wchar_t *wc, const char *s, size_t n) - if (s == NULL) - return 0; - if (n == 0) - return (size_t)-2; - if (wc != NULL) - *wc = *s; - return *s != '\0'; -} -#endif - -protected const Char * -ct_visual_string(const Char *s) -{ - static Char *buff = NULL; - static size_t buffsize = 0; - void *p; - Char *dst; - ssize_t used = 0; - if (!s) return NULL; - if (!buff) { - buffsize = CT_BUFSIZ; - buff = el_malloc(buffsize * sizeof(*buff)); - } - dst = buff; + + if (ct_conv_wbuff_resize(conv, CT_BUFSIZ) == -1) + return NULL; + + used = 0; + dst = conv->wbuff; while (*s) { - used = ct_visual_char(dst, buffsize - (size_t)(dst - buff), *s); - if (used == -1) { /* failed to encode, need more buffer space */ - used = dst - buff; - buffsize += CT_BUFSIZ; - p = el_realloc(buff, buffsize * sizeof(*buff)); - if (p == NULL) - goto out; - buff = p; - dst = buff + used; - /* don't increment s here - we want to retry it! */ + used = ct_visual_char(dst, + conv->wsize - (size_t)(dst - conv->wbuff), *s); + if (used != -1) { + ++s; + dst += used; + continue; } - else - ++s; - dst += used; + + /* failed to encode, need more buffer space */ + used = dst - conv->wbuff; + if (ct_conv_wbuff_resize(conv, conv->wsize + CT_BUFSIZ) == -1) + return NULL; + dst = conv->wbuff + used; } - if (dst >= (buff + buffsize)) { /* sigh */ - buffsize += 1; - p = el_realloc(buff, buffsize * sizeof(*buff)); - if (p == NULL) - goto out; - buff = p; - dst = buff + buffsize - 1; + + if (dst >= (conv->wbuff + conv->wsize)) { /* sigh */ + used = dst - conv->wbuff; + if (ct_conv_wbuff_resize(conv, conv->wsize + CT_BUFSIZ) == -1) + return NULL; + dst = conv->wbuff + used; } - *dst = 0; - return buff; -out: - el_free(buff); - buffsize = 0; - return NULL; + *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Sep 13 16:05:27 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4AEEE030C4; Wed, 13 Sep 2017 16:05:27 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B19D6B687; Wed, 13 Sep 2017 16:05:27 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DG5QZ9097915; Wed, 13 Sep 2017 16:05:26 GMT (envelope-from pfg@FreeBSD.org) Received: (from pfg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DG5Q0r097910; Wed, 13 Sep 2017 16:05:26 GMT (envelope-from pfg@FreeBSD.org) Message-Id: <201709131605.v8DG5Q0r097910@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: pfg set sender to pfg@FreeBSD.org using -f From: "Pedro F. Giffuni" Date: Wed, 13 Sep 2017 16:05:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323546 - in vendor/NetBSD/libedit/2017-09-05: . TEST readline X-SVN-Group: vendor X-SVN-Commit-Author: pfg X-SVN-Commit-Paths: in vendor/NetBSD/libedit/2017-09-05: . TEST readline X-SVN-Commit-Revision: 323546 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 16:05:27 -0000 Author: pfg Date: Wed Sep 13 16:05:26 2017 New Revision: 323546 URL: https://svnweb.freebsd.org/changeset/base/323546 Log: Tag import of libedit 2017-09-05. Added: vendor/NetBSD/libedit/2017-09-05/ - copied from r323542, vendor/NetBSD/libedit/dist/ vendor/NetBSD/libedit/2017-09-05/editline.7 - copied unchanged from r323545, vendor/NetBSD/libedit/dist/editline.7 vendor/NetBSD/libedit/2017-09-05/historyn.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/historyn.c vendor/NetBSD/libedit/2017-09-05/literal.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/literal.c vendor/NetBSD/libedit/2017-09-05/literal.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/literal.h vendor/NetBSD/libedit/2017-09-05/tokenizern.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/tokenizern.c Replaced: vendor/NetBSD/libedit/2017-09-05/Makefile - copied unchanged from r323545, vendor/NetBSD/libedit/dist/Makefile vendor/NetBSD/libedit/2017-09-05/TEST/Makefile - copied unchanged from r323545, vendor/NetBSD/libedit/dist/TEST/Makefile vendor/NetBSD/libedit/2017-09-05/chared.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/chared.c vendor/NetBSD/libedit/2017-09-05/chared.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/chared.h vendor/NetBSD/libedit/2017-09-05/chartype.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/chartype.c vendor/NetBSD/libedit/2017-09-05/chartype.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/chartype.h vendor/NetBSD/libedit/2017-09-05/common.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/common.c vendor/NetBSD/libedit/2017-09-05/config.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/config.h vendor/NetBSD/libedit/2017-09-05/editline.3 - copied unchanged from r323545, vendor/NetBSD/libedit/dist/editline.3 vendor/NetBSD/libedit/2017-09-05/editrc.5 - copied unchanged from r323545, vendor/NetBSD/libedit/dist/editrc.5 vendor/NetBSD/libedit/2017-09-05/el.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/el.c vendor/NetBSD/libedit/2017-09-05/el.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/el.h vendor/NetBSD/libedit/2017-09-05/eln.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/eln.c vendor/NetBSD/libedit/2017-09-05/emacs.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/emacs.c vendor/NetBSD/libedit/2017-09-05/filecomplete.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/filecomplete.c vendor/NetBSD/libedit/2017-09-05/filecomplete.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/filecomplete.h vendor/NetBSD/libedit/2017-09-05/hist.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/hist.c vendor/NetBSD/libedit/2017-09-05/hist.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/hist.h vendor/NetBSD/libedit/2017-09-05/histedit.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/histedit.h vendor/NetBSD/libedit/2017-09-05/history.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/history.c vendor/NetBSD/libedit/2017-09-05/keymacro.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/keymacro.c vendor/NetBSD/libedit/2017-09-05/keymacro.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/keymacro.h vendor/NetBSD/libedit/2017-09-05/makelist - copied unchanged from r323545, vendor/NetBSD/libedit/dist/makelist vendor/NetBSD/libedit/2017-09-05/map.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/map.c vendor/NetBSD/libedit/2017-09-05/map.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/map.h vendor/NetBSD/libedit/2017-09-05/parse.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/parse.c vendor/NetBSD/libedit/2017-09-05/parse.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/parse.h vendor/NetBSD/libedit/2017-09-05/prompt.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/prompt.c vendor/NetBSD/libedit/2017-09-05/prompt.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/prompt.h vendor/NetBSD/libedit/2017-09-05/read.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/read.c vendor/NetBSD/libedit/2017-09-05/read.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/read.h vendor/NetBSD/libedit/2017-09-05/readline.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/readline.c vendor/NetBSD/libedit/2017-09-05/readline/readline.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/readline/readline.h vendor/NetBSD/libedit/2017-09-05/refresh.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/refresh.c vendor/NetBSD/libedit/2017-09-05/refresh.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/refresh.h vendor/NetBSD/libedit/2017-09-05/search.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/search.c vendor/NetBSD/libedit/2017-09-05/search.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/search.h vendor/NetBSD/libedit/2017-09-05/sig.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/sig.c vendor/NetBSD/libedit/2017-09-05/sig.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/sig.h vendor/NetBSD/libedit/2017-09-05/sys.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/sys.h vendor/NetBSD/libedit/2017-09-05/terminal.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/terminal.c vendor/NetBSD/libedit/2017-09-05/terminal.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/terminal.h vendor/NetBSD/libedit/2017-09-05/tokenizer.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/tokenizer.c vendor/NetBSD/libedit/2017-09-05/tty.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/tty.c vendor/NetBSD/libedit/2017-09-05/tty.h - copied unchanged from r323545, vendor/NetBSD/libedit/dist/tty.h vendor/NetBSD/libedit/2017-09-05/vi.c - copied unchanged from r323545, vendor/NetBSD/libedit/dist/vi.c Copied: vendor/NetBSD/libedit/2017-09-05/Makefile (from r323545, vendor/NetBSD/libedit/dist/Makefile) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/NetBSD/libedit/2017-09-05/Makefile Wed Sep 13 16:05:26 2017 (r323546, copy of r323545, vendor/NetBSD/libedit/dist/Makefile) @@ -0,0 +1,139 @@ +# $NetBSD: Makefile,v 1.65 2017/06/30 20:26:52 kre Exp $ +# @(#)Makefile 8.1 (Berkeley) 6/4/93 + +USE_SHLIBDIR= yes + +WARNS?= 5 +LIB= edit + +LIBDPLIBS+= terminfo ${.CURDIR}/../libterminfo + +.include "bsd.own.mk" + +COPTS+= -Wunused-parameter +CWARNFLAGS.gcc+= -Wconversion +CWARNFLAGS.clang+= -Wno-cast-qual + +SRCS = chared.c chartype.c common.c el.c eln.c emacs.c filecomplete.c \ + hist.c history.c historyn.c keymacro.c literal.c map.c \ + parse.c prompt.c read.c readline.c refresh.c search.c sig.c \ + terminal.c tokenizer.c tokenizern.c tty.c vi.c + +MAN= editline.3 editrc.5 editline.7 + +MLINKS= \ +editline.3 el_deletestr.3 \ +editline.3 el_end.3 \ +editline.3 el_get.3 \ +editline.3 el_getc.3 \ +editline.3 el_gets.3 \ +editline.3 el_init.3 \ +editline.3 el_init_fd.3 \ +editline.3 el_insertstr.3 \ +editline.3 el_line.3 \ +editline.3 el_parse.3 \ +editline.3 el_push.3 \ +editline.3 el_reset.3 \ +editline.3 el_resize.3 \ +editline.3 el_set.3 \ +editline.3 el_source.3 \ +editline.3 history.3 \ +editline.3 history_end.3 \ +editline.3 history_init.3 \ +editline.3 tok_end.3 \ +editline.3 tok_init.3 \ +editline.3 tok_line.3 \ +editline.3 tok_reset.3 \ +editline.3 tok_str.3 + +MLINKS+= \ +editline.3 el_wdeletestr.3 \ +editline.3 el_wget.3 \ +editline.3 el_wgetc.3 \ +editline.3 el_wgets.3 \ +editline.3 el_winsertstr.3 \ +editline.3 el_wline.3 \ +editline.3 el_wparse.3 \ +editline.3 el_wpush.3 \ +editline.3 el_wset.3 \ +editline.3 history_w.3 \ +editline.3 history_wend.3 \ +editline.3 history_winit.3 \ +editline.3 tok_wend.3 \ +editline.3 tok_winit.3 \ +editline.3 tok_wline.3 \ +editline.3 tok_wreset.3 \ +editline.3 tok_wstr.3 + +LIBEDITDIR?=${.CURDIR} + +INCS= histedit.h +INCSDIR=/usr/include + +CLEANFILES+=common.h.tmp emacs.h.tmp fcns.h.tmp func.h.tmp +CLEANFILES+=help.h.tmp vi.h.tmp tc1.o tc1 .depend + +CPPFLAGS+=-I. -I${LIBEDITDIR} +CPPFLAGS+=-I. -I${.CURDIR} +#CPPFLAGS+=-DDEBUG_TTY -DDEBUG_KEY -DDEBUG -DDEBUG_REFRESH +#CPPFLAGS+=-DDEBUG_PASTE -DDEBUG_EDIT + +AHDR=vi.h emacs.h common.h +ASRC=${LIBEDITDIR}/vi.c ${LIBEDITDIR}/emacs.c ${LIBEDITDIR}/common.c + +DPSRCS+= ${AHDR} fcns.h func.h help.h +CLEANFILES+= ${AHDR} fcns.h func.h help.h + +SUBDIR= readline + +.depend: ${AHDR} fcns.h func.h help.h + +vi.h: vi.c makelist Makefile + ${_MKTARGET_CREATE} + ${HOST_SH} ${LIBEDITDIR}/makelist -h ${LIBEDITDIR}/vi.c \ + > ${.TARGET}.tmp && \ + mv ${.TARGET}.tmp ${.TARGET} + +emacs.h: emacs.c makelist Makefile + ${_MKTARGET_CREATE} + ${HOST_SH} ${LIBEDITDIR}/makelist -h ${LIBEDITDIR}/emacs.c \ + > ${.TARGET}.tmp && \ + mv ${.TARGET}.tmp ${.TARGET} + +common.h: common.c makelist Makefile + ${_MKTARGET_CREATE} + ${HOST_SH} ${LIBEDITDIR}/makelist -h ${LIBEDITDIR}/common.c \ + > ${.TARGET}.tmp && \ + mv ${.TARGET}.tmp ${.TARGET} + +fcns.h: ${AHDR} makelist Makefile + ${_MKTARGET_CREATE} + ${HOST_SH} ${LIBEDITDIR}/makelist -fh ${AHDR} > ${.TARGET}.tmp && \ + mv ${.TARGET}.tmp ${.TARGET} + +func.h: ${AHDR} makelist Makefile + ${_MKTARGET_CREATE} + ${HOST_SH} ${LIBEDITDIR}/makelist -fc ${AHDR} > ${.TARGET}.tmp && \ + mv ${.TARGET}.tmp ${.TARGET} + +help.h: ${ASRC} makelist Makefile + ${_MKTARGET_CREATE} + ${HOST_SH} ${LIBEDITDIR}/makelist -bh ${ASRC} > ${.TARGET}.tmp && \ + mv ${.TARGET}.tmp ${.TARGET} + +tc1.o: ${LIBEDITDIR}/TEST/tc1.c + +tc1: libedit.a tc1.o + ${_MKTARGET_LINK} + ${CC} ${LDFLAGS} ${.ALLSRC} -o ${.TARGET} libedit.a ${LDADD} -ltermlib + +.include +.include + +# XXX +.if defined(HAVE_GCC) +COPTS.editline.c+= -Wno-cast-qual +COPTS.literal.c+= -Wno-sign-conversion +COPTS.tokenizer.c+= -Wno-cast-qual +COPTS.tokenizern.c+= -Wno-cast-qual +.endif Copied: vendor/NetBSD/libedit/2017-09-05/TEST/Makefile (from r323545, vendor/NetBSD/libedit/dist/TEST/Makefile) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/NetBSD/libedit/2017-09-05/TEST/Makefile Wed Sep 13 16:05:26 2017 (r323546, copy of r323545, vendor/NetBSD/libedit/dist/TEST/Makefile) @@ -0,0 +1,13 @@ +# $NetBSD: Makefile,v 1.7 2016/03/23 22:27:48 christos Exp $ + +NOMAN=1 +PROG=wtc1 +CPPFLAGS=-I${.CURDIR}/.. +LDADD+=-ledit -ltermlib +DPADD+=${LIBEDIT} ${LIBTERMLIB} + +.ifdef DEBUG +CPPFLAGS+=-DDEBUG +.endif + +.include Copied: vendor/NetBSD/libedit/2017-09-05/chared.c (from r323545, vendor/NetBSD/libedit/dist/chared.c) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/NetBSD/libedit/2017-09-05/chared.c Wed Sep 13 16:05:26 2017 (r323546, copy of r323545, vendor/NetBSD/libedit/dist/chared.c) @@ -0,0 +1,753 @@ +/* $NetBSD: chared.c,v 1.56 2016/05/22 19:44:26 christos Exp $ */ + +/*- + * Copyright (c) 1992, 1993 + * The Regents of the University of California. All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * Christos Zoulas of Cornell University. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include "config.h" +#if !defined(lint) && !defined(SCCSID) +#if 0 +static char sccsid[] = "@(#)chared.c 8.1 (Berkeley) 6/4/93"; +#else +__RCSID("$NetBSD: chared.c,v 1.56 2016/05/22 19:44:26 christos Exp $"); +#endif +#endif /* not lint && not SCCSID */ + +/* + * chared.c: Character editor utilities + */ +#include +#include +#include + +#include "el.h" +#include "common.h" +#include "fcns.h" + +/* value to leave unused in line buffer */ +#define EL_LEAVE 2 + +/* cv_undo(): + * Handle state for the vi undo command + */ +libedit_private void +cv_undo(EditLine *el) +{ + c_undo_t *vu = &el->el_chared.c_undo; + c_redo_t *r = &el->el_chared.c_redo; + size_t size; + + /* Save entire line for undo */ + size = (size_t)(el->el_line.lastchar - el->el_line.buffer); + vu->len = (ssize_t)size; + vu->cursor = (int)(el->el_line.cursor - el->el_line.buffer); + (void)memcpy(vu->buf, el->el_line.buffer, size * sizeof(*vu->buf)); + + /* save command info for redo */ + r->count = el->el_state.doingarg ? el->el_state.argument : 0; + r->action = el->el_chared.c_vcmd.action; + r->pos = r->buf; + r->cmd = el->el_state.thiscmd; + r->ch = el->el_state.thisch; +} + +/* cv_yank(): + * Save yank/delete data for paste + */ +libedit_private void +cv_yank(EditLine *el, const wchar_t *ptr, int size) +{ + c_kill_t *k = &el->el_chared.c_kill; + + (void)memcpy(k->buf, ptr, (size_t)size * sizeof(*k->buf)); + k->last = k->buf + size; +} + + +/* c_insert(): + * Insert num characters + */ +libedit_private void +c_insert(EditLine *el, int num) +{ + wchar_t *cp; + + if (el->el_line.lastchar + num >= el->el_line.limit) { + if (!ch_enlargebufs(el, (size_t)num)) + return; /* can't go past end of buffer */ + } + + if (el->el_line.cursor < el->el_line.lastchar) { + /* if I must move chars */ + for (cp = el->el_line.lastchar; cp >= el->el_line.cursor; cp--) + cp[num] = *cp; + } + el->el_line.lastchar += num; +} + + +/* c_delafter(): + * Delete num characters after the cursor + */ +libedit_private void +c_delafter(EditLine *el, int num) +{ + + if (el->el_line.cursor + num > el->el_line.lastchar) + num = (int)(el->el_line.lastchar - el->el_line.cursor); + + if (el->el_map.current != el->el_map.emacs) { + cv_undo(el); + cv_yank(el, el->el_line.cursor, num); + } + + if (num > 0) { + wchar_t *cp; + + for (cp = el->el_line.cursor; cp <= el->el_line.lastchar; cp++) + *cp = cp[num]; + + el->el_line.lastchar -= num; + } +} + + +/* c_delafter1(): + * Delete the character after the cursor, do not yank + */ +libedit_private void +c_delafter1(EditLine *el) +{ + wchar_t *cp; + + for (cp = el->el_line.cursor; cp <= el->el_line.lastchar; cp++) + *cp = cp[1]; + + el->el_line.lastchar--; +} + + +/* c_delbefore(): + * Delete num characters before the cursor + */ +libedit_private void +c_delbefore(EditLine *el, int num) +{ + + if (el->el_line.cursor - num < el->el_line.buffer) + num = (int)(el->el_line.cursor - el->el_line.buffer); + + if (el->el_map.current != el->el_map.emacs) { + cv_undo(el); + cv_yank(el, el->el_line.cursor - num, num); + } + + if (num > 0) { + wchar_t *cp; + + for (cp = el->el_line.cursor - num; + cp <= el->el_line.lastchar; + cp++) + *cp = cp[num]; + + el->el_line.lastchar -= num; + } +} + + +/* c_delbefore1(): + * Delete the character before the cursor, do not yank + */ +libedit_private void +c_delbefore1(EditLine *el) +{ + wchar_t *cp; + + for (cp = el->el_line.cursor - 1; cp <= el->el_line.lastchar; cp++) + *cp = cp[1]; + + el->el_line.lastchar--; +} + + +/* ce__isword(): + * Return if p is part of a word according to emacs + */ +libedit_private int +ce__isword(wint_t p) +{ + return iswalnum(p) || wcschr(L"*?_-.[]~=", p) != NULL; +} + + +/* cv__isword(): + * Return if p is part of a word according to vi + */ +libedit_private int +cv__isword(wint_t p) +{ + if (iswalnum(p) || p == L'_') + return 1; + if (iswgraph(p)) + return 2; + return 0; +} + + +/* cv__isWord(): + * Return if p is part of a big word according to vi + */ +libedit_private int +cv__isWord(wint_t p) +{ + return !iswspace(p); +} + + +/* c__prev_word(): + * Find the previous word + */ +libedit_private wchar_t * +c__prev_word(wchar_t *p, wchar_t *low, int n, int (*wtest)(wint_t)) +{ + p--; + + while (n--) { + while ((p >= low) && !(*wtest)(*p)) + p--; + while ((p >= low) && (*wtest)(*p)) + p--; + } + + /* cp now points to one character before the word */ + p++; + if (p < low) + p = low; + /* cp now points where we want it */ + return p; +} + + +/* c__next_word(): + * Find the next word + */ +libedit_private wchar_t * +c__next_word(wchar_t *p, wchar_t *high, int n, int (*wtest)(wint_t)) +{ + while (n--) { + while ((p < high) && !(*wtest)(*p)) + p++; + while ((p < high) && (*wtest)(*p)) + p++; + } + if (p > high) + p = high; + /* p now points where we want it */ + return p; +} + +/* cv_next_word(): + * Find the next word vi style + */ +libedit_private wchar_t * +cv_next_word(EditLine *el, wchar_t *p, wchar_t *high, int n, + int (*wtest)(wint_t)) +{ + int test; + + while (n--) { + test = (*wtest)(*p); + while ((p < high) && (*wtest)(*p) == test) + p++; + /* + * vi historically deletes with cw only the word preserving the + * trailing whitespace! This is not what 'w' does.. + */ + if (n || el->el_chared.c_vcmd.action != (DELETE|INSERT)) + while ((p < high) && iswspace(*p)) + p++; + } + + /* p now points where we want it */ + if (p > high) + return high; + else + return p; +} + + +/* cv_prev_word(): + * Find the previous word vi style + */ +libedit_private wchar_t * +cv_prev_word(wchar_t *p, wchar_t *low, int n, int (*wtest)(wint_t)) +{ + int test; + + p--; + while (n--) { + while ((p > low) && iswspace(*p)) + p--; + test = (*wtest)(*p); + while ((p >= low) && (*wtest)(*p) == test) + p--; + } + p++; + + /* p now points where we want it */ + if (p < low) + return low; + else + return p; +} + + +/* cv_delfini(): + * Finish vi delete action + */ +libedit_private void +cv_delfini(EditLine *el) +{ + int size; + int action = el->el_chared.c_vcmd.action; + + if (action & INSERT) + el->el_map.current = el->el_map.key; + + if (el->el_chared.c_vcmd.pos == 0) + /* sanity */ + return; + + size = (int)(el->el_line.cursor - el->el_chared.c_vcmd.pos); + if (size == 0) + size = 1; + el->el_line.cursor = el->el_chared.c_vcmd.pos; + if (action & YANK) { + if (size > 0) + cv_yank(el, el->el_line.cursor, size); + else + cv_yank(el, el->el_line.cursor + size, -size); + } else { + if (size > 0) { + c_delafter(el, size); + re_refresh_cursor(el); + } else { + c_delbefore(el, -size); + el->el_line.cursor += size; + } + } + el->el_chared.c_vcmd.action = NOP; +} + + +/* cv__endword(): + * Go to the end of this word according to vi + */ +libedit_private wchar_t * +cv__endword(wchar_t *p, wchar_t *high, int n, int (*wtest)(wint_t)) +{ + int test; + + p++; + + while (n--) { + while ((p < high) && iswspace(*p)) + p++; + + test = (*wtest)(*p); + while ((p < high) && (*wtest)(*p) == test) + p++; + } + p--; + return p; +} + +/* ch_init(): + * Initialize the character editor + */ +libedit_private int +ch_init(EditLine *el) +{ + el->el_line.buffer = el_malloc(EL_BUFSIZ * + sizeof(*el->el_line.buffer)); + if (el->el_line.buffer == NULL) + return -1; + + (void) memset(el->el_line.buffer, 0, EL_BUFSIZ * + sizeof(*el->el_line.buffer)); + el->el_line.cursor = el->el_line.buffer; + el->el_line.lastchar = el->el_line.buffer; + el->el_line.limit = &el->el_line.buffer[EL_BUFSIZ - EL_LEAVE]; + + el->el_chared.c_undo.buf = el_malloc(EL_BUFSIZ * + sizeof(*el->el_chared.c_undo.buf)); + if (el->el_chared.c_undo.buf == NULL) + return -1; + (void) memset(el->el_chared.c_undo.buf, 0, EL_BUFSIZ * + sizeof(*el->el_chared.c_undo.buf)); + el->el_chared.c_undo.len = -1; + el->el_chared.c_undo.cursor = 0; + el->el_chared.c_redo.buf = el_malloc(EL_BUFSIZ * + sizeof(*el->el_chared.c_redo.buf)); + if (el->el_chared.c_redo.buf == NULL) + return -1; + el->el_chared.c_redo.pos = el->el_chared.c_redo.buf; + el->el_chared.c_redo.lim = el->el_chared.c_redo.buf + EL_BUFSIZ; + el->el_chared.c_redo.cmd = ED_UNASSIGNED; + + el->el_chared.c_vcmd.action = NOP; + el->el_chared.c_vcmd.pos = el->el_line.buffer; + + el->el_chared.c_kill.buf = el_malloc(EL_BUFSIZ * + sizeof(*el->el_chared.c_kill.buf)); + if (el->el_chared.c_kill.buf == NULL) + return -1; + (void) memset(el->el_chared.c_kill.buf, 0, EL_BUFSIZ * + sizeof(*el->el_chared.c_kill.buf)); + el->el_chared.c_kill.mark = el->el_line.buffer; + el->el_chared.c_kill.last = el->el_chared.c_kill.buf; + el->el_chared.c_resizefun = NULL; + el->el_chared.c_resizearg = NULL; + el->el_chared.c_aliasfun = NULL; + el->el_chared.c_aliasarg = NULL; + + el->el_map.current = el->el_map.key; + + el->el_state.inputmode = MODE_INSERT; /* XXX: save a default */ + el->el_state.doingarg = 0; + el->el_state.metanext = 0; + el->el_state.argument = 1; + el->el_state.lastcmd = ED_UNASSIGNED; + + return 0; +} + +/* ch_reset(): + * Reset the character editor + */ +libedit_private void +ch_reset(EditLine *el) +{ + el->el_line.cursor = el->el_line.buffer; + el->el_line.lastchar = el->el_line.buffer; + + el->el_chared.c_undo.len = -1; + el->el_chared.c_undo.cursor = 0; + + el->el_chared.c_vcmd.action = NOP; + el->el_chared.c_vcmd.pos = el->el_line.buffer; + + el->el_chared.c_kill.mark = el->el_line.buffer; + + el->el_map.current = el->el_map.key; + + el->el_state.inputmode = MODE_INSERT; /* XXX: save a default */ + el->el_state.doingarg = 0; + el->el_state.metanext = 0; + el->el_state.argument = 1; + el->el_state.lastcmd = ED_UNASSIGNED; + + el->el_history.eventno = 0; +} + +/* ch_enlargebufs(): + * Enlarge line buffer to be able to hold twice as much characters. + * Returns 1 if successful, 0 if not. + */ +libedit_private int +ch_enlargebufs(EditLine *el, size_t addlen) +{ + size_t sz, newsz; + wchar_t *newbuffer, *oldbuf, *oldkbuf; + + sz = (size_t)(el->el_line.limit - el->el_line.buffer + EL_LEAVE); + newsz = sz * 2; + /* + * If newly required length is longer than current buffer, we need + * to make the buffer big enough to hold both old and new stuff. + */ + if (addlen > sz) { + while(newsz - sz < addlen) + newsz *= 2; + } + + /* + * Reallocate line buffer. + */ + newbuffer = el_realloc(el->el_line.buffer, newsz * sizeof(*newbuffer)); + if (!newbuffer) + return 0; + + /* zero the newly added memory, leave old data in */ + (void) memset(&newbuffer[sz], 0, (newsz - sz) * sizeof(*newbuffer)); + + oldbuf = el->el_line.buffer; + + el->el_line.buffer = newbuffer; + el->el_line.cursor = newbuffer + (el->el_line.cursor - oldbuf); + el->el_line.lastchar = newbuffer + (el->el_line.lastchar - oldbuf); + /* don't set new size until all buffers are enlarged */ + el->el_line.limit = &newbuffer[sz - EL_LEAVE]; + + /* + * Reallocate kill buffer. + */ + newbuffer = el_realloc(el->el_chared.c_kill.buf, newsz * + sizeof(*newbuffer)); + if (!newbuffer) + return 0; + + /* zero the newly added memory, leave old data in */ + (void) memset(&newbuffer[sz], 0, (newsz - sz) * sizeof(*newbuffer)); + + oldkbuf = el->el_chared.c_kill.buf; + + el->el_chared.c_kill.buf = newbuffer; + el->el_chared.c_kill.last = newbuffer + + (el->el_chared.c_kill.last - oldkbuf); + el->el_chared.c_kill.mark = el->el_line.buffer + + (el->el_chared.c_kill.mark - oldbuf); + + /* + * Reallocate undo buffer. + */ + newbuffer = el_realloc(el->el_chared.c_undo.buf, + newsz * sizeof(*newbuffer)); + if (!newbuffer) + return 0; + + /* zero the newly added memory, leave old data in */ + (void) memset(&newbuffer[sz], 0, (newsz - sz) * sizeof(*newbuffer)); + el->el_chared.c_undo.buf = newbuffer; + + newbuffer = el_realloc(el->el_chared.c_redo.buf, + newsz * sizeof(*newbuffer)); + if (!newbuffer) + return 0; + el->el_chared.c_redo.pos = newbuffer + + (el->el_chared.c_redo.pos - el->el_chared.c_redo.buf); + el->el_chared.c_redo.lim = newbuffer + + (el->el_chared.c_redo.lim - el->el_chared.c_redo.buf); + el->el_chared.c_redo.buf = newbuffer; + + if (!hist_enlargebuf(el, sz, newsz)) + return 0; + + /* Safe to set enlarged buffer size */ + el->el_line.limit = &el->el_line.buffer[newsz - EL_LEAVE]; + if (el->el_chared.c_resizefun) + (*el->el_chared.c_resizefun)(el, el->el_chared.c_resizearg); + return 1; +} + +/* ch_end(): + * Free the data structures used by the editor + */ +libedit_private void +ch_end(EditLine *el) +{ + el_free(el->el_line.buffer); + el->el_line.buffer = NULL; + el->el_line.limit = NULL; + el_free(el->el_chared.c_undo.buf); + el->el_chared.c_undo.buf = NULL; + el_free(el->el_chared.c_redo.buf); + el->el_chared.c_redo.buf = NULL; + el->el_chared.c_redo.pos = NULL; + el->el_chared.c_redo.lim = NULL; + el->el_chared.c_redo.cmd = ED_UNASSIGNED; + el_free(el->el_chared.c_kill.buf); + el->el_chared.c_kill.buf = NULL; + ch_reset(el); +} + + +/* el_insertstr(): + * Insert string at cursorI + */ +int +el_winsertstr(EditLine *el, const wchar_t *s) +{ + size_t len; + + if (s == NULL || (len = wcslen(s)) == 0) + return -1; + if (el->el_line.lastchar + len >= el->el_line.limit) { + if (!ch_enlargebufs(el, len)) + return -1; + } + + c_insert(el, (int)len); + while (*s) + *el->el_line.cursor++ = *s++; + return 0; +} + + +/* el_deletestr(): + * Delete num characters before the cursor + */ +void +el_deletestr(EditLine *el, int n) +{ + if (n <= 0) + return; + + if (el->el_line.cursor < &el->el_line.buffer[n]) + return; + + c_delbefore(el, n); /* delete before dot */ + el->el_line.cursor -= n; + if (el->el_line.cursor < el->el_line.buffer) + el->el_line.cursor = el->el_line.buffer; +} + +/* el_cursor(): + * Move the cursor to the left or the right of the current position + */ +int +el_cursor(EditLine *el, int n) +{ + if (n == 0) + goto out; + + el->el_line.cursor += n; + + if (el->el_line.cursor < el->el_line.buffer) + el->el_line.cursor = el->el_line.buffer; + if (el->el_line.cursor > el->el_line.lastchar) + el->el_line.cursor = el->el_line.lastchar; +out: + return (int)(el->el_line.cursor - el->el_line.buffer); +} + +/* c_gets(): + * Get a string + */ +libedit_private int +c_gets(EditLine *el, wchar_t *buf, const wchar_t *prompt) +{ + ssize_t len; + wchar_t *cp = el->el_line.buffer, ch; + + if (prompt) { + len = (ssize_t)wcslen(prompt); + (void)memcpy(cp, prompt, (size_t)len * sizeof(*cp)); + cp += len; + } + len = 0; + + for (;;) { + el->el_line.cursor = cp; + *cp = ' '; + el->el_line.lastchar = cp + 1; + re_refresh(el); + + if (el_wgetc(el, &ch) != 1) { + ed_end_of_file(el, 0); + len = -1; + break; + } + + switch (ch) { + + case L'\b': /* Delete and backspace */ + case 0177: + if (len == 0) { + len = -1; + break; + } + len--; + cp--; + continue; + + case 0033: /* ESC */ + case L'\r': /* Newline */ + case L'\n': + buf[len] = ch; + break; + + default: + if (len >= (ssize_t)(EL_BUFSIZ - 16)) + terminal_beep(el); + else { + buf[len++] = ch; + *cp++ = ch; + } + continue; + } + break; + } + + el->el_line.buffer[0] = '\0'; + el->el_line.lastchar = el->el_line.buffer; + el->el_line.cursor = el->el_line.buffer; + return (int)len; +} + + +/* c_hpos(): + * Return the current horizontal position of the cursor + */ +libedit_private int +c_hpos(EditLine *el) +{ + wchar_t *ptr; + + /* + * Find how many characters till the beginning of this line. + */ + if (el->el_line.cursor == el->el_line.buffer) + return 0; + else { + for (ptr = el->el_line.cursor - 1; + ptr >= el->el_line.buffer && *ptr != '\n'; + ptr--) + continue; + return (int)(el->el_line.cursor - ptr - 1); + } +} + +libedit_private int +ch_resizefun(EditLine *el, el_zfunc_t f, void *a) +{ + el->el_chared.c_resizefun = f; + el->el_chared.c_resizearg = a; + return 0; +} + +libedit_private int +ch_aliasfun(EditLine *el, el_afunc_t f, void *a) +{ + el->el_chared.c_aliasfun = f; + el->el_chared.c_aliasarg = a; + return 0; +} Copied: vendor/NetBSD/libedit/2017-09-05/chared.h (from r323545, vendor/NetBSD/libedit/dist/chared.h) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/NetBSD/libedit/2017-09-05/chared.h Wed Sep 13 16:05:26 2017 (r323546, copy of r323545, vendor/NetBSD/libedit/dist/chared.h) @@ -0,0 +1,155 @@ +/* $NetBSD: chared.h,v 1.30 2016/05/22 19:44:26 christos Exp $ */ + +/*- + * Copyright (c) 1992, 1993 + * The Regents of the University of California. All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * Christos Zoulas of Cornell University. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)chared.h 8.1 (Berkeley) 6/4/93 + */ + +/* + * el.chared.h: Character editor interface + */ +#ifndef _h_el_chared +#define _h_el_chared + +/* + * This is an issue of basic "vi" look-and-feel. Defining VI_MOVE works + * like real vi: i.e. the transition from command<->insert modes moves + * the cursor. + * + * On the other hand we really don't want to move the cursor, because + * all the editing commands don't include the character under the cursor. + * Probably the best fix is to make all the editing commands aware of + * this fact. + */ +#define VI_MOVE + +/* + * Undo information for vi - no undo in emacs (yet) + */ +typedef struct c_undo_t { + ssize_t len; /* length of saved line */ + int cursor; /* position of saved cursor */ + wchar_t *buf; /* full saved text */ +} c_undo_t; + +/* redo for vi */ +typedef struct c_redo_t { + wchar_t *buf; /* redo insert key sequence */ + wchar_t *pos; + wchar_t *lim; + el_action_t cmd; /* command to redo */ + wchar_t ch; /* char that invoked it */ + int count; + int action; /* from cv_action() */ +} c_redo_t; + +/* + * Current action information for vi + */ +typedef struct c_vcmd_t { + int action; + wchar_t *pos; +} c_vcmd_t; + +/* *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***