Date: Wed, 13 Sep 2017 10:45:49 +0000 (UTC) From: Andriy Gapon <avg@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r323530 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua vendor-sys/illumo... Message-ID: <201709131045.v8DAjnv0063855@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: avg Date: Wed Sep 13 10:45:49 2017 New Revision: 323530 URL: https://svnweb.freebsd.org/changeset/base/323530 Log: 7431 ZFS Channel Programs illumos/illumos-gate@dfc115332c94a2f62058ac7f2bce7631fbd20b3d https://github.com/illumos/illumos-gate/commit/dfc115332c94a2f62058ac7f2bce7631fbd20b3d https://www.illumos.org/issues/7431 ZFS channel programs (ZCP) adds support for performing compound ZFS administrative actions via Lua scripts in a sandboxed environment (with time and memory limits). This initial commit includes both base support for running ZCP scripts, and a small initial library of API calls which support getting properties and listing, destroying, and promoting datasets. Testing: in addition to the included unit tests, channel programs have been in use at Delphix for several months for batch destroying filesystems. The dsl_destroy_snaps_nvl() call has also been replaced with For reference, the new zfs-program manpage is included below. ZFS-PROGRAM(1M) 1M ZFS-PROGRAM(1M) NAME zfs program – executes ZFS channel programs SYNOPSIS zfs program [-t timeout] [-m memory-limit] pool script DESCRIPTION The ZFS channel program interface allows ZFS administrative operations to be run programmatically as a Lua script. The entire script is executed atomically, with no other administrative operations taking effect concurrently. A library of ZFS calls is made available to channel program scripts. Channel programs may only be run with root privileges. A modified version of the Lua 5.2 interpreter is used to run channel program scripts. The Lua 5.2 manual can be found at: http://www.lua.org/manual/5.2/ ... Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: John Kennedy <john.kennedy@delphix.com> Reviewed by: Dan Kimmel <dan.kimmel@delphix.com> Approved by: Garrett D'Amore <garrett@damore.org> Author: Chris Williamson <chris.williamson@delphix.com> Added: vendor/illumos/dist/man/man1m/zfs-program.1m Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/lib/libzpool/common/kernel.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h vendor/illumos/dist/man/man1m/zfs.1m Changes in other areas also in this revision: Added: vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ vendor-sys/illumos/dist/uts/common/fs/zfs/lua/README.zfs vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lapi.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lapi.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lauxlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lauxlib.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lbaselib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lbitlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcode.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcode.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcompat.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lcorolib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lctype.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lctype.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldebug.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldebug.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldo.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldo.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ldump.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lfunc.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lfunc.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lgc.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llex.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/llimits.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lmem.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lmem.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lobject.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lobject.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lopcodes.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lopcodes.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lparser.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lparser.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstate.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstate.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstring.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstring.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lstrlib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltable.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltablib.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltm.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/ltm.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lua.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/luaconf.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lualib.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lundump.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lundump.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lvm.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lvm.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lzio.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/lua/lzio.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_global.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_iter.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zcp_prop.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_get.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_global.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_iter.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/zcp_synctask.c (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dataset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_destroy.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dataset.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_destroy.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_dir.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_vfsops.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c ============================================================================== --- vendor/illumos/dist/cmd/zfs/zfs_main.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/cmd/zfs/zfs_main.c Wed Sep 13 10:45:49 2017 (r323530) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. * Copyright 2012 Milan Jurik. All rights reserved. * Copyright (c) 2012, Joyent, Inc. All rights reserved. * Copyright (c) 2011-2012 Pawel Jakub Dawidek. All rights reserved. @@ -50,6 +50,7 @@ #include <grp.h> #include <pwd.h> #include <signal.h> +#include <sys/debug.h> #include <sys/list.h> #include <sys/mkdev.h> #include <sys/mntent.h> @@ -106,6 +107,7 @@ static int zfs_do_holds(int argc, char **argv); static int zfs_do_release(int argc, char **argv); static int zfs_do_diff(int argc, char **argv); static int zfs_do_bookmark(int argc, char **argv); +static int zfs_do_channel_program(int argc, char **argv); /* * Enable a reasonable set of defaults for libumem debugging on DEBUG builds. @@ -153,6 +155,7 @@ typedef enum { HELP_RELEASE, HELP_DIFF, HELP_BOOKMARK, + HELP_CHANNEL_PROGRAM, } zfs_help_t; typedef struct zfs_command { @@ -180,6 +183,7 @@ static zfs_command_t command_table[] = { { "promote", zfs_do_promote, HELP_PROMOTE }, { "rename", zfs_do_rename, HELP_RENAME }, { "bookmark", zfs_do_bookmark, HELP_BOOKMARK }, + { "program", zfs_do_channel_program, HELP_CHANNEL_PROGRAM }, { NULL }, { "list", zfs_do_list, HELP_LIST }, { NULL }, @@ -324,6 +328,10 @@ get_usage(zfs_help_t idx) "[snapshot|filesystem]\n")); case HELP_BOOKMARK: return (gettext("\tbookmark <snapshot> <bookmark>\n")); + case HELP_CHANNEL_PROGRAM: + return (gettext("\tprogram [-t <instruction limit>] " + "[-m <memory limit (b)>] <pool> <program file> " + "[lua args...]\n")); } abort(); @@ -352,6 +360,18 @@ safe_malloc(size_t size) return (data); } +void * +safe_realloc(void *data, size_t size) +{ + void *newp; + if ((newp = realloc(data, size)) == NULL) { + free(data); + nomem(); + } + + return (newp); +} + static char * safe_strdup(char *str) { @@ -6945,6 +6965,190 @@ zfs_do_bookmark(int argc, char **argv) dgettext(TEXT_DOMAIN, err_msg)); } + return (ret != 0); + +usage: + usage(B_FALSE); + return (-1); +} + +static int +zfs_do_channel_program(int argc, char **argv) +{ + int ret, fd; + char c; + char *progbuf, *filename, *poolname; + size_t progsize, progread; + nvlist_t *outnvl; + uint64_t instrlimit = ZCP_DEFAULT_INSTRLIMIT; + uint64_t memlimit = ZCP_DEFAULT_MEMLIMIT; + zpool_handle_t *zhp; + + /* check options */ + while (-1 != + (c = getopt(argc, argv, "t:(instr-limit)m:(memory-limit)"))) { + switch (c) { + case 't': + case 'm': { + uint64_t arg; + char *endp; + + errno = 0; + arg = strtoull(optarg, &endp, 0); + if (errno != 0 || *endp != '\0') { + (void) fprintf(stderr, gettext( + "invalid argument " + "'%s': expected integer\n"), optarg); + goto usage; + } + + if (c == 't') { + if (arg > ZCP_MAX_INSTRLIMIT || arg == 0) { + (void) fprintf(stderr, gettext( + "Invalid instruction limit: " + "%s\n"), optarg); + return (1); + } else { + instrlimit = arg; + } + } else { + ASSERT3U(c, ==, 'm'); + if (arg > ZCP_MAX_MEMLIMIT || arg == 0) { + (void) fprintf(stderr, gettext( + "Invalid memory limit: " + "%s\n"), optarg); + return (1); + } else { + memlimit = arg; + } + } + break; + } + case '?': + (void) fprintf(stderr, gettext("invalid option '%c'\n"), + optopt); + goto usage; + } + } + + argc -= optind; + argv += optind; + + if (argc < 2) { + (void) fprintf(stderr, + gettext("invalid number of arguments\n")); + goto usage; + } + + poolname = argv[0]; + filename = argv[1]; + if (strcmp(filename, "-") == 0) { + fd = 0; + filename = "standard input"; + } else if ((fd = open(filename, O_RDONLY)) < 0) { + (void) fprintf(stderr, gettext("cannot open '%s': %s\n"), + filename, strerror(errno)); + return (1); + } + + if ((zhp = zpool_open(g_zfs, poolname)) == NULL) { + (void) fprintf(stderr, gettext("cannot open pool '%s'"), + poolname); + return (1); + } + zpool_close(zhp); + + /* + * Read in the channel program, expanding the program buffer as + * necessary. + */ + progread = 0; + progsize = 1024; + progbuf = safe_malloc(progsize); + do { + ret = read(fd, progbuf + progread, progsize - progread); + progread += ret; + if (progread == progsize && ret > 0) { + progsize *= 2; + progbuf = safe_realloc(progbuf, progsize); + } + } while (ret > 0); + + if (fd != 0) + (void) close(fd); + if (ret < 0) { + free(progbuf); + (void) fprintf(stderr, + gettext("cannot read '%s': %s\n"), + filename, strerror(errno)); + return (1); + } + progbuf[progread] = '\0'; + + /* + * Any remaining arguments are passed as arguments to the lua script as + * a string array: + * { + * "argv" -> [ "arg 1", ... "arg n" ], + * } + */ + nvlist_t *argnvl = fnvlist_alloc(); + fnvlist_add_string_array(argnvl, ZCP_ARG_CLIARGV, argv + 2, argc - 2); + + ret = lzc_channel_program(poolname, progbuf, instrlimit, memlimit, + argnvl, &outnvl); + + if (ret != 0) { + /* + * On error, report the error message handed back by lua if one + * exists. Otherwise, generate an appropriate error message, + * falling back on strerror() for an unexpected return code. + */ + char *errstring = NULL; + if (nvlist_exists(outnvl, ZCP_RET_ERROR)) { + (void) nvlist_lookup_string(outnvl, + ZCP_RET_ERROR, &errstring); + if (errstring == NULL) + errstring = strerror(ret); + } else { + switch (ret) { + case EINVAL: + errstring = + "Invalid instruction or memory limit."; + break; + case ENOMEM: + errstring = "Return value too large."; + break; + case ENOSPC: + errstring = "Memory limit exhausted."; + break; + case ETIME: + errstring = "Timed out."; + break; + case EPERM: + errstring = "Permission denied. Channel " + "programs must be run as root."; + break; + default: + errstring = strerror(ret); + } + } + (void) fprintf(stderr, + gettext("Channel program execution failed:\n%s\n"), + errstring); + } else { + (void) printf("Channel program fully executed "); + if (nvlist_empty(outnvl)) { + (void) printf("with no return value.\n"); + } else { + (void) printf("with return value:\n"); + dump_nvlist(outnvl, 4); + } + } + + free(progbuf); + fnvlist_free(outnvl); + fnvlist_free(argnvl); return (ret != 0); usage: Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Wed Sep 13 10:45:49 2017 (r323530) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. * Copyright (c) 2012 by Frederik Wessels. All rights reserved. * Copyright (c) 2013 by Prasad Joshi (sTec). All rights reserved. * Copyright 2016 Igor Kozhukhov <ikozhukhov@gmail.com>. @@ -5159,6 +5159,11 @@ get_history_one(zpool_handle_t *zhp, void *data) (void) printf(" output:\n"); dump_nvlist(fnvlist_lookup_nvlist(rec, ZPOOL_HIST_OUTPUT_NVL), 8); + } + if (nvlist_exists(rec, ZPOOL_HIST_ERRNO)) { + (void) printf(" errno: %lld\n", + fnvlist_lookup_int64(rec, + ZPOOL_HIST_ERRNO)); } } else { if (!cb->internal) Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Sep 13 10:45:49 2017 (r323530) @@ -2323,6 +2323,74 @@ zfs_get_clones_nvl(zfs_handle_t *zhp) } /* + * Accepts a property and value and checks that the value + * matches the one found by the channel program. If they are + * not equal, print both of them. + */ +void +zcp_check(zfs_handle_t *zhp, zfs_prop_t prop, uint64_t intval, + const char *strval) +{ + if (!zhp->zfs_hdl->libzfs_prop_debug) + return; + int error; + char *poolname = zhp->zpool_hdl->zpool_name; + const char *program = + "args = ...\n" + "ds = args['dataset']\n" + "prop = args['property']\n" + "value, setpoint = zfs.get_prop(ds, prop)\n" + "return {value=value, setpoint=setpoint}\n"; + nvlist_t *outnvl; + nvlist_t *retnvl; + nvlist_t *argnvl = fnvlist_alloc(); + + fnvlist_add_string(argnvl, "dataset", zhp->zfs_name); + fnvlist_add_string(argnvl, "property", zfs_prop_to_name(prop)); + + error = lzc_channel_program(poolname, program, + 10 * 1000 * 1000, 10 * 1024 * 1024, argnvl, &outnvl); + + if (error == 0) { + retnvl = fnvlist_lookup_nvlist(outnvl, "return"); + if (zfs_prop_get_type(prop) == PROP_TYPE_NUMBER) { + int64_t ans; + error = nvlist_lookup_int64(retnvl, "value", &ans); + if (error != 0) { + (void) fprintf(stderr, "zcp check error: %u\n", + error); + return; + } + if (ans != intval) { + (void) fprintf(stderr, + "%s: zfs found %lld, but zcp found %lld\n", + zfs_prop_to_name(prop), + (longlong_t)intval, (longlong_t)ans); + } + } else { + char *str_ans; + error = nvlist_lookup_string(retnvl, "value", &str_ans); + if (error != 0) { + (void) fprintf(stderr, "zcp check error: %u\n", + error); + return; + } + if (strcmp(strval, str_ans) != 0) { + (void) fprintf(stderr, + "%s: zfs found %s, but zcp found %s\n", + zfs_prop_to_name(prop), + strval, str_ans); + } + } + } else { + (void) fprintf(stderr, + "zcp check failed, channel program error: %u\n", error); + } + nvlist_free(argnvl); + nvlist_free(outnvl); +} + +/* * Retrieve a property from the given object. If 'literal' is specified, then * numbers are left as exact values. Otherwise, numbers are converted to a * human-readable form. @@ -2368,6 +2436,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char &t) == 0) (void) snprintf(propbuf, proplen, "%llu", val); } + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_MOUNTPOINT: @@ -2436,7 +2505,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char /* 'legacy' or 'none' */ (void) strlcpy(propbuf, str, proplen); } - + zcp_check(zhp, prop, NULL, propbuf); break; case ZFS_PROP_ORIGIN: @@ -2444,6 +2513,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char if (str == NULL) return (-1); (void) strlcpy(propbuf, str, proplen); + zcp_check(zhp, prop, NULL, str); break; case ZFS_PROP_CLONES: @@ -2458,7 +2528,6 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char if (get_numeric_property(zhp, prop, src, &source, &val) != 0) return (-1); - /* * If quota or reservation is 0, we translate this into 'none' * (unless literal is set), and indicate that it's the default @@ -2477,6 +2546,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char else zfs_nicenum(val, propbuf, proplen); } + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_FILESYSTEM_LIMIT: @@ -2501,6 +2571,8 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char } else { zfs_nicenum(val, propbuf, proplen); } + + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_REFRATIO: @@ -2510,6 +2582,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char (void) snprintf(propbuf, proplen, "%llu.%02llux", (u_longlong_t)(val / 100), (u_longlong_t)(val % 100)); + zcp_check(zhp, prop, val, NULL); break; case ZFS_PROP_TYPE: @@ -2530,6 +2603,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char abort(); } (void) snprintf(propbuf, proplen, "%s", str); + zcp_check(zhp, prop, NULL, propbuf); break; case ZFS_PROP_MOUNTED: @@ -2555,6 +2629,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char * consumers. */ (void) strlcpy(propbuf, zhp->zfs_name, proplen); + zcp_check(zhp, prop, NULL, propbuf); break; case ZFS_PROP_MLSLABEL: @@ -2604,26 +2679,33 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char if (get_numeric_property(zhp, prop, src, &source, &val) != 0) return (-1); (void) snprintf(propbuf, proplen, "%llu", (u_longlong_t)val); + zcp_check(zhp, prop, val, NULL); break; default: switch (zfs_prop_get_type(prop)) { case PROP_TYPE_NUMBER: if (get_numeric_property(zhp, prop, src, - &source, &val) != 0) + &source, &val) != 0) { return (-1); - if (literal) + } + + if (literal) { (void) snprintf(propbuf, proplen, "%llu", (u_longlong_t)val); - else + } else { zfs_nicenum(val, propbuf, proplen); + } + zcp_check(zhp, prop, val, NULL); break; case PROP_TYPE_STRING: str = getprop_string(zhp, prop, &source); if (str == NULL) return (-1); + (void) strlcpy(propbuf, str, proplen); + zcp_check(zhp, prop, NULL, str); break; case PROP_TYPE_INDEX: @@ -2632,7 +2714,9 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char return (-1); if (zfs_prop_index_to_string(prop, val, &strval) != 0) return (-1); + (void) strlcpy(propbuf, strval, proplen); + zcp_check(zhp, prop, NULL, strval); break; default: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h Wed Sep 13 10:45:49 2017 (r323530) @@ -79,6 +79,7 @@ struct libzfs_handle { libzfs_fru_t **libzfs_fru_hash; libzfs_fru_t *libzfs_fru_list; char libzfs_chassis_id[256]; + boolean_t libzfs_prop_debug; }; struct zfs_handle { Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_util.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Wed Sep 13 10:45:49 2017 (r323530) @@ -651,6 +651,10 @@ libzfs_init(void) zpool_feature_init(); libzfs_mnttab_init(hdl); + if (getenv("ZFS_PROP_DEBUG") != NULL) { + hdl->libzfs_prop_debug = B_TRUE; + } + return (hdl); } Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Wed Sep 13 10:45:49 2017 (r323530) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 RackTop Systems. @@ -152,7 +152,15 @@ lzc_ioctl(zfs_ioc_t ioc, const char *name, } while (ioctl(g_fd, ioc, &zc) != 0) { - if (errno == ENOMEM && resultp != NULL) { + /* + * If ioctl exited with ENOMEM, we retry the ioctl after + * increasing the size of the destination nvlist. + * + * Channel programs that exit with ENOMEM probably ran over the + * lua memory sandbox; they should not be retried. + */ + if (errno == ENOMEM && resultp != NULL && + ioc != ZFS_IOC_CHANNEL_PROGRAM) { free((void *)(uintptr_t)zc.zc_nvlist_dst); zc.zc_nvlist_dst_size *= 2; zc.zc_nvlist_dst = (uint64_t)(uintptr_t) @@ -864,6 +872,57 @@ lzc_destroy_bookmarks(nvlist_t *bmarks, nvlist_t **err pool[strcspn(pool, "/#")] = '\0'; error = lzc_ioctl(ZFS_IOC_DESTROY_BOOKMARKS, pool, bmarks, errlist); + + return (error); +} + +/* + * Executes a channel program. + * + * If this function returns 0 the channel program was successfully loaded and + * ran without failing. Note that individual commands the channel program ran + * may have failed and the channel program is responsible for reporting such + * errors through outnvl if they are important. + * + * This method may also return: + * + * EINVAL The program contains syntax errors, or an invalid memory or time + * limit was given. No part of the channel program was executed. + * If caused by syntax errors, 'outnvl' contains information about the + * errors. + * + * ECHRNG The program was executed, but encountered a runtime error, such as + * calling a function with incorrect arguments, invoking the error() + * function directly, failing an assert() command, etc. Some portion + * of the channel program may have executed and committed changes. + * Information about the failure can be found in 'outnvl'. + * + * ENOMEM The program fully executed, but the output buffer was not large + * enough to store the returned value. No output is returned through + * 'outnvl'. + * + * ENOSPC The program was terminated because it exceeded its memory usage + * limit. Some portion of the channel program may have executed and + * committed changes to disk. No output is returned through 'outnvl'. + * + * ETIME The program was terminated because it exceeded its Lua instruction + * limit. Some portion of the channel program may have executed and + * committed changes to disk. No output is returned through 'outnvl'. + */ +int +lzc_channel_program(const char *pool, const char *program, uint64_t instrlimit, + uint64_t memlimit, nvlist_t *argnvl, nvlist_t **outnvl) +{ + int error; + nvlist_t *args; + + args = fnvlist_alloc(); + fnvlist_add_string(args, ZCP_ARG_PROGRAM, program); + fnvlist_add_nvlist(args, ZCP_ARG_ARGLIST, argnvl); + fnvlist_add_uint64(args, ZCP_ARG_INSTRLIMIT, instrlimit); + fnvlist_add_uint64(args, ZCP_ARG_MEMLIMIT, memlimit); + error = lzc_ioctl(ZFS_IOC_CHANNEL_PROGRAM, pool, args, outnvl); + fnvlist_free(args); return (error); } Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Wed Sep 13 10:45:49 2017 (r323530) @@ -20,7 +20,7 @@ */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 RackTop Systems. */ @@ -85,6 +85,9 @@ boolean_t lzc_exists(const char *); int lzc_rollback(const char *, char *, int); int lzc_rollback_to(const char *, const char *); + +int lzc_channel_program(const char *, const char *, uint64_t, uint64_t, + nvlist_t *, nvlist_t **); #ifdef __cplusplus } Modified: vendor/illumos/dist/lib/libzpool/common/kernel.c ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzpool/common/kernel.c Wed Sep 13 10:45:49 2017 (r323530) @@ -702,6 +702,7 @@ vpanic(const char *fmt, va_list adx) char buf[512]; (void) vsnprintf(buf, 512, fmt, adx); assfail(buf, NULL, 0); + abort(); /* necessary to make vpanic meet noreturn requirements */ } void Modified: vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h ============================================================================== --- vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Wed Sep 13 10:41:47 2017 (r323529) +++ vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Wed Sep 13 10:45:49 2017 (r323530) @@ -62,6 +62,7 @@ extern "C" { #include <time.h> #include <procfs.h> #include <pthread.h> +#include <setjmp.h> #include <sys/debug.h> #include <libsysevent.h> #include <sys/note.h> @@ -107,8 +108,8 @@ extern void dprintf_setup(int *argc, char **argv); extern void cmn_err(int, const char *, ...); extern void vcmn_err(int, const char *, __va_list); -extern void panic(const char *, ...); -extern void vpanic(const char *, __va_list); +extern void panic(const char *, ...) __NORETURN; +extern void vpanic(const char *, __va_list) __NORETURN; #define fm_panic panic @@ -317,6 +318,7 @@ extern void kstat_runq_back_to_waitq(kstat_io_t *); #define KM_SLEEP UMEM_NOFAIL #define KM_PUSHPAGE KM_SLEEP #define KM_NOSLEEP UMEM_DEFAULT +#define KM_NORMALPRI 0 /* not needed with UMEM_DEFAULT */ #define KMC_NODEBUG UMC_NODEBUG #define KMC_NOTOUCH 0 /* not needed for userland caches */ #define kmem_alloc(_s, _f) umem_alloc(_s, _f) Added: vendor/illumos/dist/man/man1m/zfs-program.1m ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/illumos/dist/man/man1m/zfs-program.1m Wed Sep 13 10:45:49 2017 (r323530) @@ -0,0 +1,499 @@ +.\" This file and its contents are supplied under the terms of the +.\" Common Development and Distribution License ("CDDL"), version 1.0. +.\" You may only use this file in accordance with the terms of version +.\" 1.0 of the CDDL. +.\" +.\" A full copy of the text of the CDDL should have accompanied this +.\" source. A copy of the CDDL is also available via the Internet at +.\" http://www.illumos.org/license/CDDL. +.\" +.\" +.\" Copyright (c) 2016 by Delphix. All Rights Reserved. +.\" +.Dd January 21, 2016 +.Dt ZFS-PROGRAM 1M +.Os +.Sh NAME +.Nm zfs program +.Nd executes ZFS channel programs +.Sh SYNOPSIS +.Cm zfs program +.Op Fl t Ar instruction-limit +.Op Fl m Ar memory-limit +.Ar pool +.Ar script +.\".Op Ar optional arguments to channel program +.Sh DESCRIPTION +The ZFS channel program interface allows ZFS administrative operations to be +run programmatically as a Lua script. +The entire script is executed atomically, with no other administrative +operations taking effect concurrently. +A library of ZFS calls is made available to channel program scripts. +Channel programs may only be run with root privileges. +.Pp +A modified version of the Lua 5.2 interpreter is used to run channel program +scripts. +The Lua 5.2 manual can be found at: +.Bd -centered -offset indent +.Lk http://www.lua.org/manual/5.2/ +.Ed +.Pp +The channel program given by +.Ar script +will be run on +.Ar pool , +and any attempts to access or modify other pools will cause an error. +.Sh OPTIONS +.Bl -tag -width "-t" +.It Fl t Ar instruction-limit +Execution time limit, in number of Lua instructions to execute. +If a channel program executes more than the specified number of instructions, +it will be stopped and an error will be returned. +The default limit is 10 million instructions, and it can be set to a maximum of +100 million instructions. +.It Fl m Ar memory-limit +Memory limit, in bytes. +If a channel program attempts to allocate more memory than the given limit, it +will be stopped and an error returned. +The default memory limit is 10 MB, and can be set to a maximum of 100 MB. +.El +.Pp +All remaining argument strings will be passed directly to the Lua script as +described in the +.Sx LUA INTERFACE +section below. +.Sh LUA INTERFACE +A channel program can be invoked either from the command line, or via a library +call to +.Fn lzc_channel_program . +.Ss Arguments +Arguments passed to the channel program are converted to a Lua table. +If invoked from the command line, extra arguments to the Lua script will be +accessible as an array stored in the argument table with the key 'argv': +.Bd -literal -offset indent +args = ... +argv = args["argv"] +-- argv == {1="arg1", 2="arg2", ...} +.Ed +.Pp +If invoked from the libZFS interface, an arbitrary argument list can be +passed to the channel program, which is accessible via the same +"..." syntax in Lua: +.Bd -literal -offset indent +args = ... +-- args == {"foo"="bar", "baz"={...}, ...} +.Ed +.Pp +Note that because Lua arrays are 1-indexed, arrays passed to Lua from the +libZFS interface will have their indices incremented by 1. +That is, the element +in +.Va arr[0] +in a C array passed to a channel program will be stored in +.Va arr[1] +when accessed from Lua. +.Ss Return Values +Lua return statements take the form: +.Bd -literal -offset indent +return ret0, ret1, ret2, ... +.Ed +.Pp +Return statements returning multiple values are permitted internally in a +channel program script, but attempting to return more than one value from the +top level of the channel program is not permitted and will throw an error. +However, tables containing multiple values can still be returned. +If invoked from the command line, a return statement: +.Bd -literal -offset indent +a = {foo="bar", baz=2} +return a +.Ed +.Pp +Will be output formatted as: +.Bd -literal -offset indent +Channel program fully executed with return value: + return: + baz: 2 + foo: 'bar' +.Ed +.Ss Fatal Errors +If the channel program encounters a fatal error while running, a non-zero exit +status will be returned. +If more information about the error is available, a singleton list will be +returned detailing the error: +.Bd -literal -offset indent +error: "error string, including Lua stack trace" +.Ed +.Pp +If a fatal error is returned, the channel program may have not executed at all, +may have partially executed, or may have fully executed but failed to pass a +return value back to userland. +.Pp +If the channel program exhausts an instruction or memory limit, a fatal error +will be generated and the program will be stopped, leaving the program partially +executed. +No attempt is made to reverse or undo any operations already performed. +Note that because both the instruction count and amount of memory used by a +channel program are deterministic when run against the same inputs and +filesystem state, as long as a channel program has run successfully once, you +can guarantee that it will finish successfully against a similar size system. +.Pp +If a channel program attempts to return too large a value, the program will +fully execute but exit with a nonzero status code and no return value. +.Pp +.Em Note: +ZFS API functions do not generate Fatal Errors when correctly invoked, they +return an error code and the channel program continues executing. +See the +.Sx ZFS API +section below for function-specific details on error return codes. +.Ss Lua to C Value Conversion +When invoking a channel program via the libZFS interface, it is necessary to +translate arguments and return values from Lua values to their C equivalents, +and vice-versa. +.Pp +There is a correspondence between nvlist values in C and Lua tables. +A Lua table which is returned from the channel program will be recursively +converted to an nvlist, with table values converted to their natural +equivalents: +.Bd -literal -offset indent +string -> string +number -> int64 +boolean -> boolean_value +nil -> boolean (no value) +table -> nvlist +.Ed +.Pp +Likewise, table keys are replaced by string equivalents as follows: +.Bd -literal -offset indent +string -> no change +number -> signed decimal string ("%lld") +boolean -> "true" | "false" +.Ed +.Pp +Any collision of table key strings (for example, the string "true" and a +true boolean value) will cause a fatal error. +.Pp +Lua numbers are represented internally as signed 64-bit integers. +.Sh LUA STANDARD LIBRARY +The following Lua built-in base library functions are available: +.Bd -literal -offset indent +assert rawlen +collectgarbage rawget +error rawset +getmetatable select +ipairs setmetatable +next tonumber +pairs tostring +rawequal type +.Ed +.Pp +All functions in the +.Em coroutine , +.Em string , +and +.Em table +built-in submodules are also available. +A complete list and documentation of these modules is available in the Lua +manual. +.Pp +The following functions base library functions have been disabled and are +not available for use in channel programs: +.Bd -literal -offset indent +dofile +loadfile +load +pcall +print +xpcall +.Ed +.Sh ZFS API +.Ss Function Arguments +Each API function takes a fixed set of required positional arguments and +optional keyword arguments. +For example, the destroy function takes a single positional string argument +(the name of the dataset to destroy) and an optional "defer" keyword boolean +argument. +When using parentheses to specify the arguments to a Lua function, only +positional arguments can be used: +.Bd -literal -offset indent +zfs.sync.destroy("rpool@snap") +.Ed +.Pp +To use keyword arguments, functions must be called with a single argument that +is a Lua table containing entries mapping integers to positional arguments and +strings to keyword arguments: +.Bd -literal -offset indent +zfs.sync.destroy({1="rpool@snap", defer=true}) +.Ed +.Pp +The Lua language allows curly braces to be used in place of parenthesis as +syntactic sugar for this calling convention: +.Bd -literal -offset indent +zfs.sync.snapshot{"rpool@snap", defer=true} +.Ed +.Ss Function Return Values +If an API function succeeds, it returns 0. +If it fails, it returns an error code and the channel program continues +executing. +API functions do not generate Fatal Errors except in the case of an +unrecoverable internal file system error. +.Pp +In addition to returning an error code, some functions also return extra +details describing what caused the error. +This extra description is given as a second return value, and will always be a +Lua table, or Nil if no error details were returned. +Different keys will exist in the error details table depending on the function +and error case. +Any such function may be called expecting a single return value: +.Bd -literal -offset indent +errno = zfs.sync.promote(dataset) +.Ed +.Pp +Or, the error details can be retrieved: +.Bd -literal -offset indent +errno, details = zfs.sync.promote(dataset) +if (errno == EEXIST) then + assert(details ~= Nil) + list_of_conflicting_snapshots = details +end +.Ed +.Pp +The following global aliases for API function error return codes are defined +for use in channel programs: +.Bd -literal -offset indent +EPERM ECHILD ENODEV ENOSPC +ENOENT EAGAIN ENOTDIR ESPIPE +ESRCH ENOMEM EISDIR EROFS +EINTR EACCES EINVAL EMLINK +EIO EFAULT ENFILE EPIPE +ENXIO ENOTBLK EMFILE EDOM +E2BIG EBUSY ENOTTY ERANGE +ENOEXEC EEXIST ETXTBSY EDQUOT +EBADF EXDEV EFBIG +.Ed +.Ss API Functions +For detailed descriptions of the exact behavior of any zfs administrative +operations, see the main +.Xr zfs 1 +manual page. +.Bl -tag -width "xx" +.It Em zfs.debug(msg) +Record a debug message in the zfs_dbgmsg log. +A log of these messages can be printed via mdb's "::zfs_dbgmsg" command, or +can be monitored live by running: +.Bd -literal -offset indent + dtrace -n 'zfs-dbgmsg{trace(stringof(arg0))}' +.Ed +.Pp +msg (string) +.Bd -ragged -compact -offset "xxxx" +Debug message to be printed. +.Ed +.It Em zfs.get_prop(dataset, property) +Returns two values. +First, a string, number or table containing the property value for the given +dataset. +Second, a string containing the source of the property (i.e. the name of the +dataset in which it was set or nil if it is readonly). +Throws a Lua error if the dataset is invalid or the property doesn't exist. +Note that Lua only supports int64 number types whereas ZFS number properties +are uint64. +This means very large values (like guid) may wrap around and appear negative. +.Pp +dataset (string) +.Bd -ragged -compact -offset "xxxx" +Filesystem or snapshot path to retrieve properties from. +.Ed +.Pp +property (string) +.Bd -ragged -compact -offset "xxxx" +Name of property to retrieve. +All filesystem, snapshot and volume properties are supported except +for 'mounted' and 'iscsioptions.' +Also supports the 'written@snap' and 'written#bookmark' properties and +the '<user|group><quota|used>@id' properties, though the id must be in numeric +form. +.Ed +.El +.Bl -tag -width "xx" +.It Sy zfs.sync submodule +The sync submodule contains functions that modify the on-disk state. +They are executed in "syncing context". +.Pp +The available sync submodule functions are as follows: +.Bl -tag -width "xx" +.It Em zfs.sync.destroy(dataset, [defer=true|false]) +Destroy the given dataset. +Returns 0 on successful destroy, or a nonzero error code if the dataset could +not be destroyed (for example, if the dataset has any active children or +clones). +.Pp +dataset (string) +.Bd -ragged -compact -offset "xxxx" +Filesystem or snapshot to be destroyed. +.Ed +.Pp +[optional] defer (boolean) +.Bd -ragged -compact -offset "xxxx" +Valid only for destroying snapshots. +If set to true, and the snapshot has holds or clones, allows the snapshot to be +marked for deferred deletion rather than failing. +.Ed +.It Em zfs.sync.promote(dataset) +Promote the given clone to a filesystem. *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201709131045.v8DAjnv0063855>