Date: Sun, 28 Dec 2014 23:31:35 -0600 From: "James R. Van Artsdalen" <james-freebsd-fs2@jrv.org> To: Steven Hartland <killing@multiplay.co.uk> Cc: "freebsd-fs@FreeBSD.ORG" <freebsd-fs@FreeBSD.ORG>, d@delphij.net Subject: Re: ZFS: FreeBSD 10.1 can't import/mount FreeBSD 9 pool Message-ID: <54A0E737.6000101@jrv.org> In-Reply-To: <549EC457.6010509@multiplay.co.uk> References: <549DF2B1.3030909@jrv.org> <549DF7EB.1080308@multiplay.co.uk> <549E18AB.8060708@jrv.org> <549E1C4F.7090400@multiplay.co.uk> <A9BE5EB9-1C79-47DE-8164-D6FE63EA622A@nanoimagingservices.com> <549EC457.6010509@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
It's not clear what upstream had in mind. This actually breaks pools that worked. Worse, it doesn't seem to have occurred to anyone that archival pools perform just fine at 99%+ capacity: I often went to 99.9%, leaving no more than 20 GB free. The performance analysis was incomplete. Moreover, the solution *already existed* - not a line of new code was needed. For myself I'm setting the tunable to 63 (effectively off) since my write pools already have a filesystem with these properties: # zfs get used,readonly,reservation BIGTEX/UNIX/var/empty NAME PROPERTY VALUE SOURCE BIGTEX/UNIX/var/empty used 628K - BIGTEX/UNIX/var/empty readonly on local BIGTEX/UNIX/var/empty reservation 25G local # PS. I consider the upstream change actually broken. My pool that did not import properly due to lack of free space was not exactly full: # zpool list SAS01 NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT SAS01 43.5T 42.6T 948G - 0% 97% 1.00x ONLINE - # PPS. If the tunable is changed be sure *every* pool is protected with a reservation since it really does happen that full pools can jam themselves into a permanent read-only state that can't be recovered. On 12/27/2014 8:38 AM, Steven Hartland wrote: > This was an upstream change so I couldn't comment on the details. > > The intention is clear, to prevent a pool getting into a state where > it has no space and from which it can't recover, but why so much of > the pool is reserved is unclear particularly for large pools e.g. on a > 8 TB pool 256GB is reserved. > > I can seen the benefit of making this configurable on a pool by pool > basis, or at least capping it to a reasonable value, as well as making > it backwards compatible so scenarios like this don't occur but there > may be implementation details which prevent this I'm not sure as I've > not looked into the details. > > If this is where your mind is too then I would suggest looking to > raise issue upstream. > > > On 27/12/2014 05:04, Craig Yoshioka wrote: >> I brought this up before, but I will again. This was not a great >> change. I also have archival drives which can now give me problems. >> Why was this reserved space not implemented as a user configurable FS >> option? >> >> Sent from my iPhone >> >>> On Dec 26, 2014, at 8:41 PM, Steven Hartland >>> <killing@multiplay.co.uk> wrote: >>> >>> It was introduced by: >>> https://svnweb.freebsd.org/base?view=revision&revision=268473 >>> >>> Tuning of it was added by: >>> https://svnweb.freebsd.org/base?view=revision&revision=274674 >>> >>> Hope this helps. >>> >>> Regards >>> Steve >>> >>>> On 27/12/2014 02:25, James R. Van Artsdalen wrote: >>>> Oops - this will break every single one of my archival pools. >>>> >>>> If there is no userland ability to enable backwards compatibility, can >>>> you tell me where it is in the source or about when it was added? >>>> >>>>> On 12/26/2014 6:06 PM, Steven Hartland wrote: >>>>> Later versions reserve space for deletions etc, so if your volume is >>>>> too full could fail in this manor. >>>>> >>>>> The fix would be to clear down space so this is no longer an issue. >>>>> >>>>>> On 26/12/2014 23:43, James R. Van Artsdalen wrote: >>>>>> FreeBSD bigtex.housenet.jrv 10.1-PRERELEASE FreeBSD >>>>>> 10.1-PRERELEASE #2 >>>>>> r273476M: Thu Oct 23 20:39:40 CDT 2014 >>>>>> james@bigtex.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 >>>>>> >>>>>> A pool created by a FreeBSD 9 system was imported into FreeBSD >>>>>> 10.1 but >>>>>> failed to create the recursive mountpoints as shown below. >>>>>> >>>>>> What's especially interesting is that the free space reported by >>>>>> zpool(1) and zfs(1) are wildly different, even though there are no >>>>>> reservations. >>>>>> >>>>>> Note that I was able to do a zpool upgrade, but that zfs upgrade >>>>>> failed >>>>>> on the children datasets. >>>>>> >>>>>> # zpool import SAS01 >>>>>> cannot mount '/SAS01/t03': failed to create mountpoint >>>>>> cannot mount '/SAS01/t04': failed to create mountpoint >>>>>> cannot mount '/SAS01/t05': failed to create mountpoint >>>>>> cannot mount '/SAS01/t06': failed to create mountpoint >>>>>> cannot mount '/SAS01/t07': failed to create mountpoint >>>>>> cannot mount '/SAS01/t08': failed to create mountpoint >>>>>> cannot mount '/SAS01/t12': failed to create mountpoint >>>>>> cannot mount '/SAS01/t13': failed to create mountpoint >>>>>> # zpool list SAS01 >>>>>> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH >>>>>> ALTROOT >>>>>> SAS01 43.5T 42.6T 948G - 0% 97% 1.00x >>>>>> ONLINE - >>>>>> # zfs list -p SAS01 >>>>>> NAME USED AVAIL REFER MOUNTPOINT >>>>>> SAS01 33279222543840 0 314496 /SAS01 >>>>>> # zpool get all SAS01 >>>>>> NAME PROPERTY VALUE >>>>>> SOURCE >>>>>> SAS01 size >>>>>> 43.5T - >>>>>> SAS01 capacity >>>>>> 97% - >>>>>> SAS01 altroot - >>>>>> default >>>>>> SAS01 health >>>>>> ONLINE - >>>>>> SAS01 guid 1341452135 >>>>>> default >>>>>> SAS01 version - >>>>>> default >>>>>> SAS01 bootfs - >>>>>> default >>>>>> SAS01 delegation on >>>>>> default >>>>>> SAS01 autoreplace off >>>>>> default >>>>>> SAS01 cachefile - >>>>>> default >>>>>> SAS01 failmode wait >>>>>> default >>>>>> SAS01 listsnapshots off >>>>>> default >>>>>> SAS01 autoexpand off >>>>>> default >>>>>> SAS01 dedupditto 0 >>>>>> default >>>>>> SAS01 dedupratio >>>>>> 1.00x - >>>>>> SAS01 free >>>>>> 948G - >>>>>> SAS01 allocated >>>>>> 42.6T - >>>>>> SAS01 readonly >>>>>> off - >>>>>> SAS01 comment - >>>>>> default >>>>>> SAS01 expandsize >>>>>> - - >>>>>> SAS01 freeing 0 >>>>>> default >>>>>> SAS01 fragmentation >>>>>> 0% - >>>>>> SAS01 leaked 0 >>>>>> default >>>>>> SAS01 feature@async_destroy enabled >>>>>> local >>>>>> SAS01 feature@empty_bpobj active >>>>>> local >>>>>> SAS01 feature@lz4_compress active >>>>>> local >>>>>> SAS01 feature@multi_vdev_crash_dump enabled >>>>>> local >>>>>> SAS01 feature@spacemap_histogram active >>>>>> local >>>>>> SAS01 feature@enabled_txg active >>>>>> local >>>>>> SAS01 feature@hole_birth active >>>>>> local >>>>>> SAS01 feature@extensible_dataset enabled >>>>>> local >>>>>> SAS01 feature@embedded_data active >>>>>> local >>>>>> SAS01 feature@bookmarks enabled >>>>>> local >>>>>> SAS01 feature@filesystem_limits enabled >>>>>> local >>>>>> # zfs get all SAS01 >>>>>> NAME PROPERTY VALUE SOURCE >>>>>> SAS01 type filesystem - >>>>>> SAS01 creation Tue Dec 23 2:51 2014 - >>>>>> SAS01 used 30.3T - >>>>>> SAS01 available 0 - >>>>>> SAS01 referenced 307K - >>>>>> SAS01 compressratio 1.00x - >>>>>> SAS01 mounted yes - >>>>>> SAS01 quota none default >>>>>> SAS01 reservation none default >>>>>> SAS01 recordsize 128K default >>>>>> SAS01 mountpoint /SAS01 default >>>>>> SAS01 sharenfs off default >>>>>> SAS01 checksum on default >>>>>> SAS01 compression off default >>>>>> SAS01 atime on default >>>>>> SAS01 devices on default >>>>>> SAS01 exec on default >>>>>> SAS01 setuid on default >>>>>> SAS01 readonly off default >>>>>> SAS01 jailed off default >>>>>> SAS01 snapdir hidden default >>>>>> SAS01 aclmode discard default >>>>>> SAS01 aclinherit restricted default >>>>>> SAS01 canmount on default >>>>>> SAS01 xattr off temporary >>>>>> SAS01 copies 1 default >>>>>> SAS01 version 5 - >>>>>> SAS01 utf8only off - >>>>>> SAS01 normalization none - >>>>>> SAS01 casesensitivity sensitive - >>>>>> SAS01 vscan off default >>>>>> SAS01 nbmand off default >>>>>> SAS01 sharesmb off default >>>>>> SAS01 refquota none default >>>>>> SAS01 refreservation none default >>>>>> SAS01 primarycache all default >>>>>> SAS01 secondarycache all default >>>>>> SAS01 usedbysnapshots 0 - >>>>>> SAS01 usedbydataset 307K - >>>>>> SAS01 usedbychildren 30.3T - >>>>>> SAS01 usedbyrefreservation 0 - >>>>>> SAS01 logbias latency default >>>>>> SAS01 dedup off default >>>>>> SAS01 mlslabel - >>>>>> SAS01 sync standard default >>>>>> SAS01 refcompressratio 1.00x - >>>>>> SAS01 written 307K - >>>>>> SAS01 logicalused 30.2T - >>>>>> SAS01 logicalreferenced 12K - >>>>>> SAS01 volmode default default >>>>>> SAS01 filesystem_limit none default >>>>>> SAS01 snapshot_limit none default >>>>>> SAS01 filesystem_count none default >>>>>> SAS01 snapshot_count none default >>>>>> SAS01 redundant_metadata all default >>>>>> # zpool status SAS01 >>>>>> pool: SAS01 >>>>>> state: ONLINE >>>>>> scan: scrub repaired 0 in 20h26m with 0 errors on Thu Dec 25 >>>>>> 20:57:34 2014 >>>>>> config: >>>>>> >>>>>> NAME STATE READ WRITE CKSUM >>>>>> SAS01 ONLINE 0 0 0 >>>>>> raidz2-0 ONLINE 0 0 0 >>>>>> da45 ONLINE 0 0 0 >>>>>> da44 ONLINE 0 0 0 >>>>>> da47 ONLINE 0 0 0 >>>>>> da43 ONLINE 0 0 0 >>>>>> da42 ONLINE 0 0 0 >>>>>> da46 ONLINE 0 0 0 >>>>>> da41 ONLINE 0 0 0 >>>>>> da40 ONLINE 0 0 0 >>>>>> >>>>>> errors: No known data errors >>>>>> # zfs upgrade -r SAS01 >>>>>> cannot set property for 'SAS01/t03': out of space >>>>>> cannot set property for 'SAS01/t04': out of space >>>>>> cannot set property for 'SAS01/t05': out of space >>>>>> cannot set property for 'SAS01/t06': out of space >>>>>> cannot set property for 'SAS01/t07': out of space >>>>>> cannot set property for 'SAS01/t08': out of space >>>>>> cannot set property for 'SAS01/t12': out of space >>>>>> cannot set property for 'SAS01/t13': out of space >>>>>> 0 filesystems upgraded >>>>>> 1 filesystems already at this version >>>>>> # >>>>>> _______________________________________________ >>>>>> freebsd-fs@freebsd.org mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>> To unsubscribe, send any mail to >>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54A0E737.6000101>