Date: Tue, 14 Oct 2014 09:14:54 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: "K. Macy" <kmacy@freebsd.org> Cc: "freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org>, mark <Mark.Martinec@ijs.si>, FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Message-ID: <A7771879317F4194A1D5E4921CD33593@multiplay.co.uk> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <DE7DD7A94E9B4F1FBB3AFF57EDB47C67@multiplay.co.uk> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <A5BA41116A7F4B23A9C9E469C4146B99@multiplay.co.uk> <CAHM0Q_N%2BC=3qgUnyDkEugOFcL=J8gBjbTg8v45Vz3uT=e=Fn2g@mail.gmail.com> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <CAHM0Q_OpV2sAQQAH6Cj_=yJWAOt8pTPWQ-m45JSiXDpBwT6WTA@mail.gmail.com> <E2E24A91B8B04C2DBBBC7E029A12BD05@multiplay.co.uk> <CAHM0Q_Oeka25-kdSDRC2evS1R8wuQ0_XgbcdZCjS09aXJ9_WWQ@mail.gmail.com> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <E3C3C359999140B48943A0E1A04F83A9@multiplay.co.uk> <CAHM0Q_O7LNBiQAEjygANa%2B0rqm9cywjTPbNXabB4TePfEHAZsA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
From: "K. Macy" <kmacy@freebsd.org>
>>> Thank you both for analysis and effort!
>>>
>>> I can't rule out the possibility that my main system pool
>>> on a SSD was low on space at some point in time, but the
>>> three 4 GiB cloned pools (sys1boot and its brothers) were all
>>> created as a zfs send / receive copies of the main / (root)
>>> file system and I haven't noticed anything unusual during
>>> syncing. This syncing was done manually (using zxfer) and
>>> independently from the upgrade on the system - on a steady/quiet
>>> system, when the source file system definitely had sufficient
>>> free space.
>>>
>>> The source file system now shows 1.2 GiB of usage shown
>>> by df:
>>> shiny/ROOT 61758388 1271620 60486768 2% /
>>> Seems unlikely that the 1.2 GiB has grown to 4 GiB space
>>> on a cloned filesystem.
>>>
>>> Will try to import the main two pools after re-creating
>>> a sane boot pool...
>>
>>
>> Yer zfs list only shows around 2-3GB used too but zpool list
>> shows the pool is out of space. Cant rule out an accounting
>> issue though.
>>
>
> What is using the extra space in the pool? Is there an unmounted
> dataset or snapshot? Do you know how to easily tell? Unlike txg and
> zio processing I don't have the luxury of having just read that part
> of the codebase.
Its not clear but I believe it could just be fragmention even though
its ashift=9.
I sent the last snapshot to another pool of the same size and it
resulted in:
NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT
sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE -
sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE -
I believe FRAG is 0% as the feature wasn't enabled for the lifetime of
the pool hence its simply not showing a valid value.
zfs list -t all -r sys1boot
NAME USED AVAIL REFER MOUNTPOINT
sys1boot 1.76G 2.08G 11K /sys1boot
sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT
sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-08-19_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-08-20_04.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-21_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-22_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-23_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-24_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-26_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-27_04.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-28_04.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-08-29_04.00 128K - 1.19G -
sys1boot/ROOT@auto-2014-08-31_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-01_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-02_04.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-03_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-04_04.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-05_04.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-07_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-08_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-09_04.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-10_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-10_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-10_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-10_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-10_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-10_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-11_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-11_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-11_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-11_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-11_16.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-11_20.00 84.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-12_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-12_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-12_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-12_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-12_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-12_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-13_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-13_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-13_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-13_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-13_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-13_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-14_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-14_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-14_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-14_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-14_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-14_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-15_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-15_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-15_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-15_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-15_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-15_20.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-16_00.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-16_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-16_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-16_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-16_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-16_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-17_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-17_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-17_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-17_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-17_16.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-17_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_16.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_20.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-18_23.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_00.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_01.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_02.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_03.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_04.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_05.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_06.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_07.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_08.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_09.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_10.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_11.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_12.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_13.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_14.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_15.00 1K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_16.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_17.00 85.5K - 1.19G -
sys1boot/ROOT@auto-2014-09-19_18.00 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_18.40 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_18.50 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_19.00 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_19.10 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_19.20 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_19.30 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_19.40 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_19.50 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_20.00 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_20.10 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_20.20 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_20.30 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_20.40 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_20.50 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_21.00 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_21.10 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_21.20 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_21.30 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_21.40 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_21.50 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_22.00 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_22.10 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_22.20 1K - 1.20G -
sys1boot/ROOT@auto-2014-09-19_22.30 0 - 1.20G -
Regards
Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A7771879317F4194A1D5E4921CD33593>