Date: Fri, 18 May 2018 15:42:05 +0000 From: Paul Esson <paul.esson@redstor.com> To: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Unexpected zvol usage Message-ID: <HE1PR0102MB25880FD0731B56770F19D7F29E900@HE1PR0102MB2588.eurprd01.prod.exchangelabs.com>
next in thread | raw e-mail | index | archive | help
Hi Folks,
I have an 11.1-RELEASE system being used as a host for a bhyve guest. Ther=
e is a large zpool on the host created from 12 x 10TB HDDs using raidz2 red=
undancy with ashift12. I have created a sparse zvol within the pool using =
default settings and presented that to the bhyve vm as an ahci-hd disk type=
. The guest has a zpool and filesystem dataset built on this disk. When I=
start to write to the filesystem on the guest I am finding that the used/r=
eferenced on the host's zvol are more than double those on the guest. The =
logicalused/referenced values on the host zvol are more in line with the eq=
uivalent guest values, but my problem is that the host zvol is likely to fi=
ll before I have written all intended data to the guest.
I have included below information from both the host and guest before and a=
fter writing. This output shows that the zvol uses a default 8K blocksize =
and that the guest zfs is therefore ashift13. I also tried creating the zv=
ol with a 4K blocksize and the guest zfs ashift12 so that 4K blocks were co=
nsistent across hosts and guest, but still saw the amplification on writes =
to the zvol.
Any insight greatly appreciated.
HOST
Zpool
RAIDZ2 12 x HDDs, ashift 12
NAME PROPERTY VALUE SOURCE
dc1-hn-01 type filesystem -
dc1-hn-01 creation Mon Apr 23 14:35 2018 -
dc1-hn-01 used 32.0G -
dc1-hn-01 available 78.2T -
dc1-hn-01 referenced 201K -
dc1-hn-01 compressratio 1.00x -
dc1-hn-01 mounted yes -
dc1-hn-01 quota none default
dc1-hn-01 reservation none default
dc1-hn-01 recordsize 128K default
dc1-hn-01 mountpoint /export/data/dc1-hn-01 local
dc1-hn-01 sharenfs off default
dc1-hn-01 checksum on default
dc1-hn-01 compression off default
dc1-hn-01 atime on default
dc1-hn-01 devices on default
dc1-hn-01 exec on default
dc1-hn-01 setuid on default
dc1-hn-01 readonly off default
dc1-hn-01 jailed off default
dc1-hn-01 snapdir hidden default
dc1-hn-01 aclmode discard default
dc1-hn-01 aclinherit restricted default
dc1-hn-01 canmount on default
dc1-hn-01 xattr off temporary
dc1-hn-01 copies 1 default
dc1-hn-01 version 5 -
dc1-hn-01 utf8only off -
dc1-hn-01 normalization none -
dc1-hn-01 casesensitivity sensitive -
dc1-hn-01 vscan off default
dc1-hn-01 nbmand off default
dc1-hn-01 sharesmb off default
dc1-hn-01 refquota none default
dc1-hn-01 refreservation none default
dc1-hn-01 primarycache all default
dc1-hn-01 secondarycache all default
dc1-hn-01 usedbysnapshots 0 -
dc1-hn-01 usedbydataset 201K -
dc1-hn-01 usedbychildren 32.0G -
dc1-hn-01 usedbyrefreservation 0 -
dc1-hn-01 logbias latency default
dc1-hn-01 dedup off default
dc1-hn-01 mlslabel -
dc1-hn-01 sync standard default
dc1-hn-01 refcompressratio 1.00x -
dc1-hn-01 written 201K -
dc1-hn-01 logicalused 2.89G -
dc1-hn-01 logicalreferenced 36.5K -
dc1-hn-01 volmode default default
dc1-hn-01 filesystem_limit none default
dc1-hn-01 snapshot_limit none default
dc1-hn-01 filesystem_count none default
dc1-hn-01 snapshot_count none default
dc1-hn-01 redundant_metadata all default
NAME AVAIL USED USEDSNAP USEDDS USEDREFR=
ESERV USEDCHILD
dc1-hn-01 78.2T 32.0G 0 201K =
0 32.0G
dc1-hn-01/vm 78.2T 31.9G 0 990M =
0 30.9G
dc1-hn-01/vm/dc1-olbp-sn-11 78.2T 30.9G 0 238K =
0 30.9G
dc1-hn-01/vm/dc1-olbp-sn-11/disk0 78.2T 30.9G 0 4.35G =
26.6G 0
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 78.2T 4.50M 0 4.50M =
0 0
Sparse ZVOL - baseline
NAME PROPERTY VALUE =
SOURCE
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 15:36 2=
018 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 4.50M =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.2T =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 4.50M =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T =
local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 4.50M =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 4.50M =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 1.89M =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 1.89M =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev =
local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all =
default
GUEST - baseline
1 x vdisk from host ZVOL ashift 13
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHIL=
D
dc1-sn-11 26.9T 632K 0 176K 0 456=
K
NAME PROPERTY VALUE SOURCE
dc1-sn-11 type filesystem -
dc1-sn-11 creation Fri May 18 15:40 2018 -
dc1-sn-11 used 632K -
dc1-sn-11 available 26.9T -
dc1-sn-11 referenced 176K -
dc1-sn-11 compressratio 1.00x -
dc1-sn-11 mounted yes -
dc1-sn-11 quota none default
dc1-sn-11 reservation none default
dc1-sn-11 recordsize 128K default
dc1-sn-11 mountpoint /export/data/dc1-sn-11 local
dc1-sn-11 sharenfs off default
dc1-sn-11 checksum on default
dc1-sn-11 compression off default
dc1-sn-11 atime on default
dc1-sn-11 devices on default
dc1-sn-11 exec on default
dc1-sn-11 setuid on default
dc1-sn-11 readonly off default
dc1-sn-11 jailed off default
dc1-sn-11 snapdir hidden default
dc1-sn-11 aclmode discard default
dc1-sn-11 aclinherit restricted default
dc1-sn-11 canmount on default
dc1-sn-11 xattr off temporary
dc1-sn-11 copies 1 default
dc1-sn-11 version 5 -
dc1-sn-11 utf8only off -
dc1-sn-11 normalization none -
dc1-sn-11 casesensitivity sensitive -
dc1-sn-11 vscan off default
dc1-sn-11 nbmand off default
dc1-sn-11 sharesmb off default
dc1-sn-11 refquota none default
dc1-sn-11 refreservation none default
dc1-sn-11 primarycache all default
dc1-sn-11 secondarycache all default
dc1-sn-11 usedbysnapshots 0 -
dc1-sn-11 usedbydataset 176K -
dc1-sn-11 usedbychildren 456K -
dc1-sn-11 usedbyrefreservation 0 -
dc1-sn-11 logbias latency default
dc1-sn-11 dedup off default
dc1-sn-11 mlslabel -
dc1-sn-11 sync standard default
dc1-sn-11 refcompressratio 1.00x -
dc1-sn-11 written 176K -
dc1-sn-11 logicalused 49K -
dc1-sn-11 logicalreferenced 11.5K -
dc1-sn-11 volmode default default
dc1-sn-11 filesystem_limit none default
dc1-sn-11 snapshot_limit none default
dc1-sn-11 filesystem_count none default
dc1-sn-11 snapshot_count none default
dc1-sn-11 redundant_metadata all default
After writing some data to the guest
HOST ZVOL
NAME PROPERTY VALUE =
SOURCE
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 15:36 2=
018 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 99.7G =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.1T =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 99.7G =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T =
local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 99.7G =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 99.7G =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 43.6G =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 43.6G =
-
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev =
local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none =
default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all =
default
GUEST ZFS
NAME PROPERTY VALUE SOURCE
dc1-sn-11 type filesystem -
dc1-sn-11 creation Fri May 18 15:40 2018 -
dc1-sn-11 used 44.3G -
dc1-sn-11 available 26.8T -
dc1-sn-11 referenced 176K -
dc1-sn-11 compressratio 1.00x -
dc1-sn-11 mounted no -
dc1-sn-11 quota none default
dc1-sn-11 reservation none default
dc1-sn-11 recordsize 128K default
dc1-sn-11 mountpoint /export/data/dc1-sn-11 local
dc1-sn-11 sharenfs off default
dc1-sn-11 checksum on default
dc1-sn-11 compression off default
dc1-sn-11 atime on default
dc1-sn-11 devices on default
dc1-sn-11 exec on default
dc1-sn-11 setuid on default
dc1-sn-11 readonly off default
dc1-sn-11 jailed off default
dc1-sn-11 snapdir hidden default
dc1-sn-11 aclmode discard default
dc1-sn-11 aclinherit restricted default
dc1-sn-11 canmount on default
dc1-sn-11 xattr on default
dc1-sn-11 copies 1 default
dc1-sn-11 version 5 -
dc1-sn-11 utf8only off -
dc1-sn-11 normalization none -
dc1-sn-11 casesensitivity sensitive -
dc1-sn-11 vscan off default
dc1-sn-11 nbmand off default
dc1-sn-11 sharesmb off default
dc1-sn-11 refquota none default
dc1-sn-11 refreservation none default
dc1-sn-11 primarycache all default
dc1-sn-11 secondarycache all default
dc1-sn-11 usedbysnapshots 0 -
dc1-sn-11 usedbydataset 176K -
dc1-sn-11 usedbychildren 44.3G -
dc1-sn-11 usedbyrefreservation 0 -
dc1-sn-11 logbias latency default
dc1-sn-11 dedup off default
dc1-sn-11 mlslabel -
dc1-sn-11 sync standard default
dc1-sn-11 refcompressratio 1.00x -
dc1-sn-11 written 176K -
dc1-sn-11 logicalused 44.2G -
dc1-sn-11 logicalreferenced 11.5K -
dc1-sn-11 volmode default default
dc1-sn-11 filesystem_limit none default
dc1-sn-11 snapshot_limit none default
dc1-sn-11 filesystem_count none default
dc1-sn-11 snapshot_count none default
dc1-sn-11 redundant_metadata all default
Regards,
Paul Esson
t +44 (0)118 951 5235 | m +44 (0)776 690 6514
e paul.esson@redstor.com<mailto:paul.esson@redstor.com>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?HE1PR0102MB25880FD0731B56770F19D7F29E900>