Date: Sat, 2 May 2009 09:39:04 +1000 From: nf <nf@wh3rd.net> To: freebsd-questions@freebsd.org Subject: Poor ZFS performance Message-ID: <e2fbd2a10905011639x475c1fbex92508493449bff51@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello!
I've got a ZFS-based file server running FreeBSD 7.1-STABLE, with two
raidz1's of 3 x 500gb.
Recently my system has exhibited exceptionally poor performance in a
way that confuses me. None of the drives are in a degraded state, and
zpool iostat reports typical performance figures, while actual
applications (dd, in this case) only receive data at much lower rates.
My basic benchmark, and once that I'd used before, is to dd a big file
and output it to /dev/null. What's occurring now is mystifying, as dd
is reporting much lower figures than zpool iostat.
--
nf@ice2:~>uname -a
FreeBSD ice2 7.1-STABLE FreeBSD 7.1-STABLE #0: Sun Feb 8 20:31:17 EST
2009 root@ice2:/usr/obj/usr/src/sys/GENERIC amd64
nf@ice2:/media/nf/videos>dd if=sph-kingok.avi of=/dev/null
1433288+0 records in
1433288+0 records out
733843456 bytes transferred in 61.124812 secs (12005656 bytes/sec)
(and, while that's running:)
nf@ice2:~>sudo zpool iostat -v 2
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
data 1.19T 1.53T 13 29 990K 153K
raidz1 1.01T 361G 9 11 641K 58.8K
ad4s1d - - 3 8 283K 30.2K
ad8s1d - - 3 8 282K 30.2K
ad10s1d - - 3 8 284K 30.2K
raidz1 191G 1.17T 4 17 349K 94.5K
ad6s1d - - 1 12 138K 48.3K
ad10s1e - - 1 12 134K 48.3K
ad12s1d - - 2 12 141K 48.3K
----------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
data 1.19T 1.53T 1.41K 0 179M 0
raidz1 1.01T 361G 1.40K 0 179M 0
ad4s1d - - 676 0 72.1M 0
ad8s1d - - 453 0 45.1M 0
ad10s1d - - 605 0 61.7M 0
raidz1 191G 1.17T 4 0 127K 0
ad6s1d - - 0 0 49.9K 0
ad10s1e - - 1 0 74.4K 0
ad12s1d - - 1 0 92.1K 0
----------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
data 1.19T 1.53T 1.19K 102 152M 517K
raidz1 1.01T 361G 1.19K 42 152M 149K
ad4s1d - - 369 16 39.2M 75.4K
ad8s1d - - 436 17 48.7M 77.2K
ad10s1d - - 563 19 65.3M 76.2K
raidz1 191G 1.17T 5 59 116K 368K
ad6s1d - - 0 32 63.9K 187K
ad10s1e - - 2 34 146K 188K
ad12s1d - - 1 34 114K 190K
----------- ----- ----- ----- ----- ----- -----
^C
--
So I presume you discount the first output of iostat as a measuring
error, and interpret the subsequent outputs as cumulative over the
repeat interval (in this case each two seconds). What I'm seeing is
the ZFS pool being read from at 70-90mb/sec, while dd is only getting
data at 12mb/sec. Where is all the data that's being read from the
disk going?
As I said before, on previous attempts iostat and dd reported figures
that corresponded with each other. I'm not sure what's changed between
those attempts.
(There is nothing else reading from disk at the same time. As soon as
the dd concludes, zpool iostat reports the arrays are idle.)
FWIW, here's what top reports while I'm running the dd:
--
CPU: 0.4% user, 0.0% nice, 14.3% system, 4.5% interrupt, 80.8% idle
Mem: 1950M Active, 868M Inact, 537M Wired, 238M Cache, 363M Buf, 87M Free
Swap: 1024M Total, 101M Used, 923M Free, 9% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
8402 nf 1 50 0 4604K 788K zio->i 0 0:03 8.59% dd
--
If anyone can shed some light on what's going on here, I'd be most appreciated.
Thanks,
nf
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e2fbd2a10905011639x475c1fbex92508493449bff51>