Date: Thu, 1 Sep 2011 11:50:26 -0400 From: Dave Cundiff <syshackmin@gmail.com> To: stable@freebsd.org Subject: ZFS V28 on 8.2-RELEASE write behavior Message-ID: <CAKHEz2bpHTaW=AezA0=KAGbsFF52h5BjDDNFy6HGtosnm_y%2BpQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, I'm running ZFS V28 with 8.2-RELEASE. Its a stock system patched with the following http://people.freebsd.org/~mm/patches/zfs/v28/releng-8.2-zfsv28-20110616.patch.xz https://www.illumos.org/attachments/292/txg.c.patch The zfsv28 patch did have a failed hunk in /usr/src/sys/cddl/compat/opensolaris/sys/sysmacros.h but it appeared that was because the file already existed identically in my src tree. I have not upgraded the zpool, it remains at v15 in case I need to drop back to the old kernel. I've removed all zfs related loader/sysctl options I had in from v15. What I'm seeing is strange write behavior. When I first set it up I'd see writes in small bursts of 100 megs or so every 5 seconds. Now I'm seeing small writes and reads are stalling. This is what it looks like when IO is horrible: capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- san 3.97T 8.66T 269 228 6.54M 21.9M san 3.97T 8.66T 2 142 131K 1.18M san 3.97T 8.66T 952 0 52.7M 0 san 3.97T 8.66T 527 149 24.7M 1.22M san 3.97T 8.66T 2 181 192K 1.35M san 3.97T 8.66T 1.17K 0 47.9M 0 san 3.97T 8.66T 452 247 23.6M 3.40M san 3.97T 8.66T 702 268 23.8M 3.38M san 3.97T 8.66T 0 0 0 0 san 3.97T 8.66T 686 255 30.7M 3.39M san 3.97T 8.66T 379 229 19.3M 2.82M san 3.97T 8.66T 94 0 5.59M 0 san 3.97T 8.66T 1004 243 52.4M 3.32M san 3.97T 8.66T 3 270 195K 3.48M san 3.97T 8.66T 250 0 14.6M 0 san 3.97T 8.66T 439 250 15.5M 3.42M san 3.97T 8.66T 1 257 128K 3.65M It gets better for short periods and looks like this san 3.95T 8.68T 1004 55 53.1M 360K san 3.95T 8.68T 341 533 10.6M 42.4M san 3.95T 8.68T 783 0 43.5M 0 san 3.95T 8.68T 409 374 17.1M 35.5M san 3.95T 8.68T 423 117 18.4M 3.83M san 3.95T 8.68T 1.01K 136 43.2M 2.26M san 3.95T 8.68T 447 454 16.8M 35.4M san 3.95T 8.68T 991 0 46.3M 0 san 3.95T 8.68T 420 394 21.8M 32.9M san 3.95T 8.68T 1.16K 241 54.4M 6.00M san 3.95T 8.68T 575 0 20.4M 0 san 3.95T 8.68T 284 328 9.77M 30.3M san 3.95T 8.68T 663 203 33.6M 5.15M san 3.95T 8.68T 319 0 10.0M 0 san 3.95T 8.68T 318 453 10.0M 24.0M san 3.95T 8.68T 1.06K 299 30.9M 25.0M I also see very high CPU usage on a few ZFS related threads 38 root -8 - 0K 7536K zio->i 3 64:23 25.20% {txg_thread_enter} 0 root -16 0 0K 2704K sm->sm 0 66:48 21.39% {zio_write_issue_} 0 root -16 0 0K 2704K sm->sm 1 66:36 18.16% {zio_write_issue_} 0 root -16 0 0K 2704K sm->sm 1 67:50 13.77% {zio_write_issue_} 0 root -16 0 0K 2704K CPU3 3 67:28 12.16% {zio_write_issue_} These would be a 1-2% each prior. Any help tracking this down would be greatly appreciated. Thanks, -- Dave Cundiff System Administrator A2Hosting, Inc http://www.a2hosting.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKHEz2bpHTaW=AezA0=KAGbsFF52h5BjDDNFy6HGtosnm_y%2BpQ>