Date: Sat, 8 Aug 2009 05:02:48 -0400 From: grarpamp <grarpamp@gmail.com> To: freebsd-performance@freebsd.org Subject: RELENG_7 heavy disk = system crawls Message-ID: <d2e731a10908080202i2b82f2aaia567b740058281e@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
> Are you *sure* that the cpu is your bottleneck?
Well, I've got a gig free in /usr/local which is ufs2+softdeps. So
I just dd if=dev/zero bs=1m of=zero there. Disk was 100% busy, cpu
was 10-15% system, 10% user, about 16MiB/sec.
Of course since that is my system spindle, and it was busied out
by dd, I had to wait 10 sec or so for vi to load to write this note
during it :) But the rest of the interface was responsive. Though
I've never run RELENG_4 on this box, I'd venture it would feel
similar in this case.
I can dd if=/dev/ad[n].eli of=/dev/null bs=1m and use 75% system
all in geli, 27% disk busy, 20MiB/sec. Interface was slower but
reasonable.
Now do a dd if=file of=/dev/null bs=1m where file is on zfs on the
geli spindle above and the system melts down. Half cpu in geli,
half spread over about 8 spa_zio's, 4MiB/sec, all system time, disk
100% busy.
I'm not sure yet how to isolate cpu from i/o under my geli+zfs
setup. I think they're mated together.
Curiously, I've got about 104 spa_zio procs all in tq->tq DL state.
Only about 20 or so have more than a little system time on them.
But about 9 zfs mounts so that may be ok, don't know.
Don't get me wrong, FreeBSD has been my primary os since RELENG_2_2.
And it's been great, still is. I recommend and use it all the time.
It's just that this workload has really put the screws to things
and I don't see a way out. I'd like to deploy geli+zfs everywhere
but if I can't login remotely because some user has it busied out
on something I've no knobs to control, umm, yeah :)
I'm curious to see what others running geli_aes128+zfs_sha256 are
seeing in this regard. And I'd love to see what a fast dual or more
core amd64 system would be like under this workload.
As to your i/o thing, I think back in RELENG_4 that if all the
spindles were on the same pata controller/interrupt, monopolistic
loads could occur. Seemed to be a hardware thing, not BSD. IE: At
the moment I've got a half dozen spindles and filesystems spread
out under RELENG_4 all happily doing find | xargs sha1 at once, no
problems. That hardware is set for update to RELENG_7 or RELENG_8
in a few weeks.
> we need a way to nice i/o up/down
That would be handy for sure. User spindles, system spindles,
storage, net, keyboard, etc.
# systime spread
11 root 1 171 ki31 0K 8K RUN 46.5H 88.18% idle: cpu0
3215 root 1 -8 - 0K 8K geli:w 737:30 1.76% g_eli[0] ad6
607 root 1 -8 - 0K 8K geli:w 158:12 0.00% g_eli[0] ad4
3235 root 1 -16 - 0K 24K tq->tq 69:41 0.00% spa_zio
3229 root 1 -16 - 0K 24K tq->tq 69:40 0.00% spa_zio
3228 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio
3234 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio
3233 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio
3232 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio
3231 root 1 -16 - 0K 24K tq->tq 69:38 0.00% spa_zio
3230 root 1 -16 - 0K 24K tq->tq 69:37 0.00% spa_zio
1135 user 1 44 0 169M 152M select 56:02 0.00% XFree86
954 root 1 -16 - 0K 24K tq->tq 17:10 0.00% spa_zio
958 root 1 -16 - 0K 24K tq->tq 17:10 0.00% spa_zio
956 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio
953 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio
957 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio
952 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio
951 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio
955 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio
613 root 1 -8 - 0K 8K geli:w 16:12 0.00% g_eli[0] ad7
3 root 1 -8 - 0K 8K - 15:05 0.00% g_up
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d2e731a10908080202i2b82f2aaia567b740058281e>