Date: Wed, 31 Aug 2011 15:37:24 +0400 From: Lev Serebryakov <lev@serebryakov.spb.ru> To: Jeremy Chadwick <freebsd@jdc.parodius.com> Cc: freebsd-fs@freebsd.org Subject: Re: Very inconsistent (read) speed on UFS2 Message-ID: <981083303.20110831153724@serebryakov.spb.ru> In-Reply-To: <20110831101211.GA98865@icarus.home.lan> References: <1945418039.20110830231024@serebryakov.spb.ru> <317753422.20110830231815@serebryakov.spb.ru> <20110831004251.GA89979@icarus.home.lan> <147623060.20110831123623@serebryakov.spb.ru> <20110831101211.GA98865@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, Jeremy. You wrote 31 =E0=E2=E3=F3=F1=F2=E0 2011 =E3., 14:12:11: > This benchmark data is more or less unhelpful due to the fact that there > are writes occurring during the middle of your reads. There's another Yep :( > spun-off portion of this thread that is discussing how you're > benchmarking these things (specifically some code you wrote?). I don't > know what else to say in this regard. It would really help if you could > use something like bonnie++ and make sure the filesystem is not being > used by ANYTHING during your benchmarks. I'll try bonnie++, Ok. My code is really as simple as it could be: fd =3D open(fileName, O_RDONLY | O_DIRECT); gettimeofday(&start, NULL); /* s_BufferSize is 128KiB */ while ((rd =3D read(fd, s_Buffer, s_BufferSize)) > 0) size +=3D rd; gettimeofday(&end, NULL); close(fd); > Anyway, the data is interesting because from an aggregate total > perspective, you're hitting some arbitrary limit on all of your devices > which almost indicates memory bus throttling or something along those > lines; CPU time? I really don't know. Aggregate write speeds > respectively: > 43138.8 + 43138.8 + 43044.7 + 43232.9 + 43138.8 =3D=3D 215694.0 KByte/sec > 10515.9 + 10547.2 + 10703.7 + 10484.6 + 10265.5 =3D=3D 52516.9 KByte/sec > 56583.1 + 56677.2 + 56489.0 + 56614.5 + 56739.9 =3D=3D 283103.7 KByte/sec > 41001.3 + 40969.9 + 40844.5 + 41001.3 + 40875.9 =3D=3D 204692.9 KByte/sec > 15660.2 + 15503.6 + 15566.2 + 15785.5 + 15566.2 =3D=3D 78081.7 KByte/sec > The totals are "all over the place", but what interests me the most is > that the total aggregate never exceeds an amount that's slightly under > 300MBytes/sec.. That number has some relevance if, say, you're using a > port multiplier (5 devices aggregated across one SATA300 port). No. All drives are on separate ports of ICH9R chipset controller. And, yes, sustained and constant 300MiB/s is my dream :) Keywords: sustained and constant. > Despite these being WD20EARS drives (4 platters, ugh!), these individual As ffar as I understand, 4 platters are slightly better in linear access than 3 platters, but worse in random access, as it read more data without heads movement. > devices should be able to push 75-90MBytes/sec writes, and slightly > higher reads. Read is about 110MiB/s at beginning of drive. > Here's an idea: can you stop using the filesystem for a bit and instead > do raw dd's from all of the /dev/adaX entries to /dev/null > simultaneously (pick something like bs=3D64k or bs=3D256k), then run your > iostats? I'm basically trying to figure out if the bad speeds are > actually the devices themselves or if it's the geom_raid5 stuff. You > get where I'm going with this. Not a problem! FS is unmounted, and after that: # for d in 1 2 3 4 5 ; do dd if=3D/dev/ada$d of=3D/dev/null bs=3D64k & done # iostat -c 999999 -dx ada1 ada2 ada3 ada4 ada5 device r/s w/s kr/s kw/s wait svc_t %b ada1 1849.1 0.0 118343.7 0.0 1 0.5 93 ada2 1920.3 0.0 122900.2 0.0 0 0.5 94 ada3 1874.5 0.0 119966.6 0.0 1 0.5 94 ada4 1794.5 0.0 114848.4 0.0 1 0.5 94 ada5 1893.0 0.0 121152.5 0.0 1 0.5 93 It is very typical data, speed slightly goes up and down for all HDDs without any visible fastest or slowest drive. > If 5 simultaneously dds reading from the drives is very fast (way faster > than the above) and there aren't sporadic drops in performance which > aren't caused by writes (hence my "stop using the filesystem" comment), > then I think we've narrowed down where the issue lies -- not the drives. Yep. It seems to be exactly like this. > The dd method I describe should absolutely not induce writes, hence my > recommendation. If writes are seen during the dd's, then either the > filesystem is mounted and FreeBSD is doing something "interesting" on a > filesystem or vfs level, or your system is actually an izbushka..... > Maybe softupdates are somehow responsible? Not sure. I have one ide about geom_raid5 writes... I need to check it. --=20 // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?981083303.20110831153724>