Date: Mon, 06 Jul 2009 20:00:49 +0300 From: Alexander Motin <mav@FreeBSD.org> To: Bruce Evans <brde@optusnet.com.au> Cc: freebsd-arch@freebsd.org Subject: Re: DFLTPHYS vs MAXPHYS Message-ID: <4A522DC1.2080908@FreeBSD.org> In-Reply-To: <20090707011217.O43961@delplex.bde.org> References: <4A4FAA2D.3020409@FreeBSD.org> <20090705100044.4053e2f9@ernst.jennejohn.org> <4A50667F.7080608@FreeBSD.org> <20090705223126.I42918@delplex.bde.org> <4A50BA9A.9080005@FreeBSD.org> <20090706005851.L1439@besplex.bde.org> <4A50DEE8.6080406@FreeBSD.org> <20090706034250.C2240@besplex.bde.org> <4A50F619.4020101@FreeBSD.org> <20090707011217.O43961@delplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Bruce Evans wrote: > On Sun, 5 Jul 2009, Alexander Motin wrote: >> In this tests you've got almost only negative side of effect, as you >> have said, due to cache misses. > > No, I got negative and positive for the regular file (due to cache misses > for large block sizes and too many transactions for very small block sizes > (< 16K), and only positive for the disk file (due to cache misses not > being tested). No, I mean that you didn't get any benefit from increasing from disk I/O transaction sizes. You still were limited by 64K. >> But with 64K MAXPHYS you just didn't get any benefit from using bigger >> block size. > > MAXPHYS is 128K. The ata driver has a limit of 64K so anything larger > than 64K wouldn't do much except increase cache misses. In physio(), > it would just causes physio() to ask the driver to read 64K at a time. > My claim is partly that 64K such a large size that the extra CPU caused > by splitting up into 64K-blocks is insignificant. ATA subsystem allows drivers to have different transaction sizes. At least AHCI driver can do more. What is about insignificant - I have shown example, when it is not completely so. > Here are better results for the disk file test, with cache accesses and > misses counted by perfmon: > > Cache misses are minimized here using a user buffer size of 64K. > > Cache accesses are minimized here using a user buffer size of 1M. > > Cache misses increased by a factor of 5 going from user buffer size > 2M to 4M. > > Cache misses increased by a another factor of 5 going from user buffer > size 4M to 128M. I can't explain why there are as many as 13.8 million > -- I would have expected 2*256M/64 = 8M only, but in more cases. 8 > million cache misses in only 4.8 seconds is a lot, and you would get > that many in only 1.3 seconds at 200MB/S. Of course, 128M is a silly > buffer size, but I would expect the cache effects to show up at about > half the L2 size under more realistic loads. > > Cache accesses varied significantly, between 146 million (block size > 16384), 37 million (block size 1M) and 138 million (block size 128M). > I can only partly explain this. I think the minimum number is > 2*256M/16 = 32M (for fetching from L2 to L1 16 bytes at a time). > 128M might result from fetching 4 bytes at a time or thrashing causing > the equivalent. I think on small transaction size cache misses could be caused not by transfered data itself, but by different variables addressed by code. Growing number of misses on bigger blocks is also predictable. Working with regular file could giva a different results, as soon as data will not be read into the same memory, but over the all buffer cache. And once more I want to say that you are testing not the same I was speaking about. I agree that enormous block size in user-level will affect cache efficiency negatively, just because of large am mounts of data moved by CPU. What I wanted to say is that IMHO allowing device to transfer data with bigger blocks, when needed, will give positive effect for both I/O hardware and CPU usage effectiveness, without significant affect for caching, as caches are mostly trashed not there, but in completely different places. -- Alexander Motin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A522DC1.2080908>