Date: Thu, 23 Jan 2014 00:04:10 -0500 From: J David <j.david.lists@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-net@freebsd.org Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <CABXB=RToav%2B%2BV38pOorVPWpgZSuYmL-x7e8oxd3ayJCmAtLn-g@mail.gmail.com> In-Reply-To: <1891524918.14888294.1390450374695.JavaMail.root@uoguelph.ca> References: <CABXB=RQ2ck=kc7AH9GLcmVuKyTAfiDbSZ9N6XQ4A%2Bw-q9NqSmA@mail.gmail.com> <1891524918.14888294.1390450374695.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 22, 2014 at 11:12 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote: > So, do you consider the 32K results as reasonable or terrible performance? > (They are obviously much better than 64K, except for the reread case.) It's the 64k numbers that prompted the "terrible" thread title. A 196K/sec write speed on a 10+ Gbit network is pretty disastrous. The 32k numbers are, as you say, better. Possibly reasonable, but I'm not sure if they're optimal. It's hard to tell the latency of the virtual network, which would be needed to make that determination. It would be best if FreeBSD out of the box blows the doors of Debian out of the box and FreeBSD tuned to the gills blows the doors off of Debian tuned to the gills. Right now, Debian seems to be the one with the edge and with FreeBSD's illustrious history as the NFS performance king for so many years, that just won't do. :) > Btw, I don't think you've mentioned what network device driver gets used > for this virtual environment. That might be useful, in case the maintainer > of that driver is aware of some issue/patch. KVM uses virtio. >> 00:38:07.932732 IP (tos 0x0, ttl 64, id 38912, offset 0, flags [DF], >> proto TCP (6), length 53628) >> > I don't know why this would be so large. A 32K write should be under > 33Kbytes in size, not 53Kbytes. I suspect tcpdump is confused? Since TCP is stream oriented, is there a reason to expect 1:1 correlation between NFS writes and TCP packets? > Well, it seems Debian is doing 4096 byte writes, which won't have anywhere > near the effect on the network driver/virtual hardware that a 64K (about > 45 IP datagrams) in one NFS RPC will. Debian's kernel says it is doing 64k reads/writes on that mount. So again, possibly an expectation of 1:1 correlation between NFS writes and TCP packets is not being satisfied. However, iozone is doing 4k reads/writes for these tests, so it's also possible that Debian is not coalescing them at all (which FreeBSD apparently is) and the 4k writes are hitting the virtual wire as-is. Also, both sides have TSO and LRO, so it would be surprising (and incorrect?) behavior if a 64k packet were actually fragmented into 45 IP datagrams. Although if something is happening to temporarily provoke exactly that behavior, it might explain the 1500 byte packets, so that's definitely a lead. Maybe it would be possible for me to peek at the stream from various different points and establish who is doing the fragmenting. It could be that if Debian is basically disregarding the 64k setting and using only 4k packets, it's simply not hitting whatever large-packet bad behavior that is harming FreeBSD. However it also performs better in the server role, with the client requesting the larger packets. So that's not definitive. > Yea, looking at this case in wireshark might make what is going on > apparent. Possibly, but that would likely have to be done by someone with more NFS protocol familiarity than I. Also, the incorrect checksums on outbound packets are normal because the interface supports checksum offloading. The checksum simply hasn't been calculated yet when tcpdump sees it. Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABXB=RToav%2B%2BV38pOorVPWpgZSuYmL-x7e8oxd3ayJCmAtLn-g>