Date: Thu, 23 Jan 2014 21:34:10 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: J David <j.david.lists@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <1858487115.15503128.1390530850665.JavaMail.root@uoguelph.ca> In-Reply-To: <CABXB=RToav%2B%2BV38pOorVPWpgZSuYmL-x7e8oxd3ayJCmAtLn-g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
J. David wrote: > On Wed, Jan 22, 2014 at 11:12 PM, Rick Macklem <rmacklem@uoguelph.ca> > wrote: > > So, do you consider the 32K results as reasonable or terrible > > performance? > > (They are obviously much better than 64K, except for the reread > > case.) > > It's the 64k numbers that prompted the "terrible" thread title. A > 196K/sec write speed on a 10+ Gbit network is pretty disastrous. > > The 32k numbers are, as you say, better. Possibly reasonable, but > I'm > not sure if they're optimal. It's hard to tell the latency of the > virtual network, which would be needed to make that determination. > It > would be best if FreeBSD out of the box blows the doors of Debian out > of the box and FreeBSD tuned to the gills blows the doors off of > Debian tuned to the gills. Right now, Debian seems to be the one > with > the edge and with FreeBSD's illustrious history as the NFS > performance > king for so many years, that just won't do. :) > Btw, did you see this recent thread, where disabling TSO resolved the problem. http://docs.FreeBSD.org/cgi/mid.cgi?2C287272-7B57-4AAD-B22F-6A65D9F8677B rick > > Btw, I don't think you've mentioned what network device driver gets > > used > > for this virtual environment. That might be useful, in case the > > maintainer > > of that driver is aware of some issue/patch. > > KVM uses virtio. > > >> 00:38:07.932732 IP (tos 0x0, ttl 64, id 38912, offset 0, flags > >> [DF], > >> proto TCP (6), length 53628) > >> > > I don't know why this would be so large. A 32K write should be > > under > > 33Kbytes in size, not 53Kbytes. I suspect tcpdump is confused? > > Since TCP is stream oriented, is there a reason to expect 1:1 > correlation between NFS writes and TCP packets? > > > Well, it seems Debian is doing 4096 byte writes, which won't have > > anywhere > > near the effect on the network driver/virtual hardware that a 64K > > (about > > 45 IP datagrams) in one NFS RPC will. > > Debian's kernel says it is doing 64k reads/writes on that mount. So > again, possibly an expectation of 1:1 correlation between NFS writes > and TCP packets is not being satisfied. > > However, iozone is doing 4k reads/writes for these tests, so it's > also > possible that Debian is not coalescing them at all (which FreeBSD > apparently is) and the 4k writes are hitting the virtual wire as-is. > > Also, both sides have TSO and LRO, so it would be surprising (and > incorrect?) behavior if a 64k packet were actually fragmented into 45 > IP datagrams. Although if something is happening to temporarily > provoke exactly that behavior, it might explain the 1500 byte > packets, > so that's definitely a lead. Maybe it would be possible for me to > peek at the stream from various different points and establish who is > doing the fragmenting. > > It could be that if Debian is basically disregarding the 64k setting > and using only 4k packets, it's simply not hitting whatever > large-packet bad behavior that is harming FreeBSD. However it also > performs better in the server role, with the client requesting the > larger packets. So that's not definitive. > > > Yea, looking at this case in wireshark might make what is going on > > apparent. > > Possibly, but that would likely have to be done by someone with more > NFS protocol familiarity than I. > > Also, the incorrect checksums on outbound packets are normal because > the interface supports checksum offloading. The checksum simply > hasn't been calculated yet when tcpdump sees it. > > Thanks! > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1858487115.15503128.1390530850665.JavaMail.root>