Date: Mon, 28 Jun 2010 07:56:00 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: "Rick C. Petty" <rick-freebsd2009@kiwi-computer.com> Cc: Rick Macklem <rmacklem@uoguelph.ca>, freebsd-stable@freebsd.org Subject: Re: Why is NFSv4 so slow? Message-ID: <20100628145600.GA92144@icarus.home.lan> In-Reply-To: <20100628142025.GB52174@kay.kiwi-computer.com> References: <20100627221607.GA31646@kay.kiwi-computer.com> <Pine.GSO.4.63.1006271949220.3233@muncher.cs.uoguelph.ca> <20100628031401.GA45282@kay.kiwi-computer.com> <20100628034741.GA45748@kay.kiwi-computer.com> <20100628045852.GA75380@icarus.home.lan> <20100628142025.GB52174@kay.kiwi-computer.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jun 28, 2010 at 09:20:25AM -0500, Rick C. Petty wrote: > > > > > > Again, my ports tree is mounted as FSType nfs with option nfsv4. > > > FreeBSD/amd64 8.1-PRERELEASE r208408M GENERIC kernel. > > > > This sounds like NFSv4 is "tickling" some kind of bug in your NIC driver > > but I'm not entirely sure. Can you provide output from: > > > > 1) ifconfig -a (you can X out the IPs + MACs if you want) > > > On one of the clients: > > re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC> > ether e0:cb:4e:cd:d3:XX > inet 172.XX.XX.9 netmask 0xffffff00 broadcast 172.XX.XX.255 > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=3<RXCSUM,TXCSUM> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet6 ::1 prefixlen 128 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > > > 2) netstat -m > > server: > > 1739/1666/3405 mbufs in use (current/cache/total) > 257/1257/1514/25600 mbuf clusters in use (current/cache/total/max) > 256/547 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/405/405/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 948K/4550K/5499K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > client: > > 264/2046/2310 mbufs in use (current/cache/total) > 256/1034/1290/25600 mbuf clusters in use (current/cache/total/max) > 256/640 mbuf+clusters out of packet secondary zone in use (current/cache) > 3/372/375/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 590K/4067K/4657K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > > 3) vmstat -i > > Server: > > interrupt total rate > irq1: atkbd0 24 0 > irq18: atapci1 1883933 0 > irq20: nfe0 ohci1 1712603504 793 > cpu0: timer 4315963536 1999 > irq256: hdac0 12 0 > irq257: ahci0 139934363 64 > cpu2: timer 4315960172 1999 > cpu1: timer 4315960172 1999 > cpu3: timer 4315960172 1999 > Total 19118265888 8858 > > Client: > > interrupt total rate > irq1: atkbd0 1063022 0 > irq16: hdac0 16013959 6 > irq17: atapci0+++ 6 0 > irq18: ohci0 ohci1* 5324486 2 > irq19: atapci1 7500968 2 > irq20: ahc0 19 0 > irq21: ahc1 112390 0 > cpu0: timer 5125670841 1999 > irq256: hdac1 2 0 > irq257: re0 742537149 289 > cpu1: timer 5125664297 1999 > Total 11023887139 4301 > > > 4) prtconf -lvc (only need the Ethernet-related entries) > > I'll assume you meant to type "pciconf", on the server: Yes sorry -- I spend my days at work dealing with Solaris (which is where where prtconf comes from :-) ). Three other things to provide output from if you could (you can X out IPs and MACs too), from both client and server: 6) netstat -idn 7) sysctl hw.pci | grep msi 8) Contents of /etc/sysctl.conf Thanks. > server, immediately after restarting all of nfs scripts (rpcbind > nfsclient nfsuserd nfsserver mountd nfsd statd lockd nfscbd): > > Jun 27 18:04:44 rpcbind: cannot get information for udp6 > Jun 27 18:04:44 rpcbind: cannot get information for tcp6 These two usually indicate you removed IPv6 support from the kernel, except your ifconfig output (I've remove it) on the server shows you do have IPv6 support. I've been trying to get these warnings removed for quite some time (PR kern/96242). They're harmless, but the inconsistency here is a little weird -- are you explicitly disabling IPv6 on nfe0? The remaining messages in your kernel log Rick can probably explain. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100628145600.GA92144>