Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 28 Jun 2010 09:20:25 -0500
From:      "Rick C. Petty" <rick-freebsd2009@kiwi-computer.com>
To:        Jeremy Chadwick <freebsd@jdc.parodius.com>
Cc:        Rick Macklem <rmacklem@uoguelph.ca>, freebsd-stable@freebsd.org
Subject:   Re: Why is NFSv4 so slow?
Message-ID:  <20100628142025.GB52174@kay.kiwi-computer.com>
In-Reply-To: <20100628045852.GA75380@icarus.home.lan>
References:  <20100627221607.GA31646@kay.kiwi-computer.com> <Pine.GSO.4.63.1006271949220.3233@muncher.cs.uoguelph.ca> <20100628031401.GA45282@kay.kiwi-computer.com> <20100628034741.GA45748@kay.kiwi-computer.com> <20100628045852.GA75380@icarus.home.lan>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Jun 27, 2010 at 09:58:53PM -0700, Jeremy Chadwick wrote:
> > 
> > Again, my ports tree is mounted as FSType nfs with option nfsv4.
> > FreeBSD/amd64 8.1-PRERELEASE r208408M GENERIC kernel.
> 
> This sounds like NFSv4 is "tickling" some kind of bug in your NIC driver
> but I'm not entirely sure.  Can you provide output from:
> 
> 1) ifconfig -a  (you can X out the IPs + MACs if you want)

On the NFSv4 server:

nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8010b<RXCSUM,TXCSUM,VLAN_MTU,TSO4,LINKSTATE>
	ether 00:22:15:b4:2d:XX
	inet 172.XX.XX.4 netmask 0xffffff00 broadcast 172.XX.XX.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1 prefixlen 64 scopeid 0x2 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

On one of the clients:

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
	ether e0:cb:4e:cd:d3:XX
	inet 172.XX.XX.9 netmask 0xffffff00 broadcast 172.XX.XX.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

> 2) netstat -m

server:

1739/1666/3405 mbufs in use (current/cache/total)
257/1257/1514/25600 mbuf clusters in use (current/cache/total/max)
256/547 mbuf+clusters out of packet secondary zone in use (current/cache)
0/405/405/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
948K/4550K/5499K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

client:

264/2046/2310 mbufs in use (current/cache/total)
256/1034/1290/25600 mbuf clusters in use (current/cache/total/max)
256/640 mbuf+clusters out of packet secondary zone in use (current/cache)
3/372/375/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
590K/4067K/4657K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

> 3) vmstat -i

Server:

interrupt                          total       rate
irq1: atkbd0                          24          0
irq18: atapci1                   1883933          0
irq20: nfe0 ohci1             1712603504        793
cpu0: timer                   4315963536       1999
irq256: hdac0                         12          0
irq257: ahci0                  139934363         64
cpu2: timer                   4315960172       1999
cpu1: timer                   4315960172       1999
cpu3: timer                   4315960172       1999
Total                        19118265888       8858

Client:

interrupt                          total       rate
irq1: atkbd0                     1063022          0
irq16: hdac0                    16013959          6
irq17: atapci0+++                      6          0
irq18: ohci0 ohci1*              5324486          2
irq19: atapci1                   7500968          2
irq20: ahc0                           19          0
irq21: ahc1                       112390          0
cpu0: timer                   5125670841       1999
irq256: hdac1                          2          0
irq257: re0                    742537149        289
cpu1: timer                   5125664297       1999
Total                        11023887139       4301

> 4) prtconf -lvc  (only need the Ethernet-related entries)

I'll assume you meant to type "pciconf", on the server:

nfe0@pci0:0:10:0:	class=0x020000 card=0x82f21043 chip=0x076010de rev=0xa2 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'NForce Network Controller (MCP78 NIC)'
    class      = network
    subclass   = ethernet
    cap 01[44] = powerspec 2  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 16 messages, 64 bit, vector masks 
    cap 08[6c] = HT MSI fixed address window disabled at 0xfee00000

client:

re0@pci0:1:0:0:	class=0x020000 card=0x84321043 chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor'
    device     = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[70] = PCI-Express 2 endpoint IRQ 2 max data 128(256) link x1(x1)
    cap 11[b0] = MSI-X supports 4 messages in map 0x20
    cap 03[d0] = VPD

> 5) sysctl dev.XXX.N  (ex. for em0, XXX=em, N=0)

server:

dev.nfe.0.%desc: NVIDIA nForce MCP77 Networking Adapter
dev.nfe.0.%driver: nfe
dev.nfe.0.%location: slot=10 function=0 handle=\_SB_.PCI0.NMAC
dev.nfe.0.%pnpinfo: vendor=0x10de device=0x0760 subvendor=0x1043 subdevice=0x82f2 class=0x020000
dev.nfe.0.%parent: pci0
dev.nfe.0.process_limit: 192
dev.nfe.0.stats.rx.frame_errors: 0
dev.nfe.0.stats.rx.extra_bytes: 0
dev.nfe.0.stats.rx.late_cols: 0
dev.nfe.0.stats.rx.runts: 0
dev.nfe.0.stats.rx.jumbos: 0
dev.nfe.0.stats.rx.fifo_overuns: 0
dev.nfe.0.stats.rx.crc_errors: 0
dev.nfe.0.stats.rx.fae: 0
dev.nfe.0.stats.rx.len_errors: 0
dev.nfe.0.stats.rx.unicast: 1762645090
dev.nfe.0.stats.rx.multicast: 1
dev.nfe.0.stats.rx.broadcast: 7608
dev.nfe.0.stats.tx.octets: 2036479975330
dev.nfe.0.stats.tx.zero_rexmits: 2090186021
dev.nfe.0.stats.tx.one_rexmits: 0
dev.nfe.0.stats.tx.multi_rexmits: 0
dev.nfe.0.stats.tx.late_cols: 0
dev.nfe.0.stats.tx.fifo_underuns: 0
dev.nfe.0.stats.tx.carrier_losts: 0
dev.nfe.0.stats.tx.excess_deferrals: 0
dev.nfe.0.stats.tx.retry_errors: 0
dev.nfe.0.stats.tx.unicast: 0
dev.nfe.0.stats.tx.multicast: 0
dev.nfe.0.stats.tx.broadcast: 0
dev.nfe.0.wake: 0

client:

c: RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet
dev.re.0.%driver: re
dev.re.0.%location: slot=0 function=0
dev.re.0.%pnpinfo: vendor=0x10ec device=0x8168 subvendor=0x1043 subdevice=0x8432 class=0x020000
dev.re.0.%parent: pci1

> check "dmesg" to see if there's any messages the kernel has
> been spitting out which look relevant?  Thanks.

server, immediately after restarting all of nfs scripts (rpcbind nfsclient nfsuserd nfsserver mountd nfsd statd lockd nfscbd):

Jun 27 18:04:44 rpcbind: cannot get information for udp6
Jun 27 18:04:44 rpcbind: cannot get information for tcp6
NLM: failed to contact remote rpcbind, stat = 5, port = 28416
Jun 27 18:05:12 amanda kernel: NLM: failed to contact remote rpcbind, stat = 5, port = 28416

client, when noticing the mounting-over-directories problem:

NLM: failed to contact remote rpcbind, stat = 5, port = 28416
nfsv4 client/server protocol prob err=10020
nfsv4 client/server protocol prob err=10020
...

No other related messages were found in /var/log/messages either.

-- 


-- Rick C. Petty



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100628142025.GB52174>