Date: Sun, 16 Jan 2011 16:55:24 -0800 From: Pyun YongHyeon <pyunyh@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-net@freebsd.org, Ronald Klop <ronald-freebsd8@klop.yi.org> Subject: Re: bogus 0 len IP packet, was: Hang in VOP_LOCK1_APV on 8-STABLE with NFS. Message-ID: <20110117005524.GA1305@michelle.cdnetworks.com> In-Reply-To: <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca> References: <op.vpekz9uz8527sy@212-123-145-58.ip.telfort.nl> <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Jan 16, 2011 at 08:54:59AM -0500, Rick Macklem wrote: > Ronald has reported having a problem with the FreeBSD NFS client using > 8.2-prerelease. I've redirected it here, since it looks like there is > a TCP/IP issue that is causing it. > > > > > >> > > >> These are the links to the dumps: > > >> http://klop.ws/~ronald/nfs-problem/procstat.nolockd > > >> http://klop.ws/~ronald/nfs-problem/ps.nolockd > > >> http://klop.ws/~ronald/nfs-problem/linux5.nfs.nolockd.dump.gz > > >> http://klop.ws/~ronald/nfs-problem/linux5.nfs.with_rpc_patch.dump.gz > > >> > > I looked at the last of these via wireshark and it seems that the FreeBSD > client is sending bogus TCP/IP packets with a IP length == 0. (If you look > at the above dump (...with_rpc_patch.dump.gz), the first one is at packet #46, > then #3024, then persistently starting at #3234.) Basically the packet looks > like: > frame len: 4446 > MAC: dst 00:0d:56:70:b7:6c src b8:ac:6f:47:73:6e type: 08 00 (IP) > IP > version: 4 > header length: 20 > differentiated services field: 0x00 > total length: 0 > - followed by what looks like a legitimate TCP/IP packet > > Here's the first bytes of the raw packet data: > 00 0d 56 70 b7 6c b8 ac 6f 47 73 6e 08 00 > 45 00 00 00 ... > > After this packet is sent to the Linux server, it replies with a TCP ack, > which gets ACK'd from FreeBSD as well. For the persistent case, it just > keeps doing this (bogus 0 length packet from FreeBSD -> Linux server, > followed by the two TCP ack packets) over and over and over again, to > the end of the dump. > > So, does anyone have an idea why the IP length field would be set to 0 > for these TCP/IP packets? > > Here's some info from Ronald w.r.t. his hardware. (All I can think of is > that he could try disabling TSO, etc?) > > Thanks in advance for any help with this, rick > It seems that issue came from TSO. Driver will set ip_len and ip_sum field to 0 before passing the TCP segment to controller. The failed length were 4446, 5858, 3034 and 4310 and the total number of such frames are more than 35k within 90 seconds. Since failed length 4310 is continuously repeated I guess there is edge case where em(4) didn't free failed TCP segment for TSO. I remember there was commit to HEAD(r217295) which could be related with this issue. > > > > I just looked at the last dump and there seems to be a network > > > issue. > > > (It first shows up at packet #46, then again at #3025, then > > > persistently > > > starting at #3234.) > > > > > > I'd like to post on freebsd-net@ to see if anyone more conversant > > > with > > > TCP/IP can look, but first I'd like to get a little more info on > > > your > > > hardware/software config. > > > > > > In particular, what network hardware does the FreeBSD client use? > > > > > > And I assume the server is some variant of Linux? > > > > > > Thanks for creating the tcpdumps, rick > > > ps: If you look on wireshark, the problem seems to start with a > > > badly formed IP datagram that then causes acks in both > > > directions. > > > > We are getting off list now. I don't know how good that is. > > > I've redirected it to freebsd-net@ in the hopes that networking folks > can help. > > > But here is some info. I also noticed the bad packets. And my computer > > is > > resending the same info at quite a high rate (MB/s). > > The server is an up-to-date Linux Debian 5 with a 2.6.26-1-686-bigmem > > kernel. Colleagues using Linux clients don't have these problems. > > > > dmesg | grep em0 > > em0: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0xece0-0xecff > > mem > > 0xf7fe0000-0xf7ffffff,0xf7fd9000-0xf7fd9fff irq 21 at device 25.0 on > > pci0 > > em0: Using an MSI interrupt > > em0: [FILTER] > > em0: Ethernet address: b8:ac:6f:47:73:6e > > > > pciconf -lv > > em0@pci0:0:25:0: class=0x020000 card=0x02761028 chip=0x10de8086 > > rev=0x02 hdr=0x00 > > vendor = 'Intel Corporation' > > device = 'Intel Gigabit network connection (82567LM-3 )' > > class = network > > subclass = ethernet > > > > > > [root@ronald ~]# ifconfig em0 > > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > > 1500 > > options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC> > > ether b8:ac:6f:47:73:6e > > inet 10.1.20.49 netmask 0xffffff00 broadcast 10.1.20.255 > > media: Ethernet autoselect (100baseTX <full-duplex>) > > status: active > > > > Thanks for looking into this. If you need more just let me know. I can > > also reproduce the problem and send nfsstat or netstat output or .... > > > > Thanks, > > Ronald.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110117005524.GA1305>