Date: Tue, 27 Apr 1999 02:35:05 -0700 (PDT) From: Matthew Dillon <dillon@apollo.backplane.com> To: David Greenman <dg@root.com>, Julian Elischer <julian@whistle.com>, luoqi@watermarkgroup.com, alc@cs.rice.edu, hackers@freebsd.org Subject: Found TCP NFS bug in packet trace. Looks like nfs_realign() is broken Message-ID: <199904270935.CAA00567@apollo.backplane.com>
next in thread | raw e-mail | index | archive | help
I direct everyone's attention to the '30fb 64XX' sequence in the below
packet trace. This demonstrates why TCP NFS links get stuck.
The TCP packet is encompassing *two* requests rather then one. This is
correct. However, the server is screwing up on it.
Note that in the NFS request, the NFS xid is '30fb 64dc' for the first
request and '30fb 64dd' for the second.
Note that in the NFS reply, *BOTH* xid's in the reply are 30fb 64dd.
30fb 64dc got destroyed! In fact, it looks like the server is sending
two copies of the 30fb 64dd response instead of a 30fb 64dc and 30fb 64dd
response.
I tracked it down to nfs_realign(). The routine tries to realign data
contained in an mbuf. It completely destroys one of the commands in the
mbuf while attempting to realign the other. The destruction occurs due
to the shared-data nature of mbufs when ( as far as I can tell ) the
mbuf is flagged M_EXT. If I turn nfs_realign() into a NOP, TCP mounts
appear to work again ( at least better then they did before. There is
still occassional weirdness ).
Doh!
I'll have a patch to fix this properly later on tuesday. This brings up
another problem, though - unaligned mbufs appear to be a regular occurance
w/NFS but the data *appears* to be aligned in the tcpdump. Is the MAC
header or TCP options causing the packet data to be unaligned?
I'm guessing that NFS is doing a lot of buffer realigning and its probably
having a negative effect on performance. I can rewrite the alignment code
to allocate fresh mbufs, but I would kinda like to know why the below tcp
packets appear to be unaligned from the point of view of the NFS code.
there might be a quick fix we can make somewhere to increase performance.
-Matt
Matthew Dillon
<dillon@backplane.com>
23:48:09.209937 209.157.86.12.1021 > 209.157.86.2.2049: P 100656:100920(264) ack 60585 win 17200 (DF)
4500 0130 1436 4000 4006 d648 d19d 560c
d19d 5602 03fd 0801 186c d0c5 7371 bbce
5018 4330 3ab3 0000 8000 0080 30fb 64dc <<<<<<<<<
0000 0000 0000 0002 0001 86a3 0000 0003
0000 0003 0000 0001 0000 0030 0000 0000
0000 0000 0000 0000 0000 0000 0000 0007
0000 0000 0000 0002 0000 0003 0000 0004
0000 0005 0000 0014 0000 001f 0000 0000
0000 0000 0000 001c 0700 0000 4216 9c77
0c00 0000 8e45 0300 6922 8679 0000 0000
0000 0000 0000 0004 6c69 623a 8000 0080
>>>>>>>> 30fb 64dd 0000 0000 0000 0002 0001 86a3
0000 0003 0000 0003 0000 0001 0000 0030
0000 0000 0000 0000 0000 0000 0000 0000
0000 0007 0000 0000 0000 0002 0000 0003
0000 0004 0000 0005 0000 0014 0000 001f
0000 0000 0000 0000 0000 001c 0700 0000
4216 9c77 0c00 0000 8e45 0300 6922 8679
0000 0000 0000 0000 0000 0003 6c69 6200
23:48:09.214561 209.157.86.2.2049 > 209.157.86.12.1021: P 60749:60989(240) ack 101116 win 33176 (DF)
4500 0118 7cdc 4000 4006 6dba d19d 5602
d19d 560c 0801 03fd 7371 bc72 186c d291
5018 8198 7102 0000 8000 0074 30fb 64dd <<<<<<<<
0000 0001 0000 0000 0000 0000 0000 0000
0000 0000 0000 0002 0000 0001 0000 0002
0000 01ed 0000 0003 0000 0000 0000 0000
0000 0000 0000 0200 0000 0000 0000 0400
0000 00c6 000d 0016 0000 0000 0000 0007
0000 0000 0003 458e 3725 5d9d 0000 0000
3725 5d9d 0000 0000 3725 5d9d 0000 0000
>>>>>>>> 8000 0074 30fb 64dd 0000 0001 0000 0000
0000 0000 0000 0000 0000 0000 0000 0002
0000 0001 0000 0002 0000 01ed 0000 0003
0000 0000 0000 0000 0000 0000 0000 0200
0000 0000 0000 0400 0000 00c6 000d 0016
0000 0000 0000 0007 0000 0000 0003 458e
3725 5d9d 0000 0000 3725 5d9d 0000 0000
3725 5d9d 0000 0000
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199904270935.CAA00567>