Date: Sun, 02 Dec 2007 20:28:09 +0100 From: Vitezslav Novy <vnovy@vnovy.net> To: freebsd-stable@freebsd.org Subject: Re: 7.0 BETA3 - slow TCP upload (TSO related?) Message-ID: <47530749.7030809@vnovy.net> In-Reply-To: <2a41acea0711301012j4472074rbc4dadd2cf661173@mail.gmail.com> References: <474FB9E5.9090208@vnovy.net> <2a41acea0711300016w1aa144fel1641b3dbf075e043@mail.gmail.com> <474FD1CB.3020601@FreeBSD.org> <2a41acea0711301012j4472074rbc4dadd2cf661173@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>>> On Nov 29, 2007 11:21 PM, Vitezslav Novy <vnovy@vnovy.net> wrote: >>>> Hello, >>>> >>>> my configuration is >>>> >>>> kernel GENERIC >>>> >>>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>>> options=18b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM,TSO4> >>>> ether 00:19:d1:0f:1c:18 >>>> inet 86.49.14.16 netmask 0xffffff00 broadcast 86.49.14.255 >>>> media: Ethernet autoselect (100baseTX <full-duplex>) >>>> status: active >>>> >>>> and standard "open" ipfw firewall and >>>> natd -u -s -m -d -dynamic -n em0 >>>> >>>> I experience very slow TCP upload from this host - cca 50kbps. >>>> I have some debug prints in kernel (mostly in ip_output and ipfw log) >>>> and I see: >>>> >>>> 1/ outgoing packet appears in ip_output with ip_len 2924 and >>>> m->pkthdr.csum_flags=1 >>>> 2/ is diverted by firewall >>>> 3/ Packet appears immediately again in ip_output with ip_len 2924 and >>>> m->pkthdr.csum_flags=1 >>>> 4/ Packet is accepted by firewall and dropped by ip_output with error 40 >>>> EMSGSIZE >>>> 5/ After cca 0.4s (tcp retransmit timeout?) new packet appears in >>>> ip_output with ip_len 1488 and m->pkthdr.csum_flags=1 >>>> 6/ is successfully diverted and accepted by ipfw and sent to wire. >>>> 7/ after tcp ack is received new packet appears in ip_output with ip_len >>>> 2924 and everything repeats >>>> >>>> >>>> Packets are not changed by natd, beacause have src address of em0. >>>> nat >>>> >>>> >>>> Upload has normal speed (512kbps) if >>>> I unset TSO on interface OR set net.inet.tcp.tso=0 OR (strange thing) >>>> delete ipfw divert rule >>>> >>>> If necessary I will collect and send more info. >>> TSO is silly at 100Mb, turn it off :) After more debugging everything looks clear. Problem is TSO+divert related. TCP layer sends large packet with CSUM_TSO set, packet is diverted and ip_output returns 0 to TCP layer. When packet is reinjected into ip_output CSUM_TSO flag is lost. Packet is dropped by ip_output with error EMSGSIZE, but this error is propagated to natd, which cannot do anything with it. After retransmit timeout, TCP layer send packet again and because it is retransmit, TSO is not used and packet is successfully sent . Because TCP layer has no feedback about problem with TSO, next packet is sent with TSO flag again. I'm not sure if it is possible to protect CSUM_TSO flag during divert process. I tested simple patch which makes ipfw to refuse divert packet with CSUM_TSO flag a returns EMSGSIZE immediately. It works well for me. Maybe it can be direction where solution can be found, but it can break use of divert for purposes other than natd. vita
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47530749.7030809>