Date: Thu, 08 May 2008 10:32:59 +0200 From: Andre Oppermann <andre@freebsd.org> To: Deng XueFeng <dengxf@gmail.com> Cc: freebsd-net@freebsd.org, Mark Hills <mark@pogo.org.uk> Subject: Re: read() returns ETIMEDOUT on steady TCP connection Message-ID: <4822BABB.4020407@freebsd.org> In-Reply-To: <20080508120410.70E4.B627C632@gmail.com> References: <20080506133208.C2EC.B627C632@gmail.com> <48223324.6070203@freebsd.org> <20080508120410.70E4.B627C632@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Deng XueFeng wrote: > hi, > the patch can not apply to 6.2, cound do a new patch for 6.2 or 6.3 ? The logging function is not (yet) present in RELENG_6. I'll post the patch when I've backported the functionality. However it's an important information that it happens on 6.2 too. That means the source of the trouble wasn't introduced with 7.0. -- Andre >> I've looked at the code paths again. There are two possibilities: >> >> a) the mbuf allocator has some anomaly where it rejects memory requests >> but doesn't update the statistics (the code is there however). >> >> b) the error doesn't come from the mbuf allocation but from ip_output() >> and further down the chain. >> >> To differentiate please try this updated patch and report the log output >> again (don't forget to set net.inet.tcp.log_debug=1): >> >> http://people.freebsd.org/~andre/tcp_output-error-log.diff >> >> -- >> Andre >> >> Deng XueFeng wrote: >>> hi >>> I'am also meet this problem in my mss server(missey streaming server). >>> one encoder push stream to mss, then run 100 client player playing the >>> sream, as the client number increase, mss will occur this error sooner or later >>> like this: >>> I'am using kqueue, and will got a event with EV_EOF and fflags = >>> ETIMEDOUT, >>> if i ignore EV_EOF flag, then ETIMEDOUT will be return by read(2), >>> >>> and the tcpdump also show that server will send RST packet to encoder. >>> >>> >>>> Hello, >>>> >>>> I'm are having a trouble with TCP connections being dropped with "read: >>>> Operation timed out". What is unusual is that this is happening right in >>>> the middle of sending a steady stream of data with no network congestion. >>>> >>>> The system is FreeBSD 7 and a bespoke streaming server with 1Gbit >>>> connection. The server receives a 192kbps inbound stream over TCP, and >>>> broadcasts it over a large number of TCP streams. >>>> >>>> With no visible or obvious pattern, the inbound read() fails with >>>> ETIMEDOUT. The likelihood of this happening seems to increase as the >>>> number of audience connections increases. It's happens every few minutes >>>> even with a small audience (eg. 300 outbound connections and about >>>> 60mbit). >>>> >>>> It doesn't cough and splutter -- steady data is coming in, then it just >>>> drops the connection. >>>> >>>> systat doesn't show problems inbound; all packets received are delivered >>>> to the upper layer. But on outbound, there is consistent 'output drops': >>>> >>>> IP Output >>>> 7028 total packets sent >>>> 7028 - generated locally >>>> 314 - output drops >>>> >>>> As the number of outbound connections increases, the 'output drops' >>>> increases to around 10% of the total packets sent and maintains that >>>> ratio. There's no problems with network capacity. >>>> >>>> I've tried different servers, different network interfaces (bge, em), >>>> different kernel (7-RELEASE, 7-STABLE). Have also checked dev.bge.0.stats >>>> and dev.em.0.stats for CRC errors etc. which show no problems. 'netstat >>>> -m' doesn't show any reaching of mbuf and sbuf limits. The problem is seen >>>> in a dedicated, uncontended test environment. >>>> >>>> Can anyone explain why the packets are being dropped outbound, and how >>>> this could affect inbound TCP data in such an abrupt way? What can I do to >>>> solve this? >>>> >>>> Thanks, >>>> >>>> Mark >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4822BABB.4020407>