From owner-freebsd-net@FreeBSD.ORG Thu May 8 08:32:59 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 77F461065674 for ; Thu, 8 May 2008 08:32:59 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id C02188FC21 for ; Thu, 8 May 2008 08:32:58 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 68376 invoked from network); 8 May 2008 07:34:38 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 8 May 2008 07:34:38 -0000 Message-ID: <4822BABB.4020407@freebsd.org> Date: Thu, 08 May 2008 10:32:59 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Deng XueFeng References: <20080506133208.C2EC.B627C632@gmail.com> <48223324.6070203@freebsd.org> <20080508120410.70E4.B627C632@gmail.com> In-Reply-To: <20080508120410.70E4.B627C632@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Mark Hills Subject: Re: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 May 2008 08:32:59 -0000 Deng XueFeng wrote: > hi, > the patch can not apply to 6.2, cound do a new patch for 6.2 or 6.3 ? The logging function is not (yet) present in RELENG_6. I'll post the patch when I've backported the functionality. However it's an important information that it happens on 6.2 too. That means the source of the trouble wasn't introduced with 7.0. -- Andre >> I've looked at the code paths again. There are two possibilities: >> >> a) the mbuf allocator has some anomaly where it rejects memory requests >> but doesn't update the statistics (the code is there however). >> >> b) the error doesn't come from the mbuf allocation but from ip_output() >> and further down the chain. >> >> To differentiate please try this updated patch and report the log output >> again (don't forget to set net.inet.tcp.log_debug=1): >> >> http://people.freebsd.org/~andre/tcp_output-error-log.diff >> >> -- >> Andre >> >> Deng XueFeng wrote: >>> hi >>> I'am also meet this problem in my mss server(missey streaming server). >>> one encoder push stream to mss, then run 100 client player playing the >>> sream, as the client number increase, mss will occur this error sooner or later >>> like this: >>> I'am using kqueue, and will got a event with EV_EOF and fflags = >>> ETIMEDOUT, >>> if i ignore EV_EOF flag, then ETIMEDOUT will be return by read(2), >>> >>> and the tcpdump also show that server will send RST packet to encoder. >>> >>> >>>> Hello, >>>> >>>> I'm are having a trouble with TCP connections being dropped with "read: >>>> Operation timed out". What is unusual is that this is happening right in >>>> the middle of sending a steady stream of data with no network congestion. >>>> >>>> The system is FreeBSD 7 and a bespoke streaming server with 1Gbit >>>> connection. The server receives a 192kbps inbound stream over TCP, and >>>> broadcasts it over a large number of TCP streams. >>>> >>>> With no visible or obvious pattern, the inbound read() fails with >>>> ETIMEDOUT. The likelihood of this happening seems to increase as the >>>> number of audience connections increases. It's happens every few minutes >>>> even with a small audience (eg. 300 outbound connections and about >>>> 60mbit). >>>> >>>> It doesn't cough and splutter -- steady data is coming in, then it just >>>> drops the connection. >>>> >>>> systat doesn't show problems inbound; all packets received are delivered >>>> to the upper layer. But on outbound, there is consistent 'output drops': >>>> >>>> IP Output >>>> 7028 total packets sent >>>> 7028 - generated locally >>>> 314 - output drops >>>> >>>> As the number of outbound connections increases, the 'output drops' >>>> increases to around 10% of the total packets sent and maintains that >>>> ratio. There's no problems with network capacity. >>>> >>>> I've tried different servers, different network interfaces (bge, em), >>>> different kernel (7-RELEASE, 7-STABLE). Have also checked dev.bge.0.stats >>>> and dev.em.0.stats for CRC errors etc. which show no problems. 'netstat >>>> -m' doesn't show any reaching of mbuf and sbuf limits. The problem is seen >>>> in a dedicated, uncontended test environment. >>>> >>>> Can anyone explain why the packets are being dropped outbound, and how >>>> this could affect inbound TCP data in such an abrupt way? What can I do to >>>> solve this? >>>> >>>> Thanks, >>>> >>>> Mark >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >