From owner-freebsd-performance@FreeBSD.ORG Wed Apr 9 16:07:49 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3DCB837B401; Wed, 9 Apr 2003 16:07:49 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id B13EC43F75; Wed, 9 Apr 2003 16:07:48 -0700 (PDT) (envelope-from sean@perrin.int.nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1001) id 7ABF320F00; Wed, 9 Apr 2003 16:07:33 -0700 (PDT) Date: Wed, 9 Apr 2003 16:07:33 -0700 From: Sean Chittenden To: "Jin Guojun [DSD]" Message-ID: <20030409230733.GX79923@perrin.int.nxad.com> References: <3E94A22D.174321F0@lbl.gov> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3E94A22D.174321F0@lbl.gov> User-Agent: Mutt/1.4i X-PGP-Key: finger seanc@FreeBSD.org X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341 X-Web-Homepage: http://sean.chittenden.org/ cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: tcp_output starving -- is due to mbuf get delay? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2003 23:07:49 -0000 > When testing GigE path that has 67 ms RTT, the maximum TCP throughput is > limited at 250 Mb/s. By tracing the problem, I found that tcp_output() is > starving > where snd_wnd and snd_cwnd are fully open. The snd_cc is never filled beyond > the 4.05MB even though the snd_hiwat is 10MB and snd_sbmax is 8MB. That is, > sosend never stopped at sbwait. So only place can slow down is the mbuf > allocation > in sosend(). The attached trace file shows that each MGET and MCLGET takes > significant time -- around 8 us at slow start time, and gradually increasing > after that > in an range 18 to 648. > Each packet Tx on GigE takes 12 us. It average mbuf allocation takes 18 us, then > > the performance will be reduced to 40%, in fact it is down to 25%, which means > higher average delay. > > I have change NMBCLUSTER from 2446 to 6566 to 10240, and nothing is improved. > > Any one can tell what factors would cause MGET / MCLGET to wait? > Is there any way to make MGET/MCLGET not to wait? Luigi posted a patch about this a while back (last summer sometime, iirc). http://people.freebsd.org/~seanc/patches/#o1_mbuf_lookup I updated his patch but haven't had a chance to test it. If you're feeling brave, see if applying this patch fixes this bottle neck. -sc -- Sean Chittenden