From owner-freebsd-current@FreeBSD.ORG Fri Mar 5 18:41:14 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE3EC106564A for ; Fri, 5 Mar 2010 18:41:14 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id 4B02A8FC0A for ; Fri, 5 Mar 2010 18:41:13 +0000 (UTC) Received: by fxm23 with SMTP id 23so2924260fxm.3 for ; Fri, 05 Mar 2010 10:41:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=YOufFhZajYTufitynUmCExnb3xNDE/nnTXP38GtYjDM=; b=KfQl0Z2toK+7gUE3wbPhRaz9bUggDL6P7shRtcE83wpGWraY+qaYYOyYirOSEY4UVe pLZ+l4NX1p6HMkbeT0O30EpPHd7vcXcEd4AkGva4FUQ1lJ8MhExfVRjdbnhainYh7xMl oTYtRa7uIJz3jApXnl/eZCyyHU0MoxIhtLoeg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=TvLCffEYZF0t0MFkk8G0Kvcf8PYBH5D1tsp78Qv2mEGmi8TavE4D9KvXexdcRmHQCN Vix8bKFmmmvU9QxwREZ6R2bpgASyI+VoZWlTvcGpmSk5xJ/YuATbgJdGGBccKWYPQPJa /ehQKfXkw9z5R2Ti7xnbj/fzObTw0Cn1YeBPQ= Received: by 10.87.20.13 with SMTP id x13mr930407fgi.67.1267814464400; Fri, 05 Mar 2010 10:41:04 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id l19sm1081437fgb.11.2010.03.05.10.41.01 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 05 Mar 2010 10:41:02 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 5 Mar 2010 10:40:46 -0800 From: Pyun YongHyeon Date: Fri, 5 Mar 2010 10:40:46 -0800 To: Ian FREISLICH Message-ID: <20100305184046.GD14818@michelle.cdnetworks.com> References: <20100305175639.GB14818@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 18:41:14 -0000 On Fri, Mar 05, 2010 at 08:16:31PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Fri, Mar 05, 2010 at 01:20:57PM +0200, Ian FREISLICH wrote: > > > Hi > > > > > > I have a system that is experiencing mild to severe packet loss. > > > The interfaces are configured as follows: > > > > > > lagg0: bce0, bce1, bce2, bce3 lagproto lacp > > > > > > lagg0 then is used as the hwdev for the vlan interfaces. > > > > > > I have pf with a few queues for bandwidth management. > > > > > > There isn't that much traffic on it (200-500Mbit/s). > > > > > > I see only the following suspect for packet loss: > > > > > > dev.bce.0.com_no_buffers: 140151466 > > > dev.bce.1.com_no_buffers: 514723247 > > > dev.bce.2.com_no_buffers: 10454050 > > > dev.bce.3.com_no_buffers: 369371 > > > > > > Most of the time, these numbers are static, but every once in a > > > while they increase massively by several thousand, but only on 2 > > > interfaces. The 1 minute average rate on those interfaces is 266/s > > > and 123/s. > > > > > > Does anyone think this is related to the packet loss or are these > > > counters just a red herring? Is there anything that can be done > > > to reduce this count? > > > > > > > I think this sysctl node indicates number of dropped frames in > > completion processor of NetXtreme II. The counter is incremented > > when the processor received a frame successfully but it couldn't > > pass the frame to system as there are no available RX buffers so > > completion processor dopped the received frame. > > If you see mbuf shortage from netstat that would be normal. But if > > system has a lot of free mbuf resources it may indicate other > > issue. bce(4) may not be able to replenish controller with RX > > buffer if system is suffering from high load. > > I don't think I've ever seen an mbuf shortage on this host, and > load isn't that high, typically 12% CPU or 88% idle. That's just > on 2 (of 16) cores busy. There's tons of free memory (~12G) if I > need to increase the number of buffers available, but I'm not sure > which tunable to use to do that. The routing table also isn't large > at about 4000 prefixes. > > [firewall1.jnb1] ~ # netstat -m > 4118/7147/11265 mbufs in use (current/cache/total) > 3092/6850/9942/131072 mbuf clusters in use (current/cache/total/max) > 2060/4212 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/678/678/65536 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/32768 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/16384 16k jumbo clusters in use (current/cache/total/max) > 7214K/18198K/25412K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > I currently set the following in loader.conf: > > net.isr.maxthreads="8" > net.isr.direct=0 > if_igb_load="yes" > kern.ipc.nmbclusters="131072" > kern.maxusers="1024" > Would you show me the output of dmesg(bce(4)/brgphy(4) only) and the output of "pciconf -lcbv" for the controller?