From owner-freebsd-current@FreeBSD.ORG Mon Mar 8 17:50:00 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9CD33106566B for ; Mon, 8 Mar 2010 17:50:00 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id 2883D8FC18 for ; Mon, 8 Mar 2010 17:49:59 +0000 (UTC) Received: by fxm23 with SMTP id 23so5028505fxm.3 for ; Mon, 08 Mar 2010 09:49:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=334g2wsQbw29WWPSBruXjuQfbWDVaIY6mr+SSzMW/bs=; b=vHVwXjYHN3s6Pg27Eqe+l7VACDOXBFfk6n/mM10hqS2aW/ywcJ3GXnArBxsqWPiSWm aOgUz/FazGltETrw+saZDLpiKvWX7K8fnwYXmG5g04xThc0aIYIC6AUVRskdVjWcFp5n PqAWHlTsSAguXGG1SRJyU9wWDZF3e/5oMBAqA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=MqqkmA44Wd6codKB5VqWrjf2fs555wctxXOYSKVoaSdAd1RAX+sxxInwiRlfoFFokq OZi3dDVFEEtruIcS/8C3+Kv6tbjoFdfcPCXsC/CaehRZMQt1Yv9Vy1wTexAjWb7wqcGS xPFvOMMPYkSwa9t+adfwXka19cmVTxIBiKpxg= Received: by 10.87.15.18 with SMTP id s18mr7284624fgi.18.1268070598913; Mon, 08 Mar 2010 09:49:58 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 3sm5034334fge.0.2010.03.08.09.49.52 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 08 Mar 2010 09:49:54 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 8 Mar 2010 09:49:49 -0800 From: Pyun YongHyeon Date: Mon, 8 Mar 2010 09:49:49 -0800 To: Ian FREISLICH Message-ID: <20100308174949.GA1311@michelle.cdnetworks.com> References: <20100305215539.GG14818@michelle.cdnetworks.com> <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Mar 2010 17:50:00 -0000 On Mon, Mar 08, 2010 at 04:45:20PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > > > Pyun YongHyeon wrote: > > > > Thanks for the info. Frankly, I have no idea how to explain the > > > > issue given that you have no heavy load. > > > > > > How many cores would be involved in handling the traffic and runnig > > > PF rules on this machine? There are 4x > > > CPU: Quad-Core AMD Opteron(tm) Processor 8354 (2194.51-MHz K8-class CPU) > > > In this server. I'm also using carp extensively. > > > > > > > pf(4) uses a single lock for processing, number of core would have > > no much benefit. > > What's interesting is the effect on CPU utilisation and interrupt > generation that net.inet.ip.fastforwarding has: > > net.inet.ip.fastforwarding=1 > interrupt rate is around 10000/s per bce interface > cpu 8.0% interrupt > Yes, this is one of intentional change of the patch. Stock bce(4) seems to generate too much interrupts on BCM5709 so I rewrote interrupt handling with the help of David. sysctl nodes are also exported to control interrupt moderation so you can change them if you want. Default value was tuned to generate interrupts less than 10k per second and try to minimize latencies. > net.inet.ip.fastforwarding=0 > interrupt rate is around 5000/s per bce interface > cpu 13.0% interrupt > It also appears to not drop packets, but I'll have to watch it for longer. > Hmm, actually that's not what I originally expected. :-) The patch replaced some suspicious memory barrier instructions with bus_dmamap_sync(9) and you may see the effect.