From owner-freebsd-net@FreeBSD.ORG Sun Aug 22 22:27:50 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 615D410656A9 for ; Sun, 22 Aug 2010 22:27:50 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2F6F18FC16 for ; Sun, 22 Aug 2010 22:27:49 +0000 (UTC) Received: by pvg4 with SMTP id 4so2285586pvg.13 for ; Sun, 22 Aug 2010 15:27:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=XcZjRPOVuG7lLCLY7boumanDYARaqJh1vaG5+MP1zS4=; b=OMIPC9Hi2zQubyFsHimbjMP64dJWz6KkKx5W0aUeaL6mefjegZx0bk+PLu0oIEfYxF w8jW1Jt1lVOMvDPsLRyl+TRg5sivrQwhqZEMc+ZD5/vGYFSUG0njbL15Qd+PVDG6sbTP g7sYJ1YZpeKg05Hp0bxtpNKK6aWA859QQXEjU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=KG3ybSodF+fzvHXnZ1SW1tMPXE/ryy00j6id5RLlkXuZP005kvimP8w7cM2EZ6o178 qIGC1ZPH4ASLroJeOciskhKjgk5rbKs8IHnSQ6V83qvySRaY+5dB7fUwDstRj38oN7Wj HZsKnBgdo0r6sMI/v+kwVL4Ls7MfhmDGhOyN0= Received: by 10.114.12.14 with SMTP id 14mr5007900wal.23.1282516069713; Sun, 22 Aug 2010 15:27:49 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id d35sm11119733waa.9.2010.08.22.15.27.47 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 22 Aug 2010 15:27:48 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Sun, 22 Aug 2010 15:27:46 -0700 From: Pyun YongHyeon Date: Sun, 22 Aug 2010 15:27:46 -0700 To: Adrian Chadd Message-ID: <20100822222746.GC6013@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: 8.0-RELEASE-p3: 4k jumbo mbuf cluster exhaustion X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Aug 2010 22:27:50 -0000 On Sun, Aug 22, 2010 at 05:40:30PM +0800, Adrian Chadd wrote: > I disabled tso, tx chksum and rx chksum. This fixed the 4k jumbo > allocation growth. > I recall there was SIOCSIFCAP ioctl handling bug in bce(4) on 8.0 so it might also disable IFCAP_TSO4/IFCAP_TXCSUM/IFCAP_RXCSUM when yo disabled RX checksum offloading. But I can't explain how checksum offloading could be related with the growth of 4k jumbo buffers. > > Turning on tso on a live proxy didn't affect jumbo allocations. > Turning on txcsum caused jumbo allocations to begin growing again. > DIsabling txcsum again caused jumbo allocations to stop increasing, > but it doesn't seem to be decreasing back to the steady state (~ 8k.) > Turning on rxcsum didn't affect jumbo allocations. > > So it seems txcsum is the culprit here. > There was a lot of changes in bce(4) since 8.0-RELEASE. I vaguely guess your issue could be related with header split feature of bce(4) which was now disabled. Are you using jumbo frame/ZERO_COPY_SOCKETS with bce(4)? > > > Adrian > > On 22 August 2010 16:11, Adrian Chadd wrote: > > Hi, > > > > I've got a Squid/Lusca server on 8.0-RELEASE-p3 which is exhibiting > > some very strange behaviour. > > > > After a few minutes uptime, the 4k mbuf cluster zone fills up and > > Squid/Lusca spends almost all of it's time sleeping in "keglimit". > > > > I've bumped kern.ipc.nmbclusters to 262144 and kern.ipc.jumbop to > > 32768 but the system will slowly crawl towards filling that zone. > > > > The box has a bce on-board NIC and is using ipfw to handle redirecting > > traffic to/from the box for transparent TCP interception. It's > > handling around ~30,000 concurrent connections at the moment. > > > > I have other very busy proxies on FreeBSD-7.x pushing a few hundred > > megabits without any issues. This box falls over after ~ 20 mbit. > > > > If I bypass redirection and/or kill squid, the 4k cluster count drops > > back down to < 500 and stays there. > > > > Does anyone have any ideas on where to begin debugging this? > > > > Thanks, > > > > > > Adrian > >