From owner-freebsd-net@FreeBSD.ORG Sun Aug 22 09:40:31 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7266B1065696 for ; Sun, 22 Aug 2010 09:40:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3B7C48FC15 for ; Sun, 22 Aug 2010 09:40:30 +0000 (UTC) Received: by iwn36 with SMTP id 36so5325301iwn.13 for ; Sun, 22 Aug 2010 02:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=8AA4xc57oUdeOsWoX+qrESZcKfyjGYrUwvTcVff1viI=; b=YVyb3Bg1TnTInBUX5EMDE3GnlqmMBvZNa5Zj6XZSCFVOcRgLIXLHCplqBsJgzYm9xE h8NMiIXlVntXHo9BI1egw/8NJhuckZlhvZqLrxjjOUcj68dhtXgww5EvwjaCJTXGPBMX eFk+cPoC6uWhwauisU/crRHQ7HaSjqdreBIrA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=sY2Azj634kjKp+7y7VQTqgYyJDajn/k9KAmS9LSrh5aN5PhHf79tFxD7FeT/5cdNs0 DXNchBfc6PoQ1R5RzppxNZAv5mJ3p9jpZKdFgkuPKLngtOE0iivzzibBXfJuAKgsTiNE Ez6QVNbermfNSElBRpYc3bhhc0Nn5LIe958Sc= MIME-Version: 1.0 Received: by 10.231.148.195 with SMTP id q3mr4482456ibv.199.1282470030503; Sun, 22 Aug 2010 02:40:30 -0700 (PDT) Received: by 10.231.168.14 with HTTP; Sun, 22 Aug 2010 02:40:30 -0700 (PDT) In-Reply-To: References: Date: Sun, 22 Aug 2010 17:40:30 +0800 Message-ID: From: Adrian Chadd To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: 8.0-RELEASE-p3: 4k jumbo mbuf cluster exhaustion X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Aug 2010 09:40:31 -0000 I disabled tso, tx chksum and rx chksum. This fixed the 4k jumbo allocation growth. Turning on tso on a live proxy didn't affect jumbo allocations. Turning on txcsum caused jumbo allocations to begin growing again. DIsabling txcsum again caused jumbo allocations to stop increasing, but it doesn't seem to be decreasing back to the steady state (~ 8k.) Turning on rxcsum didn't affect jumbo allocations. So it seems txcsum is the culprit here. Adrian On 22 August 2010 16:11, Adrian Chadd wrote: > Hi, > > I've got a Squid/Lusca server on 8.0-RELEASE-p3 which is exhibiting > some very strange behaviour. > > After a few minutes uptime, the 4k mbuf cluster zone fills up and > Squid/Lusca spends almost all of it's time sleeping in "keglimit". > > I've bumped kern.ipc.nmbclusters to 262144 and kern.ipc.jumbop to > 32768 but the system will slowly crawl towards filling that zone. > > The box has a bce on-board NIC and is using ipfw to handle redirecting > traffic to/from the box for transparent TCP interception. It's > handling around ~30,000 concurrent connections at the moment. > > I have other very busy proxies on FreeBSD-7.x pushing a few hundred > megabits without any issues. This box falls over after ~ 20 mbit. > > If I bypass redirection and/or kill squid, the 4k cluster count drops > back down to < 500 and stays there. > > Does anyone have any ideas on where to begin debugging this? > > Thanks, > > > Adrian >