Date: Sun, 22 Aug 2010 15:27:46 -0700 From: Pyun YongHyeon <pyunyh@gmail.com> To: Adrian Chadd <adrian.chadd@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: 8.0-RELEASE-p3: 4k jumbo mbuf cluster exhaustion Message-ID: <20100822222746.GC6013@michelle.cdnetworks.com> In-Reply-To: <AANLkTikYMU=wML_z=HDnkUF1PGYMVa1q-QWTrkxD%2B7EP@mail.gmail.com> References: <AANLkTikrbCFHz-CnuYcgH2JzpeH5hob0Aa2y5dwn3Hvv@mail.gmail.com> <AANLkTikYMU=wML_z=HDnkUF1PGYMVa1q-QWTrkxD%2B7EP@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Aug 22, 2010 at 05:40:30PM +0800, Adrian Chadd wrote: > I disabled tso, tx chksum and rx chksum. This fixed the 4k jumbo > allocation growth. > I recall there was SIOCSIFCAP ioctl handling bug in bce(4) on 8.0 so it might also disable IFCAP_TSO4/IFCAP_TXCSUM/IFCAP_RXCSUM when yo disabled RX checksum offloading. But I can't explain how checksum offloading could be related with the growth of 4k jumbo buffers. > > Turning on tso on a live proxy didn't affect jumbo allocations. > Turning on txcsum caused jumbo allocations to begin growing again. > DIsabling txcsum again caused jumbo allocations to stop increasing, > but it doesn't seem to be decreasing back to the steady state (~ 8k.) > Turning on rxcsum didn't affect jumbo allocations. > > So it seems txcsum is the culprit here. > There was a lot of changes in bce(4) since 8.0-RELEASE. I vaguely guess your issue could be related with header split feature of bce(4) which was now disabled. Are you using jumbo frame/ZERO_COPY_SOCKETS with bce(4)? > > > Adrian > > On 22 August 2010 16:11, Adrian Chadd <adrian.chadd@gmail.com> wrote: > > Hi, > > > > I've got a Squid/Lusca server on 8.0-RELEASE-p3 which is exhibiting > > some very strange behaviour. > > > > After a few minutes uptime, the 4k mbuf cluster zone fills up and > > Squid/Lusca spends almost all of it's time sleeping in "keglimit". > > > > I've bumped kern.ipc.nmbclusters to 262144 and kern.ipc.jumbop to > > 32768 but the system will slowly crawl towards filling that zone. > > > > The box has a bce on-board NIC and is using ipfw to handle redirecting > > traffic to/from the box for transparent TCP interception. It's > > handling around ~30,000 concurrent connections at the moment. > > > > I have other very busy proxies on FreeBSD-7.x pushing a few hundred > > megabits without any issues. This box falls over after ~ 20 mbit. > > > > If I bypass redirection and/or kill squid, the 4k cluster count drops > > back down to < 500 and stays there. > > > > Does anyone have any ideas on where to begin debugging this? > > > > Thanks, > > > > > > Adrian > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100822222746.GC6013>