From owner-freebsd-net Mon Dec 30 11:22:31 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4FF637B401 for ; Mon, 30 Dec 2002 11:22:29 -0800 (PST) Received: from patrocles.silby.com (d73.as8.nwbl0.wi.voyager.net [169.207.132.73]) by mx1.FreeBSD.org (Postfix) with ESMTP id 00F9C43EA9 for ; Mon, 30 Dec 2002 11:22:28 -0800 (PST) (envelope-from silby@silby.com) Received: from patrocles.silby.com (localhost [127.0.0.1]) by patrocles.silby.com (8.12.6/8.12.6) with ESMTP id gBUJTT8U023109; Mon, 30 Dec 2002 13:29:29 -0600 (CST) (envelope-from silby@silby.com) Received: from localhost (silby@localhost) by patrocles.silby.com (8.12.6/8.12.6/Submit) with ESMTP id gBUJTNQm023106; Mon, 30 Dec 2002 13:29:24 -0600 (CST) X-Authentication-Warning: patrocles.silby.com: silby owned process doing -bs Date: Mon, 30 Dec 2002 13:29:22 -0600 (CST) From: Mike Silbersack To: Jeff Behl Cc: freebsd-net Subject: Re: when are mbuf clusters released? In-Reply-To: <3E108C31.9060403@expertcity.com> Message-ID: <20021230132145.N22880-100000@patrocles.silby.com> References: <3E108C31.9060403@expertcity.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Mon, 30 Dec 2002, Jeff Behl wrote: > 5066/52544/256000 mbufs in use (current/peak/max): > 5031/50612/64000 mbuf clusters in use (current/peak/max) > > is there some strange interaction going on between apace2 and bsd? > killing apache caused the mbuf clusters to start draining, but only > slowly. will clusters still be allocated in FIN_WAIT_? states? TIME_WAIT? Before I answer your question, let me explain how clusters are allocated. The first number above shows how many are in use at the moment. The second number shows how many have been used, and are currently allocated. The third is the limit you have set. What this means is that once an mbuf (or cluster) has been allocated, it is never truly freed, only returned to the free list. As a result, after your spike in mbuf usage, you never really get the memory back. However, this may be OK if you have plenty of ram. > This maching was serving a couple hundred connections a second...which > doesn't seem like it should have taxed it much (p3 1.2 gHz). CPU util > was low. > > Any help appreciated. > > Jeff Now, on to why the value spiked. Yes, connections in FIN_WAIT* states still hang on to mbuf clusters relating to the data they have been asked to send. There was a DoS script going around which intentionally stuck many sockets on a server in the FIN_WAIT_2 state until enough had been stuck to cause mbuf cluster exhaustion. To determine if this is the case, just run netstat -n and look at the sendq value; if you see high sendq values on a lot of sockets, this may be your answer. The other possibility is that you're being hit with lots of IP fragments... currently, the IP reassembly code allows too many unassembled packets to sit around. There's no way to inspect the IP reassembly queue actively, but you could use netstat -s to see "fragments received" - if the number is high, then it's likely something is up. Mike "Silby" Silbersack To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message