From owner-freebsd-net@FreeBSD.ORG Mon Apr 21 11:43:51 2008 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8139510656C2; Mon, 21 Apr 2008 11:43:51 +0000 (UTC) (envelope-from is@rambler-co.ru) Received: from relay0.rambler.ru (relay0.rambler.ru [81.19.66.187]) by mx1.freebsd.org (Postfix) with ESMTP id 3C7C28FC1F; Mon, 21 Apr 2008 11:43:51 +0000 (UTC) (envelope-from is@rambler-co.ru) Received: from localhost (is1.park.rambler.ru [81.19.64.121]) by relay0.rambler.ru (Postfix) with ESMTP id 0825F5C23; Mon, 21 Apr 2008 15:28:32 +0400 (MSD) Date: Mon, 21 Apr 2008 15:27:53 +0400 From: Igor Sysoev To: gnn@freebsd.org Message-ID: <20080421112753.GD44915@rambler-co.ru> References: <20080420102827.U67663@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.13 (2006-08-11) Cc: Robert Watson , net@freebsd.org Subject: Re: zonelimit issues... X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Apr 2008 11:43:51 -0000 On Mon, Apr 21, 2008 at 05:16:28PM +0900, gnn@freebsd.org wrote: > At Mon, 21 Apr 2008 16:46:00 +0900, > gnn@freebsd.org wrote: > > > > At Sun, 20 Apr 2008 10:32:25 +0100 (BST), > > rwatson wrote: > > > > > > > > > On Fri, 18 Apr 2008, gnn@freebsd.org wrote: > > > > > > > I am wondering why this patch was never committed? > > > > > > > > http://people.freebsd.org/~delphij/misc/patch-zonelimit-workaround > > > > > > > > It does seem to address an issue I'm seeing where processes get into the > > > > zonelimit state through the use of mbufs (a high speed UDP packet receiver) > > > > but even after network pressure is reduced/removed the process never gets > > > > out of that state again. Applying the patch fixed the issue, but I'd like > > > > to have some discussion as to the general merits of the approach. > > > > > > > > Unfortunately the test that currently causes this is tied very tightly to > > > > code at work that I can't share, but I will hopefully be improving mctest to > > > > try to exhibit this behavior. > > > > > > When you take all load off the system, do mbufs and clusters get properly > > > freed back to UMA (as visible in netstat -m)? If not, continuing to bump up > > > against the zonelimit would suggest an mbuf/cluster leak, in which case we > > > need to track that bug. > > > > > > > This is unclear as the process that creates the issue opens 50 UDP > > multicast sockets with very large socket buffers. I am investigating > > this aspect some more. > > > > OK, yes, the clusters etc. go back to normal when the incoming > pressure is released. I do not believe we have a cluster/mbuf leak. There is no cluster/mbuf leak. The problem that FreeBSD has small KVA space: only 2G even on amd64 32G machines. So with vm.kmem_size=1G # 64M KVA kern.maxbcache=64M # 4M KVA kern.ipc.maxpipekva=4M I can use something like this: # 256M KVA/KVM kern.ipc.nmbjumbop=64000 # 216M KVA/KVM kern.ipc.nmbclusters=98304 # 162M KVA/KVM kern.ipc.maxsockets=163840 # 8M KVA/KVM net.inet.tcp.maxtcptw=163840 # 24M KVA/KVM kern.maxfiles=204800 -- Igor Sysoev http://sysoev.ru/en/