From owner-freebsd-current@FreeBSD.ORG Thu Jan 3 06:12:03 2008 Return-Path: Delivered-To: current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D420816A418; Thu, 3 Jan 2008 06:12:03 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from canonware.com (canonware.com [64.183.146.166]) by mx1.freebsd.org (Postfix) with ESMTP id B46E713C43E; Thu, 3 Jan 2008 06:12:03 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from [192.168.168.201] (unknown [192.168.168.201]) by canonware.com (Postfix) with ESMTP id 9AFB91298C5; Wed, 2 Jan 2008 22:12:51 -0800 (PST) Message-ID: <477C7CB6.8080701@freebsd.org> Date: Wed, 02 Jan 2008 22:12:06 -0800 From: Jason Evans User-Agent: Thunderbird 1.5.0.12 (X11/20071018) MIME-Version: 1.0 To: Joe Marcus Clarke References: <1199314166.9913.63.camel@shumai.marcuscom.com> <477C47BC.1020101@freebsd.org> <1199340028.64371.9.camel@shumai.marcuscom.com> In-Reply-To: <1199340028.64371.9.camel@shumai.marcuscom.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Robert Watson , current Subject: Re: Memory problem with latest malloc.c X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jan 2008 06:12:03 -0000 Joe Marcus Clarke wrote: > On Wed, 2008-01-02 at 18:26 -0800, Jason Evans wrote: >> It would be really helpful to me if you run your program with >> MALLOC_OPTIONS=dM and monitor memory usage. These flags cause mmap to >> be used instead of sbrk, and we can find out from that how much memory >> you really need. If peak memory usage is substantially different when >> using mmap versus sbrk, there's probably a malloc bug. > > Memory climbed up to 976 MB SZ, 974 MB RSS MB with dM > -> /etc/malloc.conf. The file was eventually generated without error. > Again, with Aj -> /etc/malloc.conf, the python2.5 process operating on > the same file planed out at 504 MB SZ, 501 MB RSS. Okay, that indicates that there is not a problem with malloc; you're running into the data segment resource limit. It isn't possible to increase the data segment beyond 512 MB on i386, so your best bet is to use MALLOC_OPTIONS=DM for the memory-intensive program. That will cause the program use all available space in the data segment, then start using mmap as necessary. I'm sorta thinking that MALLOC_OPTIONS=DM should be the default. Robert Watson is the person who talked me into this change, so feel free to give him a hard time about the extra configuration you have to do in order to get work done. =) Thanks, Jason