Date: Mon, 21 Mar 2011 23:13:04 +1100 From: Andrew MacIntyre <andymac@bullseye.apana.org.au> To: freebsd-python@freebsd.org Subject: Re: Python malloc issue? Message-ID: <4D8740D0.1020005@bullseye.andymac.org> In-Reply-To: <AANLkTikifWXgzzjm-P84d3Zq-ufZO4g5TugZsaQhBdmV@mail.gmail.com> References: <AANLkTikifWXgzzjm-P84d3Zq-ufZO4g5TugZsaQhBdmV@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Amit Dev wrote:
> I'm observing a strange memory usage pattern with python strings on
> Freebsd. Consider
> the following session. Idea is to create a list which holds some
> strings so that cumulative characters in the list is 100MB.
>
>>>> l = []
>>>> for i in xrange(100000):
> ... l.append(str(i) * (1000/len(str(i))))
>
> This uses around 100MB of memory as expected and 'del l' will clear that.
>
>>>> for i in xrange(20000):
> ... l.append(str(i) * (5000/len(str(i))))
>
> This is using 165MB of memory. I really don't understand where the
> additional memory usage is coming from. [Size of both lists are same]
>
> Python 2.6.4 on FreeBSD 7.2. On Linux and windows both uses around
> 100mb memory only.
A lot depends on the behavior of the system malloc(), and also the
reporting tools.
Python's internal allocator is only used for objects up to 256 bytes;
over that the request is passed to the platform malloc().
At a guess, a lot of your requests in the second case are over 4096
bytes (a single page of memory) and so the allocator is probably padding
the requests out to 2 pages, which is roughly in proportion with the
memory consumption you report.
jemalloc's behaviour can be tailored a bit - you might want to read the
man page.
--
-------------------------------------------------------------------------
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.org.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D8740D0.1020005>