From owner-freebsd-current@FreeBSD.ORG  Thu Jun  9 19:15:57 2005
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
X-Original-To: freebsd-current@freebsd.org
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 7397B16A41C;
	Thu,  9 Jun 2005 19:15:57 +0000 (GMT)
	(envelope-from phk@critter.freebsd.dk)
Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 20E4E43D1F;
	Thu,  9 Jun 2005 19:15:56 +0000 (GMT)
	(envelope-from phk@critter.freebsd.dk)
Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk
	[83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id 733C11EC338;
	Thu,  9 Jun 2005 21:15:55 +0200 (CEST)
Received: from critter.freebsd.dk (localhost [127.0.0.1])
	by critter.freebsd.dk (8.13.4/8.13.3) with ESMTP id j59JFap8009020;
	Thu, 9 Jun 2005 21:15:37 +0200 (CEST)
	(envelope-from phk@critter.freebsd.dk)
To: Bakul Shah <bakul@BitBlocks.com>
From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
In-Reply-To: Your message of "Thu, 09 Jun 2005 12:11:15 PDT."
	<200506091911.j59JBGQn096385@gate.bitblocks.com> 
Date: Thu, 09 Jun 2005 21:15:36 +0200
Message-ID: <9019.1118344536@critter.freebsd.dk>
Sender: phk@critter.freebsd.dk
Cc: freebsd-fs@freebsd.org, =?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?= <des@des.no>,
	FreeBSD Current <freebsd-current@freebsd.org>
Subject: Re: you are in an fs with millions of small files 
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 09 Jun 2005 19:15:57 -0000

In message <200506091911.j59JBGQn096385@gate.bitblocks.com>, Bakul Shah writes:

>Start with a small array.  When it gets full double it (or
>grow by half if you want to waste less memory).

If I might make an observation...

You do not waste malloc'ed memory until you access it.  You waste some
page table entries etc in the kernel, but the actual pages do not
become part of the memory-pressure mob-rule until you touch it the
first time.

This is a very important point which people still (after 20 years of
virtual memory systems and 10 years of phkmalloc) still do not seem
to have fully understood.

I would advice a simple doubling and a terminal realloc to cut down
to actual size.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.