Date: Sun, 3 Feb 2008 16:15:50 +0100 From: Erik Trulsson <ertr1013@student.uu.se> To: Ed Schouten <ed@fxq.nl> Cc: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= <des@des.no>, hackers@freebsd.org Subject: Re: sort(1) memory usage Message-ID: <20080203151550.GA67020@owl.midgard.homeip.net> In-Reply-To: <20080203131322.GK1179@hoeg.nl> References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 03, 2008 at 02:13:22PM +0100, Ed Schouten wrote: > * Dag-Erling Sm=F8rgrav <des@des.no> wrote: > > I've been trying to figure out why some periodic scripts consume so much > > memory. I've narrowed it down to sort(1). > >=20 > > At first, I thought the scripts were using it inefficiently, feeding it > > more data than was really needed. Then I discovered this: > >=20 > > des@ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort) > > [1] 66024 > > 66024 des 1 -8 5 54796K 52680K piperd 1 0:00 0.88% sort > >=20 > > That's right - sort(1) consumes 50+ MB of memory doing *nothing*. > >=20 > > (roughly half that on a 32-bit box) > >=20 > > Something is rotten in the state of GNU... >=20 > On my i386 box it spends 27M, but when I replace `sort' with `sed', > without any arguments, it's only 1.4 MB. I tried this on RELENG_6. I can > also reproduce this on Linux. >=20 Yep, it seems that GNU sort allocates a quite large buffer by default when the size of the input is unknown (such as when it reads input from stdin.) A quick check in the source code indicates that it tries to size this buffer according to how much memory the system has (and according to any limits set on how much memory the process is allowed to use.) The size of this buffer can be controlled with the --buffer-size option to sort(1). --=20 <Insert your favourite quote here.> Erik Trulsson ertr1013@student.uu.se
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080203151550.GA67020>