Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 3 Feb 2008 18:16:44 +0100
From:      Erik Trulsson <ertr1013@student.uu.se>
To:        Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= <des@des.no>
Cc:        hackers@freebsd.org, Ed Schouten <ed@fxq.nl>
Subject:   Re: sort(1) memory usage
Message-ID:  <20080203171644.GA67839@owl.midgard.homeip.net>
In-Reply-To: <86lk62kqeh.fsf@ds4.des.no>
References:  <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 03, 2008 at 04:31:34PM +0100, Dag-Erling Sm=F8rgrav wrote:
> Dag-Erling Sm=F8rgrav <des@des.no> writes:
> > Erik Trulsson <ertr1013@student.uu.se> writes:
> > > Yep, it seems that GNU sort allocates a quite large buffer by default=
 when
> > > the size of the input is unknown (such as when it reads input from st=
din.)
> > > A quick check in the source code indicates that it tries to size this=
 buffer
> > > according to how much memory the system has (and according to any lim=
its set
> > > on how much memory the process is allowed to use.)
> >
> > Uh, OK.  This scaling doesn't seem to work correctly.  It seems to
> > allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines,
> > regardless of memory size.

I said it *tries* to the size the buffer according the amount of memory
available.  I didn't say it succeded in doing so, or that it even made a
good attempty at it.

Those 27MB/54MB is probably because it hits some kind of limit.
On a machine having only 64MB RAM, sort(1) "only" allocated 21MB adress spa=
ce.

I suspect the scaling algorithm was designed for older machines which
rarely, if ever, had more than maybe 64MB RAM (and usually less than that),
and that little thought was given to multi-gigabyte machines like those
common today.


>=20
> Looking at the code, it seems to go to extreme lengths to get it
> absolutely wrong.  For instance, if hw.physmem / 8 > hw.usermem, it will
> pick the former, which means it's pretty much guaranteed to either fail
> or hose your system (or both).
>=20
> In the immortal words of Blazing Star: YOU FAIL IT
>=20
> Count this as a vote for ditching GNU sort in favor of a BSD-licensed
> implementation (from {Net,Open}BSD for instance).
>=20

If any such implementation was a true drop-in replacement of GNU sort
(supporting all the same options etc.) and did not have noticably worse
performance, then I certainly would not raise any objections to that.



--=20
<Insert your favourite quote here.>
Erik Trulsson
ertr1013@student.uu.se



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080203171644.GA67839>