From owner-freebsd-stable@FreeBSD.ORG Tue Sep 13 00:51:48 2005 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9225D16A41F for ; Tue, 13 Sep 2005 00:51:48 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from fileserver.fields.utoronto.ca (fileserver.fields.utoronto.ca [128.100.216.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1207743D46 for ; Tue, 13 Sep 2005 00:51:47 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from fields.fields.utoronto.ca (fields.localdomain [192.168.216.11]) by fileserver.fields.utoronto.ca (8.12.8/8.12.8/Fields 6.0) with ESMTP id j8D0pl0r023954 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 12 Sep 2005 20:51:47 -0400 Received: from obsecurity.dyndns.org (localhost.localdomain [127.0.0.1]) by fields.fields.utoronto.ca (8.12.8/8.12.8/Fields WS 6.0) with ESMTP id j8D0pj6P003875; Mon, 12 Sep 2005 20:51:46 -0400 Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E016551206; Mon, 12 Sep 2005 20:51:44 -0400 (EDT) Date: Mon, 12 Sep 2005 20:51:44 -0400 From: Kris Kennaway To: Nikolai Schupbach Message-ID: <20050913005144.GA46178@xor.obsecurity.org> References: <4326158B.8010502@net24.co.nz> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IJpNTDwzlM2Ie8A6" Content-Disposition: inline In-Reply-To: <4326158B.8010502@net24.co.nz> User-Agent: Mutt/1.4.2.1i Cc: freebsd-stable@freebsd.org Subject: Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Sep 2005 00:51:48 -0000 --IJpNTDwzlM2Ie8A6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: > Hello, >=20 > We have been trying to migrate to FreeBSD 5.4 from an older 4.x release= =20 > for one of our busy mail servers. But we have encountered problems with= =20 > directory listings on 5.4. >=20 > Our /var/mail directory contains approximately 8,000 files doing a long= =20 > directory listing (ls -l) takes approximately 5 min and during this time= =20 > the CPU is running near 100%, on a FSBD 4 box the same directory=20 > contents takes less than 20 seconds to list. Even on a directory with=20 > 200 files, each file with a different owner, it will still take at least= =20 > 4-5 seconds to list. The problems only seems to occur when the directory= =20 > contains files from many different users. (as in /var/mail). If I chown= =20 > all the files in the /var/mail directory to a single user the directory= =20 > listing is near instant. >=20 > It appears it has something to do with 'ls' looking up the id's in the=20 > password database, because if I instruct ls to display numeric IDs (ls=20 > -ln), rather than converting to user and group names, the directory with= =20 > 8,000 files, with 8,000 different owners will list instantly. >=20 > The reason this concerns me so much is we are also having a problem with= =20 > our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while=20 > operating on small and even 0 byte mailboxes, when there are approx five= =20 > or more concurrent POP3 or IMAP sessions. And I can't help thinking that= =20 > the two problems are related. >=20 > Does anyone have any ideas? Has anyone else noticed this problem? If I=20 > can't resolve it I'm most likely going to revert to using 4.11, but I'd= =20 > really like to know what is going on. (and yes I'm using UFS_DIRHASH) Sounds like the real problem is that you have >8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list. Kris --IJpNTDwzlM2Ie8A6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFDJiKgWry0BWjoQKURAti+AJ9PH0eCgZr0H2im7SOBkdAnd0/LBwCgt27s WnPqJANxqvJg8cIklPGmB60= =+OT5 -----END PGP SIGNATURE----- --IJpNTDwzlM2Ie8A6--