From owner-freebsd-current@FreeBSD.ORG Thu Dec 25 13:58:41 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A8CA16A4CE for ; Thu, 25 Dec 2003 13:58:41 -0800 (PST) Received: from obh.snafu.de (obh.snafu.de [213.73.92.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id 44BB843D1F for ; Thu, 25 Dec 2003 13:58:39 -0800 (PST) (envelope-from ob@gruft.de) Received: from ob by obh.snafu.de with local (Exim 3.36 #1) id 1AZdV8-000Ng2-00 for freebsd-current@freebsd.org; Thu, 25 Dec 2003 22:58:38 +0100 Date: Thu, 25 Dec 2003 22:58:38 +0100 From: Oliver Brandmueller To: freebsd-current@freebsd.org Message-ID: <20031225215838.GB68589@e-Gitt.NET> Mail-Followup-To: freebsd-current@freebsd.org References: <20031224154121.GA83770@e-Gitt.NET> <20031225204626.GA68589@e-Gitt.NET> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20031225204626.GA68589@e-Gitt.NET> User-Agent: Mutt/1.5.5.1i Sender: Oliver Brandmueller Subject: Re: file descriptor leak in 5.2-RC X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Dec 2003 21:58:41 -0000 Hello again. On Thu, Dec 25, 2003 at 09:46:26PM +0100, Oliver Brandmueller wrote: > - No Linuxulator. > - Tried different Threading libs. > - I do not have any unusual exits (since upgrading the RAM to a > resonable high size). Forgot something: I have no /etc/libmap.conf - I installed a libmap.conf that changed libc_r to libkse during one test (it was correctly parsed and used as I saw with clamd). I saw no difference. > I saw in high traffic times about 6000-8000 kern.openfiles after 10-12 > minutes of filtering, that sums up to about 10 per second. I will count > the number of mails fed through the system next time I try and will see > if I can see any relation (same number, twice or unrelated). I can do > the next test within the next few hours. # sysctl kern.openfiles kern.openfiles: 3473 # fgrep -c 'R=amavis' /var/log/exim/mainlog 1715 # fgrep -c 'H=localhost' /var/log/exim/mainlog 3463 Here is the interesting point! fstat shows for sh and init (with out itself) 10 file descriptors. 3463 + 10 = 3473 This means for every localhost connect related to amavisd we have one leaking file descriptor! amavisd listens on localhost port 10024 and feeds everything back to exim on port 10025 in this setup. Since exim gets connects on other ports also and the number is exactly equal to the number of connects to and from amavisd, it's very unlikely that it's related to exim. Since amavisd feeds every mail to clamd (via socket) only once and seems to use SpamAssassins libs directly, it's very likely in the way, amavisd handles the connections to/from exim. amavisd (perl script) uses - at least for outgoing SMTP - standard perl modules, in this Net::SMTP which it self includes "use Socket 1.3" and also "use IO::Socket". Most likely I'd say that perl 5.6.1 plus something in FBSD 5 together produce this leaking of file descriptors. I will take some time and consider updating to perl5.8 to see if it still happens. The problem seems - as far as I can tell - not related to the threading then. On one side I used different rhreading libs, on the other I think perl 5.6.1 does not use threading at all and I could not find anything that looks like that in amavisd (in fact it spawns simply a whole bunch of daemons). Since Michal Mertl detected the problem together with Mozilla as he wrote, it seems not to be related to perl only - which would be bad enough, but more a general problem somewhere. - Oliver -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! |