From owner-freebsd-questions@FreeBSD.ORG Mon Nov 28 02:50:14 2011 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F3F8106566C for ; Mon, 28 Nov 2011 02:50:14 +0000 (UTC) (envelope-from roberthuff@rcn.com) Received: from smtp02.lnh.mail.rcn.net (smtp02.lnh.mail.rcn.net [207.172.157.102]) by mx1.freebsd.org (Postfix) with ESMTP id 456AC8FC13 for ; Mon, 28 Nov 2011 02:50:13 +0000 (UTC) Received: from mr16.lnh.mail.rcn.net ([207.172.157.36]) by smtp02.lnh.mail.rcn.net with ESMTP; 27 Nov 2011 21:50:13 -0500 Received: from smtp04.lnh.mail.rcn.net (smtp04.lnh.mail.rcn.net [207.172.157.104]) by mr16.lnh.mail.rcn.net (MOS 4.3.3-GA) with ESMTP id BLC36750; Sun, 27 Nov 2011 21:50:12 -0500 Received: from 209-6-86-84.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com (HELO jerusalem.litteratus.org.litteratus.org) ([209.6.86.84]) by smtp04.lnh.mail.rcn.net with ESMTP; 27 Nov 2011 21:50:12 -0500 From: Robert Huff MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20178.63204.458573.817143@jerusalem.litteratus.org> Date: Sun, 27 Nov 2011 21:50:12 -0500 To: questions@freebsd.org X-Junkmail-Whitelist: YES (by domain whitelist at mr16.lnh.mail.rcn.net) Cc: Subject: too many open files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2011 02:50:14 -0000 Earlier tonight, I had a machine running FreeBSD 9.0-BETA1 #0: Tue Aug 23 10:07:23 EDT 2011 amd64 become unusable because kern.openfiles hit kern.maxfiles. I found this ... unexpected. After rebooting, I have been keeping track of kern.openfiles. Just after boot, it was about 575. Two hours later, it is over 3100 and climbing slowly but steadily. (There are moments when the number drops by <5 ... but then goes up again.) How do I figure out what is doing this? Looking at the process list (appended) I see no obvious candidates. Robert Huff PID TT STAT TIME COMMAND 0 ?? DLs 1:12.16 [kernel] 1 ?? SLs 0:00.03 /sbin/init -s 2 ?? DL 0:00.00 [sctp_iterator] 3 ?? DL 0:00.00 [xpt_thrd] 4 ?? DL 0:00.02 [pagedaemon] 5 ?? DL 0:00.00 [vmdaemon] 6 ?? DL 0:00.00 [pagezero] 7 ?? DL 0:00.08 [bufdaemon] 8 ?? DL 0:00.08 [vnlru] 9 ?? DL 0:01.68 [syncer] 10 ?? DL 0:00.00 [audit] 11 ?? RL 341:53.55 [idle] 12 ?? WL 1:36.09 [intr] 13 ?? DL 0:09.19 [geom] 14 ?? DL 0:01.37 [yarrow] 15 ?? DL 0:10.44 [usb] 16 ?? DL 0:00.41 [softdepflush] 17 ?? DL 0:01.78 [flowcleaner] 128 ?? Is 0:00.00 adjkerntz -i 355 ?? Is 0:00.00 dhclient: em0 [priv] (dhclient) 397 ?? Is 0:00.03 dhclient: em0 (dhclient) 1270 ?? Is 0:00.46 /usr/sbin/moused -p /dev/ums0 -t auto -I /var/run/mou 1297 ?? Is 0:00.01 /usr/sbin/moused -p /dev/ums2 -t auto -I /var/run/mou 1336 ?? Is 0:00.00 /sbin/devd 1630 ?? Ss 0:00.09 /usr/sbin/syslogd -c 1646 ?? Ss 0:03.83 /usr/sbin/named -u bind -c /etc/namedb/named.conf -u 1658 ?? Ss 0:00.02 /usr/sbin/rpcbind 1753 ?? Is 0:00.00 /usr/sbin/mountd -r 1755 ?? Is 0:00.02 nfsd: master (nfsd) 1756 ?? S 0:00.06 nfsd: server (nfsd) 1762 ?? Ss 0:00.01 /usr/sbin/rpc.statd 1768 ?? Ss 0:00.02 /usr/sbin/rpc.lockd 1778 ?? Is 0:18.40 /usr/local/sbin/apcupsd --kill-on-powerfail 1837 ?? Ss 0:00.53 /usr/sbin/ntpd -g -c /etc/ntp.conf -p /var/run/ntpd.p 1966 ?? Ss 0:00.02 /usr/local/sbin/dhcpd -q -cf /usr/local/etc/jerusalem 1976 ?? Is 0:00.04 /usr/local/sbin/cupsd -C /usr/local/etc/cups/cupsd.co 1997 ?? Is 0:00.05 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf 2009 ?? Is 0:00.17 /usr/local/sbin/spamass-milter -f -p /var/run/spamass 2012 ?? I 0:00.00 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf 2026 ?? Ss 0:03.10 /usr/local/bin/spamd -u spamd -H /var/spool/spamd -d 2027 ?? I 0:05.54 spamd child (perl) 2028 ?? I 0:00.02 spamd child (perl) 2039 ?? Is 0:00.02 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-f 2134 ?? I 0:04.05 [mysqld] 2182 ?? Is 0:00.01 /usr/local/bin/dbus-daemon --system 2205 ?? Ss 0:00.32 /usr/local/bin/gw6c 2285 ?? Is 0:03.84 /usr/local/sbin/clamd 2295 ?? Is 0:01.31 /usr/local/bin/freshclam --daemon -p /var/run/clamav/ 2302 ?? Is 0:00.11 /usr/local/sbin/clamav-milter -c /usr/local/etc/clama 2314 ?? Ss 0:00.21 sendmail: accepting connections (sendmail) 2318 ?? Is 0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/client 2339 ?? Is 0:00.07 /usr/local/bin/fetchmail -f /home/huff/.fetchmailrc - 2360 ?? S 0:02.27 /usr/local/bin/python2.7 /usr/local/bin/denyhosts.py 2373 ?? I 0:00.04 avahi-daemon: running [jerusalem.local] (avahi-daemon 2383 ?? I 0:00.00 /usr/local/sbin/avahi-dnsconfd -D 2443 ?? Ss 0:00.34 /usr/local/sbin/httpd -DNOHTTPACCEPT 2445 ?? I 0:00.00 /usr/local/sbin/cronolog /var/log/httpd-errors.%Y-%m. 2447 ?? I 0:00.01 /usr/local/sbin/cronolog /var/log/httpd-ssl_error.%Y- 2450 ?? I 0:00.00 /usr/local/sbin/cronolog /var/log/httpd-access.%Y-%m. 2451 ?? I 0:00.00 /usr/local/sbin/cronolog /var/log/httpd-ssl_access.%Y 2466 ?? Ss 0:00.24 /usr/sbin/sshd 2471 ?? Is 0:00.03 /usr/sbin/cron -s 2498 ?? Is 0:00.00 /usr/sbin/inetd -wW -C 60 2521 ?? Is 0:00.00 /usr/libexec/ftpd -D 2549 ?? I 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT 2550 ?? I 0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT 2551 ?? S 0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT 2552 ?? I 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT 2553 ?? I 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT 2558 ?? Is 0:01.80 /usr/local/sbin/hald 2560 ?? I 0:00.03 /usr/local/sbin/console-kit-daemon --no-daemon 2562 ?? I 0:00.03 /usr/local/libexec/polkitd 2564 ?? I 0:00.01 /usr/local/libexec/gam_server 2565 ?? I 0:00.05 hald-runner 2586 ?? I 0:00.01 hald-addon-mouse-sysmouse: /dev/ums0 (hald-addon-mous 2596 ?? I 0:00.01 hald-addon-mouse-sysmouse: /dev/ums2 (hald-addon-mous 2619 ?? S 0:01.51 hald-addon-storage: /dev/acd0 (hald-addon-storage) 2631 ?? SN 118:57.90 setiathome-6.12.amd64-portbld-freebsd 2632 ?? SN 118:35.10 setiathome-6.12.amd64-portbld-freebsd 2633 ?? SN 118:57.34 setiathome-6.12.amd64-portbld-freebsd 2671 ?? Is 0:00.00 xdm 2673 ?? Ss 3:08.62 /usr/local/bin/X :0 -auth /var/lib/xdm/authdir/authfi 2676 ?? Is 0:00.11 xdm: :0 (xdm) 2682 ?? I 0:00.02 xconsole -geometry 480x130-0-0 -daemon -notify -verbo 2701 ?? Is 0:00.02 /bin/sh -x /home/huff/.xinitrc.pt2 2703 ?? Ss 0:00.03 /usr/bin/ssh-agent /bin/sh -c /usr/bin/ssh-add; /home 2709 ?? S 0:16.31 fvwm2 (fvwm) 2712 ?? S 0:05.68 xemacs -geometry 80x30+2+2 -fn 10x20 -bg black -fg go 2714 ?? I 0:00.04 xterm -fn 10x20 -geometry 80x35+5+5 -name xterm (root 2715 ?? I 0:00.04 xterm -fn 10x20 -geometry 80x35+5+5 -name xterm (norm 2717 ?? SN 0:00.09 xbuffy -mail /var/mail/huff -fn 10x20 2718 ?? I 0:00.02 /usr/local/bin/xlogout 2731 ?? I 0:00.02 /usr/local/libexec/fvwm/2.6.3/FvwmAuto 7 4 none 0 8 5 2732 ?? S 0:00.35 /usr/local/libexec/fvwm/2.6.3/FvwmPager 9 4 none 0 8 2734 ?? S 0:01.49 /usr/local/libexec/fvwm/2.6.3/FvwmPager 11 4 none 0 8 2736 ?? I 0:00.24 xterm -geometry 80x35+2+2 -fn 10x20 -fg gold -bg blac 2752 ?? I 0:00.10 xterm -geometry 80x35+2+2 -fn 10x20 -fg gold -bg blac 2759 ?? S 1:10.70 /usr/local/bin/xmms 2782 ?? I 0:00.00 dbus-launch --autolaunch=dd18d15812f7f4854bce586a0012 2783 ?? Is 0:00.00 /usr/local/bin/dbus-daemon --fork --print-pid 5 --pri 2785 ?? S 0:04.08 /usr/local/libexec/gconfd-2 2787 ?? S 7:52.10 /usr/local/lib/seamonkey/seamonkey-bin -geometry 900x 2805 ?? Z 0:00.05 2812 ?? I 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT 3665 ?? Ss 0:00.02 sshd: [accepted] (sshd) 3666 ?? S 0:00.00 sshd: [net] (sshd) 3667 ?? Ss 0:00.01 /bin/tcsh -c ps -ax 3670 ?? R 0:00.00 ps -ax 2157 v0- SN 15:14.17 /usr/local/bin/mlnet-real 2368 v0- S 0:02.61 /usr/local/bin/boinc_client --dir /var/db/boinc 2541 v0 Is 0:00.01 login [pam] (login) 2610 v0 I 0:00.03 -tcsh (tcsh) 2663 v0 I 0:00.01 su 2664 v0 I+ 0:00.02 _su (csh) 2542 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1 2543 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2 2544 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4 2545 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5 2546 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6 2547 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7 2548 v8 Is+ 0:00.00 /usr/libexec/getty Pc ttyv8 2722 1 Is+ 0:00.02 tcsh 2721 2 Is+ 0:00.02 tcsh 2738 3 Is 0:00.02 tcsh 2743 3 I 0:00.01 su 2744 3 I+ 0:00.06 _su (csh) 2754 4 Is+ 0:00.07 tcsh