Date: Mon, 04 Apr 2011 13:48:08 +0700 From: Eugene Grosbein <egrosbein@rdtc.ru> To: "net@freebsd.org" <net@freebsd.org> Subject: mbuf clusters exhaustion & keglimit Message-ID: <4D9969A8.1060701@rdtc.ru>
next in thread | raw e-mail | index | archive | help
Hi! I'm running several loaded PPPoE access servers based on FreeBSD 8.2-STABLE/amd64 with em and igb network interfaces and 4GB RAM. No memory-intensive tasks other than routing about 2Gbit/s (1G "in" and a bit less "out"). kern.ipc.nmbclusters is set to 100000 in /etc/sysctl.conf and several months I had no problems with mbufs. Last week one of the routes stopped serviceing users for several hours but responded to pings and console was alive. Outgoing ping worked fine too but any process trying to use TCP or UDP kernel service got stuck in "keglimit" state. I've dropped to KDB from console, ran "call doadump", got full crashdump, returned from KDB, saved crashdump and tried to reboot cleanly. mpd5 failed to stop within 30 seconds timeout but file systems were unmounted cleanly and system rebooted. "vmstat -z -M vmcore" says that system was out of mbuf clusters: ITEM SIZE LIMIT USED FREE REQUESTS FAILURES mbuf_cluster: 2048, 100000, 100000, 0, 18897242, 317691 After that I've created graphs of mbuf cluster usage for all my routers and see no apparent leaks. The question is: how much kernel memory is it safe to dedicate to mbuf clusters? This system still runs with 100000 mbuf clusters maximum: Mem: 65M Active, 2759M Inact, 455M Wired, 31M Cache, 398M Buf, 435M Free It seems, 100000 mbuf clusters take only 207MB (2048+256 bytes for each), do they? Eugene Grosbein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D9969A8.1060701>