Date: Fri, 01 Aug 2003 12:48:05 -0700 From: Steve Francis <steve@expertcity.com> To: freebsd-net@freebsd.org Subject: mbuf clusters exhausted w/o reaching max? Message-ID: <3F2AC3F5.3010804@expertcity.com>
next in thread | raw e-mail | index | archive | help
I have a FreeBSD 4.8-RELEASE #5 system that reported:
Aug 1 11:50:39 rack2-101 /kernel: All mbuf clusters exhausted, please see tuning(7).
Aug 1 11:50:39 rack2-101 /kernel: All mbufs exhausted, please see tuning(7).
Yet its not close to the max allowed for clusters.
rack2-101.nyc# netstat -m
1338/4240/131072 mbufs in use (current/peak/max):
1338 mbufs allocated to data
709/3366/32768 mbuf clusters in use (current/peak/max)
7792 Kbytes allocated to network (7% of mb_map in use)
50 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
rack2-101.nyc#
I assume the problem is related to "50 requests for memory denied".
However, this system has 1G RAM, and about half of it free, as far as I can tell. (ps -axlm ouput below, as well as systat and top.)
So from all three tools, I see over 400M of inactive pages. I.e. pages that could be freed for use, and presumably allocated as mbuf clusters, but need to be flushed first.
So is this an error in the mbuf cluster allocation code, the vm system not freeing pages fast enough to keep up with the demand for the mbufs, or something else again?
Any ideas appreciated.
Thanks
ps -axlm
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
1002 87815 87814 75 2 0 390252 386472 - R ?? 668:11.73 ./bin/cCom_f
0 87829 1 0 2 0 4420 3764 select S p0- 9:50.28 /usr/bin/per
0 87823 1 1 2 0 4408 3752 select S p0- 13:29.65 /usr/bin/per
1002 87817 1 1 2 0 3344 1516 poll S ?? 2:31.43 BrokerLogger
0 87813 1 13 10 0 3256 2596 wait Is ?? 0:00.00 /usr/bin/per
0 62540 62538 0 18 0 1328 840 pause Ss p0 0:00.04 -csh (csh)
0 388 257 0 3 0 1312 712 ttyin I+ d1 0:00.01 -csh (csh)
0 115 1 0 2 0 3048 1756 select Ss ?? 0:05.88 sendmail: ac
0 155 1 83 2 0 3932 2748 select I d1- 13:28.21 /usr/local/s
25 118 1 0 18 0 2920 1576 pause Is ?? 0:00.47 sendmail: Qu
0 87814 87813 12 10 0 632 204 wait I ?? 0:00.00 sh -c ./bin/
0 1 0 0 10 0 552 112 wait ILs ?? 0:00.01 /sbin/init -
0 101 1 0 2 0 1312 784 select Ss ?? 0:06.99 /usr/sbin/nt
0 142 1 177 10 0 2076 1532 nanslp S d1- 1:09.31 /usr/bin/per
0 62538 110 0 2 0 5708 2180 select S ?? 0:00.21 sshd: root@t
0 64191 62540 0 28 0 428 248 - R+ p0 0:00.00 ps -axlm
0 110 1 0 2 0 3008 1568 select Is ?? 0:03.09 /usr/sbin/ss
0 108 1 0 10 0 1024 660 nanslp Ss ?? 0:00.47 /usr/sbin/cr
1000 168 1 0 10 0 968 528 nanslp S d1- 0:06.88 /usr/local/b
0 257 1 0 10 0 1268 824 wait Is d1 0:00.03 login [pam]
0 23 1 71 18 0 212 44 pause Is ?? 0:00.00 adjkerntz -i
0 97 1 0 2 0 940 580 select Ss ?? 0:00.41 /usr/sbin/sy
0 112 1 0 2 0 924 468 select Ss ?? 0:00.24 /usr/sbin/us
0 249 1 0 3 0 952 544 ttyin Is+ v1 0:00.00 /usr/libexec
0 248 1 0 3 0 952 544 ttyin Is+ v0 0:00.00 /usr/libexec
0 250 1 0 3 0 952 544 ttyin Is+ v2 0:00.00 /usr/libexec
0 251 1 0 3 0 952 544 ttyin Is+ v3 0:00.00 /usr/libexec
0 252 1 0 3 0 952 544 ttyin Is+ v4 0:00.00 /usr/libexec
0 253 1 0 3 0 952 544 ttyin Is+ v5 0:00.00 /usr/libexec
0 254 1 0 3 0 952 544 ttyin Is+ v6 0:00.00 /usr/libexec
0 255 1 0 3 0 952 544 ttyin Is+ v7 0:00.00 /usr/libexec
0 256 1 0 3 0 948 516 siodcd I ?? 0:00.00 /usr/libexec
0 0 0 3 -18 0 0 0 sched DLs ?? 0:00.00 (swapper)
0 6 0 0 -2 0 0 0 vlruwt DL ?? 0:00.61 (vnlru)
0 4 0 0 -18 0 0 0 psleep DL ?? 0:00.63 (bufdaemon)
0 3 0 3 18 0 0 0 psleep DL ?? 0:00.00 (vmdaemon)
0 2 0 0 -18 0 0 0 psleep DL ?? 0:00.24 (pagedaemon
0 5 0 0 18 0 0 0 syncer DL ?? 0:45.58 (syncer)
rack2-101.nyc#
systat -vmstat:
2 users Load 0.65 0.65 0.61 Aug 1 12:43
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 402424 6116 413732 5916 55372 count
All 1026416 6792 3985912 7120 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt 1 cow 8151 total
2 10 6222 617086499 8151 110 11 155608 wire stray irq7
400952 act 7921 mux irq9
16.4%Sys 19.0%Intr 26.3%User 0.0%Nice 38.3%Idl 416452 inact 2 ata0 irq14
| | | | | | | | | | 53404 cache sio0 irq4
========++++++++++>>>>>>>>>>>>> 1968 free sio1 irq3
daefr 100 clk irq0
Namei Name-cache Dir-cache 6 prcfr 128 rtc irq8
Calls hits % hits % react
756 756 100 pdwake
8 zfod pdpgs
Disks ad0 md0 3 ofod intrn
KB/t 16.00 0.00 35 %slo-z 114496 buf
tps 2 0 15 tfree 17 dirtybuf
MB/s 0.03 0.00 70438 desiredvnodes
% busy 0 0 17609 numvnodes
3657 freevnodes
top:
last pid: 64296; load averages: 0.64, 0.65, 0.61 up 2+19:12:20 12:43:41
31 processes: 3 running, 28 sleeping
CPU states: 22.1% user, 0.0% nice, 18.0% system, 22.8% interrupt, 37.1% idle
Mem: 391M Active, 407M Inact, 152M Wired, 52M Cache, 112M Buf, 1984K Free
Swap: 1000M Total, 1000M Free
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3F2AC3F5.3010804>