From owner-freebsd-net@FreeBSD.ORG Fri Jan 2 21:30:11 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78CD7106567C for ; Fri, 2 Jan 2009 21:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 624F88FC12 for ; Fri, 2 Jan 2009 21:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n02LUBXf010190 for ; Fri, 2 Jan 2009 21:30:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n02LUB9x010187; Fri, 2 Jan 2009 21:30:11 GMT (envelope-from gnats) Date: Fri, 2 Jan 2009 21:30:11 GMT Message-Id: <200901022130.n02LUB9x010187@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: "Jason DiCioccio" Cc: Subject: Re: kern/130059: [panic] Leaking 50k mbufs/hour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Jason DiCioccio List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jan 2009 21:30:11 -0000 The following reply was made to PR kern/130059; it has been noted by GNATS. From: "Jason DiCioccio" To: bug-followup@freebsd.org Cc: Subject: Re: kern/130059: [panic] Leaking 50k mbufs/hour Date: Fri, 2 Jan 2009 13:02:47 -0800 ------=_Part_175155_3729370.1230930167365 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Adding a bit more info.. Here's the netstat -m and vmstat -z output after the mbufs have had a while to build up: -- netstat -m -- 959510/235/959745 mbufs in use (current/cache/total) 257/133/390/25600 mbuf clusters in use (current/cache/total/max) 257/127 mbuf+clusters out of packet secondary zone in use (current/cache) 0/117/117/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 240391K/792K/241184K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/152/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 2716 requests for I/O initiated by sendfile 0 calls to protocol drain routines -- vmstat -z -- ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 128, 0, 99, 21, 99, 0 UMA Zones: 120, 0, 99, 21, 99, 0 UMA Slabs: 64, 0, 1132, 48, 15590, 0 UMA RCntSlabs: 104, 0, 312, 21, 856, 0 UMA Hash: 128, 0, 4, 26, 7, 0 16 Bucket: 76, 0, 21, 29, 64, 0 32 Bucket: 140, 0, 32, 24, 107, 0 64 Bucket: 268, 0, 34, 8, 142, 0 128 Bucket: 524, 0, 628, 9, 45698, 5387 VM OBJECT: 124, 0, 9136, 32404, 20553589, 0 MAP: 140, 0, 7, 49, 7, 0 KMAP ENTRY: 68, 56560, 17, 487, 153898, 0 MAP ENTRY: 68, 0, 15360, 2336, 53481139, 0 DP fakepg: 72, 0, 0, 0, 0, 0 mt_zone: 72, 0, 194, 71, 194, 0 16: 16, 0, 1996, 643, 81641919, 0 32: 32, 0, 2542, 283, 9470304, 0 64: 64, 0, 4584, 7039, 61275174, 0 128: 128, 0, 1101, 549, 6280868, 0 256: 256, 0, 1299, 366, 20697494, 0 512: 512, 0, 145, 279, 79529, 0 1024: 1024, 0, 173, 71, 532899, 0 2048: 2048, 0, 104, 146, 14198, 0 4096: 4096, 0, 266, 63, 1242097, 0 Files: 76, 0, 1595, 655, 16940144, 0 TURNSTILE: 76, 0, 295, 41, 295, 0 umtx pi: 52, 0, 0, 0, 0, 0 PROC: 696, 0, 229, 46, 541535, 0 THREAD: 552, 0, 283, 11, 952, 0 UPCALL: 44, 0, 0, 0, 0, 0 SLEEPQUEUE: 32, 0, 295, 157, 295, 0 VMSPACE: 232, 0, 196, 59, 541501, 0 cpuset: 40, 0, 2, 182, 2, 0 mbuf_packet: 256, 0, 258, 126, 20757971, 0 mbuf: 256, 0, 960618, 33, 116817015, 0 mbuf_cluster: 2048, 25600, 384, 6, 384, 0 mbuf_jumbo_pagesize: 4096, 12800, 0, 117, 19593255, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 406, 26807, 0 ACL UMA zone: 388, 0, 0, 20, 65073710, 0 g_bio: 132, 0, 12, 423, 10043465, 0 ata_request: 192, 0, 3, 337, 3212743, 0 ata_composite: 184, 0, 0, 0, 0, 0 cryptop: 60, 0, 0, 0, 0, 0 cryptodesc: 56, 0, 0, 0, 0, 0 VNODE: 276, 0, 19330, 33618, 3235279, 0 VNODEPOLL: 64, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 28, 38128632, 0 S VFS Cache: 68, 0, 20134, 23490, 10825579, 0 L VFS Cache: 291, 0, 27, 207, 175327, 0 NFSMOUNT: 560, 0, 0, 0, 0, 0 NFSNODE: 452, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 1946, 170, 28110, 0 pipe: 396, 0, 89, 31, 229902, 0 ksiginfo: 80, 0, 248, 40, 248, 0 itimer: 220, 0, 0, 36, 1, 0 KNOTE: 68, 0, 175, 217, 2157098, 0 socket: 416, 25605, 404, 127, 3609347, 0 unpcb: 168, 25622, 278, 136, 1352116, 0 ipq: 32, 904, 0, 226, 42, 0 udpcb: 180, 25608, 13, 75, 1774380, 0 inpcb: 180, 25608, 155, 197, 482839, 0 tcpcb: 464, 25600, 92, 60, 482839, 0 tcptw: 52, 5184, 63, 297, 96034, 0 syncache: 104, 15392, 9, 102, 474336, 0 hostcache: 76, 15400, 626, 124, 4537, 0 tcpreass: 20, 1690, 0, 169, 92838, 0 sackhole: 20, 0, 0, 169, 7789, 0 sctp_ep: 804, 25600, 0, 0, 0, 0 sctp_asoc: 1412, 40000, 0, 0, 0, 0 sctp_laddr: 24, 80040, 0, 145, 28, 0 sctp_raddr: 400, 80000, 0, 0, 0, 0 sctp_chunk: 92, 400008, 0, 0, 0, 0 sctp_readq: 76, 400000, 0, 0, 0, 0 sctp_stream_msg_out: 60, 400050, 0, 0, 0, 0 sctp_asconf: 24, 400055, 0, 0, 0, 0 sctp_asconf_ack: 24, 400055, 0, 0, 0, 0 ripcb: 180, 25608, 1, 43, 2, 0 rtentry: 124, 0, 78, 46, 3549, 0 SWAPMETA: 276, 121576, 158, 108, 19610, 0 Mountpoints: 716, 0, 6, 4, 6, 0 FFS inode: 128, 0, 19299, 19191, 3235170, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 19299, 16011, 3235170, 0 pfsrctrpl: 124, 10013, 0, 0, 0, 0 pfrulepl: 828, 0, 12, 8, 12, 0 pfstatepl: 284, 10010, 0, 0, 0, 0 pfaltqpl: 224, 0, 0, 0, 0, 0 pfpooladdrpl: 68, 0, 6, 106, 6, 0 pfrktable: 1240, 1002, 1, 5, 2, 0 pfrkentry: 156, 200000, 3, 47, 3, 0 pfrkentry2: 156, 0, 0, 0, 0, 0 pffrent: 16, 5075, 0, 0, 0, 0 pffrag: 48, 0, 0, 0, 0, 0 pffrcache: 48, 10062, 0, 0, 0, 0 pffrcent: 12, 50141, 0, 0, 0, 0 pfstatescrub: 28, 0, 0, 0, 0, 0 pfiaddrpl: 100, 0, 0, 0, 0, 0 pfospfen: 108, 0, 696, 24, 696, 0 pfosfp: 28, 0, 407, 228, 407, 0 ------=_Part_175155_3729370.1230930167365 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline
Adding a bit more info..  Here's the netstat -m and vmstat -z output after the mbufs have had a while to build up:

-- netstat -m --
959510/235/959745 mbufs in use (current/cache/total)
257/133/390/25600 mbuf clusters in use (current/cache/total/max)
257/127 mbuf+clusters out of packet secondary zone in use (current/cache)
0/117/117/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
240391K/792K/241184K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/152/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
2716 requests for I/O initiated by sendfile
0 calls to protocol drain routines

-- vmstat -z --

ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES

UMA Kegs:                 128,        0,       99,       21,       99,        0
UMA Zones:                120,        0,       99,       21,       99,        0
UMA Slabs:               & nbsp; 64,        0,     1132,       48,    15590,        0
UMA RCntSlabs:            104,        0,      312,       21,      856,        0
UMA Hash:                 128,        0,        4,       26,        7,        0
16 Bucket:                 76,   ;      0,       21,       29,       64,        0
32 Bucket:                140,        0,       32,       24,      107,        0
64 Bucket:                268,        0,       34,        8,      142,        0
128 Bucket:               524,   ;      0,      628,        9,    45698,     5387
VM OBJECT:                124,        0,     9136,    32404, 20553589,        0
MAP:                      140,        0,        7,       49,        7,        0
KMAP ENTRY:                68,   ;  56560,       17,      487,   153898,        0
MAP ENTRY:                 68,        0,    15360,     2336, 53481139,        0
DP fakepg:                 72,        0,        0,        0,        0,        0
mt_zone:                   72,   &nb sp;    0,      194,       71,      194,        0
16:                        16,        0,     1996,      643, 81641919,        0
32:                        32,        0,     2542,      283,  9470304,        0
64:                   &nbs p;    64,        0,     4584,     7039, 61275174,        0
128:                      128,        0,     1101,      549,  6280868,        0
256:                      256,        0,     1299,      366, 20697494,        0
512:                       512,        0,      145,      279,    79529,        0
1024:                    1024,        0,      173,       71,   532899,        0
2048:                    2048,        0,      104,      146,    14198,        0
4096:                   &n bsp;4096,        0,      266,       63,  1242097,        0
Files:                     76,        0,     1595,      655, 16940144,        0
TURNSTILE:                 76,        0,      295,       41,      295,        0
umtx pi:                   52,   &nb sp;    0,        0,        0,        0,        0
PROC:                     696,        0,      229,       46,   541535,        0
THREAD:                   552,        0,      283,       11,      952,        0
UPCALL:                    44,        0,        0,        0,        0,        0
SLEEPQUEUE:                32,        0,      295,      157,      295,        0
VMSPACE:                  232,        0,      196,       59,   541501,        0
cpuset:                    40,        0,        2,      182,        2,        0
mbuf_packet:              256,        0,      258,      126, 20757971,        0
mbuf:                     256,        0,   960618,       33, 116817015,        0
mbuf_cluster:            2048,    25600,      384, &nbs p;      6,      384,        0
mbuf_jumbo_pagesize:     4096,    12800,        0,      117, 19593255,        0
mbuf_jumbo_9k:           9216,     6400,        0,        0,        0,        0
mbuf_jumbo_16k:         16384,     3200,        0,        0,   ;      0,        0
mbuf_ext_refcnt:            4,        0,        0,      406,    26807,        0
ACL UMA zone:             388,        0,        0,       20, 65073710,        0
g_bio:                    132,       &nbs p;0,       12,      423, 10043465,        0
ata_request:              192,        0,        3,      337,  3212743,        0
ata_composite:            184,        0,        0,        0,        0,        0
cryptop:                   60,   ;      0,        0,        0,        0,        0
cryptodesc:                56,        0,        0,        0,        0,        0
VNODE:                    276,        0,    19330,    33618,  3235279,        0
VNODEPOLL:                 64,        0,        0,        0,        0,        0
NAMEI:                   1024,        0,        0,       28, 38128632,        0
S VFS Cache:               68,        0,    20134,    23490, 10825579,        0
L VFS Cache:              291,        0,     & nbsp; 27,      207,   175327,        0
NFSMOUNT:                 560,        0,        0,        0,        0,        0
NFSNODE:                  452,        0,        0,        0,        0,        0
DIRHASH:         &nbs p;       1024,        0,     1946,      170,    28110,        0
pipe:                     396,        0,       89,       31,   229902,        0
ksiginfo:                  80,        0,      248,       40,      248,        0
itimer:                   220,        0,        0,       36,        1,        0
KNOTE:                     68,        0,      175,      217,  2157098,        0
socket:                   416,    25605,      404,      127,  3609347,        0
unpcb:                    168,   &nb sp;25622,      278,      136,  1352116,        0
ipq:                       32,      904,        0,      226,       42,        0
udpcb:                    180,    25608,       13,       75,  1774380,        0
inpcb:                   & nbsp;180,    25608,      155,      197,   482839,        0
tcpcb:                    464,    25600,       92,       60,   482839,        0
tcptw:                     52,     5184,       63,      297,    96034,        0
syncache:                 104,    15392,        9,      102,   474336,        0
hostcache:                 76,    15400,      626,      124,     4537,        0
tcpreass:                  20,     1690,        0,      169,    92838,        0
sackhole:                  20,        0,        0,      169,     7789,        0
sctp_ep:                  804,    25600,        0,        0,        0,        0
sctp_asoc:               1412,    40000,        0,        0,        0,        0
sctp_laddr:                 ;24,    80040,        0,      145,       28,        0
sctp_raddr:               400,    80000,        0,        0,        0,        0
sctp_chunk:                92,   400008,        0,        0,        0,        0
sctp_readq:                76,   ; 400000,        0,        0,        0,        0
sctp_stream_msg_out:       60,   400050,        0,        0,        0,        0
sctp_asconf:               24,   400055,        0,        0,        0,        0
sctp_asconf_ack:           24,   400055,        0,   ;      0,        0,        0
ripcb:                    180,    25608,        1,       43,        2,        0
rtentry:                  124,        0,       78,       46,     3549,        0
SWAPMETA:                 276,   121576,      158,      108,    19610,        0
Mountpoints:              716,        0,        6,        4,        6,        0
FFS inode:                128,        0,    19299,    19191,  3235170,        0
FFS1 dinode:              128,        0,        0,        0,        0,        0
FFS2 dinode:              256,        0,    19299,    16011,  3235170,        0
pfsrctrpl:                124,    10013,        0,        0,        0,        0
pfrulepl:                 828,       &nbs p;0,       12,        8,       12,        0
pfstatepl:                284,    10010,        0,        0,        0,        0
pfaltqpl:                 224,        0,        0,        0,        0,        0
pfpooladdrpl:              68,        0,        6,      106,        6,        0
pfrktable:               1240,     1002,        1,        5,        2,        0
pfrkentry:                156,   200000,        3,       47,        3,        0
pfrkentry2:               156,     & nbsp;  0,        0,        0,        0,        0
pffrent:                   16,     5075,        0,        0,        0,        0
pffrag:                    48,        0,        0,        0,        0,        0
pffrcache:         &n bsp;       48,    10062,        0,        0,        0,        0
pffrcent:                  12,    50141,        0,        0,        0,        0
pfstatescrub:              28,        0,        0,        0,        0,        0
pfiaddrpl:             &nb sp;  100,        0,        0,        0,        0,        0
pfospfen:                 108,        0,      696,       24,      696,        0
pfosfp:                    28,        0,      407,      228,      407,        0



------=_Part_175155_3729370.1230930167365--