From owner-freebsd-current@FreeBSD.ORG Wed Nov 17 18:36:34 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17A1F1065673; Wed, 17 Nov 2010 18:36:34 +0000 (UTC) (envelope-from thierry.herbelot@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by mx1.freebsd.org (Postfix) with ESMTP id 486C48FC13; Wed, 17 Nov 2010 18:36:30 +0000 (UTC) Received: from mail.herbelot.nom (unknown [82.227.159.103]) by smtp6-g21.free.fr (Postfix) with ESMTP id 5BA83822F6; Wed, 17 Nov 2010 19:36:24 +0100 (CET) Received: from tulipe.herbelot.nom (tulipe.herbelot.nom [192.168.2.5]) by mail.herbelot.nom (8.14.1/8.14.1) with ESMTP id oAHIaGu9018073; Wed, 17 Nov 2010 19:36:16 +0100 (CET) From: Thierry Herbelot To: freebsd-current@freebsd.org Date: Wed, 17 Nov 2010 19:36:10 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-STABLE; KDE/4.5.3; amd64; ; ) References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011170729.41894.thierry.herbelot@free.fr> In-Reply-To: <201011170729.41894.thierry.herbelot@free.fr> MIME-Version: 1.0 Content-Type: Multipart/Mixed; boundary="Boundary-00=_aCC5MxDID7T1Y5x" Message-Id: <201011171936.10764.thierry.herbelot@free.fr> Cc: "Bjoern A. Zeeb" , FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 18:36:34 -0000 --Boundary-00=_aCC5MxDID7T1Y5x Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thierry Herbelot a =E9crit > "Bjoern A. Zeeb" a =E9crit >=20 > > On Wed, 17 Nov 2010, Thierry Herbelot wrote: > >=20 > > Hi, > >=20 > > first of all freebsd-virtualization@ is the better list for this; Cc:ed. >=20 > (in fact, I did not know where else to send this message : -net, ... ? > thanks for the CC) >=20 > > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > > problem : there seems to be a memory leak in the kernel, which > > > eventually causes a panic. > > >=20 > > > (yes, we have seen the following message : "WARNING: VIMAGE > > > (virtualized network stack) is a highly experimental feature.") > > >=20 > > > Configuring a network interface in a jail with vnet enabled, then > > > removing that jail causes these messages. In dmesg: > > >=20 > > > [...] > > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > > >=20 > > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > > (with attached VIMAGE kernel config file). > > >=20 > > > The following commands reproduce the bug: > > >=20 > > > jail -l -u root -c path=3D/ name=3Dfoo persist vnet && > > > jexec foo ifconfig lo0 127.0.0.1/8 && > > > jail -r foo > > >=20 > > > Running it too many times exhausts kernel memory and crashes the > > > machine. > > >=20 > > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > > -head. > >=20 > > The problem has been present since day 1 and is still present up to > > HEAD. This is about type stability and teardown. So far we are no > > able to actually free TCP (and UDP in SVN) states as they might still > > be accesses after "free". It's a general problem in the network stack > > and has been implemented as a measure to circumvent panics in those > > cases. > >=20 > > I am not sure if that's actually what's biting you wrt to memory or > > if it's something else. Unfortunately boot logs and kernel configs > > don't help much; enable ddb and get show uma and show malloc reports > > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > > restarts. I might have some patches for a couple of things but cannot > > (yet) help with the additional (duplicate) UMA zones showing up at > > each iteration (for the -z case). >=20 > The context for our problem is that VIMAGE jails are mant to be used "in > production" for automated tests, thus are likely to be multiple on one > physical host, and are likely to be started and stopped according to our > needs. >=20 > We have tried more "classical" virtualization solutions (qemu, kvm, ... , > normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems > to be the correct answer. (for other reasons, an i386 version of the > kernel must be used) >=20 > The point of my original email was to simplify the configuration for a te= st > setup : any machine with the attached configuration file and the above > sequence of commands creating and deleting jails, running in a loop, will > eventually panic. >=20 > Anyway, I will follow up with the logs you mentioned. As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug= on=20 a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes fo= r=20 each vnet jail creation/destruction cycle) Cheers Thierry >=20 > Cheers (and thanks for the quick feedback) >=20 > Thierry Herbelot >=20 > > /bz >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" --Boundary-00=_aCC5MxDID7T1Y5x Content-Type: text/plain; charset="ISO-8859-15"; name="ddb-log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="ddb-log" [root@waldorf ~]# while jail -l -u root -c path=/ name=foo persist vnet && jexe c foo ifconfig lo0 127.0.0.1/8 && jail -r foo ; do continue ; done lock order reversal: 1st 0xc0bed490 allprison (allprison) @ kern/kern_jail.c:914 2nd 0xc0d5c56c vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ net/vnet.c:618 KDB: stack backtrace: db_trace_self_wrapper(c0ab1220,e9867a08,c05fc285,c05ec9db,c0ab40dc,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c05ec9db,c0ab40dc,c6d2d138,c6d30c88,e9867a64,...) at kdb_backtrace+0x29 _witness_debugger(c0ab40dc,c0d5c56c,c0abf638,c6d30c88,c0abf794,...) at _witness_debugger+0x25 witness_checkorder(c0d5c56c,1,c0abf78b,26a,0,...) at witness_checkorder+0x839 _sx_slock(c0d5c56c,0,c0abf78b,26a,0,...) at _sx_slock+0x85 vnet_sysinit(c7acf000,c0bb5d00,6180,0,c0b80888,...) at vnet_sysinit+0x2b vnet_alloc(c6f73028,c0aa9de6,0,10,c0aa5dc0,...) at vnet_alloc+0x9e kern_jail_set(c77ed000,c767b380,1,c767b380,280930fc,...) at kern_jail_set+0x179a jail_set(c77ed000,e9867cf8,c0ae5dd0,c0ab4ea5,c77e87f8,...) at jail_set+0x50 syscall(e9867d38) at syscall+0x220 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (507, FreeBSD ELF32, jail_set), eip = 0x280eae1b, esp = 0xbf7feb7c, ebp = 0xbf7fec38 --- Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. panic: kmem_malloc(131072): kmem_map too small: 335409152 total allocated cpuid = 0 KDB: enter: panic [thread pid 2129 tid 100072 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> bt Tracing pid 2129 tid 100072 td 0xc771fc80 kdb_enter(c0aadf11,c0aadf11,c0ad7eba,e95fa94c,0,...) at kdb_enter+0x3a panic(c0ad7eba,20000,13fdf000,c0ad7eb4,7d0,...) at panic+0x136 kmem_malloc(c149008c,20000,102,e95fa9d8,c082951d,...) at kmem_malloc+0x280 page_alloc(0,20000,e95fa9cb,102,0,...) at page_alloc+0x27 uma_large_malloc(20000,102,102,2,1,...) at uma_large_malloc+0x4d malloc(20000,c0b90240,102,c0585588,20000,...) at malloc+0x118 flowtable_alloc(c0ac790e,8000,2,0,0,...) at flowtable_alloc+0xee ip_init(40,c7629dc0,0,e95faa80,c0617807,...) at ip_init+0x330 protosw_init(40,c0b8e754,c0b941b4,c7629dc0,e95faa8c,...) at protosw_init+0x139 domain_init(c0b93f20,e95faaa8,c068bdde,c0b93f20,0,...) at domain_init+0x27 vnet_domain_init(c0b93f20,0,c0abf78b,26a,0,...) at vnet_domain_init+0x11 vnet_sysinit(c718c000,c0bb5d00,6180,0,c0b80888,...) at vnet_sysinit+0x3e vnet_alloc(c877e028,c0aa9de6,0,10,c0aa5dc0,...) at vnet_alloc+0x9e kern_jail_set(c771fc80,c7676e80,1,c7676e80,280930fc,...) at kern_jail_set+0x179a jail_set(c771fc80,e95facf8,c0ae5dd0,c0ab4ea5,c771ad48,...) at jail_set+0x50 syscall(e95fad38) at syscall+0x220 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (507, FreeBSD ELF32, jail_set), eip = 0x280eae1b, esp = 0xbf7feb7c, ebp = 0xbf7fec38 --- db> show uma Zone Size Used Free Requests ipq 32 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 FFS2 dinode 256 397 83 420 FFS1 dinode 128 0 0 0 FFS inode 116 397 98 420 Mountpoints 644 3 15 3 SWAPMETA 276 0 0 0 ip6flow 64 0 0 0 ip4flow 40 0 0 0 selfd 28 13 622 953 rtentry 108 9 99 9 unpcb 172 4 88 41 ripcb 220 0 0 0 sctp_asconf_ack 24 0 0 0 sctp_asconf 24 0 0 0 sctp_stream_msg_out 64 0 0 0 sctp_readq 76 0 0 0 sctp_chunk 92 0 0 0 sctp_raddr 432 0 0 0 sctp_laddr 24 0 145 3 sctp_asoc 1484 0 0 0 sctp_ep 860 0 0 0 sackhole 20 0 0 0 tcpreass 20 0 0 0 hostcache 76 0 0 0 syncache 112 0 0 0 tcptw 52 0 0 0 tcpcb 632 2 10 2 tcp_inpcb 220 2 34 2 udpcb 8 2 404 215 udp_inpcb 220 2 70 215 ipq 32 0 0 0 socket 412 8 28 405 bridge_rtnode 36 20 283 24 KNOTE 72 0 0 0 itimer 220 0 0 0 ksiginfo 80 56 232 153 pipe 392 0 90 1453 NFSNODE 468 0 0 0 NFSMOUNT 524 0 0 0 DIRHASH 1024 0 48 26 NAMEI 1024 0 48 17675 L VFS Cache 292 0 0 0 S VFS Cache 72 410 226 917 VNODEPOLL 60 0 0 0 VNODE 268 422 68 447 ata_composite 180 0 0 0 ata_request 204 0 190 2955 ttyoutq 256 156 84 420 ttyinq 152 300 168 810 g_bio 140 0 224 15483 mbuf_ext_refcnt 4 0 0 0 mbuf_jumbo_16k 16384 0 0 0 mbuf_jumbo_9k 9216 0 0 0 mbuf_jumbo_page 4096 0 0 0 mbuf_cluster 2048 512 42 512 mbuf 256 1 387 3924 mbuf_packet 256 256 256 1367 audit_record 816 0 0 0 cpuset 40 2 366 148 VMSPACE 236 17 159 2112 SLEEPQUEUE 48 133 221 133 THREAD 636 113 19 113 PROC 680 35 37 2129 MAC labels 20 0 0 0 umtx pi 52 0 0 0 TURNSTILE 72 133 77 133 Files 56 40 563 8850 4096 4096 2534 28 7824 2048 2048 273 109 1052 1024 1024 68 192 3976 512 512 92 76 4304 256 256 904 131 16844 128 128 2546 484 12434 64 64 5488 707 49123 32 32 3077 426 40826 16 16 3258 1005 42186 mt_zone 2056 270 0 270 SG fakepg 76 0 0 0 DP fakepg 76 0 0 0 PDPT 32 137 428 137 MAP ENTRY 72 280 356 51654 KMAP ENTRY 72 59 471 34398 MAP 140 7 21 7 VM OBJECT 136 667 232 27531 128 Bucket 524 93 12 1366 64 Bucket 268 206 18 260 32 Bucket 140 78 34 259 16 Bucket 76 56 44 82 UMA Hash 128 4 26 4 UMA RCntSlabs 544 277 3 277 UMA Slabs 284 5388 16 18442 UMA Zones 888 237 19 1770 UMA Kegs 128 237 33 1770 db> show malloc Type InUse MemUse Requests CAM periph 2 1K 12 scsi_cd 0 0K 0 acpicmbat 0 0K 0 $PIR 0 0K 0 acpidev 68 3K 68 CAM SIM 1 1K 1 isofs_mount 0 0K 0 isofs_node 0 0K 0 isadev 8 1K 8 ast_driver 0 0K 0 afd_driver 0 0K 0 nexusdev 4 1K 4 msi 3 1K 3 mptable 0 0K 0 memdesc 1 4K 1 MCA 0 0K 0 acd_driver 1 2K 1 ar_driver 0 0K 6 legacydrv 0 0K 0 io_apic 1 1K 1 ad_driver 1 1K 1 ata_pci 0 0K 0 ata_dma 0 0K 0 ata_generic 2 2K 2 scsi_da 0 0K 0 madt_table 0 0K 0 CAM queue 3 1K 7 GEOM 66 181K 5441 apmdev 2 1K 3 mpt_user 0 0K 0 CAM dev queue 1 1K 1 pfs_vncache 0 0K 0 pfs_nodes 21 3K 21 nullfs_mount 0 0K 0 atkbddev 2 1K 2 nullfs_hash 1 1K 1 vm_pgdata 1 64K 1 nullfs_node 0 0K 0 msdosfs_mount 0 0K 0 msdosfs_fat 0 0K 0 UMAHash 0 0K 0 ufs_mount 6 93K 6 ufs_quota 0 0K 0 ufs_dirhash 9 5K 27 pagedep 1 64K 1 inodedep 1 256K 1 newblk 1 1K 1 bmsafemap 0 0K 0 allocdirect 0 0K 0 indirdep 0 0K 0 allocindir 0 0K 0 freefrag 0 0K 0 freeblks 0 0K 0 freefile 0 0K 0 diradd 0 0K 0 mkdir 0 0K 0 dirrem 0 0K 0 newdirblk 0 0K 0 savedino 0 0K 0 mactemp 0 0K 0 audit_trigger 0 0K 0 audit_pipe 0 0K 0 audit_pipeent 0 0K 0 audit_pipe_presel 0 0K 0 audit_evclass 172 3K 211 audit_bsm 0 0K 0 audit_cred 0 0K 0 audit_data 0 0K 0 audit_path 0 0K 0 audit_text 0 0K 0 audit_gidset 0 0K 0 rpc 2 5K 2 NLM 0 0K 0 nfss_srvsock 0 0K 0 nfss_srvdesc 0 0K 0 nfss_daemon 0 0K 0 NFS FHA 1 1K 1 nfsclient_lock 0 0K 0 nfsclient_nlminfo 0 0K 0 nfsclient_req 0 0K 0 nfsclient_bigfh 0 0K 0 nfsclient_diroff 0 0K 0 nfsclient_hash 0 0K 0 nfsclient_directio 0 0K 0 nfsclient_srvsock 0 0K 0 mld 10 2K 83 msdosfs_fileno 0 0K 0 msdosfs_node 0 0K 0 in6_mfilter 0 0K 0 in6_multi 12 2K 669 ip6_moptions 0 0K 0 ip6_msource 0 0K 0 DEVFSP 0 0K 0 DEVFS 16 1K 17 DEVFS_RULE 0 0K 0 fragment 0 0K 0 syncache 1 72K 74 tcplog 0 0K 0 hostcache 1 16K 74 sctp_map 0 0K 0 sctp_stri 0 0K 0 sctp_stro 0 0K 0 sctp_aadr 0 0K 0 sctp_a_it 0 0K 103 sctp_atcl 0 0K 0 sctp_atky 0 0K 0 sctp_athm 0 0 sctp_athi 0 0K 0 sctp_stre 0 0K 0 sctp_cmsg 0 0K 0 sctp_cpal 0 0K 0 sctp_vrf 1 1K 74 sctp_ifa 99 13K 223 sctp_ifn 2 1K 75 sctp_timw 0 0K 0 sctp_mvrf 0 0K 0 sctp_iter 0 0K 103 sctp_socko 0 0K 0 encap_export_host 0 0K 0 in_mfilter 0 0K 0 in_multi 2 1K 75 ip_moptions 0 0K 0 ip_msource 0 0K 0 DEVFS2 0 0K 0 DEVFS3 206 26K 330 DEVFS1 190 48K 315 ipid 0 0K 0 igmp 10 2K 83 80211scan 0 0K 0 80211ratectl 0 0K 0 80211power 0 0K 0 80211node 0 0K 0 80211nodeie 0 0K 0 80211mesh 0 0K 0 80211com 0 0K 0 80211dfs 0 0K 0 80211crypto 0 0K 0 80211vap 0 0K 0 vnet 2 1K 75 vnet_data 2 56K 75 vnet_data_free 1 1K 1 routetbl 2684 303890K 3469 CAM ccb queue 0 0K 0 ata_da 0 0K 0 scsi_ch 0 0K 0 mfibuf 0 0K 0 md_disk 0 0K 0 md_sectors 0 0K 0 LED 2 1K 2 kbdmux 7 10K 7 vlan 0 0K 0 tun 0 0K 0 amr 0 0K 0 lltable 23 6K 315 gif 0 0K 0 fw_com 0 0K 0 faith 0 0K 0 arpcom 8 1K 8 epair 4 1K 4 clone 8 32K 81 ifdescr 0 0K 0 ifnet 12 11K 158 ifaddr 93 18K 823 ether_multi 15 1K 600 BPF 10 1K 83 subr_export_host 0 0K 0 mount 50 3K 2211 vnodemarker 0 0K 385 acpisem 15 2K 15 vnodes 2 1K 2 vfs_hash 1 256K 1 export_host 0 0K 0 cl_savebuf 0 0K 0 vfscache 1 512K 1 biobuf 4 8K 6 acl 0 0K 0 soname 3 1K 89 pcb 14 79K 891 ipsbuf 0 0K 0 shmfd 1 4K 1 ksem 1 4K 1 mbuf_tag 0 0K 1 accf 0 0K 0 pts 0 0K 0 tty 36 18K 38 shm 1 12K 1 sem 4 6K 4 msg 4 25K 4 ioctlops 0 0K 13640 select 6 1K 6 iov 1 1K 745 Witness 1 104K 1 iirbuf 0 0K 0 if_fwip 0 0K 0 ata_pmp 0 0K 0 USB 45 68K 46 Unitno 11 1K 403 taskqueue 15 1K 15 stack 0 0K 2 USBdev 29 7K 29 USBHC 0 0K 0 if_fwe 0 0K 0 sglist 0 0K 0 sbuf 0 0K 2741 CAM XPT 12 2K 26 rman 233 15K 683 acpitask 1 1K 1 acpica 3082 163K 65871 Per-cpu 1 1K 1 kobj 255 510K 363 aaccam 0 0K 0 eventhandler 299 11K 299 devstat 32 65K 32 bus 1110 52K 5921 bus-sc 81 171K 2508 fwmem 0 0K 0 SWAP 0 0K 0 p1003.1b 1 1K 1 umtx 132 9K 132 callout 3 768K 3 sysctl 0 0K 1420 sysctloid 4004 122K 4124 sysctltmp 0 0K 535 firewire 0 0K 0 plimit 12 3K 355 uidinfo 2 2K 17 cred 20 2K 10282 pgrp 17 2K 472 session 15 1K 31 proc 2 8K 2 subproc 109 163K 2203 fw_xfer 0 0K 0 osd 0 0K 0 aacbuf 0 0K 0 UART 51 22K 51 mtx_pool 2 8K 2 twe_commands 0 0K 0 module 379 24K 379 twa_commands 0 0K 0 cache 0 0K 0 devbuf 5642 12692K 5728 temp 59 231K 52869 ip6opt 0 0K 0 ip6ndp 11 1K 157 acpipwr 0 0K 0 free 0 0K 0 lockf 14 1K 18 linker 108 4K 110 ddb_capture 1 48K 1 KTRACE 100 13K 100 ciss_data 0 0K 0 entropy 1024 64K 1024 prison 1 2K 74 ithread 99 8K 99 PUC 8 2K 8 Fail Points 0 0K 0 zombie 0 0K 0 proc-args 17 1K 771 ppbusdev 3 1K 3 kqueue 0 0K 0 kenv 79 7K 83 filedesc 40 14K 2572 filedesc_to_leader 0 0K 0 sigio 1 1K 1 acpi_perf 4 1K 4 SCSI SES 0 0K 0 tty console 0 0K 0 cdev 9 2K 9 pci_link 16 2K 16 SCSI sa 0 0K 0 db> --Boundary-00=_aCC5MxDID7T1Y5x--