From owner-freebsd-virtualization@FreeBSD.ORG Sun Nov 14 04:25:08 2010 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3382106566C; Sun, 14 Nov 2010 04:25:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 431448FC13; Sun, 14 Nov 2010 04:25:08 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 39F1741C734; Sun, 14 Nov 2010 05:25:07 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id 1wevgnu2x6pi; Sun, 14 Nov 2010 05:25:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 3E42641C733; Sun, 14 Nov 2010 05:25:06 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id EF70D4448F3; Sun, 14 Nov 2010 04:24:23 +0000 (UTC) Date: Sun, 14 Nov 2010 04:24:23 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Julian Elischer In-Reply-To: <4CDF0C99.5080201@freebsd.org> Message-ID: <20101114042205.G78896@maildrop.int.zabbadoz.net> References: <4CDEFC2D.4090908@freebsd.org> <20101113212800.O78896@maildrop.int.zabbadoz.net> <4CDF0C99.5080201@freebsd.org> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: virtualization@freebsd.org Subject: Re: limitations on jail style virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Nov 2010 04:25:08 -0000 On Sat, 13 Nov 2010, Julian Elischer wrote: Hi Julian, > It was only a short discussion among "non developers" during a short breakout > session. > the session was "what is this VIMAGE/jails thing"? > and was not a dev-summit meeting but an "introduction to vimage" for end > users. Ok. Thanks for the follow-up. Much appreciated. > During the discussion people were asking questions that they had. Some of the > questions > I could answer well but others resulted in discussions that ended up with > things like, > "we you could do that but that would require that you had a different > /dev/pfsync for > each jail, and we have no way to do that yet". Well, to my understanding pfsync might already be fixed but maybe it was just /dev/pf. Ermal might want to follow-up but to my understanding a) there'll be one for every jail (given you allow it to show up in devfs) and b) based on credtials on open you'll figure out the right jail. > I promised the group that after the meeting I would bring up the topic with > other interested > developers... so here we are.. Tahnks a lot! /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Sun Nov 14 04:30:09 2010 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEDC11065673; Sun, 14 Nov 2010 04:30:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 83F118FC0C; Sun, 14 Nov 2010 04:30:08 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id D594E41C705; Sun, 14 Nov 2010 05:30:07 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id 7WJA6przmHn3; Sun, 14 Nov 2010 05:30:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 2F74941C733; Sun, 14 Nov 2010 05:30:06 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 95DC04448F3; Sun, 14 Nov 2010 04:27:32 +0000 (UTC) Date: Sun, 14 Nov 2010 04:27:32 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Julian Elischer In-Reply-To: <4CDF1F7E.8000601@freebsd.org> Message-ID: <20101114042427.V78896@maildrop.int.zabbadoz.net> References: <4CDEFC2D.4090908@freebsd.org> <4CDF0D87.70700@freebsd.org> <4CDF1F7E.8000601@freebsd.org> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: virtualization@freebsd.org, Jamie Gritton Subject: Re: limitations on jail style virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Nov 2010 04:30:09 -0000 On Sat, 13 Nov 2010, Julian Elischer wrote: > On 11/13/10 2:13 PM, Julian Elischer wrote: >> On 11/13/10 1:55 PM, Brandon Gooch wrote: >>> On Sat, Nov 13, 2010 at 2:59 PM, Julian Elischer >>> wrote: >>> >>> Was this brought up in any of the discussions? >>> >>> http://www.7he.at/freebsd/vps/ >> >> no it was not brought up.. >> it was an unofficial non-planned discussion that errupted pretty much >> spontaneously an a couple of couches so it was not prepared in advance.. >> >> Since I don't know much about that project I didn't bring it up. >> However I should take the time to read it.. >> thanks for bringing it up! > > replying to myself: > > Now that I have looked at the reference above I realise that I did look > at it before but during a particularly hectic pereiod of my life so I didn't > really take it in as much as I should have. > > I really do like this and I seen no reason why it can not be all linked > together. > Since I last heard about it, it appears that they have advanced somewhat. > They are already integrating the VIMAGE code into their project, and they > have > extended the VIMAGE mechanism to suite their own purposes. > > It would be nice if they would show up in -virtualization sometimes so we > could > include them into our discussions. > >> >>> I'm not sure if the VPS project pertains directly to what you're >>> talking about, but perhaps some of the code or ideas from the project >>> might? >>> >>> Even if it doesn't, it's still an exciting project that adds a ton of >>> value to FreeBSD's light-weight virtualization strategy. What do think >>> about the VPS concept in relation to the current virtualization effort >>> being put in to jails? It seems to me that recent efforts at >>> virtualizing kernel-level objects makes VPS the future of FreeBSD's >>> virtualization, leaving jails as a great way to isolate >>> applications... >> >> it the approaches are compatible I see no reason to not combine >> but I'll know more when I have read it. > > the approaches are different but not necessarily incompatible > In particular I'd like to hear what Jamie (James Gritton) > thinks about it, as he has most of the jail direction in his head :-) > > I'd certainly like to be able do do many of the things they hope to do (or > have a start at). we had Klaus at the DevSummit in Karlsruhe and he gave a talk at EuroBSDCon as well. See the wiki and the 2010.eurobsdcon.org web page (thought the material should be on his weg page as well). So, yes, we are talking to him, even though I got busy the last weeks. As he's piggybacking on VNET/VIMAGE and there'll me more things he and we'll do, there's certainly code going to be shared (I would hope). /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Mon Nov 15 07:26:54 2010 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39FD2106566B for ; Mon, 15 Nov 2010 07:26:54 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from out-0.mx.aerioconnect.net (outx.internet-mail-service.net [216.240.47.247]) by mx1.freebsd.org (Postfix) with ESMTP id 17B248FC13 for ; Mon, 15 Nov 2010 07:26:53 +0000 (UTC) Received: from idiom.com (postfix@mx0.idiom.com [216.240.32.160]) by out-0.mx.aerioconnect.net (8.13.8/8.13.8) with ESMTP id oAF7QqWT021876; Sun, 14 Nov 2010 23:26:52 -0800 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (h-67-100-89-137.snfccasy.static.covad.net [67.100.89.137]) by idiom.com (Postfix) with ESMTP id 4C6152D6013; Sun, 14 Nov 2010 23:26:51 -0800 (PST) Message-ID: <4CE0E0BD.1040707@freebsd.org> Date: Sun, 14 Nov 2010 23:26:53 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: <4CDEFC2D.4090908@freebsd.org> <4CDF0D87.70700@freebsd.org> <4CDF1F7E.8000601@freebsd.org> <20101114042427.V78896@maildrop.int.zabbadoz.net> In-Reply-To: <20101114042427.V78896@maildrop.int.zabbadoz.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 216.240.47.51 Cc: virtualization@freebsd.org, Jamie Gritton Subject: Re: limitations on jail style virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Nov 2010 07:26:54 -0000 On 11/13/10 8:27 PM, Bjoern A. Zeeb wrote: > On Sat, 13 Nov 2010, Julian Elischer wrote: > >> On 11/13/10 2:13 PM, Julian Elischer wrote: >>> On 11/13/10 1:55 PM, Brandon Gooch wrote: >>>> On Sat, Nov 13, 2010 at 2:59 PM, Julian >>>> Elischer wrote: >>>> >>>> Was this brought up in any of the discussions? >>>> >>>> http://www.7he.at/freebsd/vps/ >>> >>> no it was not brought up.. >>> it was an unofficial non-planned discussion that errupted pretty much >>> spontaneously an a couple of couches so it was not prepared in >>> advance.. >>> >>> Since I don't know much about that project I didn't bring it up. >>> However I should take the time to read it.. >>> thanks for bringing it up! >> >> replying to myself: >> >> Now that I have looked at the reference above I realise that I did >> look >> at it before but during a particularly hectic pereiod of my life so >> I didn't >> really take it in as much as I should have. >> >> I really do like this and I seen no reason why it can not be all >> linked together. >> Since I last heard about it, it appears that they have advanced >> somewhat. >> They are already integrating the VIMAGE code into their project, >> and they have >> extended the VIMAGE mechanism to suite their own purposes. >> >> It would be nice if they would show up in -virtualization sometimes >> so we could >> include them into our discussions. >> >>> >>>> I'm not sure if the VPS project pertains directly to what you're >>>> talking about, but perhaps some of the code or ideas from the >>>> project >>>> might? >>>> >>>> Even if it doesn't, it's still an exciting project that adds a >>>> ton of >>>> value to FreeBSD's light-weight virtualization strategy. What do >>>> think >>>> about the VPS concept in relation to the current virtualization >>>> effort >>>> being put in to jails? It seems to me that recent efforts at >>>> virtualizing kernel-level objects makes VPS the future of FreeBSD's >>>> virtualization, leaving jails as a great way to isolate >>>> applications... >>> >>> it the approaches are compatible I see no reason to not combine >>> but I'll know more when I have read it. >> >> the approaches are different but not necessarily incompatible >> In particular I'd like to hear what Jamie (James Gritton) >> thinks about it, as he has most of the jail direction in his head :-) >> >> I'd certainly like to be able do do many of the things they hope to >> do (or have a start at). > > we had Klaus at the DevSummit in Karlsruhe and he gave a talk at > EuroBSDCon as well. See the wiki and the 2010.eurobsdcon.org web > page (thought the material should be on his weg page as well). > > So, yes, we are talking to him, even though I got busy the last weeks. > As he's piggybacking on VNET/VIMAGE and there'll me more things he and > we'll do, there's certainly code going to be shared (I would hope). \ this sort of dovetails into something I've been thinking about for a while, which is NUMA support. Now, you may well ask "what has this to do with NUMA?" so I will explain. It seems to me that as the speed limits for single CPUS seem to be getting reached, we are seeing the transfer from speed to breadth. i.e more and more CPUs. We will shortly see the intruduction of systems that support not 2 or 4 threads of operation but thousands of threads of operation. We are already seeing some early implementations of this. A single image OS will have great trouble handling this sort of system, but the people who are hoping to make it all work are teh people like VMWARE. They will take systems with thousands of threads and break them up in a more-or less dynamic manner, keeping the NUMA layout of the system in mind. I have been wondering if we might not be able to do something similar. where we one one BSD system which overall may not be that efficient, but break it up along the NUMA fault lines using something like VPS to present a whole bunch of independent systems that can run very efficiently (on local NUMA RAM) an yet be 'tightly coupled' by the fact that they are running in one overall system with full cross connectivity (even though it be slightly less efficient than the NUMA couplings). We could, scale our systems up to tens of thousands of CPUS if we were willing to break them up to a large number of 'jails'/VPSs, each with only 32 or so threads (or whatever the hardware effieciently supports). Then the overall system could be responsible mostly for inter-system communications and housekeeping. Duplicated OS images and seaprated data ranges might all come together to form this "cluster of systems in a single OS". Anyhow, that the way I see things progressing and the OS that can harness all these CPUs in a useful way will have a lot if demand. whether it will be ESX+OS-de-jour or Linux++ or ClusterBSD will remain to be seen. Julian > > /bz > From owner-freebsd-virtualization@FreeBSD.ORG Mon Nov 15 07:45:07 2010 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6652F1065775; Mon, 15 Nov 2010 07:45:07 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id DBA988FC15; Mon, 15 Nov 2010 07:45:06 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id E808941C6FC; Mon, 15 Nov 2010 08:45:05 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id kJqtVjA0bSNA; Mon, 15 Nov 2010 08:45:05 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 8EC8341C6F2; Mon, 15 Nov 2010 08:45:05 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 1F6004448F3; Mon, 15 Nov 2010 07:43:56 +0000 (UTC) Date: Mon, 15 Nov 2010 07:43:56 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Julian Elischer In-Reply-To: <4CE0E0BD.1040707@freebsd.org> Message-ID: <20101115073916.X24596@maildrop.int.zabbadoz.net> References: <4CDEFC2D.4090908@freebsd.org> <4CDF0D87.70700@freebsd.org> <4CDF1F7E.8000601@freebsd.org> <20101114042427.V78896@maildrop.int.zabbadoz.net> <4CE0E0BD.1040707@freebsd.org> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: virtualization@freebsd.org, Jamie Gritton Subject: Re: limitations on jail style virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Nov 2010 07:45:07 -0000 On Sun, 14 Nov 2010, Julian Elischer wrote: Julian, > this sort of dovetails into something I've been thinking about for a while, > which is NUMA support. really good thoughts, but bad timing and wrong target audience I think. 1) first of all you want the basic underlying OS support all sort of things and as we know people are working on things here and there and it's not only NUMA. 2) before 1) is rock solid I will not even think about how to leverage it for VIMAGE. It would really just be a waste of time; it will however be a good idea to make sure that the proposals people come up with in 1) will scale for VIMAGE as well, but that's a different story. I am at least trying to keep an eye on the network stack side. 3) Before VIMAGE is NOT stable we MUST NOT introduce more complexity. So if you are eager to work on this I am all for it and you should get in touch with the folks doing 1). /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Mon Nov 15 11:07:08 2010 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 003BF1065702 for ; Mon, 15 Nov 2010 11:07:07 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E0C008FC08 for ; Mon, 15 Nov 2010 11:07:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oAFB77wY086466 for ; Mon, 15 Nov 2010 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oAFB77Pf086464 for freebsd-virtualization@FreeBSD.org; Mon, 15 Nov 2010 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 15 Nov 2010 11:07:07 GMT Message-Id: <201011151107.oAFB77Pf086464@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-virtualization@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-virtualization@FreeBSD.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Nov 2010 11:07:08 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/148155 virtualization[vimage] Kernel panic with PF/IPFilter + VIMAGE kernel s kern/143808 virtualization[pf] pf does not work inside jail 2 problems total. From owner-freebsd-virtualization@FreeBSD.ORG Mon Nov 15 16:42:35 2010 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB07E106566C; Mon, 15 Nov 2010 16:42:35 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from out-0.mx.aerioconnect.net (out-0-6.mx.aerioconnect.net [216.240.47.66]) by mx1.freebsd.org (Postfix) with ESMTP id 980418FC15; Mon, 15 Nov 2010 16:42:35 +0000 (UTC) Received: from idiom.com (postfix@mx0.idiom.com [216.240.32.160]) by out-0.mx.aerioconnect.net (8.13.8/8.13.8) with ESMTP id oAFGgYge013547; Mon, 15 Nov 2010 08:42:34 -0800 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (h-67-100-89-137.snfccasy.static.covad.net [67.100.89.137]) by idiom.com (Postfix) with ESMTP id 400A32D6019; Mon, 15 Nov 2010 08:42:33 -0800 (PST) Message-ID: <4CE162FB.5070209@freebsd.org> Date: Mon, 15 Nov 2010 08:42:35 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: <4CDEFC2D.4090908@freebsd.org> <4CDF0D87.70700@freebsd.org> <4CDF1F7E.8000601@freebsd.org> <20101114042427.V78896@maildrop.int.zabbadoz.net> <4CE0E0BD.1040707@freebsd.org> <20101115073916.X24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101115073916.X24596@maildrop.int.zabbadoz.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 216.240.47.51 Cc: virtualization@freebsd.org, Jamie Gritton Subject: Re: limitations on jail style virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Nov 2010 16:42:35 -0000 On 11/14/10 11:43 PM, Bjoern A. Zeeb wrote: > On Sun, 14 Nov 2010, Julian Elischer wrote: > > Julian, > >> this sort of dovetails into something I've been thinking about for >> a while, >> which is NUMA support. > > really good thoughts, but bad timing and wrong target audience I > think. I don't think the thought was seep enough to warrant a larger audience :-) > > 1) first of all you want the basic underlying OS support all sort of > things and as we know people are working on things here and there > and it's not only NUMA. Well yes, of course but it's interesting how various different parts of what are until now completely separate projects might some day come together in a larger picture. > > 2) before 1) is rock solid I will not even think about how to leverage > it for VIMAGE. It would really just be a waste of time; it will > however be a good idea to make sure that the proposals people come > up with in 1) will scale for VIMAGE as well, but that's a different > story. I am at least trying to keep an eye on the network stack > side. > > 3) Before VIMAGE is NOT stable we MUST NOT introduce more complexity. agreed. My only thought at the moment is to try and keep an eye on the bigger picture when changing the smaller parts. > > So if you are eager to work on this I am all for it and you should get > in touch with the folks doing 1). > > /bz > From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 06:00:25 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01D4F106566B; Wed, 17 Nov 2010 06:00:09 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 052508FC17; Wed, 17 Nov 2010 06:00:09 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 56FEA41C6DB; Wed, 17 Nov 2010 07:00:07 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id Y9KXxZ-ykPnq; Wed, 17 Nov 2010 07:00:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 5C48B41C690; Wed, 17 Nov 2010 07:00:06 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 24A584448F3; Wed, 17 Nov 2010 05:57:53 +0000 (UTC) Date: Wed, 17 Nov 2010 05:57:53 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Thierry Herbelot In-Reply-To: <201011170627.28025.thierry.herbelot@free.fr> Message-ID: <20101117055208.S24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-current , FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: FreeBSD virtualization mailing list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 06:00:25 -0000 On Wed, 17 Nov 2010, Thierry Herbelot wrote: Hi, first of all freebsd-virtualization@ is the better list for this; Cc:ed. > We are using FreeBSD + VIMAGE at work, and we have seen an annoying problem : > there seems to be a memory leak in the kernel, which eventually causes a > panic. > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > network stack) is a highly experimental feature.") > > Configuring a network interface in a jail with vnet enabled, then removing > that jail causes these messages. In dmesg: > > [...] > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled (with > attached VIMAGE kernel config file). > > The following commands reproduce the bug: > > jail -l -u root -c path=/ name=foo persist vnet && > jexec foo ifconfig lo0 127.0.0.1/8 && > jail -r foo > > Running it too many times exhausts kernel memory and crashes the machine. > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN -head. The problem has been present since day 1 and is still present up to HEAD. This is about type stability and teardown. So far we are no able to actually free TCP (and UDP in SVN) states as they might still be accesses after "free". It's a general problem in the network stack and has been implemented as a measure to circumvent panics in those cases. I am not sure if that's actually what's biting you wrt to memory or if it's something else. Unfortunately boot logs and kernel configs don't help much; enable ddb and get show uma and show malloc reports after the crash (or watch vmstat -z and vmstat -m) after every 50 jail restarts. I might have some patches for a couple of things but cannot (yet) help with the additional (duplicate) UMA zones showing up at each iteration (for the -z case). /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 06:45:43 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9739D106564A for ; Wed, 17 Nov 2010 06:45:43 +0000 (UTC) (envelope-from thierry.herbelot@free.fr) Received: from smtpfb1-g21.free.fr (smtpfb1-g21.free.fr [212.27.42.9]) by mx1.freebsd.org (Postfix) with ESMTP id B9B938FC16 for ; Wed, 17 Nov 2010 06:45:40 +0000 (UTC) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by smtpfb1-g21.free.fr (Postfix) with ESMTP id 871EF2E039 for ; Wed, 17 Nov 2010 07:30:09 +0100 (CET) Received: from mail.herbelot.nom (unknown [82.227.159.103]) by smtp6-g21.free.fr (Postfix) with ESMTP id 0FDA38224B; Wed, 17 Nov 2010 07:30:00 +0100 (CET) Received: from tulipe.herbelot.nom (tulipe.herbelot.nom [192.168.2.5]) by mail.herbelot.nom (8.14.1/8.14.1) with ESMTP id oAH6TlFe020404; Wed, 17 Nov 2010 07:29:48 +0100 (CET) From: Thierry Herbelot To: freebsd-current@freebsd.org, FreeBSD virtualization mailing list , "Bjoern A. Zeeb" Date: Wed, 17 Nov 2010 07:29:41 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-STABLE; KDE/4.5.3; amd64; ; ) References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101117055208.S24596@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201011170729.41894.thierry.herbelot@free.fr> X-Mailman-Approved-At: Wed, 17 Nov 2010 07:07:34 +0000 Cc: Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 06:45:43 -0000 "Bjoern A. Zeeb" a =E9crit > On Wed, 17 Nov 2010, Thierry Herbelot wrote: >=20 > Hi, >=20 > first of all freebsd-virtualization@ is the better list for this; Cc:ed. (in fact, I did not know where else to send this message : -net, ... ? than= ks=20 for the CC) >=20 > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > problem : there seems to be a memory leak in the kernel, which > > eventually causes a panic. > >=20 > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > > network stack) is a highly experimental feature.") > >=20 > > Configuring a network interface in a jail with vnet enabled, then > > removing that jail causes these messages. In dmesg: > >=20 > > [...] > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > >=20 > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > (with attached VIMAGE kernel config file). > >=20 > > The following commands reproduce the bug: > >=20 > > jail -l -u root -c path=3D/ name=3Dfoo persist vnet && > > jexec foo ifconfig lo0 127.0.0.1/8 && > > jail -r foo > >=20 > > Running it too many times exhausts kernel memory and crashes the machin= e. > >=20 > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > -head. >=20 > The problem has been present since day 1 and is still present up to > HEAD. This is about type stability and teardown. So far we are no > able to actually free TCP (and UDP in SVN) states as they might still > be accesses after "free". It's a general problem in the network stack > and has been implemented as a measure to circumvent panics in those > cases. >=20 > I am not sure if that's actually what's biting you wrt to memory or > if it's something else. Unfortunately boot logs and kernel configs > don't help much; enable ddb and get show uma and show malloc reports > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > restarts. I might have some patches for a couple of things but cannot > (yet) help with the additional (duplicate) UMA zones showing up at > each iteration (for the -z case). The context for our problem is that VIMAGE jails are mant to be used "in=20 production" for automated tests, thus are likely to be multiple on one=20 physical host, and are likely to be started and stopped according to our=20 needs. We have tried more "classical" virtualization solutions (qemu, kvm, ... ,=20 normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems = to=20 be the correct answer. (for other reasons, an i386 version of the kernel mu= st=20 be used) The point of my original email was to simplify the configuration for a test= =20 setup : any machine with the attached configuration file and the above=20 sequence of commands creating and deleting jails, running in a loop, will=20 eventually panic. Anyway, I will follow up with the logs you mentioned. Cheers (and thanks for the quick feedback) Thierry Herbelot >=20 > /bz From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 07:25:08 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 541E7106564A for ; Wed, 17 Nov 2010 07:25:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id D02858FC0C for ; Wed, 17 Nov 2010 07:25:07 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id CBF5D41C650; Wed, 17 Nov 2010 08:25:06 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id 9s7scPrIQgCX; Wed, 17 Nov 2010 08:25:05 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id CC8E341C66F; Wed, 17 Nov 2010 08:25:05 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 4FB024448F3; Wed, 17 Nov 2010 07:23:01 +0000 (UTC) Date: Wed, 17 Nov 2010 07:23:01 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Thierry Herbelot In-Reply-To: <201011170729.41894.thierry.herbelot@free.fr> Message-ID: <20101117072103.R24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011170729.41894.thierry.herbelot@free.fr> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 07:25:08 -0000 On Wed, 17 Nov 2010, Thierry Herbelot wrote: Hi, >> The problem has been present since day 1 and is still present up to >> HEAD. This is about type stability and teardown. So far we are no >> able to actually free TCP (and UDP in SVN) states as they might still >> be accesses after "free". It's a general problem in the network stack >> and has been implemented as a measure to circumvent panics in those >> cases. >> >> I am not sure if that's actually what's biting you wrt to memory or >> if it's something else. Unfortunately boot logs and kernel configs >> don't help much; enable ddb and get show uma and show malloc reports >> after the crash (or watch vmstat -z and vmstat -m) after every 50 jail >> restarts. I might have some patches for a couple of things but cannot >> (yet) help with the additional (duplicate) UMA zones showing up at >> each iteration (for the -z case). > > The context for our problem is that VIMAGE jails are mant to be used "in > production" for automated tests, thus are likely to be multiple on one > physical host, and are likely to be started and stopped according to our > needs. > > We have tried more "classical" virtualization solutions (qemu, kvm, ... , > normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems to > be the correct answer. (for other reasons, an i386 version of the kernel must > be used) > > The point of my original email was to simplify the configuration for a test > setup : any machine with the attached configuration file and the above > sequence of commands creating and deleting jails, running in a loop, will > eventually panic. The thing you can do is start the jails persistent and then only garbage collect proceses and network configuration manually and reconfigure/restart things for the next iteration; you might still leak network stack details between runs but if that's not a problem for you the solution might work. Regards, Bjoern -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 12:57:17 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0712E1065694; Wed, 17 Nov 2010 12:57:17 +0000 (UTC) (envelope-from zec@icir.org) Received: from munja.zvne.fer.hr (munja.zvne.fer.hr [161.53.66.248]) by mx1.freebsd.org (Postfix) with ESMTP id 8882F8FC17; Wed, 17 Nov 2010 12:57:16 +0000 (UTC) Received: from sluga.fer.hr ([161.53.66.244]) by munja.zvne.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Wed, 17 Nov 2010 13:45:11 +0100 Received: from localhost ([161.53.19.8]) by sluga.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Wed, 17 Nov 2010 13:45:10 +0100 From: Marko Zec To: freebsd-virtualization@freebsd.org Date: Wed, 17 Nov 2010 13:45:06 +0100 User-Agent: KMail/1.9.10 References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101117055208.S24596@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201011171345.06789.zec@icir.org> X-OriginalArrivalTime: 17 Nov 2010 12:45:11.0037 (UTC) FILETIME=[45BA4AD0:01CB8655] Cc: Thierry Herbelot , "Bjoern A. Zeeb" , freebsd-current Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 12:57:17 -0000 On Wednesday 17 November 2010 06:57:53 Bjoern A. Zeeb wrote: > On Wed, 17 Nov 2010, Thierry Herbelot wrote: > > Hi, > > first of all freebsd-virtualization@ is the better list for this; Cc:ed. > > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > problem : there seems to be a memory leak in the kernel, which eventually > > causes a panic. > > > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > > network stack) is a highly experimental feature.") > > > > Configuring a network interface in a jail with vnet enabled, then > > removing that jail causes these messages. In dmesg: > > > > [...] > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > > > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > (with attached VIMAGE kernel config file). > > > > The following commands reproduce the bug: > > > > jail -l -u root -c path=/ name=foo persist vnet && > > jexec foo ifconfig lo0 127.0.0.1/8 && > > jail -r foo > > > > Running it too many times exhausts kernel memory and crashes the machine. > > > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > -head. > > The problem has been present since day 1 and is still present up to > HEAD. This is about type stability and teardown. So far we are no > able to actually free TCP (and UDP in SVN) states as they might still > be accesses after "free". It's a general problem in the network stack > and has been implemented as a measure to circumvent panics in those > cases. Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. Note that the original VIMAGE prototype on FreeBSD 4.x from 2002 or 2003 used (mostly) unified memory zones, so there's no technical reason why this couldn't work again in FreeBSD current or 8-stable. Cheers, Marko > I am not sure if that's actually what's biting you wrt to memory or > if it's something else. Unfortunately boot logs and kernel configs > don't help much; enable ddb and get show uma and show malloc reports > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > restarts. I might have some patches for a couple of things but cannot > (yet) help with the additional (duplicate) UMA zones showing up at > each iteration (for the -z case). > > /bz From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 13:20:17 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38BF71065779 for ; Wed, 17 Nov 2010 13:20:17 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id BB95B8FC0A for ; Wed, 17 Nov 2010 13:20:16 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id BBD1C41C75D; Wed, 17 Nov 2010 14:20:15 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id YrHo-H5OCB6K; Wed, 17 Nov 2010 14:20:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 3368E41C75C; Wed, 17 Nov 2010 14:20:06 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id BC8C94448F3; Wed, 17 Nov 2010 13:17:57 +0000 (UTC) Date: Wed, 17 Nov 2010 13:17:57 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Marko Zec In-Reply-To: <201011171345.06789.zec@icir.org> Message-ID: <20101117131438.I24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011171345.06789.zec@icir.org> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 13:20:17 -0000 On Wed, 17 Nov 2010, Marko Zec wrote: > Actually, we never seriously discussed or revisited the issue with separate > UMA pools for each vnet instance. > > My original motivation when O introduced separate UMA pools was primarily in > making it easier to spot resource leaks, and to prove the correctness of the > whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps > the time has come to move on. Having said that, and given that the current > VIMAGE resource allocation model is far from being optimal (a lot of memory > sits reserved but 99% unused, and cannot be reclaimed later on vnet > teardown), perhaps it's time that we reconsider using unified UMA pools. I think there is a misunderstanding here; it can be reclaimed by the time we have the teardown properly sorted out and it will immediately help normal non-VIMAGE systems under memory pressure as well. The problem is that, at least for TCP (and UDP in one special case as I found after lots of testing), we are no there yet. After that, when it comes to resource usage, I am still wondering how trasz' resource limits will plug into that. By the time we can see those coming together we should be able to decide whether to go left or right. /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 15:00:57 2010 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAF46106566C for ; Wed, 17 Nov 2010 15:00:57 +0000 (UTC) (envelope-from mark@exonetric.com) Received: from relay0.exonetric.net (relay0.exonetric.net [82.138.248.161]) by mx1.freebsd.org (Postfix) with ESMTP id A5CF78FC21 for ; Wed, 17 Nov 2010 15:00:57 +0000 (UTC) Received: from markimac.fairfx.local (unknown [62.244.179.66]) by relay0.exonetric.net (Postfix) with ESMTP id B2C5C57269 for ; Wed, 17 Nov 2010 14:37:35 +0000 (GMT) Message-ID: <4CE3E8AF.8010604@exonetric.com> Date: Wed, 17 Nov 2010 14:37:35 +0000 From: Mark Blackman User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.15) Gecko/20101027 SeaMonkey/2.0.10 MIME-Version: 1.0 To: freebsd-virtualization@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: global resource accounting on a per-jail basis X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 15:00:57 -0000 Hi, Are there any plans to add CPU (at least) accounting on a per jail basis, so that I can distinguish all the CPU cycles used by jail X. While sar could be useful, it only notes user id and that's not unique across jails. I know that some kind of resource limiting is planned and that would naturally require the accounting suggested. Regards, Mark Blackman Exonetric From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 15:03:40 2010 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F61F106566B for ; Wed, 17 Nov 2010 15:03:40 +0000 (UTC) (envelope-from mark@exonetric.com) Received: from relay0.exonetric.net (relay0.exonetric.net [82.138.248.161]) by mx1.freebsd.org (Postfix) with ESMTP id 093B68FC0A for ; Wed, 17 Nov 2010 15:03:39 +0000 (UTC) Received: from markimac.fairfx.local (unknown [62.244.179.66]) by relay0.exonetric.net (Postfix) with ESMTP id 3141A57269 for ; Wed, 17 Nov 2010 15:03:39 +0000 (GMT) Message-ID: <4CE3EECA.8010709@exonetric.com> Date: Wed, 17 Nov 2010 15:03:38 +0000 From: Mark Blackman User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.15) Gecko/20101027 SeaMonkey/2.0.10 MIME-Version: 1.0 To: freebsd-virtualization@FreeBSD.org References: <4CE3E8AF.8010604@exonetric.com> In-Reply-To: <4CE3E8AF.8010604@exonetric.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: global resource accounting on a per-jail basis X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 15:03:40 -0000 Mark Blackman wrote: > Hi, > > > I know that some kind of resource limiting is planned and > that would naturally require the accounting suggested. /me bothers to google and finds... http://lists.freebsd.org/pipermail/freebsd-announce/2010-July/001335.html never mind. :) Cheers, Mark From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 15:39:01 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83F6D106564A for ; Wed, 17 Nov 2010 15:39:01 +0000 (UTC) (envelope-from jamesbrandongooch@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 548398FC19 for ; Wed, 17 Nov 2010 15:39:00 +0000 (UTC) Received: by pxi1 with SMTP id 1so557474pxi.13 for ; Wed, 17 Nov 2010 07:39:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=e204mNvBvBXd5Y+7m1kHh5x+whPjXNSKfYW9tYHBmLE=; b=CoEd6iW+taj38ZwxOCNh1u2C92Ndv0Ofh2MZvErQDwI655tQxkISyJVUUH+Khyk3zq zAlfW/GOfNENu5BjnxLmlFv174QoxmcDFDTvPI0TpClPCD6c7zgxJ4+htuPHsfqyEL0C +fdfFbQNhqL0gTzIvSdTAtjBoJCKpS4vbvdMY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=LkgcbKF3/VEabmLqygYu3PjcpyslAdeIpGnQjlb3if9TKJrbscRSZu9LICVwMIa1IA SMgiwRqmeNzcCo21tidJ5LCNnYmoEtLbqjkiPbbHZ4CT4X79zRtoWRNNsl4e5CW+ZtwL ld8xf3fp85luj/RHnRaOJONkRWR4VveiIP0og= MIME-Version: 1.0 Received: by 10.142.171.20 with SMTP id t20mr7125940wfe.242.1290006855334; Wed, 17 Nov 2010 07:14:15 -0800 (PST) Received: by 10.142.245.11 with HTTP; Wed, 17 Nov 2010 07:14:15 -0800 (PST) In-Reply-To: <20101117131438.I24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011171345.06789.zec@icir.org> <20101117131438.I24596@maildrop.int.zabbadoz.net> Date: Wed, 17 Nov 2010 09:14:15 -0600 Message-ID: From: Brandon Gooch To: "Bjoern A. Zeeb" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 15:39:01 -0000 On Wed, Nov 17, 2010 at 7:17 AM, Bjoern A. Zeeb wrote: > On Wed, 17 Nov 2010, Marko Zec wrote: > >> Actually, we never seriously discussed or revisited the issue with >> separate >> UMA pools for each vnet instance. >> >> My original motivation when O introduced separate UMA pools was primaril= y >> in >> making it easier to spot resource leaks, and to prove the correctness of >> the >> whole VIMAGE / VNET thing. =A0Having more or less achieved those goals, >> perhaps >> the time has come to move on. =A0Having said that, and given that the >> current >> VIMAGE resource allocation model is far from being optimal (a lot of >> memory >> sits reserved but 99% unused, and cannot be reclaimed later on vnet >> teardown), perhaps it's time that we reconsider using unified UMA pools. > > I think there is a misunderstanding here; =A0it can be reclaimed by the > time we have the teardown properly sorted out and it will immediately > help normal non-VIMAGE systems under memory pressure as well. > The problem is that, at least for TCP (and UDP in one special case as > I found after lots of testing), we are no there yet. > > After that, when it comes to resource usage, I am still wondering how > trasz' resource limits will plug into that. =A0By the time we can see > those coming together we should be able to decide whether to go left > or right. > I've been running into this memory exhaustion as well, having a need to stop and start my VIMAGE jails frequently. I'm confident that the proper solution will be worked out, but I wonder what sort of time-frame we may be looking at -- is VIMAGE expected to be production by 9.0-RELEASE? Also, does anyone know the current status of trasz's work (which I believe is to be completed December of this year)? I hope it's still on schedule :) -Brandon From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 06:15:27 2010 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1BE11065679; Thu, 18 Nov 2010 06:15:27 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A6A878FC20; Thu, 18 Nov 2010 06:15:27 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oAI6FRud048758; Thu, 18 Nov 2010 06:15:27 GMT (envelope-from bz@freefall.freebsd.org) Received: (from bz@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oAI6FRTZ048754; Thu, 18 Nov 2010 06:15:27 GMT (envelope-from bz) Date: Thu, 18 Nov 2010 06:15:27 GMT Message-Id: <201011180615.oAI6FRTZ048754@freefall.freebsd.org> To: x86mail@gmail.com, bz@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-virtualization@FreeBSD.org From: bz@FreeBSD.org Cc: Subject: Re: kern/152047: [vimage] [panic] TUN\TAP under jail with vimage crashes system X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 06:15:27 -0000 Synopsis: [vimage] [panic] TUN\TAP under jail with vimage crashes system State-Changed-From-To: open->analyzed State-Changed-By: bz State-Changed-When: Thu Nov 18 06:13:54 UTC 2010 State-Changed-Why: Problem is well known. Responsible-Changed-From-To: freebsd-bugs->freebsd-virtualization Responsible-Changed-By: bz Responsible-Changed-When: Thu Nov 18 06:13:54 UTC 2010 Responsible-Changed-Why: Re-assign to freebsd virtualization list as it's a VNET specific problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=152047 From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 18:36:34 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17A1F1065673; Wed, 17 Nov 2010 18:36:34 +0000 (UTC) (envelope-from thierry.herbelot@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by mx1.freebsd.org (Postfix) with ESMTP id 486C48FC13; Wed, 17 Nov 2010 18:36:30 +0000 (UTC) Received: from mail.herbelot.nom (unknown [82.227.159.103]) by smtp6-g21.free.fr (Postfix) with ESMTP id 5BA83822F6; Wed, 17 Nov 2010 19:36:24 +0100 (CET) Received: from tulipe.herbelot.nom (tulipe.herbelot.nom [192.168.2.5]) by mail.herbelot.nom (8.14.1/8.14.1) with ESMTP id oAHIaGu9018073; Wed, 17 Nov 2010 19:36:16 +0100 (CET) From: Thierry Herbelot To: freebsd-current@freebsd.org Date: Wed, 17 Nov 2010 19:36:10 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-STABLE; KDE/4.5.3; amd64; ; ) References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011170729.41894.thierry.herbelot@free.fr> In-Reply-To: <201011170729.41894.thierry.herbelot@free.fr> MIME-Version: 1.0 Content-Type: Multipart/Mixed; boundary="Boundary-00=_aCC5MxDID7T1Y5x" Message-Id: <201011171936.10764.thierry.herbelot@free.fr> X-Mailman-Approved-At: Thu, 18 Nov 2010 08:13:30 +0000 Cc: "Bjoern A. Zeeb" , FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 18:36:34 -0000 --Boundary-00=_aCC5MxDID7T1Y5x Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thierry Herbelot a =E9crit > "Bjoern A. Zeeb" a =E9crit >=20 > > On Wed, 17 Nov 2010, Thierry Herbelot wrote: > >=20 > > Hi, > >=20 > > first of all freebsd-virtualization@ is the better list for this; Cc:ed. >=20 > (in fact, I did not know where else to send this message : -net, ... ? > thanks for the CC) >=20 > > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > > problem : there seems to be a memory leak in the kernel, which > > > eventually causes a panic. > > >=20 > > > (yes, we have seen the following message : "WARNING: VIMAGE > > > (virtualized network stack) is a highly experimental feature.") > > >=20 > > > Configuring a network interface in a jail with vnet enabled, then > > > removing that jail causes these messages. In dmesg: > > >=20 > > > [...] > > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > > >=20 > > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > > (with attached VIMAGE kernel config file). > > >=20 > > > The following commands reproduce the bug: > > >=20 > > > jail -l -u root -c path=3D/ name=3Dfoo persist vnet && > > > jexec foo ifconfig lo0 127.0.0.1/8 && > > > jail -r foo > > >=20 > > > Running it too many times exhausts kernel memory and crashes the > > > machine. > > >=20 > > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > > -head. > >=20 > > The problem has been present since day 1 and is still present up to > > HEAD. This is about type stability and teardown. So far we are no > > able to actually free TCP (and UDP in SVN) states as they might still > > be accesses after "free". It's a general problem in the network stack > > and has been implemented as a measure to circumvent panics in those > > cases. > >=20 > > I am not sure if that's actually what's biting you wrt to memory or > > if it's something else. Unfortunately boot logs and kernel configs > > don't help much; enable ddb and get show uma and show malloc reports > > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > > restarts. I might have some patches for a couple of things but cannot > > (yet) help with the additional (duplicate) UMA zones showing up at > > each iteration (for the -z case). >=20 > The context for our problem is that VIMAGE jails are mant to be used "in > production" for automated tests, thus are likely to be multiple on one > physical host, and are likely to be started and stopped according to our > needs. >=20 > We have tried more "classical" virtualization solutions (qemu, kvm, ... , > normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems > to be the correct answer. (for other reasons, an i386 version of the > kernel must be used) >=20 > The point of my original email was to simplify the configuration for a te= st > setup : any machine with the attached configuration file and the above > sequence of commands creating and deleting jails, running in a loop, will > eventually panic. >=20 > Anyway, I will follow up with the logs you mentioned. As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug= on=20 a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes fo= r=20 each vnet jail creation/destruction cycle) Cheers Thierry >=20 > Cheers (and thanks for the quick feedback) >=20 > Thierry Herbelot >=20 > > /bz >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" --Boundary-00=_aCC5MxDID7T1Y5x Content-Type: text/plain; charset="ISO-8859-15"; name="ddb-log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="ddb-log" [root@waldorf ~]# while jail -l -u root -c path=/ name=foo persist vnet && jexe c foo ifconfig lo0 127.0.0.1/8 && jail -r foo ; do continue ; done lock order reversal: 1st 0xc0bed490 allprison (allprison) @ kern/kern_jail.c:914 2nd 0xc0d5c56c vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ net/vnet.c:618 KDB: stack backtrace: db_trace_self_wrapper(c0ab1220,e9867a08,c05fc285,c05ec9db,c0ab40dc,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c05ec9db,c0ab40dc,c6d2d138,c6d30c88,e9867a64,...) at kdb_backtrace+0x29 _witness_debugger(c0ab40dc,c0d5c56c,c0abf638,c6d30c88,c0abf794,...) at _witness_debugger+0x25 witness_checkorder(c0d5c56c,1,c0abf78b,26a,0,...) at witness_checkorder+0x839 _sx_slock(c0d5c56c,0,c0abf78b,26a,0,...) at _sx_slock+0x85 vnet_sysinit(c7acf000,c0bb5d00,6180,0,c0b80888,...) at vnet_sysinit+0x2b vnet_alloc(c6f73028,c0aa9de6,0,10,c0aa5dc0,...) at vnet_alloc+0x9e kern_jail_set(c77ed000,c767b380,1,c767b380,280930fc,...) at kern_jail_set+0x179a jail_set(c77ed000,e9867cf8,c0ae5dd0,c0ab4ea5,c77e87f8,...) at jail_set+0x50 syscall(e9867d38) at syscall+0x220 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (507, FreeBSD ELF32, jail_set), eip = 0x280eae1b, esp = 0xbf7feb7c, ebp = 0xbf7fec38 --- Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. panic: kmem_malloc(131072): kmem_map too small: 335409152 total allocated cpuid = 0 KDB: enter: panic [thread pid 2129 tid 100072 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> bt Tracing pid 2129 tid 100072 td 0xc771fc80 kdb_enter(c0aadf11,c0aadf11,c0ad7eba,e95fa94c,0,...) at kdb_enter+0x3a panic(c0ad7eba,20000,13fdf000,c0ad7eb4,7d0,...) at panic+0x136 kmem_malloc(c149008c,20000,102,e95fa9d8,c082951d,...) at kmem_malloc+0x280 page_alloc(0,20000,e95fa9cb,102,0,...) at page_alloc+0x27 uma_large_malloc(20000,102,102,2,1,...) at uma_large_malloc+0x4d malloc(20000,c0b90240,102,c0585588,20000,...) at malloc+0x118 flowtable_alloc(c0ac790e,8000,2,0,0,...) at flowtable_alloc+0xee ip_init(40,c7629dc0,0,e95faa80,c0617807,...) at ip_init+0x330 protosw_init(40,c0b8e754,c0b941b4,c7629dc0,e95faa8c,...) at protosw_init+0x139 domain_init(c0b93f20,e95faaa8,c068bdde,c0b93f20,0,...) at domain_init+0x27 vnet_domain_init(c0b93f20,0,c0abf78b,26a,0,...) at vnet_domain_init+0x11 vnet_sysinit(c718c000,c0bb5d00,6180,0,c0b80888,...) at vnet_sysinit+0x3e vnet_alloc(c877e028,c0aa9de6,0,10,c0aa5dc0,...) at vnet_alloc+0x9e kern_jail_set(c771fc80,c7676e80,1,c7676e80,280930fc,...) at kern_jail_set+0x179a jail_set(c771fc80,e95facf8,c0ae5dd0,c0ab4ea5,c771ad48,...) at jail_set+0x50 syscall(e95fad38) at syscall+0x220 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (507, FreeBSD ELF32, jail_set), eip = 0x280eae1b, esp = 0xbf7feb7c, ebp = 0xbf7fec38 --- db> show uma Zone Size Used Free Requests ipq 32 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 rtentry 108 0 72 6 ripcb 220 0 0 0 FFS2 dinode 256 397 83 420 FFS1 dinode 128 0 0 0 FFS inode 116 397 98 420 Mountpoints 644 3 15 3 SWAPMETA 276 0 0 0 ip6flow 64 0 0 0 ip4flow 40 0 0 0 selfd 28 13 622 953 rtentry 108 9 99 9 unpcb 172 4 88 41 ripcb 220 0 0 0 sctp_asconf_ack 24 0 0 0 sctp_asconf 24 0 0 0 sctp_stream_msg_out 64 0 0 0 sctp_readq 76 0 0 0 sctp_chunk 92 0 0 0 sctp_raddr 432 0 0 0 sctp_laddr 24 0 145 3 sctp_asoc 1484 0 0 0 sctp_ep 860 0 0 0 sackhole 20 0 0 0 tcpreass 20 0 0 0 hostcache 76 0 0 0 syncache 112 0 0 0 tcptw 52 0 0 0 tcpcb 632 2 10 2 tcp_inpcb 220 2 34 2 udpcb 8 2 404 215 udp_inpcb 220 2 70 215 ipq 32 0 0 0 socket 412 8 28 405 bridge_rtnode 36 20 283 24 KNOTE 72 0 0 0 itimer 220 0 0 0 ksiginfo 80 56 232 153 pipe 392 0 90 1453 NFSNODE 468 0 0 0 NFSMOUNT 524 0 0 0 DIRHASH 1024 0 48 26 NAMEI 1024 0 48 17675 L VFS Cache 292 0 0 0 S VFS Cache 72 410 226 917 VNODEPOLL 60 0 0 0 VNODE 268 422 68 447 ata_composite 180 0 0 0 ata_request 204 0 190 2955 ttyoutq 256 156 84 420 ttyinq 152 300 168 810 g_bio 140 0 224 15483 mbuf_ext_refcnt 4 0 0 0 mbuf_jumbo_16k 16384 0 0 0 mbuf_jumbo_9k 9216 0 0 0 mbuf_jumbo_page 4096 0 0 0 mbuf_cluster 2048 512 42 512 mbuf 256 1 387 3924 mbuf_packet 256 256 256 1367 audit_record 816 0 0 0 cpuset 40 2 366 148 VMSPACE 236 17 159 2112 SLEEPQUEUE 48 133 221 133 THREAD 636 113 19 113 PROC 680 35 37 2129 MAC labels 20 0 0 0 umtx pi 52 0 0 0 TURNSTILE 72 133 77 133 Files 56 40 563 8850 4096 4096 2534 28 7824 2048 2048 273 109 1052 1024 1024 68 192 3976 512 512 92 76 4304 256 256 904 131 16844 128 128 2546 484 12434 64 64 5488 707 49123 32 32 3077 426 40826 16 16 3258 1005 42186 mt_zone 2056 270 0 270 SG fakepg 76 0 0 0 DP fakepg 76 0 0 0 PDPT 32 137 428 137 MAP ENTRY 72 280 356 51654 KMAP ENTRY 72 59 471 34398 MAP 140 7 21 7 VM OBJECT 136 667 232 27531 128 Bucket 524 93 12 1366 64 Bucket 268 206 18 260 32 Bucket 140 78 34 259 16 Bucket 76 56 44 82 UMA Hash 128 4 26 4 UMA RCntSlabs 544 277 3 277 UMA Slabs 284 5388 16 18442 UMA Zones 888 237 19 1770 UMA Kegs 128 237 33 1770 db> show malloc Type InUse MemUse Requests CAM periph 2 1K 12 scsi_cd 0 0K 0 acpicmbat 0 0K 0 $PIR 0 0K 0 acpidev 68 3K 68 CAM SIM 1 1K 1 isofs_mount 0 0K 0 isofs_node 0 0K 0 isadev 8 1K 8 ast_driver 0 0K 0 afd_driver 0 0K 0 nexusdev 4 1K 4 msi 3 1K 3 mptable 0 0K 0 memdesc 1 4K 1 MCA 0 0K 0 acd_driver 1 2K 1 ar_driver 0 0K 6 legacydrv 0 0K 0 io_apic 1 1K 1 ad_driver 1 1K 1 ata_pci 0 0K 0 ata_dma 0 0K 0 ata_generic 2 2K 2 scsi_da 0 0K 0 madt_table 0 0K 0 CAM queue 3 1K 7 GEOM 66 181K 5441 apmdev 2 1K 3 mpt_user 0 0K 0 CAM dev queue 1 1K 1 pfs_vncache 0 0K 0 pfs_nodes 21 3K 21 nullfs_mount 0 0K 0 atkbddev 2 1K 2 nullfs_hash 1 1K 1 vm_pgdata 1 64K 1 nullfs_node 0 0K 0 msdosfs_mount 0 0K 0 msdosfs_fat 0 0K 0 UMAHash 0 0K 0 ufs_mount 6 93K 6 ufs_quota 0 0K 0 ufs_dirhash 9 5K 27 pagedep 1 64K 1 inodedep 1 256K 1 newblk 1 1K 1 bmsafemap 0 0K 0 allocdirect 0 0K 0 indirdep 0 0K 0 allocindir 0 0K 0 freefrag 0 0K 0 freeblks 0 0K 0 freefile 0 0K 0 diradd 0 0K 0 mkdir 0 0K 0 dirrem 0 0K 0 newdirblk 0 0K 0 savedino 0 0K 0 mactemp 0 0K 0 audit_trigger 0 0K 0 audit_pipe 0 0K 0 audit_pipeent 0 0K 0 audit_pipe_presel 0 0K 0 audit_evclass 172 3K 211 audit_bsm 0 0K 0 audit_cred 0 0K 0 audit_data 0 0K 0 audit_path 0 0K 0 audit_text 0 0K 0 audit_gidset 0 0K 0 rpc 2 5K 2 NLM 0 0K 0 nfss_srvsock 0 0K 0 nfss_srvdesc 0 0K 0 nfss_daemon 0 0K 0 NFS FHA 1 1K 1 nfsclient_lock 0 0K 0 nfsclient_nlminfo 0 0K 0 nfsclient_req 0 0K 0 nfsclient_bigfh 0 0K 0 nfsclient_diroff 0 0K 0 nfsclient_hash 0 0K 0 nfsclient_directio 0 0K 0 nfsclient_srvsock 0 0K 0 mld 10 2K 83 msdosfs_fileno 0 0K 0 msdosfs_node 0 0K 0 in6_mfilter 0 0K 0 in6_multi 12 2K 669 ip6_moptions 0 0K 0 ip6_msource 0 0K 0 DEVFSP 0 0K 0 DEVFS 16 1K 17 DEVFS_RULE 0 0K 0 fragment 0 0K 0 syncache 1 72K 74 tcplog 0 0K 0 hostcache 1 16K 74 sctp_map 0 0K 0 sctp_stri 0 0K 0 sctp_stro 0 0K 0 sctp_aadr 0 0K 0 sctp_a_it 0 0K 103 sctp_atcl 0 0K 0 sctp_atky 0 0K 0 sctp_athm 0 0 sctp_athi 0 0K 0 sctp_stre 0 0K 0 sctp_cmsg 0 0K 0 sctp_cpal 0 0K 0 sctp_vrf 1 1K 74 sctp_ifa 99 13K 223 sctp_ifn 2 1K 75 sctp_timw 0 0K 0 sctp_mvrf 0 0K 0 sctp_iter 0 0K 103 sctp_socko 0 0K 0 encap_export_host 0 0K 0 in_mfilter 0 0K 0 in_multi 2 1K 75 ip_moptions 0 0K 0 ip_msource 0 0K 0 DEVFS2 0 0K 0 DEVFS3 206 26K 330 DEVFS1 190 48K 315 ipid 0 0K 0 igmp 10 2K 83 80211scan 0 0K 0 80211ratectl 0 0K 0 80211power 0 0K 0 80211node 0 0K 0 80211nodeie 0 0K 0 80211mesh 0 0K 0 80211com 0 0K 0 80211dfs 0 0K 0 80211crypto 0 0K 0 80211vap 0 0K 0 vnet 2 1K 75 vnet_data 2 56K 75 vnet_data_free 1 1K 1 routetbl 2684 303890K 3469 CAM ccb queue 0 0K 0 ata_da 0 0K 0 scsi_ch 0 0K 0 mfibuf 0 0K 0 md_disk 0 0K 0 md_sectors 0 0K 0 LED 2 1K 2 kbdmux 7 10K 7 vlan 0 0K 0 tun 0 0K 0 amr 0 0K 0 lltable 23 6K 315 gif 0 0K 0 fw_com 0 0K 0 faith 0 0K 0 arpcom 8 1K 8 epair 4 1K 4 clone 8 32K 81 ifdescr 0 0K 0 ifnet 12 11K 158 ifaddr 93 18K 823 ether_multi 15 1K 600 BPF 10 1K 83 subr_export_host 0 0K 0 mount 50 3K 2211 vnodemarker 0 0K 385 acpisem 15 2K 15 vnodes 2 1K 2 vfs_hash 1 256K 1 export_host 0 0K 0 cl_savebuf 0 0K 0 vfscache 1 512K 1 biobuf 4 8K 6 acl 0 0K 0 soname 3 1K 89 pcb 14 79K 891 ipsbuf 0 0K 0 shmfd 1 4K 1 ksem 1 4K 1 mbuf_tag 0 0K 1 accf 0 0K 0 pts 0 0K 0 tty 36 18K 38 shm 1 12K 1 sem 4 6K 4 msg 4 25K 4 ioctlops 0 0K 13640 select 6 1K 6 iov 1 1K 745 Witness 1 104K 1 iirbuf 0 0K 0 if_fwip 0 0K 0 ata_pmp 0 0K 0 USB 45 68K 46 Unitno 11 1K 403 taskqueue 15 1K 15 stack 0 0K 2 USBdev 29 7K 29 USBHC 0 0K 0 if_fwe 0 0K 0 sglist 0 0K 0 sbuf 0 0K 2741 CAM XPT 12 2K 26 rman 233 15K 683 acpitask 1 1K 1 acpica 3082 163K 65871 Per-cpu 1 1K 1 kobj 255 510K 363 aaccam 0 0K 0 eventhandler 299 11K 299 devstat 32 65K 32 bus 1110 52K 5921 bus-sc 81 171K 2508 fwmem 0 0K 0 SWAP 0 0K 0 p1003.1b 1 1K 1 umtx 132 9K 132 callout 3 768K 3 sysctl 0 0K 1420 sysctloid 4004 122K 4124 sysctltmp 0 0K 535 firewire 0 0K 0 plimit 12 3K 355 uidinfo 2 2K 17 cred 20 2K 10282 pgrp 17 2K 472 session 15 1K 31 proc 2 8K 2 subproc 109 163K 2203 fw_xfer 0 0K 0 osd 0 0K 0 aacbuf 0 0K 0 UART 51 22K 51 mtx_pool 2 8K 2 twe_commands 0 0K 0 module 379 24K 379 twa_commands 0 0K 0 cache 0 0K 0 devbuf 5642 12692K 5728 temp 59 231K 52869 ip6opt 0 0K 0 ip6ndp 11 1K 157 acpipwr 0 0K 0 free 0 0K 0 lockf 14 1K 18 linker 108 4K 110 ddb_capture 1 48K 1 KTRACE 100 13K 100 ciss_data 0 0K 0 entropy 1024 64K 1024 prison 1 2K 74 ithread 99 8K 99 PUC 8 2K 8 Fail Points 0 0K 0 zombie 0 0K 0 proc-args 17 1K 771 ppbusdev 3 1K 3 kqueue 0 0K 0 kenv 79 7K 83 filedesc 40 14K 2572 filedesc_to_leader 0 0K 0 sigio 1 1K 1 acpi_perf 4 1K 4 SCSI SES 0 0K 0 tty console 0 0K 0 cdev 9 2K 9 pci_link 16 2K 16 SCSI sa 0 0K 0 db> --Boundary-00=_aCC5MxDID7T1Y5x-- From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 08:17:35 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18378106566B; Thu, 18 Nov 2010 08:17:35 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id C32D78FC13; Thu, 18 Nov 2010 08:17:34 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 23C5341C710; Thu, 18 Nov 2010 09:17:34 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id d7CNhwaOADxN; Thu, 18 Nov 2010 09:17:33 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 2ADDE41C705; Thu, 18 Nov 2010 09:17:33 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 1B6584448F3; Thu, 18 Nov 2010 08:16:23 +0000 (UTC) Date: Thu, 18 Nov 2010 08:16:22 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Thierry Herbelot In-Reply-To: <201011171936.10764.thierry.herbelot@free.fr> Message-ID: <20101118081512.K24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011170729.41894.thierry.herbelot@free.fr> <201011171936.10764.thierry.herbelot@free.fr> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-current@freebsd.org, FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 08:17:35 -0000 On Wed, 17 Nov 2010, Thierry Herbelot wrote: > As promised, here are the full logs (in attachment) > > This is a serial console log showing the command loop that triggers the bug on > a debug kernel and ensuing DDB session. > > the obvious problem line is : > routetbl 2684 303890K 3469 > > (further tests showed an increase of the routetbl malloc zone by 4MBytes for > each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 10:55:13 2010 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33DEC1065679; Thu, 18 Nov 2010 10:55:13 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 697048FC0A; Thu, 18 Nov 2010 10:55:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oAIAtBXB068850; Thu, 18 Nov 2010 10:55:11 GMT (envelope-from bz@freefall.freebsd.org) Received: (from bz@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oAIAtBiQ068846; Thu, 18 Nov 2010 10:55:11 GMT (envelope-from bz) Date: Thu, 18 Nov 2010 10:55:11 GMT Message-Id: <201011181055.oAIAtBiQ068846@freefall.freebsd.org> To: bz@FreeBSD.org, bz@FreeBSD.org, freebsd-virtualization@FreeBSD.org From: bz@FreeBSD.org Cc: Subject: Re: kern/141696: [rum] [panic] rum(4)+ vimage = kernel panic X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 10:55:13 -0000 Synopsis: [rum] [panic] rum(4)+ vimage = kernel panic Responsible-Changed-From-To: bz->freebsd-virtualization Responsible-Changed-By: bz Responsible-Changed-When: Thu Nov 18 10:54:38 UTC 2010 Responsible-Changed-Why: MOve it back to the list for the moment. Easier to have all open problems at one single point. http://www.freebsd.org/cgi/query-pr.cgi?pr=141696 From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 10:56:20 2010 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B527D106564A; Thu, 18 Nov 2010 10:56:20 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8C26C8FC0A; Thu, 18 Nov 2010 10:56:20 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oAIAuKN7068915; Thu, 18 Nov 2010 10:56:20 GMT (envelope-from bz@freefall.freebsd.org) Received: (from bz@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oAIAuK0i068911; Thu, 18 Nov 2010 10:56:20 GMT (envelope-from bz) Date: Thu, 18 Nov 2010 10:56:20 GMT Message-Id: <201011181056.oAIAuK0i068911@freefall.freebsd.org> To: bz@FreeBSD.org, bz@FreeBSD.org, freebsd-virtualization@FreeBSD.org From: bz@FreeBSD.org Cc: Subject: Re: kern/147950: [vimage] [carp] VIMAGE + CARP = kernel crash X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 10:56:20 -0000 Synopsis: [vimage] [carp] VIMAGE + CARP = kernel crash Responsible-Changed-From-To: bz->freebsd-virtualization Responsible-Changed-By: bz Responsible-Changed-When: Thu Nov 18 10:55:27 UTC 2010 Responsible-Changed-Why: Move it back to the list for the moment. Easier to have all open problems at one single point. http://www.freebsd.org/cgi/query-pr.cgi?pr=147950 From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 20:29:08 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91C85106566B; Thu, 18 Nov 2010 20:29:08 +0000 (UTC) (envelope-from thierry.herbelot@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by mx1.freebsd.org (Postfix) with ESMTP id 8EDDB8FC1C; Thu, 18 Nov 2010 20:29:05 +0000 (UTC) Received: from mail.herbelot.nom (unknown [82.227.159.103]) by smtp6-g21.free.fr (Postfix) with ESMTP id DB7F782260; Thu, 18 Nov 2010 21:28:59 +0100 (CET) Received: from tulipe.herbelot.nom (tulipe.herbelot.nom [192.168.2.5]) by mail.herbelot.nom (8.14.1/8.14.1) with ESMTP id oAIKSoc2026327; Thu, 18 Nov 2010 21:28:51 +0100 (CET) From: Thierry Herbelot To: "Bjoern A. Zeeb" Date: Thu, 18 Nov 2010 21:28:44 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-STABLE; KDE/4.5.3; amd64; ; ) References: <201011170627.28025.thierry.herbelot@free.fr> <201011171936.10764.thierry.herbelot@free.fr> <20101118081512.K24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101118081512.K24596@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201011182128.44827.thierry.herbelot@free.fr> Cc: freebsd-current@freebsd.org, FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 20:29:08 -0000 "Bjoern A. Zeeb" a =E9crit > On Wed, 17 Nov 2010, Thierry Herbelot wrote: > > As promised, here are the full logs (in attachment) > >=20 > > This is a serial console log showing the command loop that triggers the > > bug on a debug kernel and ensuing DDB session. > >=20 > > the obvious problem line is : > > routetbl 2684 303890K 3469 > >=20 > > (further tests showed an increase of the routetbl malloc zone by 4MBytes > > for each vnet jail creation/destruction cycle) >=20 > Hmm, I had fixed that (somewhere). I'll see where the patch went. You > are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 AS= AP Thanks TfH >=20 > /bz From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 20:55:09 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F357106564A; Thu, 18 Nov 2010 20:55:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 26A5A8FC0A; Thu, 18 Nov 2010 20:55:08 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id DD8F541C747; Thu, 18 Nov 2010 21:55:06 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id UHOzSxamPhup; Thu, 18 Nov 2010 21:55:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 061B141C750; Thu, 18 Nov 2010 21:55:06 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 1828A4448F3; Thu, 18 Nov 2010 20:51:23 +0000 (UTC) Date: Thu, 18 Nov 2010 20:51:22 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Thierry Herbelot In-Reply-To: <201011182128.44827.thierry.herbelot@free.fr> Message-ID: <20101118203639.K24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <201011171936.10764.thierry.herbelot@free.fr> <20101118081512.K24596@maildrop.int.zabbadoz.net> <201011182128.44827.thierry.herbelot@free.fr> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-64664707-1290113482=:24596" Cc: freebsd-current@freebsd.org, FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 20:55:09 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-64664707-1290113482=:24596 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Thu, 18 Nov 2010, Thierry Herbelot wrote: > "Bjoern A. Zeeb" a =E9crit >> On Wed, 17 Nov 2010, Thierry Herbelot wrote: >>> As promised, here are the full logs (in attachment) >>> >>> This is a serial console log showing the command loop that triggers the >>> bug on a debug kernel and ensuing DDB session. >>> >>> the obvious problem line is : >>> routetbl 2684 303890K 3469 >>> >>> (further tests showed an increase of the routetbl malloc zone by 4MByte= s >>> for each vnet jail creation/destruction cycle) >> >> Hmm, I had fixed that (somewhere). I'll see where the patch went. You >> are on 8.1-RELEASE or -STABLE? > > This will be for a -release, thus 8.1 for now, but we will switch to 8.2 = ASAP Well wait; I am not sure the changes are in SVN at all. I'll get back to you. /bz --=20 Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html --0-64664707-1290113482=:24596-- From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 22:04:27 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 730AB106564A; Thu, 18 Nov 2010 22:04:27 +0000 (UTC) (envelope-from thierry.herbelot@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by mx1.freebsd.org (Postfix) with ESMTP id 3FEC48FC18; Thu, 18 Nov 2010 22:04:24 +0000 (UTC) Received: from mail.herbelot.nom (unknown [82.227.159.103]) by smtp6-g21.free.fr (Postfix) with ESMTP id 38801822A0; Thu, 18 Nov 2010 23:04:18 +0100 (CET) Received: from tulipe.herbelot.nom (tulipe.herbelot.nom [192.168.2.5]) by mail.herbelot.nom (8.14.1/8.14.1) with ESMTP id oAIM4CO3010277; Thu, 18 Nov 2010 23:04:12 +0100 (CET) From: Thierry Herbelot To: "Bjoern A. Zeeb" Date: Thu, 18 Nov 2010 23:04:06 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-STABLE; KDE/4.5.3; amd64; ; ) References: <201011170627.28025.thierry.herbelot@free.fr> <201011182128.44827.thierry.herbelot@free.fr> <20101118203639.K24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101118203639.K24596@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201011182304.06712.thierry.herbelot@free.fr> Cc: freebsd-current@freebsd.org, FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 22:04:27 -0000 "Bjoern A. Zeeb" a =E9crit > On Thu, 18 Nov 2010, Thierry Herbelot wrote: > > "Bjoern A. Zeeb" a =E9crit > >=20 > >> On Wed, 17 Nov 2010, Thierry Herbelot wrote: > >>> As promised, here are the full logs (in attachment) > >>>=20 > >>> This is a serial console log showing the command loop that triggers t= he > >>> bug on a debug kernel and ensuing DDB session. > >>>=20 > >>> the obvious problem line is : > >>> routetbl 2684 303890K 3469 > >>>=20 > >>> (further tests showed an increase of the routetbl malloc zone by > >>> 4MBytes for each vnet jail creation/destruction cycle) > >>=20 > >> Hmm, I had fixed that (somewhere). I'll see where the patch went. You > >> are on 8.1-RELEASE or -STABLE? > >=20 > > This will be for a -release, thus 8.1 for now, but we will switch to 8.2 > > ASAP >=20 > Well wait; I am not sure the changes are in SVN at all. I'll get back > to you. OK for us : we are still in the process of deploying the solution, but stil= l=20 we will test any patch you can forward ;-) TfH >=20 > /bz From owner-freebsd-virtualization@FreeBSD.ORG Thu Nov 18 19:00:07 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF0831065679 for ; Thu, 18 Nov 2010 19:00:06 +0000 (UTC) (envelope-from gofd-freebsd-virtualization@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 53A838FC1A for ; Thu, 18 Nov 2010 19:00:05 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1PJ9Tj-0005VQ-GI for freebsd-virtualization@freebsd.org; Thu, 18 Nov 2010 19:45:03 +0100 Received: from rrcs-208-105-236-250.nys.biz.rr.com ([208.105.236.250]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 18 Nov 2010 19:45:03 +0100 Received: from danmartinj by rrcs-208-105-236-250.nys.biz.rr.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 18 Nov 2010 19:45:03 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-virtualization@freebsd.org From: Dan Date: Thu, 18 Nov 2010 18:33:22 +0000 (UTC) Lines: 75 Message-ID: References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011171345.06789.zec@icir.org> <20101117131438.I24596@maildrop.int.zabbadoz.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: sea.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 208.105.236.250 (Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16) X-Mailman-Approved-At: Fri, 19 Nov 2010 06:35:30 +0000 Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2010 19:00:07 -0000 Brandon Gooch writes: > > On Wed, Nov 17, 2010 at 7:17 AM, Bjoern A. Zeeb > wrote: > > On Wed, 17 Nov 2010, Marko Zec wrote: > > > >> Actually, we never seriously discussed or revisited the issue with > >> separate > >> UMA pools for each vnet instance. > >> > >> My original motivation when O introduced separate UMA pools was primarily > >> in > >> making it easier to spot resource leaks, and to prove the correctness of > >> the > >> whole VIMAGE / VNET thing.  Having more or less achieved those goals, > >> perhaps > >> the time has come to move on.  Having said that, and given that the > >> current > >> VIMAGE resource allocation model is far from being optimal (a lot of > >> memory > >> sits reserved but 99% unused, and cannot be reclaimed later on vnet > >> teardown), perhaps it's time that we reconsider using unified UMA pools. > > > > I think there is a misunderstanding here;  it can be reclaimed by the > > time we have the teardown properly sorted out and it will immediately > > help normal non-VIMAGE systems under memory pressure as well. > > The problem is that, at least for TCP (and UDP in one special case as > > I found after lots of testing), we are no there yet. > > > > After that, when it comes to resource usage, I am still wondering how > > trasz' resource limits will plug into that.  By the time we can see > > those coming together we should be able to decide whether to go left > > or right. > > > > I've been running into this memory exhaustion as well, having a need > to stop and start my VIMAGE jails frequently. > > I'm confident that the proper solution will be worked out, but I > wonder what sort of time-frame we may be looking at -- is VIMAGE > expected to be production by 9.0-RELEASE? Also, does anyone know the > current status of trasz's work (which I believe is to be completed > December of this year)? I hope it's still on schedule :) > > -Brandon > Hey there, I have been experiencing a similar problem. I am running Freebsd8.1 64bit Release and after closing my server application I come across this type of message Freed UMA keg was not empty (672 items). Lost 4 pages of memory The more virtual nodes I use the more of these message I have. Someone told me this does not mean anything but after reading this it seems I should be worried. It does show up in my log files as well. I wonder if running a fsck will resolve any issues after I have this problem? Hopefully we can find a solution because I rely on my application heavily. Thanks, Dan From owner-freebsd-virtualization@FreeBSD.ORG Fri Nov 19 06:40:09 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11EC1106566C for ; Fri, 19 Nov 2010 06:40:09 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 8A1048FC0A for ; Fri, 19 Nov 2010 06:40:08 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id E309641C679; Fri, 19 Nov 2010 07:40:06 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id JG5MWsAByykF; Fri, 19 Nov 2010 07:40:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 01B2B41C66F; Fri, 19 Nov 2010 07:40:05 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id EAA234448F3; Fri, 19 Nov 2010 06:39:57 +0000 (UTC) Date: Fri, 19 Nov 2010 06:39:57 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Dan In-Reply-To: Message-ID: <20101119063901.V24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> <201011171345.06789.zec@icir.org> <20101117131438.I24596@maildrop.int.zabbadoz.net> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Nov 2010 06:40:09 -0000 On Thu, 18 Nov 2010, Dan wrote: Hi, > I have been experiencing a similar problem. I am running > Freebsd8.1 64bit Release and after closing > my server application I come across > > this type of message > Freed UMA keg was not empty (672 items). Lost 4 pages of > memory > > The more virtual nodes I use the more of these message I have. > Someone told me this does not mean > anything but after reading this it seems I should be worried. > It does show up in my log files as well. > I wonder if running a fsck will resolve any issues after I > have this problem? Hopefully we can find a > solution because I rely on my application heavily. No, fsck will not help you anything; this is a memory (RAM) not a disk storage/file system problem. /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html