Date: Wed, 17 Nov 2010 07:29:41 +0100 From: Thierry Herbelot <thierry.herbelot@free.fr> To: freebsd-current@freebsd.org, FreeBSD virtualization mailing list <freebsd-virtualization@freebsd.org>, "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> Subject: Re: VIMAGE: Freed UMA keg was not empty Message-ID: <201011170729.41894.thierry.herbelot@free.fr> In-Reply-To: <20101117055208.S24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net>
next in thread | previous in thread | raw e-mail | index | archive | help
"Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> a =E9crit > On Wed, 17 Nov 2010, Thierry Herbelot wrote: >=20 > Hi, >=20 > first of all freebsd-virtualization@ is the better list for this; Cc:ed. (in fact, I did not know where else to send this message : -net, ... ? than= ks=20 for the CC) >=20 > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > problem : there seems to be a memory leak in the kernel, which > > eventually causes a panic. > >=20 > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > > network stack) is a highly experimental feature.") > >=20 > > Configuring a network interface in a jail with vnet enabled, then > > removing that jail causes these messages. In dmesg: > >=20 > > [...] > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > >=20 > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > (with attached VIMAGE kernel config file). > >=20 > > The following commands reproduce the bug: > >=20 > > jail -l -u root -c path=3D/ name=3Dfoo persist vnet && > > jexec foo ifconfig lo0 127.0.0.1/8 && > > jail -r foo > >=20 > > Running it too many times exhausts kernel memory and crashes the machin= e. > >=20 > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > -head. >=20 > The problem has been present since day 1 and is still present up to > HEAD. This is about type stability and teardown. So far we are no > able to actually free TCP (and UDP in SVN) states as they might still > be accesses after "free". It's a general problem in the network stack > and has been implemented as a measure to circumvent panics in those > cases. >=20 > I am not sure if that's actually what's biting you wrt to memory or > if it's something else. Unfortunately boot logs and kernel configs > don't help much; enable ddb and get show uma and show malloc reports > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > restarts. I might have some patches for a couple of things but cannot > (yet) help with the additional (duplicate) UMA zones showing up at > each iteration (for the -z case). The context for our problem is that VIMAGE jails are mant to be used "in=20 production" for automated tests, thus are likely to be multiple on one=20 physical host, and are likely to be started and stopped according to our=20 needs. We have tried more "classical" virtualization solutions (qemu, kvm, ... ,=20 normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems = to=20 be the correct answer. (for other reasons, an i386 version of the kernel mu= st=20 be used) The point of my original email was to simplify the configuration for a test= =20 setup : any machine with the attached configuration file and the above=20 sequence of commands creating and deleting jails, running in a loop, will=20 eventually panic. Anyway, I will follow up with the logs you mentioned. Cheers (and thanks for the quick feedback) Thierry Herbelot >=20 > /bz
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201011170729.41894.thierry.herbelot>