From owner-freebsd-xen@FreeBSD.ORG Tue Feb 10 09:22:37 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 497F7790 for ; Tue, 10 Feb 2015 09:22:37 +0000 (UTC) Received: from mail-pd0-f172.google.com (mail-pd0-f172.google.com [209.85.192.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B389D5D for ; Tue, 10 Feb 2015 09:22:36 +0000 (UTC) Received: by pdev10 with SMTP id v10so15325408pde.10 for ; Tue, 10 Feb 2015 01:22:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=wdT2EsAtPrnoIr16FWDXElR10gc3gmOJPKSxt3sKNrU=; b=IBug9AzG9nFQCE2cKn/+kxxnRV2RaEUC9t56EqBNW4vwrS/gcQoEmP2UaAPN44M8tp JqqPsEtMMP7oxXnczQDxv3u974WBKrA12uoPz69ZVuA7uDw5q1ZcP/w6D34IqD0kt9mu t1HcRyfJmDNgr9RFfn6Z+koZx4et0sNfjPBYXew4anUQgvqnmHTGiAD3cUhCCMfo1UTl LX5IbSEMdssa/VaGzYxAlNvdHKTkSfI3D8IOtZIv7ALc8v0btXrlNs1og8BPzZ5tFlOk a6P5V90vtOHlkBE+Ngp3rjrJBu//kfAOa1AkIs93fF+XvCnja6j4IFetEpBvvJQJ8DVx Ly/w== X-Gm-Message-State: ALoCoQlwk8KXNY7xR4l6wzxx6nKBcEgYowksVG0Fg4j7iYlbo6Vi4LhOjJeTrh+0XgZrdQVj/gBW MIME-Version: 1.0 X-Received: by 10.70.41.231 with SMTP id i7mr35709372pdl.102.1423560155345; Tue, 10 Feb 2015 01:22:35 -0800 (PST) Sender: andy@fud.org.nz Received: by 10.70.129.238 with HTTP; Tue, 10 Feb 2015 01:22:35 -0800 (PST) In-Reply-To: References: <54D88BD5.7050703@citrix.com> Date: Tue, 10 Feb 2015 22:22:35 +1300 X-Google-Sender-Auth: VHMkaFlrY9uAcYuo0rzjRpv2F5U Message-ID: Subject: Re: xenstore memory issue From: Andrew Thompson To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Feb 2015 09:22:37 -0000 On 10 February 2015 at 16:25, Andrew Thompson wrote: > On 9 February 2015 at 23:28, Roger Pau Monn=C3=A9 = wrote: > >> Hello, >> >> El 09/02/15 a les 10.39, Andrew Thompson ha escrit: >> > Hi, >> > >> > >> > I have three VMs with Rackspace and one is behaving oddly with xenstor= e >> > memory consumption. Here are the kernel versions and vmstat -m results= . >> > >> > > As you can see the third VM is using 100MB in xenstore memory and it >> seems >> > to be climbing by 1-2MB per hour. Eventually all the processes go in t= o >> > pfault state and it grinds to a halt. >> >> That's certainly weird, are you doing something different on this VM as >> compared to the others? Did you hot-add a nic, disk or ballooned memory? >> >> Has the VM been saved/restored or migrated? >> >> Tracking down this kind of xenstore leaks can be difficult without >> having a way to reproduce them. >> >> > A bit of trial and error with dtrace has narrowed this down. I can cause > the leak by just opening /dev/xen/xenstore > > int main() { > open("/dev/xen/xenstore", O_RDWR, 0); > } > > # vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore > xenstore 8739 104797K - 56078 16,32,64,128,256,512 > xenstore 8740 104809K - 56079 16,32,64,128,256,512 > > > Using dtrace probes I can see that xs_dev_close is never called. > I think I have worked this out. Rackspace use an agent called nova-agent which keeps and open handle on /dev/xen/xenstore. Since xenstore isnt using the D_TRACKCLOSE flag it will not call d_close until the last reference is dropped. Since xenstore expects to malloc/free on open and close this assumption breaks and will leak memory. If i stop nova-agent I can see xs_dev_close being called and the memory freed with testing with xenstore-read. The correct solution seems to be to set D_TRACKCLOSE if I understand its purpose correctly. regards, Andrew