From owner-freebsd-xen@FreeBSD.ORG Mon Feb 9 09:39:54 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2544E96 for ; Mon, 9 Feb 2015 09:39:54 +0000 (UTC) Received: from mail-pd0-f172.google.com (mail-pd0-f172.google.com [209.85.192.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB18D64C for ; Mon, 9 Feb 2015 09:39:54 +0000 (UTC) Received: by pdev10 with SMTP id v10so7429784pde.10 for ; Mon, 09 Feb 2015 01:39:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:content-type; bh=l0ihp9yQ9LwkDtmtVtNhc3Q6zR2/PBPrJO28+lRXItQ=; b=jjHbLAfCJ8skhSEDJEJ30gHiyuDD8ALbIX58/qjuwCjg2hljdS7uXq0v7peILVm/tk 3KVtfzdJjrADavI9RYwsU0DwIBw9bMEI9NZPOVrcpLl12BUrsFZhWXFOIj4xonF34860 0DcWPa2rPVUWlIyBpVsiY/A+LPYGgb3V9CZvS7iKZnfXZi0s2xXy0rHkp+1begNQu21I VzhnG/61GgMtwD1J1jliN7M92S0YOcAVfroI44BUSCWM1FZqnAptbX17BRUZwvkr/v5q +LtjmnZ0Y1r7f+mqLRWvhxSUQ7fM3cGzAkEgJEGxaQr58mR6114bhXGCQo2DsahZDrXw TJWA== X-Gm-Message-State: ALoCoQmXwHnPh1/CjXSfzlPh4Hk8tjaIz8xSztaIncAuIFTgCTJKgOacoq+i0+GOamJzHF96n6V2 MIME-Version: 1.0 X-Received: by 10.68.135.37 with SMTP id pp5mr27267929pbb.105.1423474787740; Mon, 09 Feb 2015 01:39:47 -0800 (PST) Sender: andy@fud.org.nz Received: by 10.70.129.238 with HTTP; Mon, 9 Feb 2015 01:39:47 -0800 (PST) Date: Mon, 9 Feb 2015 22:39:47 +1300 X-Google-Sender-Auth: RmMQUhwHYsJR2GOUR_gtoKApb30 Message-ID: Subject: xenstore memory issue From: Andrew Thompson To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Feb 2015 09:39:55 -0000 Hi, I have three VMs with Rackspace and one is behaving oddly with xenstore memory consumption. Here are the kernel versions and vmstat -m results. FreeBSD us.e.com 10.0-RELEASE-p9 FreeBSD 10.0-RELEASE-p9 #0: Mon Sep 15 14:35:52 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 xenbus 16 2K - 86 16,32,64,256 xenstore 409 4837K - 38424052 16,32,64,128,256 xen_hvm 2 8K - 2 4096 xen_intr 25 4K - 25 128 FreeBSD uk.e.com 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov 4 05:07:17 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 xenbus 11 2K - 83 16,32,64,256 xenstore 198 2317K - 43428137 16,32,64,128,256,512 xen_hvm 2 8K - 2 4096 xen_intr 24 3K - 24 128 FreeBSD au.e.com 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov 4 05:07:17 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 xenbus 11 2K - 83 16,32,64,256 xenstore 8477 101653K - 55249 16,32,64,128,256,512 xen_hvm 2 8K - 2 4096 xen_intr 14 2K - 14 128 As you can see the third VM is using 100MB in xenstore memory and it seems to be climbing by 1-2MB per hour. Eventually all the processes go in to pfault state and it grinds to a halt. How should I be debugging this? Is it either a local leak or the Xen host is to blame? cheers, Andew