From owner-freebsd-stable@FreeBSD.ORG Tue Jan 19 17:01:04 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD0B6106566B for ; Tue, 19 Jan 2010 17:01:04 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id 52B688FC18 for ; Tue, 19 Jan 2010 17:01:03 +0000 (UTC) Received: from omta12.westchester.pa.mail.comcast.net ([76.96.62.44]) by qmta14.westchester.pa.mail.comcast.net with comcast id XT9h1d0060xGWP85EV149a; Tue, 19 Jan 2010 17:01:04 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta12.westchester.pa.mail.comcast.net with comcast id XV131d0053S48mS3YV13v8; Tue, 19 Jan 2010 17:01:04 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id E9B041E3033; Tue, 19 Jan 2010 09:01:01 -0800 (PST) Date: Tue, 19 Jan 2010 09:01:01 -0800 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Message-ID: <20100119170101.GA80917@icarus.home.lan> References: <7346c5c61001030842r7dc76199y51e4c1c90a3eea6e@mail.gmail.com> <7346c5c61001091706m45a3a2a5k3ca8bb0c4bec5ea8@mail.gmail.com> <7346c5c61001171521w1ca4738w98e8fcca24643cda@mail.gmail.com> <201001180829.48126.npapke@acm.org> <7346c5c61001190840k31466754i32b2ae833390b79b@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7346c5c61001190840k31466754i32b2ae833390b79b@mail.gmail.com> User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Re: ZFS performance degradation over time X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2010 17:01:04 -0000 On Tue, Jan 19, 2010 at 11:40:50AM -0500, Garrett Moore wrote: > I've been watching my memory usage and I have no idea what is consuming > memory as 'Active'. > > Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB > buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was > again slow (<1MB/s). I tried killing rTorrent and no significant amount of > memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any > significant amount of memory, and I was definitely nowhere near 6500MB > usage. > > I tried running the perl oneliner again to hog a bunch of memory, and almost > all of the Active memory was IMMEDIATELY marked as Free, and my performance > was excellent again. > > I'm not sure what in userland could be causing the issue. The only things > I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash, > Python, Perl, and SABNZBd. There is nothing that *should* be consuming any > serious amount of memory. I've two recommendations: 1) Have you considered "upgrading" to RELENG_8 (e.g. 8.0-STABLE) instead of sticking with 8.0-RELEASE? There's been a recent MFC to RELENG_8 which pertain to ARC drainage. I'm referring to the commit labelled revision 1.22.2.2 (RELENG_8): http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c 2) Have you tried using vfs.zfs.arc_max in loader.conf to limit the ARC size? I'd recommend picking something like 1GB as a cap (your machine has 8GB total at present, if I remember right). I believe long ago someone said this isn't an explicit hard limit on the maximum size of the ARC, but I believe this was during the RELENG_7 days and the ARC "stuff" on FreeBSD has changed since then. I wish the tunables were better documented, or at least explained in detail (hello Wiki!). Finally: Does anyone have reservations about me crossposting this thread to freebsd-fs@freebsd.org, to get some additional eyes? Reports like this often scare/worry those of us who run servers. :-) > On Mon, Jan 18, 2010 at 11:29 AM, Norbert Papke wrote: > > > On January 17, 2010, Garrett Moore wrote: > > > I upgraded my system to 8GB of ram to see if that would help. It hasn't > > > made much of a difference. After having rTorrent running for a while, my > > > performance again tanked. Around 6.5GB of memory was showing as 'Active' > > > according to top. > > > > 6.5GB of "active" memory seems to imply that a user process is growing or a > > large number of user processes are being created. I would expect ZFS's > > cache > > to increase the size of "wired" memory. > > > > Sorry, I have not followed this thread closely. Are you sure that the > > degradation is ZFS related? Could it be caused by, for instance, a > > userland > > memory leak? What happens to active memory when you restart rtorrent? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |