From owner-freebsd-current@FreeBSD.ORG Mon Apr 20 05:25:36 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B05C106566B; Mon, 20 Apr 2009 05:25:36 +0000 (UTC) (envelope-from ben@wanderview.com) Received: from mail.wanderview.com (mail.wanderview.com [66.92.166.102]) by mx1.freebsd.org (Postfix) with ESMTP id CCF288FC12; Mon, 20 Apr 2009 05:25:35 +0000 (UTC) (envelope-from ben@wanderview.com) Received: from harkness.in.wanderview.com (harkness.in.wanderview.com [10.76.10.150]) (authenticated bits=0) by mail.wanderview.com (8.14.3/8.14.3) with ESMTP id n3K5PQQm002671 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 20 Apr 2009 05:25:26 GMT (envelope-from ben@wanderview.com) Message-Id: <8AF79B5A-3D10-4344-BA2F-02DF84BB3F8A@wanderview.com> From: Ben Kelly To: current@freebsd.org In-Reply-To: <6535218D-6292-4F84-A8BA-FFA9B2E47F80@wanderview.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Mon, 20 Apr 2009 01:25:26 -0400 References: <20090417145024.205173ighmwi4j0o@webmail.leidinger.net> <20090418094821.00002e67@unknown> <6535218D-6292-4F84-A8BA-FFA9B2E47F80@wanderview.com> X-Mailer: Apple Mail (2.930.3) X-Spam-Score: -1.44 () ALL_TRUSTED X-Scanned-By: MIMEDefang 2.64 on 10.76.20.1 Cc: fs@freebsd.org Subject: Re: ZFS: unlimited arc cache growth? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Apr 2009 05:25:36 -0000 On Apr 18, 2009, at 5:17 PM, Ben Kelly wrote: > After the rsync completed my machine slowly evicts buffers until its > back down to about twice arc_c. There was one case, however, where > I saw it stop at about four times arc_c. In that case it was > failing to evict buffers due to a missed lock. Its not clear yet if > it was a buffer lock or hash lock. When this happens you'll see the > arcstats.mutex_missed sysctl go up. I'm going to see if I can track > down why this is occuring under idle conditions. That seems > suspicious to me. Sorry to reply to my own mail, but I found some more information I thought I would share. First, the missed mutex problem was an error on my part. I had accidentally deleted a rather important line when I was instrumenting the code earlier. Once this was replaced that missed mutex count dropped back to a more reasonable level. Next, the arcstats.size value is not strictly the amount of cached data. It represents a combination of cached buffers, actively referenced buffers, and "other" data. In this case "other" data is things like dnode structures that are directly allocated using kmem_cache_alloc() and simply tacked on to the ARC accounting variable using arc_space_consume(). At this point I don't think the ARC has a way of signaling these "other" data users of memory pressure. The actual amount of memory the ARC has cached that can actually be freed is limited to buffers it internally allocated that have zero active references. This consists of the data and metadata lists for the MRU and MFU caches. On my server right now I have an arc_c_max of about 40MB. After running a simple find(1) over /usr/src I ended up with the following memory usage: arcstats.size = 132MB anonymous inflight buffers = 212KB MRU referenced buffers = 80MB MFU referenced buffers = 1KB dbuf structure "other" data = 8MB dnode structure "other" data = 25MB unknown "other" data (probably dbuf related) ~= 18MB evictable buffer data = 3KB So right now the ARC has done the best it can to free up data. If you define the cache as storing only inactive data, then basically the ARC has emptied the cache completely. This just isn't visible from the exported arcstats.size variable. I guess there is some question as to whether data is being referenced longer than it needs to be by outside consumers. Anyway, just thought I would share what I found. At this point it doesn't look like tweaking limits will really help. Also, my previous idea that the inactive buffers were being prevented from eviction for too long was incorrect. If anyone is interested I can put together a patch that exports the amount of evictable data in the cache. - Ben