From owner-freebsd-current@FreeBSD.ORG  Mon Jan  6 11:15:05 2014
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 2F3614BD
 for <freebsd-current@freebsd.org>; Mon,  6 Jan 2014 11:15:05 +0000 (UTC)
Received: from frv197.fwdcdn.com (frv197.fwdcdn.com [212.42.77.197])
 (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id D9B5613AF
 for <freebsd-current@freebsd.org>; Mon,  6 Jan 2014 11:15:04 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net;
 s=ffe; 
 h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-Id:Cc:To:Subject:From:Date;
 bh=4sd2F/5jRS1UgY/VDnf7jphFsJegs/saLoyJTcE0bCI=; 
 b=FKLjwhbxWVG2QhpUvgq54pnkLz2kJniBzJQ0iRwkZwKNtYOCc2YPQ2m1JuSYuFBg9YndDLkTmFWuGelAQ18sJGlqScxKQ/Ui3t0SrOmDXiGKOmp6+7ltwxyCbUWEXx1675Q+j6fLtRm/MPsljMt8XLCLA3vSboA2oLRlsMjsCXE=;
Received: from [10.10.10.45] (helo=frv45.ukr.net)
 by frv197.fwdcdn.com with smtp ID 1W088y-0004xh-K5
 for freebsd-current@freebsd.org; Mon, 06 Jan 2014 13:14:52 +0200
Date: Mon, 06 Jan 2014 13:14:52 +0200
From: Vladimir Sharun <atz@ukr.net>
Subject: Re[2]: ARC "pressured out", how to control/stabilize ? (reformatted
 to text/plain)
To: Andriy Gapon <avg@freebsd.org>
X-Mailer: mail.ukr.net 5.0
Message-Id: <1389005433.815055146.2dcjke36@frv45.ukr.net>
In-Reply-To: <52C93E4D.1050100@FreeBSD.org>
References: <1388839805.123581691.q97ijp8l@frv45.ukr.net>
 <52C93E4D.1050100@FreeBSD.org>
MIME-Version: 1.0
Received: from atz@ukr.net by frv45.ukr.net; Mon, 06 Jan 2014 13:14:52 +0200
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: binary
Content-Disposition: inline
Cc: Current FreeBSD <freebsd-current@freebsd.org>
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 06 Jan 2014 11:15:05 -0000

Dear Andriy and FreeBSD community,

I got the few minutes run for this dtrace hook; here's the output for 15 minutes run:

http://pastebin.com/pKm9kLwa

Does it explain something ?

 
> on 04/01/2014 14:50 Vladimir Sharun said the following:
> [snip]
> > ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other
> [snip]
> > ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
> [snip]
> > zio_data_buf_131072: 131072,      0,  488217,       9,287155442,   0,   0
> 
> I noticed a particular discrepancy between reported ARC usage and sizes of UMA
> zones used by ZFS code:
> 
> 488217 * 131072 = ~59GB right there.
> 
> There are several possibilities for this discrepancy:
> - bad accounting or reporting of ARC stats
> - those 128K buffers being used in a special way and thus not accounted as ARC
> - some sort of resource leak
> 
> You could try to use DTrace to gather the stacks of all code paths that lead to
> allocation of those buffers.  Something like:
> 
> fbt::zio_data_buf_alloc:entry
> /arg0 == 131072/
> {
> @[stack()] = count();
> }
> 
> This could be a start for understanding the issue.
>