From owner-freebsd-current@FreeBSD.ORG  Sun Jan  5 11:16:57 2014
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 85C8EC08
 for <freebsd-current@FreeBSD.org>; Sun,  5 Jan 2014 11:16:57 +0000 (UTC)
Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140])
 by mx1.freebsd.org (Postfix) with ESMTP id D497C1663
 for <freebsd-current@FreeBSD.org>; Sun,  5 Jan 2014 11:16:56 +0000 (UTC)
Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua
 [212.40.38.100])
 by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA08001;
 Sun, 05 Jan 2014 13:14:13 +0200 (EET) (envelope-from avg@FreeBSD.org)
Received: from localhost ([127.0.0.1])
 by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD))
 id 1Vzlen-000N9q-AN; Sun, 05 Jan 2014 13:14:13 +0200
Message-ID: <52C93E4D.1050100@FreeBSD.org>
Date: Sun, 05 Jan 2014 13:13:17 +0200
From: Andriy Gapon <avg@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:24.0) Gecko/20100101 Thunderbird/24.2.0
MIME-Version: 1.0
To: Vladimir Sharun <atz@ukr.net>,
 Current FreeBSD <freebsd-current@FreeBSD.org>
Subject: Re: ARC "pressured out", how to control/stabilize ? (reformatted
 to text/plain)
References: <1388839805.123581691.q97ijp8l@frv45.ukr.net>
In-Reply-To: <1388839805.123581691.q97ijp8l@frv45.ukr.net>
X-Enigmail-Version: 1.6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 05 Jan 2014 11:16:57 -0000

on 04/01/2014 14:50 Vladimir Sharun said the following:
[snip]
> ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other
[snip]
> ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
[snip]
> zio_data_buf_131072: 131072,      0,  488217,       9,287155442,   0,   0

I noticed a particular discrepancy between reported ARC usage and sizes of UMA
zones used by ZFS code:

488217 * 131072 = ~59GB right there.

There are several possibilities for this discrepancy:
- bad accounting or reporting of ARC stats
- those 128K buffers being used in a special way and thus not accounted as ARC
- some sort of resource leak

You could try to use DTrace to gather the stacks of all code paths that lead to
allocation of those buffers.  Something like:

fbt::zio_data_buf_alloc:entry
/arg0 == 131072/
{
	@[stack()] = count();
}

This could be a start for understanding the issue.

-- 
Andriy Gapon