From owner-freebsd-fs@FreeBSD.ORG Tue Jan 29 23:42:30 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5A31C248 for ; Tue, 29 Jan 2013 23:42:30 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by mx1.freebsd.org (Postfix) with ESMTP id AF4BFF98 for ; Tue, 29 Jan 2013 23:42:29 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id j14so1406798lbo.24 for ; Tue, 29 Jan 2013 15:42:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=4dOb06WmFFmGv//BJ/aIa1Nknh5ufuQM5mi+Ga3VnSE=; b=VyQiUBfj5DJ0MCTh5ethc/8eZZuncYteMgWu0H8X7PCl78Y7pzMfYyXwZ+l7O40Vse iZgMM7fbnRR08NnOmm5vS9wpHgJcZ9CwOvTLTTSluKwfCiOF7mcuCz6txlsO9Y0FKTXf bCRm4ogQG+8dhgn9AMjeYySEXJEdpgLPIIn1E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:x-gm-message-state; bh=4dOb06WmFFmGv//BJ/aIa1Nknh5ufuQM5mi+Ga3VnSE=; b=PnT4pYXsrcWQeDvcvIszTQ9ElSCtOau6utyEFYLaFrXm7LkVhy9u961J+LHl/LDtJE MS6qTL6x4DtbstAAJkawT1Zf6wv6gfQxAc7yhFRW9TpU9EPgDA0jboYaLlYL1lbsdGfK oG8GNZuSS6eg+sxn2kXm1V9Jqbel5X7YQ5J0aah8mD89ml5oDjAvR9QFjxf5kL05RpCj EfY4DgJw4DWxOvfrUyx91eSe4c+qz0VRhN7wZjznIaVznyU6/YRGtOeIg0vEQfx836z6 QpJ/S43a1EYgBgh2OnCuHEJUlCfhsRndM9IdPIn1zXfiO2cIt9GXtwahJHNWVUhaLIyW DNGg== MIME-Version: 1.0 X-Received: by 10.152.144.202 with SMTP id so10mr2721142lab.9.1359502948362; Tue, 29 Jan 2013 15:42:28 -0800 (PST) Received: by 10.114.68.109 with HTTP; Tue, 29 Jan 2013 15:42:28 -0800 (PST) In-Reply-To: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> Date: Tue, 29 Jan 2013 15:42:28 -0800 Message-ID: Subject: Re: Improving ZFS performance for large directories From: Matthew Ahrens To: Kevin Day X-Gm-Message-State: ALoCoQmoJx68nM8xURko1bIRspmB/I2cz6arNabSxA3deWbDWGQe5KWlDHVA5r1lc+FVZokLVdpR Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2013 23:42:30 -0000 On Tue, Jan 29, 2013 at 3:20 PM, Kevin Day wrote: > I'm prepared to try an L2arc cache device (with secondarycache=metadata), You might first see how long it takes when everything is cached. E.g. by doing this in the same directory several times. This will give you a lower bound on the time it will take (or put another way, an upper bound on the improvement available from a cache device). > but I'm having trouble determining how big of a device I'd need. We've got > >30M inodes now on this filesystem, including some files with extremely > long names. Is there some way to determine the amount of metadata on a ZFS > filesystem? For a specific filesystem, nothing comes to mind, but I'm sure you could cobble something together with zdb. There are several tools to determine the amount of metadata in a ZFS storage pool: - "zdb -bbb " but this is unreliable on pools that are in use - "zpool scrub ; ; echo '::walk spa|::zfs_blkstats' | mdb -k" the scrub is slow, but this can be mitigated by setting the global variable zfs_no_scrub_io to 1. If you don't have mdb or equivalent debugging tools on freebsd, you can manually look at ->spa_dsl_pool->dp_blkstats. In either case, the "LSIZE" is the size that's required for caching (in memory or on a l2arc cache device). At a minimum you will need 512 bytes for each file, to cache the dnode_phys_t. --matt