From owner-freebsd-stable@FreeBSD.ORG Sun Aug 7 09:20:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF65E1065672 for ; Sun, 7 Aug 2011 09:20:21 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4F9B28FC15 for ; Sun, 7 Aug 2011 09:20:20 +0000 (UTC) Received: by wyh21 with SMTP id 21so342986wyh.13 for ; Sun, 07 Aug 2011 02:20:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=kn8hsmznXiXc6+JFpmxGmDfaWMRRO5A8b5LOpQfBKZ4=; b=PUO6ghOMNKBhrx9wMCeUpmiw9GyIO5Ss36+MLPsYrShUC75Fxs1pUddFnO28CeV3od h3ffuSDmz8/MoCLy1UUOXbHgYgM9P8ooEKlPGmMZqFKhMJVsxP9QvdNev92z2gYfP781 Jc4Xj0lYW3OKCPhKT/ojRII4HWMN+IzBJBW+k= Received: by 10.216.170.8 with SMTP id o8mr899217wel.101.1312708819803; Sun, 07 Aug 2011 02:20:19 -0700 (PDT) Received: from nox.fritz.box (87-198-213-102.static.ptr.magnet.ie [87.198.213.102]) by mx.google.com with ESMTPS id m38sm2831386weq.21.2011.08.07.02.20.16 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 07 Aug 2011 02:20:17 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: text/plain; charset=us-ascii From: Sean Rees In-Reply-To: <20110806062415.GB88904@in-addr.com> Date: Sun, 7 Aug 2011 10:20:15 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <010350C0-B3B3-44FC-8D94-A111C579860C@gmail.com> References: <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> <6E45CE57-491E-4077-B14C-751C73647EFC@gsoft.com.au> <4E3CBB74.9020208@FreeBSD.org> <20110806062415.GB88904@in-addr.com> To: Gary Palmer X-Mailer: Apple Mail (2.1244.3) Cc: Doug Barton , freebsd-stable@freebsd.org Subject: Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2011 09:20:21 -0000 On Aug 6, 2011, at 07:24, Gary Palmer wrote: > On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: >> On 08/05/2011 20:38, Daniel O'Connor wrote: >>=20 >>> Ahh, but OP had moved these files away and performance was still = poor.. _that_ is the bug. >>=20 >> I'm no file system expert, but it seems to me the key questions are; = how >> long does it take the system to recover from this condition, and if = it's >> more than N $periods is that a problem? We can't stop users from = doing >> wacky stuff, but the system should be robust in the face of this. >=20 > Its been quite a while since I worked on the filesystem stuff in any > detail but I believe, at least for UFS, it doesn't GC the directory, > just truncate it if enough of the entries at the end are deleted > to free up at least one fragment or block. If you create N files and > then a directory and move the N files into the directory, the = directory > entry will still be N+1 records into the directory and the only way to > "recover" is to recreate the directory that formerly contained the N > files. It is theoretically possible to compat the directory but since=20= > the code to do that wasn't written when I last worked with UFS I = suspect > its non trivial. >=20 > I don't know what ZFS does in this situation It sounds like it does something similar. I re-ran the experiment to see if I could narrow down the problem. % mkdir foo % cd foo && for i in {1..1000}; do touch $i; done % ls > list % for file in $(cat list); do rm -f $file; done % time ls (slow!) % rm -f list % time ls (slow!) I would like to dig into this a bit more, I suppose it's probably a good = enough reason to explore how DTrace works :) Sean=