From owner-freebsd-current@FreeBSD.ORG Sun Jul 22 01:42:39 2007 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DAA916A418; Sun, 22 Jul 2007 01:42:39 +0000 (UTC) (envelope-from tridge@samba.org) Received: from lists.samba.org (mail.samba.org [66.70.73.150]) by mx1.freebsd.org (Postfix) with ESMTP id 3FD9713C459; Sun, 22 Jul 2007 01:42:39 +0000 (UTC) (envelope-from tridge@samba.org) Received: by lists.samba.org (Postfix, from userid 603) id 74EAB162AD4; Sun, 22 Jul 2007 01:23:11 +0000 (GMT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18082.45557.77020.398761@samba.org> Date: Sun, 22 Jul 2007 11:25:09 +1000 To: "Timur I. Bakeyev" In-Reply-To: <20070721215822.GA1874@com.bat.ru> References: <46633B27.50601@dva.dyndns.org> <1c5c32890707031732s195a97c3vd29fb46323f28fae@mail.gmail.com> <46644820.6020609@dva.dyndns.org> <1c5c32890707041057x75712a20vef9800a7ddef7a6a@mail.gmail.com> <1183674495.75595.14.camel@worf> <1c5c32890707051739t6621e2d4ude73ce5d096ea72e@mail.gmail.com> <1183698637.55166.58.camel@shumai.marcuscom.com> <20070721215822.GA1874@com.bat.ru> X-Mailer: VM 7.19 under Emacs 22.0.95.1 From: tridge@samba.org X-Mailman-Approved-At: Sun, 22 Jul 2007 02:05:03 +0000 Cc: current , Brian Donnell , "Boris S." , Pascal Hofstee Subject: Re: ZFS vs Samba Debugging Results ... Need Help. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: tridge@samba.org List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2007 01:42:39 -0000 Timur, > This is an old standing problem, came as a workaround for some > assumptions about the opendir/telldir/seekdir behaviour in the POSIX > systems, which made Andrew to write this workaround yes, this is a long standing problem. Basically we need telldir()/seekdir() to actually work, and not consume unlimited amounts of memory. The original problems were: - when you use seekdir() after having deleted a file in the directory, you end up in the wrong place in the directory stream. - on some systems each telldir() allocates some memory, adding it to a linked list. So if you do it a few million times then you lose a lot of memory, and the closedir() takes an extraordinary amount of time as it slowly frees the linked list Here are our current attempts to fix this problem: http://samba.org/ftp/unpacked/samba4/source/lib/replace/repdir_getdents.c http://samba.org/ftp/unpacked/samba4/source/lib/replace/repdir_getdirentries.c and here is some test code that demonstrates the problem: http://samba.org/ftp/unpacked/samba4/source/lib/replace/test/os2_delete.c It's called os2_delete as this is the pattern of calls that OS/2 clients use against Samba when they want to delete all files in a large directory. It is an insane pattern of calls, but it should work, and it does work if seekdir()/telldir() work correctly. I know this is a difficult problem to fix on some filesystems because of the way directories are stored on disk. The thing is, it is even harder to solve in application code where you have absolutely no knowledge of the directory layout, and are trying to second guess everthing. There are many possible "quick fixes" in Samba. Unfortunately they tend to have O(n^2) time or O(dirsize) memory usage, or both. That's no good unfortunately. Cheers, Tridge