Date: Sun, 22 Jul 2007 11:25:09 +1000 From: tridge@samba.org To: "Timur I. Bakeyev" <timur@com.bat.ru> Cc: current <current@freebsd.org>, Brian Donnell <bdonnell@gmail.com>, "Boris S." <bst2006@dva.dyndns.org>, Pascal Hofstee <caelian@gmail.com> Subject: Re: ZFS vs Samba Debugging Results ... Need Help. Message-ID: <18082.45557.77020.398761@samba.org> In-Reply-To: <20070721215822.GA1874@com.bat.ru> References: <d8a0b7620707031331w5b0fcae2jd96cb7ff8b3f34d6@mail.gmail.com> <d8a0b7620707031359wc178fdfocf2387820fcacbf5@mail.gmail.com> <46633B27.50601@dva.dyndns.org> <1c5c32890707031732s195a97c3vd29fb46323f28fae@mail.gmail.com> <46644820.6020609@dva.dyndns.org> <1c5c32890707041057x75712a20vef9800a7ddef7a6a@mail.gmail.com> <1183674495.75595.14.camel@worf> <1c5c32890707051739t6621e2d4ude73ce5d096ea72e@mail.gmail.com> <1183698637.55166.58.camel@shumai.marcuscom.com> <20070721215822.GA1874@com.bat.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Timur,
> This is an old standing problem, came as a workaround for some
> assumptions about the opendir/telldir/seekdir behaviour in the POSIX
> systems, which made Andrew to write this workaround
yes, this is a long standing problem. Basically we need
telldir()/seekdir() to actually work, and not consume unlimited
amounts of memory.
The original problems were:
- when you use seekdir() after having deleted a file in the
directory, you end up in the wrong place in the directory
stream.
- on some systems each telldir() allocates some memory, adding it to
a linked list. So if you do it a few million times then you lose a
lot of memory, and the closedir() takes an extraordinary amount of
time as it slowly frees the linked list
Here are our current attempts to fix this problem:
http://samba.org/ftp/unpacked/samba4/source/lib/replace/repdir_getdents.c
http://samba.org/ftp/unpacked/samba4/source/lib/replace/repdir_getdirentries.c
and here is some test code that demonstrates the problem:
http://samba.org/ftp/unpacked/samba4/source/lib/replace/test/os2_delete.c
It's called os2_delete as this is the pattern of calls that OS/2
clients use against Samba when they want to delete all files in a
large directory. It is an insane pattern of calls, but it should work,
and it does work if seekdir()/telldir() work correctly.
I know this is a difficult problem to fix on some filesystems because
of the way directories are stored on disk. The thing is, it is even
harder to solve in application code where you have absolutely no
knowledge of the directory layout, and are trying to second guess
everthing.
There are many possible "quick fixes" in Samba. Unfortunately they
tend to have O(n^2) time or O(dirsize) memory usage, or both. That's
no good unfortunately.
Cheers, Tridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18082.45557.77020.398761>