Date: Sat, 04 May 2002 08:15:14 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: utsl@quic.net Cc: Bakul Shah <bakul@bitblocks.com>, Scott Hess <scott@avantgo.com>, "Vladimir B. Grebenschikov" <vova@sw.ru>, fs@FreeBSD.ORG Subject: Re: Filesystem Message-ID: <3CD3FB02.3EC1DA29@mindspring.com> References: <200205040019.UAA13780@illustrious.cnchost.com> <3CD32F43.327CDA46@mindspring.com> <20020504041936.GA19646@quic.net>
next in thread | previous in thread | raw e-mail | index | archive | help
utsl@quic.net wrote: [ ... linear directory search times on the majority of systems ... ] > OTOH, I've seen a very large application (it ran on a Sun E10K) that did > absolutely nothing about it. It was designed to put some ~1-2k files > into a spool directory, and rotate every day. Unfortunately, the > application didn't ever get redesigned to handle the scale it was being > used for. So when I dealt with it, they had a filesystem that had > 800,000 to 1M files in 15-16 directories. (Varied from day to day.) I > found out about it when I was asked to figure out why the incremental > backups for that filesystem never completed. They would run for ~35-40 > hours and then crash. If I remember right, the backup program was > running out of address space. 8-) > > Even if the filesystem had used btrees, the backup program would still > have crashed. It was trying to make a list in memory of all the files > it needed to backup. It never actually wrote anything to tape... I don't > know if all backup software does incrementals that way, but I'd bet most > of them do. > > So there can be other disadvantages to having lots of files in a > directory besides slow directory lookups. I wasn't really trying to exhasutively list all the reasons that it was bad to put a bunch of files in a large directory. There are an incredibly large number of reasons for it to be bad, and I have better things to do than spending the rest of time pointing out impedence mismatches in algorithms. 8-). My take on an application that doesn't scale is that "fixing" the application by changing the behaviour of the underlying system is just propping up bad code. Bad code deserves to lose. So if someone wrote an application like that, it's just as well that the programmer who failed to consider scaling issues lose out to the programmer who considered them. After all, it's very likely that the failure to consider scaling issues is more of an "all or nothing" thing, and that the failure to consider one means that solving it in the OS will just expose the next one. There's really no way you can make the OS behave perfectly for all applications. At some point, applications programmers will have to learn how to program, or all bets are off. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CD3FB02.3EC1DA29>