Date: Tue, 15 Feb 2000 19:22:44 -0600 (CST) From: Joe Greco <jgreco@ns.sol.net> To: dillon@apollo.backplane.com (Matthew Dillon) Cc: hackers@freebsd.org Subject: Re: Filesystem size limit? Message-ID: <200002160122.TAA93538@aurora.sol.net> In-Reply-To: <200002160041.QAA46418@apollo.backplane.com> from Matthew Dillon at "Feb 15, 2000 4:41:44 pm"
next in thread | previous in thread | raw e-mail | index | archive | help
> :Speaking of which, I'd like to compliment you on the overall design of the > :Diablo system. It has scaled very well to handle a hundred million articles > :on-spool. > : > :Dumping, hash table 67108864 entries, record size 28 <== :-) :-) :-) > :@268435472 > :diload: 104146775/104146944 entries loaded > :History file trim succeeded: > :-rw-r--r-- 1 news news 3184549904 Feb 15 05:53 /news/dhistory > :-rw-r--r-- 1 news news 3184549904 Feb 15 02:45 /news/dhistory.bak > : > :3 hours to rebuild dhistory on a SMP machine. Sigh. > : > :/dev/vinum/news 14154136 8491456 5662680 60% /news > :/dev/vinum/n0 31805976 26465776 5340200 83% /news/spool/news/N.00 > :/dev/vinum/n1 31805976 26754544 5051432 84% /news/spool/news/N.01 > :/dev/vinum/n2 31805976 27787840 4018136 87% /news/spool/news/N.02 > :/dev/vinum/n3 31805976 26834120 4971856 84% /news/spool/news/N.03 > :/dev/vinum/n4 31805976 27609456 4196520 87% /news/spool/news/N.04 > :/dev/vinum/n5 31805976 26771072 5034904 84% /news/spool/news/N.05 > :/dev/vinum/n6 31805976 27396296 4409680 86% /news/spool/news/N.06 > :/dev/vinum/n7 31805976 26801120 5004856 84% /news/spool/news/N.07 > :/dev/vinum/n8 31805976 8 31805968 0% /news/spool/news/N.08 > : > :Yeah, I'm not using that last spool, so I could probably squeeze 120 million > :articles on here. No binaries obviously. > > I have one word for this: "YowZeR!". > > I assume you bumped up the default hash table size... of course you > must have! You missed the line above. 64m hash. Required a fix to Diablo to work right. :-) I actually discovered _that_ the hard way: Diablo was ignoring the config file setting, and was creating dhistory with the default hash size, which caused lots of extra disk I/O (but it still worked fairly well!) I didn't really have a reason to track it down until I hit the 60 million article mark, when the slowdown was causing the massive NNTP STAT commands that DSRS likes to throw at servers was resulting in noticeable slowdowns (under fifty lookups per second). And don't tell me that 64m is too big. I know it is probably too big. :-) > Your welcome! > > Yes, I designed the Diablo reader *SPECIFICALLY* to be able to do > (multiple) remote spool serving. The idea was to be able to supply spool > redundancy so you could take a spool down without taking the > whole system down, and so you could run the 'frontend' readers on > pure-cpu boxes. It's predicated on the concept that a history lookup > is supposed to be cheap, which is usually true. You need something to use to retrieve it, and that's a lowest common denominator, portable, etc. It also suits the idea of server-as-network- appliance, suitable for a specific need, rather than trying to take the old hammer and bludgeon NFS into doing what is needed. And I've actually had people _argue_ with me that NFS is still the right way to do it, that the mount problems can be solved, that the chattiness isn't an issue, it blows me away. > And, of course, the reader uses a multi-fork/multi-thread design, > resulting in an extremely optimal footprint. I hate your threads. Still trying to see some parts of the bigger picture. But it was a reasonable design decision, I'll grant. > The spools are supposed to be pure message respositories based > on the message-id and I've contemplated using the technology to back > other uses, such as a web-based messaging system or even email inboxes. > The article storage format is very robust in terms of crash recovery > though in retrospect I should have added a magic cookie + binary size > header to the beginning of each article to make recovery more reliable. "Squid, Squid..." :-) I keep looking at that storage API thing in the newest Squid's. My Chicago open server pumps out 8K/sec rate limited connections to 1200 simultaneous clients, that's about 30-40 megabits of traffic 24/7. One box. If I could make a Squid server perform like that, instead of a twentieth of that, I'd be in heaven. And Squid's issue is I/O bottlenecking. > I'm glad that you and others are taking over source management of > the project, I have no time to do it any more :-(. I was actually despairing a bit, late last year, no one else seemed to care about Diablo. I am already committed to the model, though, so I would have done it all myself if I had been forced to. You'll be pleased to know that there are even more people than you know who are actively interested in what's going on, though, including some folks like Russell who have some really similar visions to mine as to where things need to go in the future. Diablo is going places. Thanks for a great base to start from. ... Joe ------------------------------------------------------------------------------- Joe Greco - Systems Administrator jgreco@ns.sol.net Solaria Public Access UNIX - Milwaukee, WI 414/342-4847 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200002160122.TAA93538>