Date: Tue, 16 Feb 1999 10:59:59 +1030 From: Greg Lehey <grog@lemis.com> To: "Bryn Wm. Moslow" <bryn@spacemonster.org>, freebsd-isp@FreeBSD.ORG Cc: shimon@simon-shapiro.org Subject: Re: DPT 3334UW RAID-5 Slowness / Weird FS problems Message-ID: <19990216105959.P2207@lemis.com> In-Reply-To: <36C88CC6.E1621F6F@spacemonster.org>; from Bryn Wm. Moslow on Mon, Feb 15, 1999 at 01:08:22PM -0800 References: <36C88CC6.E1621F6F@spacemonster.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, 15 February 1999 at 13:08:22 -0800, Bryn Wm. Moslow wrote: > I recently installed a DPT 3334UW with 64MB cache in a mail server > running RAID-5 with a 32K stripe on an external case on which the user > mail spool is mounted. The array is comprised of 6 Seagate Ultra Wide > 4.5GB SCA drives. The system is a P2 300 with 384MB and uses an > additional Seagate UW drive for boot, /usr, /var, swap, and staff home > directories. It doesn't go into swap often but if it does it only hits > about 5 to 10 MB. The system is running FreeBSD 2.2.8. I don't know the DPT controllers, but 32 kB stripes are far too small. For better performance, you should increase them to between 256 kB and 512 kB. Small stripe sizes create many more I/O requests at the drive level. > My first problem was that I initially tried to do 16K per inode to > speed things up a little bit (also, I didn't need the millions of > inodes that came with a default newfs on 43GB =).) I might be missing something, but I don't see any performance improvement by changing the number of inodes. > However when trying to mount, fsck, find, umount, or otherwise > manipulate the filesystem about 1 times out of 5 the system would > hang to the point that I had to use the reset switch to get it back. What hung? The FreeBSD system or the DPT subsystem? > If I format with the default settings to newfs, everything works > fine. The same hang also occurs if I try to do 16K block size. I > haven't tried anything bigger as I had an extremely tight window in > which to get the machine online. You should consider that once you have set the stripe size, you're stuck with it. Unless the DPTs have a good reason (like "not supported"), take a 256 kB stripe size. > Does FreeBSD have a problem with non-default fs settings? Not that I know of. > Has anyone else tried this sort of thing on such a large filesystem? I know of some people who have made file systems of this size with vinum. It took a while to create the file systems, but it worked. > I ended up having to use the default settings for newfs to get the > system to work, wasting millions of inodes and bringing me to my > next problem: Under load the filesystem is horribly slow. I expected > some of this with the RAID-5 overhead but it's actually slower than > a CCD I just moved from that was using 5 regular 2GB fast SCSI-2 > drives, much slower. What read to write ratio do you have? Writes are slow on RAID-5, but reads should be the same as for a striped organization. > When running ktrace on the processes (qpopper and mail.local mainly) and > watching top I can see that most of the processes are waiting for disk > access. I've tried enabling/disabling various DPT options in the kernel > but it's all about the same. I'd really like to stick with RAID-5 so > using 0 or 1 just isn't what I'm looking for. > > The user directories for delivery are broken out into 1st letter, 1st > two letters, username (i.e.: /home/u/us/username) to speed up dir > lookups already. I'd guess that these would end up in cache anyway, so you shouldn't see much improvement with this technique. > Any suggestions on how else to speed things up? This wasn't a > problem on my old CCD, however. > Lastly, I tried to find another RAID controller besides DPT that was > compatible with FreeBSD 2.2.x with no luck. Upgrading to 3.1 is not > an option at the moment, at least until things are more stable. 3.1-RELEASE has just come out (or is in the process of being packaged). But I don't think this is the problem. > Is anyone using anything in a host-based adapter (PCI) that is > non-DPT? There's a Compaq driver out there. It seems to have some strangenesses which suggest that it'll need a lot of work before it can be incorporated into the source tree. > The only reason I ask is that I've seen debate recently about > whether there is a problem with the DPT losing interrupts This wouldn't be your problem. > or the FreeBSD serial code is "broken". I'm not sure what relation this has with the DPT controller. I'm copying Shimon Shapiro on this reply. He's the author of the DPT driver, and he may have more insight. Greg -- When replying to this message, please copy the original recipients. For more information, see http://www.lemis.com/questions.html See complete headers for address, home page and phone numbers finger grog@lemis.com for PGP public key To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19990216105959.P2207>