Date: Thu, 26 Jun 2008 22:33:14 -0700 From: Jeremy Chadwick <koitsu@FreeBSD.org> To: Danny Carroll <fbsd@dannysplace.net> Cc: freebsd-hardware@freebsd.org Subject: Re: new server motherboard with SATA II Message-ID: <20080627053314.GA24239@eos.sc1.parodius.com> In-Reply-To: <4864769C.4050002@dannysplace.net> References: <486450DB.4000907@dannysplace.net> <20080627040545.GA21856@eos.sc1.parodius.com> <4864769C.4050002@dannysplace.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jun 27, 2008 at 03:11:56PM +1000, Danny Carroll wrote: > Jeremy, thanks for your response. > > Jeremy Chadwick wrote: >> SATA150 and SATA300 both work just fine on FreeBSD, but its dependent >> upon what chipset you go with. I would strongly recommend you go with a >> board/system that uses Intel's ICH7, 8, or 9 southbridge. I have >> extensive experience using these in production environments, and they >> are very reliable, plus fast. FreeBSD works quite well with them. > > I do have a board with an ICH10 chipset but the SATA drives are detected > as UDMA-33. This comes as no surprise, as there is no ICH10 knowledge in the PCI and ATA code at this time. ICH10 is ***very*** new, as in probably within the past 6 weeks, no? > I guess the ICH* chipsets would not support AMD64, being an intel chip. You're greatly confused with regards to what "amd64" means. FreeBSD's 32-bit operating system is called i386. FreeBSD's 64-bit operating system is called amd64. It has absolutely **nothing** to do with processor types (AMD vs. Intel). >> Second, I wouldn't bother considering using Intel MatrixRAID (which all >> of the above chipsets support) for any sort of failover for your root/OS >> disk, in case you're tempted to try it. FreeBSD has bugs pertaining to >> such support (see below Wiki URL for some examples). > > Yeah, I'm not so keen of the modern trend to have on-board raid. I'd > rather keep it simple and let FreeBSD handle it. Root disk will not be > raid at all. FreeBSD's ability to boot off of FreeBSD-managed-RAID'd volumes is horrible. Administrators have to go through a bunch of rigmarole to accomplish something that should be an absolute simplicity/necessity in this day and age. This is one reason why I myself have considered using Intel MatrixRAID, because then that "layer" is generally transparent to FreeBSD. In fact, ZFS on a root filesystem is still a "pain", yet Solaris has it down pat. >> Third, I cannot recommend nVidia chipsets, because there have been >> numerous reports recently and in the past where the SATA disks are being >> detected as UDMA33. I believe there are some ATI/AMD chipsets which are >> doing the same. There is a rumour that the operational speed of the >> disks is still SATA150/300, and just that FreeBSD is labelling the >> negotiated speed wrong, but my recommendation is not to risk it. > > Hmmm, some people say nforce4 chipsets are cool, some not... It's hard > to know which way to go. They're incredibly old, and you probably won't find them any more. >> Fourth, because you'll likely have multiple disks in a ZFS zpool, you >> should probably be aware of the problem that haunts some users from time >> to time (re: DMA errors). > > I've seen it on old ATA hardware. There's a chance you'll see it on brand new SATA300 disks with all sorts of different SATA controller hardware (not limited to just one vendor), too. The problem is somewhere within FreeBSD, and my Wiki page documents a workaround/patch that the FreeNAS guys came up with, which has sat ignored for over a year by the FreeBSD ATA maintainer. Not a good sign. >> http://wiki.freebsd.org/JeremyChadwick/ATA_issues_and_troubleshooting >> >>> I'd be willing to go with intel arch although from a ZFS perspective >>> it sounds like AMD64 is better. >> >> There was a recent discussion on developers@ (which is private) about >> some topics, which eventually lead into a discussion about ZFS, tuning, >> and a 2GB kmem limit in FreeBSD (which affects amd64 too). I can't copy >> the conversation/thread because developers@ has a strict "do not >> disclose" policy. > > I thought that the 2gb limit was less of a problem for AMD64 because of > the addressing used. You're confusing two issues (general maximum memory limit of x86/PC hardware and kmem). There is a known limit **in FreeBSD** that affects i386 and amd64 both, limiting the available amount of kmem to 2GB. You cannot increase it past that. It's a code issue, and is presently being addressed in HEAD (-CURRENT). ZFS uses kmem heavily. So, it doesn't matter if you have 16GB of RAM installed and are running amd64; kmem is still limited to 2GB. That means that effectively ZFS will only be able to use up to 1.5GB of your RAM (not 2GB, because you need to leave some available for other kernel related things; 512MB is enough). >> Just be aware you ***will*** need to tune ZFS on FreeBSD to make it >> as reliable as possible. > > We'll like I said, I'd be willing to jump on a list and provide info etc > about my setup. I plan to have it running on a test bench with lots of > IO for a week or so before I start using it. Even then the data will > not be critical so if it breaks then I can rebuild without hassle. Keep in mind that there have been some reports of ZFS on FreeBSD behaving incorrectly/oddly when a disk goes back, or a checksum error is found by ZFS. The disk will resilver for no apparent reason. > System disk will be UFS2 to keep it simple... > > I've got it running on desktop hardware (ASUS P5Q board with ICH5) while > I wait for a decision on a permanent Motherboard. With this setup I see > about 60mb write speeds on ZFS across 5 disks. I've done the basic > tuning suggested in the Wiki. One thing I notice is that the CPU is > used for 30% on Interrupts. It was firewire first, so I disabled it in > the BIOS, then it went to UHCI so I disabled all USB ports. Now it is > on the ATA controller. All of this was on the same interrupt (19). IRQ sharing is generally a "thing of the past", and interrupt conflicts are something from days prior to APICs being available on every motherboard under the sun. Meaning: "swapping around" IRQs for onboard devices rarely does anything in this day and age. > I'm thinking of getting a couple of Promise SATA-300 TX4 IO cards > (non-raid). Perhaps that will offload the CPU. I don't see how that's going to help with heavy interrupt usage. 30% interrupt usage across 5 disks doesn't sound too odd, and going with a Promise controller that doesn't have its own dedicated driver (read: it will use ata(4)) won't address that. You might be better off getting actual server hardware rather than "hacked up to be server" desktop hardware. Consider Supermicro stuff. I can personally recommend their PDSMi+ motherboard, and many other FreeBSD users rely heavily on their hardware. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080627053314.GA24239>