From owner-freebsd-smp Sun Jun 20 11:46:31 1999 Delivered-To: freebsd-smp@freebsd.org Received: from cannon.ma.ikos.com (cannon.ma.ikos.com [137.103.105.19]) by hub.freebsd.org (Postfix) with ESMTP id A722214C17 for ; Sun, 20 Jun 1999 11:46:20 -0700 (PDT) (envelope-from tich@cannon.ma.ikos.com) Received: from isolation.ma.ikos.com (isolation [137.103.105.11]) by cannon.ma.ikos.com (8.9.1/8.8.8) with ESMTP id OAA08059; Sun, 20 Jun 1999 14:42:48 -0400 (EDT) From: Richard Cownie Received: (from tich@localhost) by isolation.ma.ikos.com (8.8.8/8.8.8) id OAA18071; Sun, 20 Jun 1999 14:40:25 -0400 (EDT) Date: Sun, 20 Jun 1999 14:40:25 -0400 (EDT) Message-Id: <199906201840.OAA18071@isolation.ma.ikos.com> To: freebsd-smp@FreeBSD.ORG, ohartman@ipamzlx.physik.uni-mainz.de Subject: Re: SMP, 4GB RAM, 4x CPU Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Since I'm the source of recent messages about problems with 4 cpu's and 4GB dram, I'll respond. 1) To the best of my knowledge, 3.2-STABLE works fine with 4 cpu's, but does not yet support 4GB DRAM. Not sure if it works with 3GB ? 2) To get the use of 4GB dram, I am running 19990604-CURRENT. This has problems with 4GB dram (but I've submitted a fix), and also with 4 cpu's. However, in the nature of software development, if you're running -current then things will be broken from time to time. This seems to be particularly true of 4GB/4cpu stuff because very few people have these kinds of systems. If you can cope with the dram limits in 3.2-STABLE, then that's probably what you should plan to use to get real work done. 3) The Linux alternative is interesting and I have myself given this serious consideration. For my particular application, the show-stopper is that Linux will only handle 2GB of DRAM; also it's unclear how big a single process can be - I believe there are problems around 1GB. With FreeBSD I can use 4GB of DRAM (actually 3998MB for arcane chipset reasons), and a single process can get >2.5GB - with further tuning of kernel configuration I believe a single process can be >3.5GB. So if your applications need large virtual memory OR large physical memory, Linux just doesn't cut it. 4) The biggest system we have here is an Intel SC450NX from SAG Electronics (www.sagelec.com) with 4 x Xeon-500MHz, 4GB DRAM, 6 x 9GB disk. Total cost approx $21K. This machine is very very fast and I'm happy with it (running FreeBSD). 5) There are two different issues which may cause trouble with performance scaling to 4 cpus: a) Memory bandwidth. For the EDA applications we're running, cache locality even with 512K cache is pretty good, my rough estimate is that each extra cpu slows down the other's by 5%, so total throughput for 1,2,3,4 jobs is about 1.0, 1.9, 3.2, 3.4 But the machine so fast that we rarely have 4 jobs running simultaneously. b) Contention for kernel resources. I believe the locking in the FreeBSD kernel is not very fine-grained as yet, so if several cpu's are trying to make system calls all at once, they'll be largely serialized ? However, if your application spend 90% of their time in user space (and if you're doing Fortran floating-point number crunching then that's probably true), this is probably not an issue. 6) If you can get your work done with 2way SMP systems with <= 1GB dram, then these are better value than the 4way. The motherboards are cheaper, the dram is cheaper, and overclocked Celerons are an amazing bargain, e.g. see the Power2U Dude at www.computernerd.com. And of course with 2 2way systems you don't suffer the same SMP inefficiency as with 1 4way system. Good luck Richard Cownie To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message