Date: Fri, 20 Sep 2013 11:05:35 +0200 From: Vincent Schut <schut@sarvision.nl> To: freebsd-questions@freebsd.org Subject: Re: this 48-core box... Message-ID: <20130920110535.42c984da@sarvision.nl> In-Reply-To: <alpine.BSF.2.00.1309191157410.56073@btw.pki2.com> References: <52388C9B.6030205@foxbatcapital.com> <20130919155327.115e7344@sarvision.nl> <alpine.BSF.2.00.1309191157410.56073@btw.pki2.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 19 Sep 2013 12:05:14 -0700 (PDT) Dennis Glatting <dg@pki2.com> wrote: > > > On Thu, 19 Sep 2013, Vincent Schut wrote: > > > On Tue, 17 Sep 2013 12:08:43 -0500 > > Michael Chen <michael@foxbatcapital.com> wrote: > > > >> I'm considering bidding on this 48-core box: > >> > >> http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc > >> > >> Does anyone have experience with it and can I use all the cores? > >> > >> Thanks! > >> _______________________________________________ > >> freebsd-questions@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions > >> To unsubscribe, send any mail to > >> "freebsd-questions-unsubscribe@freebsd.org" > >> > > > > I recently bought one like that (48 cores but 'only' 96 Gb ram). It > > was meant to play a double role as both zfs file server and data > > processing server (we do lots of satellite image processing), > > running FreeBSD 9.1. It connects with a SAN and we'll use it to > > process about 36TB of satellite data in the next months. (In a > > couple of weeks we will probably have budget to split those roles, > > and buy a dedicated file server.) After several weeks of tweaking > > and testing, I can say that: > > - the zfs/file server part runs without problems > > - the satellite data processing had problems scaling to all 48 > > cores, I got max performance when running about 18 processes in > > parallel, scaling up more would lower the overall performance. > > However, this (sorry guys) appeared to be a FreeBSD problem, and > > not a hardware problem. As a test I switched to linux with ZoL (ZFS > > on Linux), and, though zfs performance is less compared to freebsd, > > data processing is much much better, like a factor 12 or so. > > > > I've noticed this same scaling problem on 32+ core servers but > haven't had a chance to look into the detail. From the performance > graphs I am confused whether my problems are processing problems or a > data I/O problem. I have done some (light) investigation as I did need the processing power. In my case the bottleneck was definitely not data I/O. Bonnie+ rates from and to the dataset were as expected, top and atop and other utils did not show any stress on the I/O system, and the algorithm which did not scale should not be IO bound, rather cpu or memory (or both). I've heard/read rumors (when I was investigating the extreme long compile time of openblas on freebsd compared to linux) about bsd being less well optimized in e.g. using the processor's L2 cache. Things like this can play an important role in the processing we do (many numerical calculations on lots of data in memory). Most of the calculations were done by quite optimized software for numerical processing (numpy/scipy using openblas (yes I did make sure openblas used only 1 thread when scaling up)). The fact that the problems disappeared when running the same under linux also point in the cpu/memory direction rather that I/O, as the ZFS on Linux performance is still behind that of ZFS on BSD. > > > > Conclusion: the hardware is alright, however when needed to do lots > > of heavy calculations on terabytes of data, the combination with > > FreeBSD appears not ideal. > > > > Of course it is you get what you pay for. Decent, OK working > > hardware, but none of the special handy-dandy features expensive > > brands will give you. If you don't need them, in my experience it > > is decent hardware for a good price. > > > > regards, > > Vincent. > > > > _______________________________________________ > > freebsd-questions@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > To unsubscribe, send any mail to > > "freebsd-questions-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130920110535.42c984da>