From owner-freebsd-questions@FreeBSD.ORG Wed Dec 25 02:01:09 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 608157EB for ; Wed, 25 Dec 2013 02:01:09 +0000 (UTC) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 103CB1517 for ; Wed, 25 Dec 2013 02:01:08 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.7/8.14.5) with ESMTP id rBP20wMO063586; Tue, 24 Dec 2013 18:00:58 -0800 (PST) (envelope-from freebsd@pki2.com) Subject: Re: InfiniBand in 10.0? From: Dennis Glatting To: iamatt In-Reply-To: References: <1387915155.71651.118.camel@btw.pki2.com> <1387920273.71651.119.camel@btw.pki2.com> <1387923358.71651.122.camel@btw.pki2.com> Content-Type: text/plain; charset="ISO-8859-1" Date: Tue, 24 Dec 2013 18:00:58 -0800 Message-ID: <1387936858.71651.137.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: rBP20wMO063586 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com Cc: FreeBSD Mailing List X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Dec 2013 02:01:09 -0000 On Tue, 2013-12-24 at 19:42 -0600, iamatt wrote: > And then you need to see what mpi are you going to use. I'm curious > which one you plan on using and how well it works on FreeBSD. > I've used MPIch2 under FreeBSD with John The Ripper about a year ago. It eventually worked but it was a struggle to get there. I have not been able to get OpenMPI to work at my former job (also FreeBSD). I've twice came close but in each case the project was turned off before I could finish. I'm running oclHascat under CentOS 6.5 in my home lab where I have >50T (uncompressed) of words. Most of my servers are FreeBSD of various versions. > > On Dec 24, 2013 4:16 PM, "Dennis Glatting" wrote: > On Tue, 2013-12-24 at 15:46 -0600, iamatt wrote: > > A good batch of mellanox cables. Melanic had a bad run of > FDR early > > in the year. Most FDR centers have been plagued with them. > Even the > > back end of our DDN lustres has had issues. They typically > do not > > just fail but downgrade to QDR rate. > > > > Ah. I though they were stripping the technology InfinBand. > > > > > On Dec 24, 2013 3:24 PM, "Dennis Glatting" > wrote: > > On Tue, 2013-12-24 at 14:43 -0600, iamatt wrote: > > > Not sure if I would even bother with running IB on > BSD. We > > have enough > > > problems just getting things stable with Linux and > mellanox > > ofed . Add all > > > of the documented build problems with the mellanox > FDR > > cables and recalls > > > and you are guaranteed a nightmare. TACC, navy, > and almost > > every center I > > > am aware of has replaced all or most or some of > their FDR, > > including us. > > > > > > > Replaced them with what? > > > > > > > As far as cards go it's all mellanox. Qlogic lost > the game, > > still dealing > > > with a 12800 qlogic switch which is a total > nightmare. > > Stick with > > > mellanox. Different vendors used to make their > own fabric > > managers in the > > > early time of IB but now its best to stick with > open ibd. > > > > > > FreeBSD and IB gives me the chills just thinking > about it. > > > On Dec 24, 2013 1:59 PM, "Dennis Glatting" > > wrote: > > > > > > > > > > > This site appears dated: > > > > > > > > https://wiki.freebsd.org/InfiniBand > > > > > > > > I have a couple of questions. > > > > > > > > 1) Under 10.0-RC2, is InfiniBand a part of > GENERIC or did > > I still have > > > > to modify the conf as specified in the wiki? I > don't see > > any of the > > > > options from the wiki in 10.0-RC2 GENERIC. > > > > > > > > 2) Any recommendation on cards? What works, what > doesn't? > > > > > > > > 3) What things to watch out for? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > freebsd-questions@freebsd.org mailing list > > > > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > > To unsubscribe, send any mail to " > > > > freebsd-questions-unsubscribe@freebsd.org" > > > > > > > _______________________________________________ > > > freebsd-questions@freebsd.org mailing list > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > To unsubscribe, send any mail to > > "freebsd-questions-unsubscribe@freebsd.org" > > > > > >