Date: Tue, 8 May 2012 07:07:24 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Gergely CZUCZY <gergely.czuczy@harmless.hu> Cc: jeff@freebsd.org, jhb@freebsd.org, net@freebsd.org Subject: Re: OFED stack, RDMA, ipoib help needed Message-ID: <1217430168.59507.1336475244392.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20120508121120.00003ea3@unknown>
next in thread | previous in thread | raw e-mail | index | archive | help
Gergely CZUCZY wrote: > Hello, > > I'd like to ask a few question in order to get some hardware to work > we've got recently. > > The hardwares are the following: > - 2x dualport Mellanox ConnectX-3 VPI cards, with 56Gbps ports > - 4 computing modules with a singleport Mellanox MT27500-family > ConnectX-3 port. > > The 2 dualport cards are in a storage box, and the 4 singleport ones > are integrated on blade-like computing nodes (4 boxes in 2U). The > storage is running FreeBSD 9-STABLE, 2012-05-07 cvsup, and the > computing nodes are running linux. > > So far we had been able to bring up the subnet-manager on the FreeBSD > node, and one of the links got into Active state, which is quite good. > We had been able to ibping between the nodes. The FreeBSD kernel > config, in addition to GENERIC, is the following: > > options OFED > options SDP > device ipoib > options IPOIB_CM > device mlx4ib > device mthca > device mlxen > > Right now we're having problems with the following issues, situations: > > 1) we assigned IP addresses to both ib interfaces (fbsd, linux side), > but weren't able to ping over IP. We've seen icmp-echo-requests > leaving > the box on the linux box, but haven't seen any incoming traffic on the > freebsd one. On the freebsd side, we had several issues: > - no incoming packets seen by tcpdump on the ib interface > - when trying to ping the other side, we've got "no route to host", > but the routing entry existed in the routing table. > - we had a few of these messages in our messages: "ib2: timing out; 0 > sends N recieves not completed", where started at 22,34 and was > growing. > > 2) We're unable to find any resources on how to do RDMA on the FreeBSD > side. We'd like to use SRP (SCSI RDMA Protocol) communication, and/or > NFS-over-RDMA for our storage link between the boxes. Where could we > find any info on this? > NFS-over-RDMA requires sessions, which are a part of NFSv4.1. There is no NFSv4.1 server support at this time. I know diddly about infiniband, so I can't help w.r.t. the rest. Good luck with it, rick > 3) Enabling connected-mode, we weren't able to find a way to specify > or > query the port that connected mode is using. Could someone please > point > us to the right direction regarding this minor issue? > > 4) We were also unable to find how to switch these dual-personality > cards between infiniband and ethernet modes. Could we also get some > pointers regarding this please? > > Basically any help would be welcome which could help making infiniband > work. > > As a side question, I've seen a comming for OFED in HEAD by jhb, > fixing > a few things, may I ask when will that get MFC'd to RELENG-9? > > Thanks in advance. > > Best regards, > Gergely CZUCZY > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1217430168.59507.1336475244392.JavaMail.root>