From owner-freebsd-infiniband@FreeBSD.ORG Mon Feb 11 21:28:56 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F40DBE10 for ; Mon, 11 Feb 2013 21:28:55 +0000 (UTC) (envelope-from tamer@polycoresoftware.com) Received: from biz112.inmotionhosting.com (biz112.inmotionhosting.com [173.247.254.188]) by mx1.freebsd.org (Postfix) with ESMTP id D5D07F39 for ; Mon, 11 Feb 2013 21:28:52 +0000 (UTC) Received: from c-174-62-105-164.hsd1.ca.comcast.net ([174.62.105.164]:56222 helo=[192.168.77.106]) by biz112.inmotionhosting.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80) (envelope-from ) id 1U4ze5-0002ra-RB for freebsd-infiniband@freebsd.org; Mon, 11 Feb 2013 12:06:33 -0800 From: Tamer Abuelata Subject: OFED setup and Mellanox cards Message-Id: <33AE1DB1-7BA9-4329-8AF0-EC7FEDCFBE7F@polycoresoftware.com> Date: Mon, 11 Feb 2013 12:06:41 -0800 To: freebsd-infiniband@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - biz112.inmotionhosting.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - polycoresoftware.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2013 21:28:56 -0000 Hi, I'm trying to get OFED up and running on a FreeBSD 9.1 machine with 2 = Mellanox cards (MT26428). We have compiled the Kernel with the following options (per = https://wiki.freebsd.org/InfiniBand ): options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib device ipoib # IP over IB devices device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards I gave an IP to each card. # ifconfig ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.d9 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 nd6 options=3D29 ib1: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.f1 inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 nd6 options=3D29 After doing a build world and build kernel I try to run an example = application and get the following warnings/errors: # ./server=20 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 listening on port 29823. Then I launch the client from another terminal: # ./client 10.0.0.2 29823 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 [=85 shows several more times =85] Segmentation fault (core dumped) So the sample app compiles fine but I get this runtime error which seems = to come from the libibverbs. This /sys/class/infiniband_verbs folder = does not exist. I'm not sure if I'm missing configuration files. However = I have devices at /dev/uverbs0 and /dev/uverbs1. The example app I am using is from: = http://thegeekinthecorner.wordpress.com/2010/08/14/building-an-rdma-capabl= e-application-with-ib-verbs-part-2-the-server/ The code is linked at the bottom of the page. Direct link: = https://sites.google.com/a/bedeir.com/home/basic-rdma-client-server.tar.gz= ?attredirects=3D0&d=3D1 I would appreciate any help or pointers from someone who managed to get = OFED to run on FreeBSD. Best regards, Tamer=