From owner-freebsd-infiniband@FreeBSD.ORG Mon Feb 11 21:28:56 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F40DBE10 for ; Mon, 11 Feb 2013 21:28:55 +0000 (UTC) (envelope-from tamer@polycoresoftware.com) Received: from biz112.inmotionhosting.com (biz112.inmotionhosting.com [173.247.254.188]) by mx1.freebsd.org (Postfix) with ESMTP id D5D07F39 for ; Mon, 11 Feb 2013 21:28:52 +0000 (UTC) Received: from c-174-62-105-164.hsd1.ca.comcast.net ([174.62.105.164]:56222 helo=[192.168.77.106]) by biz112.inmotionhosting.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80) (envelope-from ) id 1U4ze5-0002ra-RB for freebsd-infiniband@freebsd.org; Mon, 11 Feb 2013 12:06:33 -0800 From: Tamer Abuelata Subject: OFED setup and Mellanox cards Message-Id: <33AE1DB1-7BA9-4329-8AF0-EC7FEDCFBE7F@polycoresoftware.com> Date: Mon, 11 Feb 2013 12:06:41 -0800 To: freebsd-infiniband@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - biz112.inmotionhosting.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - polycoresoftware.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2013 21:28:56 -0000 Hi, I'm trying to get OFED up and running on a FreeBSD 9.1 machine with 2 = Mellanox cards (MT26428). We have compiled the Kernel with the following options (per = https://wiki.freebsd.org/InfiniBand ): options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib device ipoib # IP over IB devices device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards I gave an IP to each card. # ifconfig ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.d9 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 nd6 options=3D29 ib1: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.f1 inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 nd6 options=3D29 After doing a build world and build kernel I try to run an example = application and get the following warnings/errors: # ./server=20 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 listening on port 29823. Then I launch the client from another terminal: # ./client 10.0.0.2 29823 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs1 libibverbs: Warning: no userspace device-specific driver found for = /sys/class/infiniband_verbs/uverbs0 [=85 shows several more times =85] Segmentation fault (core dumped) So the sample app compiles fine but I get this runtime error which seems = to come from the libibverbs. This /sys/class/infiniband_verbs folder = does not exist. I'm not sure if I'm missing configuration files. However = I have devices at /dev/uverbs0 and /dev/uverbs1. The example app I am using is from: = http://thegeekinthecorner.wordpress.com/2010/08/14/building-an-rdma-capabl= e-application-with-ib-verbs-part-2-the-server/ The code is linked at the bottom of the page. Direct link: = https://sites.google.com/a/bedeir.com/home/basic-rdma-client-server.tar.gz= ?attredirects=3D0&d=3D1 I would appreciate any help or pointers from someone who managed to get = OFED to run on FreeBSD. Best regards, Tamer= From owner-freebsd-infiniband@FreeBSD.ORG Tue Feb 12 16:43:11 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7AD3B312 for ; Tue, 12 Feb 2013 16:43:11 +0000 (UTC) (envelope-from accornehl@gmail.com) Received: from mail-ee0-f46.google.com (mail-ee0-f46.google.com [74.125.83.46]) by mx1.freebsd.org (Postfix) with ESMTP id 0039B117 for ; Tue, 12 Feb 2013 16:43:10 +0000 (UTC) Received: by mail-ee0-f46.google.com with SMTP id e49so152323eek.19 for ; Tue, 12 Feb 2013 08:43:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=XK1oow+2a5oHwtr4JxwcLX5jLAsI1n+nP1qdYVnBnRs=; b=PFv+j85bxcwPBtKsGI84enXG7/AELPNOOWhHuwKAEF9kEML6Z6cjtt3MV0Xf+trNXI 09yNiU3glFUAJwt63UtyXLDmWP/AVIVJ7RH/4cJFJmAvbylT75cQOlzQPbF124b/gEOu lDQb3JY/nu6C0EaJwZXPASwvpRHCIZOVJhjhQqlm4N2ItxWtaYvj8bsaOCV4qEvi/Vwb NrlJLnYtcHr6o22r9YcXqGF4QejtV1oC6MiyCjQSGBbdAmVvkTABgrgcuuW9PF6cTVCl sL3ay5vjTWmp/wKwmAT4rHFEXQS7vYaFR3gozdGRJUM3ybbDQ8yJFrpFkYI1FmUN6w/g Gefg== MIME-Version: 1.0 X-Received: by 10.14.218.71 with SMTP id j47mr64239245eep.28.1360686980876; Tue, 12 Feb 2013 08:36:20 -0800 (PST) Received: by 10.223.18.14 with HTTP; Tue, 12 Feb 2013 08:36:20 -0800 (PST) Received: by 10.223.18.14 with HTTP; Tue, 12 Feb 2013 08:36:20 -0800 (PST) In-Reply-To: <33AE1DB1-7BA9-4329-8AF0-EC7FEDCFBE7F@polycoresoftware.com> References: <33AE1DB1-7BA9-4329-8AF0-EC7FEDCFBE7F@polycoresoftware.com> Date: Tue, 12 Feb 2013 08:36:20 -0800 Message-ID: Subject: Re: OFED setup and Mellanox cards From: Anthony Cornehl To: Tamer Abuelata Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2013 16:43:11 -0000 The /sys folder is abstracted to live in the sysctls. There is some abstraction in the Linux-compat layer, in /usr/src/sys/ofed/include, that can allow the use of folder and sysctl in your program from Linux to FreeBSD. I'll get a test setup this week. On Feb 11, 2013 1:29 PM, "Tamer Abuelata" wrote: > Hi, > > I'm trying to get OFED up and running on a FreeBSD 9.1 machine with 2 > Mellanox cards (MT26428). > We have compiled the Kernel with the following options (per > https://wiki.freebsd.org/InfiniBand ): > > options OFED # Infiniband protocol stack and support > options SDP # Sockets Direct Protocol for infiniband > options IPOIB_CM # Use connect mode ipoib > > device ipoib # IP over IB devices > device mlx4ib # ConnectX Infiniband support > device mlxen # ConnectX Ethernet support > device mthca # Infinihost cards > > I gave an IP to each card. > # ifconfig > ib0: flags=3D8043 metric 0 mtu 65520 > options=3D80018 > lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.d9 > inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 > nd6 options=3D29 > ib1: flags=3D8043 metric 0 mtu 65520 > options=3D80018 > lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.f1 > inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 > nd6 options=3D29 > > After doing a build world and build kernel I try to run an example > application and get the following warnings/errors: > > # ./server > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs1 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs1 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs1 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > listening on port 29823. > > Then I launch the client from another terminal: > # ./client 10.0.0.2 29823 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs1 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs1 > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > [=E2=80=A6 shows several more times =E2=80=A6] > Segmentation fault (core dumped) > > So the sample app compiles fine but I get this runtime error which seems > to come from the libibverbs. This /sys/class/infiniband_verbs folder does > not exist. I'm not sure if I'm missing configuration files. However I hav= e > devices at /dev/uverbs0 and /dev/uverbs1. > > The example app I am using is from: > > http://thegeekinthecorner.wordpress.com/2010/08/14/building-an-rdma-capab= le-application-with-ib-verbs-part-2-the-server/ > > The code is linked at the bottom of the page. Direct link: > > https://sites.google.com/a/bedeir.com/home/basic-rdma-client-server.tar.g= z?attredirects=3D0&d=3D1 > > I would appreciate any help or pointers from someone who managed to get > OFED to run on FreeBSD. > > Best regards, > Tamer > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to " > freebsd-infiniband-unsubscribe@freebsd.org" >