From owner-freebsd-infiniband@freebsd.org Wed Apr 13 07:14:03 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C36DAB0E6EC for ; Wed, 13 Apr 2016 07:14:03 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9111C183D; Wed, 13 Apr 2016 07:14:03 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 2FD831FE028; Wed, 13 Apr 2016 09:13:54 +0200 (CEST) Subject: Re: OFED v3.2 update To: Navdeep Parhar , "freebsd-infiniband@freebsd.org" References: <56FCF309.1070406@selasky.org> <56FD7DF8.7030702@FreeBSD.org> From: Hans Petter Selasky Message-ID: <570DF271.7010200@selasky.org> Date: Wed, 13 Apr 2016 09:17:05 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56FD7DF8.7030702@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 07:14:03 -0000 Hi, Some more details. On 03/31/16 21:43, Navdeep Parhar wrote: > Can you provide the OFA/linux-rdma/k.org version that matches the verb > APIs/KPIs in this import? This import is based on Linux kernel v3.19. On top of that, we added several experimental verbs that we are trying to submit to the Linux kernel. --HPS From owner-freebsd-infiniband@freebsd.org Sat Apr 16 08:39:44 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10270B10459 for ; Sat, 16 Apr 2016 08:39:44 +0000 (UTC) (envelope-from daichi@ongs.co.jp) Received: from pivot.bsdconsulting.co.jp (183.177.135.10.ap.gmobb-fix.jp [183.177.135.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E22CA167A for ; Sat, 16 Apr 2016 08:39:43 +0000 (UTC) (envelope-from daichi@ongs.co.jp) Received: from [192.168.1.39] (dullmdaler.ongs.co.jp [202.216.246.94]) by pivot.bsdconsulting.co.jp (Postfix) with ESMTPSA id C6D173C01C5; Sat, 16 Apr 2016 17:39:03 +0900 (JST) From: Daichi GOTO Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Mellanox MCB193A-FCAT Date: Sat, 16 Apr 2016 17:39:34 +0900 Message-Id: <49951A96-E904-4F7C-864C-F0CD953813A2@ongs.co.jp> To: freebsd-infiniband@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 08:39:44 -0000 Hi infiniband folks, this is daichi. I am considering to buy some Infiniband cards. A reseller of IB asked me I can buy some Mellanox MCB193A-FCAT cards for now. Mellanox MCB193A-FCAT = http://www.mellanoxstore.com/products/mellanox-mcb193a-fcat-connect-ib-tm-= host-channel-adapter-single-port-qsfp-fdr-56gb-s-pcie3-0-x16-tall-bracket-= rohs-r6.html Can I use this card with FreeBSD 10.3-RELEASE? Or do I need to use HEAD with OFED v3.2 patches? Please help me. Thank you. -- Daichi GOTO CEO | ONGS Inc. 81-42-316-7945 | daichi@ongs.co.jp | http://www.ongs.co.jp LinkedIn: http://linkedin.com/in/daichigoto From owner-freebsd-infiniband@freebsd.org Sat Apr 16 11:15:17 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AA13B1043F for ; Sat, 16 Apr 2016 11:15:17 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D643012DC for ; Sat, 16 Apr 2016 11:15:16 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1arOBy-0001bd-Te for freebsd-infiniband@freebsd.org; Sat, 16 Apr 2016 11:15:13 +0000 From: Justin Clift Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: How to force IB cards to come up in IB mode? Message-Id: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> Date: Sat, 16 Apr 2016 12:15:11 +0100 To: freebsd-infiniband@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 11:15:17 -0000 Hi all, I'm trying to get a new FreeBSD 10.3-STABLE install to work well with my MHGH28-XTC (dual port ConnectX DDR) cards. The problem is they're coming up in eth mode, not ib mode, and I can't seem to get that fixed. :( =46rom boot: # ifconfig mlxen0: flags=3D8802 metric 0 mtu 1500 = options=3D6d07bb ether 00:02:c9:00:ca:cd nd6 options=3D29 media: Ethernet autoselect status: no carrier mlxen1: flags=3D8802 metric 0 mtu 1500 = options=3D6d07bb ether 00:02:c9:00:ca:ce nd6 options=3D29 media: Ethernet autoselect status: no carrier My loader.conf file seems like it should be correct: # cat /boot/loader.conf sys.device.mlx4_core0.mlx4_port1=3D"ib" sys.device.mlx4_core0.mlx4_port2=3D"ib" As background, this is a new install (coming from CentOS 6.x, where it all works well), so I know there's no problem with the cards. = They're plugged into a 24 port DDR switch (Voltaire ISR 9024D-M) using active optical cables. Any ideas? Regards and best wishes, Justin Clift -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi From owner-freebsd-infiniband@freebsd.org Sat Apr 16 14:56:05 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20D7FAED7C0 for ; Sat, 16 Apr 2016 14:56:05 +0000 (UTC) (envelope-from menyy@mellanox.com) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0056.outbound.protection.outlook.com [157.56.112.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 744C51BB5 for ; Sat, 16 Apr 2016 14:56:03 +0000 (UTC) (envelope-from menyy@mellanox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=JYueaUnkcxYMm8nLwVqRVZzwY3RZM/d2mGNH43UcqF0=; b=G1zubrPotEqZFQEUBvgJkguYMpsOt8FoeUqfOPp/FTPQhSZW8ZTNLrRmI3h2cz9ziUm5XQPPmlO11a0IbhJRQcsMXR6HYWqPOvDzqmhtZlxSN1IFgzxsHclwBQCgexysjpRqNMIfjiCWy6xW23FPDVGXjGzGI815Y9OGoGMJAa8= Received: from DB5PR05MB1765.eurprd05.prod.outlook.com (10.165.7.23) by DB5PR05MB1302.eurprd05.prod.outlook.com (10.162.157.151) with Microsoft SMTP Server (TLS) id 15.1.466.19; Sat, 16 Apr 2016 14:55:54 +0000 Received: from DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) by DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) with mapi id 15.01.0453.031; Sat, 16 Apr 2016 14:55:54 +0000 From: Meny Yossefi To: "owner-freebsd-infiniband@freebsd.orgOn" , "freebsd-infiniband@freebsd.org" , Hans Petter Selasky , Yuval Bason Subject: RE: How to force IB cards to come up in IB mode? Thread-Topic: How to force IB cards to come up in IB mode? Thread-Index: AQHRl9FGkcYw8QOXCky3Yxke7zEzY5+McuvlgAA2bcA= Date: Sat, 16 Apr 2016 14:55:53 +0000 Message-ID: References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> In-Reply-To: <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.orgOn; dkim=none (message not signed) header.d=none; freebsd.orgOn; dmarc=none action=none header.from=mellanox.com; x-originating-ip: [79.178.38.82] x-ms-office365-filtering-correlation-id: 665fc7ce-4c0f-4d7d-5173-08d3660735fc x-microsoft-exchange-diagnostics: 1; DB5PR05MB1302; 5:6MApQlxtNNbWH/80byvbnRiX1ENC62tJJXM+10STD6IQL2PjTRCWMgXG+WLxB1u7Ng/FpdQw904zuqdWNvHtD4+TO0AkzFZEujoWnKFiTsFzhE9CH/Z6Xq2g4BZAHlHsa5oYth3Q2Pz6HKKJ4GnwjjoNFoI/yn+fzgWdBg08WE4pxcTxeduf/a8iKbZP5L9Q; 24:w+X0EIBmlXru+Pzpah+c/c4/vv791vWsV5fDZIGdlEiEYKHd3fhBicm7CWiZGNy7o9Bm/cRp7mKrjwcUt+FY0oFeBBV5JTJsPO9glWmMfpY=; 7:dm1DpcqEIcmCiLOp038GtTeSRnZe22XKUKpAKMYzoDcR4RwZuSiBBmVsDFkJz4OuEFsRImXIqelLrtaJhzeexi+Ib7IrdBzxnfqdgtzgUjLJ+qUlIcktCy6CTZ7n1+Mea0i5Mfj58FcJcQL31SN/FWNy9vgjDD7gJkdVm/w8r8wNCm7TZ0moSvMv/G1O0CKHG1QzkT1L5Z5+T8l9BX5/1w== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB5PR05MB1302; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(9101521026)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001); SRVR:DB5PR05MB1302; BCL:0; PCL:0; RULEID:; SRVR:DB5PR05MB1302; x-forefront-prvs: 09144DB0F7 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(53754006)(377454003)(3660700001)(5008740100001)(5004730100002)(11100500001)(33656002)(110136002)(189998001)(81166005)(107886002)(66066001)(19617315012)(19625215002)(106116001)(2906002)(16236675004)(5003600100002)(19300405004)(3280700002)(5002640100001)(92566002)(87936001)(50986999)(1096002)(1220700001)(2900100001)(3846002)(790700001)(6116002)(102836003)(122556002)(15975445007)(76176999)(2950100001)(10400500002)(2501003)(76576001)(4001450100002)(9686002)(19580405001)(19580395003)(77096005)(86362001)(586003)(74316001)(54356999); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR05MB1302; H:DB5PR05MB1765.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Apr 2016 14:55:53.5747 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1302 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 14:56:05 -0000 Hi Justin, It seems as the pre-boot configuration didn't work. loader.conf doesn't work on all sysctl params AFAIK. Is it a pre-boot only issue? What is the current sysctl output for those ports? Have you tried switching to IB after system was loaded? If I remember correctly, the driver was set for Eth configuration by defaul= t. Of course, this is reversible.. Regards, Meny Yossefi | SW Manager | FreeBSD Team Mellanox Technologies Ltd Work: +972-74-7129121, Cell: +972-52-8379557 ________________________________________ From: owner-freebsd-infiniband@freebsd.orgOn Behalf OfJustin Clift Sent: Saturday, April 16, 2016 11:15:11 AM (UTC) Monrovia, Reykjavik To: freebsd-infiniband@freebsd.org Subject: How to force IB cards to come up in IB mode? Hi all, I'm trying to get a new FreeBSD 10.3-STABLE install to work well with my MH= GH28-XTC (dual port ConnectX DDR) cards. The problem is they're coming up in eth mode, not ib mode, and I can't seem= to get that fixed. :( >From boot: # ifconfig mlxen0: flags=3D8802 metric 0 mtu 1500 options=3D6d07bb ether 00:02:c9:00:ca:cd nd6 options=3D29 media: Ethernet autoselect status: no carrier mlxen1: flags=3D8802 metric 0 mtu 1500 options=3D6d07bb ether 00:02:c9:00:ca:ce nd6 options=3D29 media: Ethernet autoselect status: no carrier My loader.conf file seems like it should be correct: # cat /boot/loader.conf sys.device.mlx4_core0.mlx4_port1=3D"ib" sys.device.mlx4_core0.mlx4_port2=3D"ib" As background, this is a new install (coming from CentOS 6.x, where it all = works well), so I know there's no problem with the cards. They're plugged = into a 24 port DDR switch (Voltaire ISR 9024D-M) using active optical cable= s. Any ideas? Regards and best wishes, Justin Clift -- "My grandfather once told me that there are two kinds of people: those who = work and those who take the credit. He told me to try to be in the first gr= oup; there was less competition there." - Indira Gandhi _______________________________________________ freebsd-infiniband@freebsd.org maili= ng list https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@freebsd.org Sat Apr 16 15:22:27 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EBB2DB102D0 for ; Sat, 16 Apr 2016 15:22:27 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF8CE1B6F for ; Sat, 16 Apr 2016 15:22:27 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1arS3A-0007UF-Qf; Sat, 16 Apr 2016 15:22:24 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: How to force IB cards to come up in IB mode? From: Justin Clift In-Reply-To: Date: Sat, 16 Apr 2016 16:22:18 +0100 Cc: "freebsd-infiniband@freebsd.org" , Hans Petter Selasky , Yuval Bason Content-Transfer-Encoding: quoted-printable Message-Id: References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> To: Meny Yossefi X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 15:22:28 -0000 Manually setting the ports via sysctl after boot works, although in a kind of strange fashion. When using the sysctl command to set the port type, the first attempt (regardless of which port) always fails: # sysctl sys.device.mlx4_core0.mlx4_port1=3D"ib" sys.device.mlx4_core0.mlx4_port1: eth sysctl: sys.device.mlx4_core0.mlx4_port1=3Dib: Invalid argument The 2nd attempt, to the other port, then succeeds: # sysctl sys.device.mlx4_core0.mlx4_port2=3D"ib" sys.device.mlx4_core0.mlx4_port2: eth -> ib It doesn't matter which order or which ports this is done in.=20 The 1st attempt always fails, the second attempt to "the other" port will then work. After the 2nd port has been changed, and IB mode becomes active, the IP addressing set in rc.conf seems to automatically take effect. eg: ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.cd inet 172.16.1.1 netmask 0xffffff00 broadcast 172.17.1.255 nd6 options=3D29 ib1: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.ce inet 172.16.2.1 netmask 0xffffff00 broadcast 172.17.1.255=20 nd6 options=3D29 While that's ok-ish for a once off thing, having some way for the cards to know which mode to start in seems like a better approach. Is the some way to get loader.conf (or anything really) to make that happen? :) Regards and best wishes, Justin Clift On 16 Apr 2016, at 15:55, Meny Yossefi wrote: > Hi Justin, >=20 > It seems as the pre-boot configuration didn't work. >=20 > loader.conf doesn't work on all sysctl params AFAIK. >=20 > Is it a pre-boot only issue? >=20 > What is the current sysctl output for those ports? >=20 > Have you tried switching to IB after system was loaded? >=20 > If I remember correctly, the driver was set for Eth configuration by = default. >=20 > Of course, this is reversible.. >=20 >=20 >=20 > Regards, > Meny Yossefi | SW Manager | FreeBSD Team > Mellanox Technologies Ltd > Work: +972-74-7129121, Cell: +972-52-8379557 >=20 >=20 >=20 > ________________________________________ >=20 > From: = owner-freebsd-infiniband@freebsd.orgOn Behalf OfJustin Clift >=20 > Sent: Saturday, April 16, 2016 11:15:11 AM (UTC) Monrovia, Reykjavik >=20 > To: = freebsd-infiniband@freebsd.org >=20 > Subject: How to force IB cards to come up in IB mode? >=20 >=20 >=20 > Hi all, >=20 >=20 >=20 > I'm trying to get a new FreeBSD 10.3-STABLE install to work well with = my MHGH28-XTC (dual port ConnectX DDR) cards. >=20 >=20 >=20 > The problem is they're coming up in eth mode, not ib mode, and I can't = seem to get that fixed. :( >=20 >=20 >=20 > =46rom boot: >=20 >=20 >=20 > # ifconfig >=20 > mlxen0: flags=3D8802 metric 0 mtu 1500 >=20 > = options=3D6d07bb >=20 > ether 00:02:c9:00:ca:cd >=20 > nd6 options=3D29 >=20 > media: Ethernet autoselect >=20 > status: no carrier >=20 > mlxen1: flags=3D8802 metric 0 mtu 1500 >=20 > = options=3D6d07bb >=20 > ether 00:02:c9:00:ca:ce >=20 > nd6 options=3D29 >=20 > media: Ethernet autoselect >=20 > status: no carrier >=20 >=20 >=20 > My loader.conf file seems like it should be correct: >=20 >=20 >=20 > # cat /boot/loader.conf >=20 > sys.device.mlx4_core0.mlx4_port1=3D"ib" >=20 > sys.device.mlx4_core0.mlx4_port2=3D"ib" >=20 >=20 >=20 > As background, this is a new install (coming from CentOS 6.x, where it = all works well), so I know there's no problem with the cards. They're = plugged into a 24 port DDR switch (Voltaire ISR 9024D-M) using active = optical cables. >=20 >=20 >=20 > Any ideas? >=20 >=20 >=20 > Regards and best wishes, >=20 >=20 >=20 > Justin Clift >=20 >=20 >=20 > -- >=20 > "My grandfather once told me that there are two kinds of people: those = who work and those who take the credit. He told me to try to be in the = first group; there was less competition there." >=20 > - Indira Gandhi >=20 >=20 >=20 > _______________________________________________ >=20 > freebsd-infiniband@freebsd.org = mailing list = https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >=20 > To unsubscribe, send any mail to = "freebsd-infiniband-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to = "freebsd-infiniband-unsubscribe@freebsd.org" -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi From owner-freebsd-infiniband@freebsd.org Sat Apr 16 19:43:40 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BC7DAEEEFD for ; Sat, 16 Apr 2016 19:43:40 +0000 (UTC) (envelope-from menyy@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0066.outbound.protection.outlook.com [157.55.234.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 87E5A1182 for ; Sat, 16 Apr 2016 19:43:38 +0000 (UTC) (envelope-from menyy@mellanox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=rprrcELpHESJCp0K71jlTtnE2gBmrrIDanfpxrpXSA0=; b=NNGRV2SWMFRjiatz1yHczoZ0loEACfwtwNIWtc5M24m5nVuEZAF0evx7lsVrOvm3725rLomEHELW8rgJS7fDNEom9nhZWSKf9U7PNEYvdTSUn0egQcTY5z8XKSFwYEKcTQpeFWEI5eg5uS3Dstyt9v/awFK2+431nB6ANVvZIhg= Received: from DB5PR05MB1765.eurprd05.prod.outlook.com (10.165.7.23) by DB5PR05MB1383.eurprd05.prod.outlook.com (10.162.157.29) with Microsoft SMTP Server (TLS) id 15.1.466.19; Sat, 16 Apr 2016 15:10:40 +0000 Received: from DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) by DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) with mapi id 15.01.0453.031; Sat, 16 Apr 2016 15:10:40 +0000 From: Meny Yossefi To: Mark Bloch , Yuval Bason , "freebsd-infiniband@freebsd.org" Subject: RE: Mellanox MCB193A-FCAT Thread-Topic: Mellanox MCB193A-FCAT Thread-Index: AQHRl7uM97Bqoue55EecjN8nLKRA2p+MR6GbgABpqfA= Date: Sat, 16 Apr 2016 15:10:40 +0000 Message-ID: References: <49951A96-E904-4F7C-864C-F0CD953813A2@ongs.co.jp> <111b47284ead41c3a30ea7c1ebc7f2ed@AMSPR05MB0580.eurprd05.prod.outlook.com> In-Reply-To: <111b47284ead41c3a30ea7c1ebc7f2ed@AMSPR05MB0580.eurprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: mellanox.com; dkim=none (message not signed) header.d=none;mellanox.com; dmarc=none action=none header.from=mellanox.com; x-originating-ip: [79.178.38.82] x-ms-office365-filtering-correlation-id: 76c064a1-4fb4-4a2b-ec8d-08d36609463f x-microsoft-exchange-diagnostics: 1; DB5PR05MB1383; 5:AjZ0j+ikeFQzm3PeB3+f10ETUGqi3DUU2ylhxxxkircxHCFRwxWzsVFyg4Fri8/L0FqAica0CcisoDQUwMu64vgASp8Ckm0iT2JADdu+LhUMMPGaNUhozbpi0KtgxYEV+s0bMX3Ikmjw1vmDgYAq9EBMsrND+KaQQHLzsVs/RoNdNHiJo94gMMKj9rEShMV1; 24:hElMeu7qngTz/YG8dHL0w6gB8Dr6pFhBfaQZR/HVtjxfKxS64R9FLBPEHbYzMsM1CiShADIRr0Yppgjz/acX/Mu5KWkWdrATRRbwJjibCFo=; 7:66jOIW251cHto2NrOntFqjPueCTjst/2emlp/bLCHwBTd0X1qAHhvYdXgVaNDWubeNwVNtegAeib9OSUi8arWST1iWKzh+/KOfOHLeCQ+HWsvyqdpsaPozMS16EPCgoyrYo7/5T0piQOjL7UWhEpB1qstgZAF7G2MKi1gbMXwG51qTzNAASA6TOV9ZmBs/fRLibusFFFXFLbxZf4FUHLng== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB5PR05MB1383; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(9101521026)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046); SRVR:DB5PR05MB1383; BCL:0; PCL:0; RULEID:; SRVR:DB5PR05MB1383; x-forefront-prvs: 09144DB0F7 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(13464003)(52044002)(57704003)(377454003)(2906002)(86362001)(3660700001)(81166005)(5002640100001)(3280700002)(10400500002)(74316001)(16601075003)(9686002)(106116001)(5001770100001)(3900700001)(586003)(189998001)(1220700001)(107886002)(1096002)(19617315012)(5003600100002)(6116002)(102836003)(66066001)(3846002)(790700001)(122556002)(15975445007)(19625215002)(33656002)(19300405004)(5008740100001)(92566002)(450100001)(54356999)(11100500001)(5004730100002)(16236675004)(15395725005)(2950100001)(2900100001)(76576001)(77096005)(87936001)(2501003)(50986999)(76176999)(19580405001)(19580395003); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR05MB1383; H:DB5PR05MB1765.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Apr 2016 15:10:40.0735 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1383 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 19:43:40 -0000 Hi Daichi, Your IB NIC (Connect-IB) requires our mlx5 driver. Our upstream solution which also exists in 10.3 is Eth only. We are currently working on the IB side. Please refer to the "OFED v3.2 update" mailing list item. For any other questions, feel free to use the Mellanox support mail. support@mellanox.com or freebsd-drivers@mellanox.com Best regards, Meny Yossefi | SW Manager | FreeBSD Team Mellanox Technologies Ltd Work: +972-74-7129121, Cell: +972-52-8379557 -----Original Message----- From: freebsd-commits-tracker Sent: Saturday, April 16, 2016 11:40 AM To: Mark Bloch ; Meny Yossefi ; Yuv= al Bason Subject: FW: Mellanox MCB193A-FCAT ________________________________________ From: owner-freebsd-infiniband@freebsd.orgOn Behalf OfDaichi GOTO Sent: Saturday, April 16, 2016 8:39:34 AM (UTC) Monrovia, Reykjavik To: freebsd-infiniband@freebsd.org Subject: Mellanox MCB193A-FCAT Hi infiniband folks, this is daichi. I am considering to buy some Infiniband cards. A reseller of IB asked me I = can buy some Mellanox MCB193A-FCAT cards for now. Mellanox MCB193A-FCAT http://www.mellanoxstore.com/products/mellanox-mcb193a-fcat-connect= -ib-tm-host-channel-adapter-single-port-qsfp-fdr-56gb-s-pcie3-0-x16-tall-br= acket-rohs-r6.html Can I use this card with FreeBSD 10.3-RELEASE? Or do I need to use HEAD wi= th OFED v3.2 patches? Please help me. Thank you. -- Daichi GOTO CEO | ONGS Inc. 81-42-316-7945 | daichi@ongs.co.jp | http://www.o= ngs.co.jp LinkedIn: http://linkedin.com/in/daichigoto _______________________________________________ freebsd-infiniband@freebsd.org maili= ng list https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@freebsd.org Sat Apr 16 20:59:37 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3143B118C6 for ; Sat, 16 Apr 2016 20:59:37 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A6D7019FF for ; Sat, 16 Apr 2016 20:59:37 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1arXJS-0000IH-M0; Sat, 16 Apr 2016 20:59:33 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: How to force IB cards to come up in IB mode? From: Justin Clift In-Reply-To: Date: Sat, 16 Apr 2016 21:59:27 +0100 Cc: "freebsd-infiniband@freebsd.org" , Hans Petter Selasky , Yuval Bason Content-Transfer-Encoding: quoted-printable Message-Id: <037CBF0B-10E4-47FB-97A1-B281E8BE5151@postgresql.org> References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> To: Meny Yossefi X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 20:59:38 -0000 On 16 Apr 2016, at 15:55, Meny Yossefi wrote: > Hi Justin, >=20 >=20 > It seems as the pre-boot configuration didn't work. >=20 > loader.conf doesn't work on all sysctl params AFAIK. >=20 >=20 > Is it a pre-boot only issue? >=20 > What is the current sysctl output for those ports? >=20 > Have you tried switching to IB after system was loaded? >=20 >=20 > If I remember correctly, the driver was set for Eth configuration by = default. >=20 > Of course, this is reversible.. Looking at the 10.3 code, this seems to be the port selection clause: = https://github.com/freebsd/freebsd/blob/master/sys/ofed/drivers/net/mlx4/m= ain.c#L706-L736 In the comments inside that clause, it says: "If IB and ETH are supported, we set the port type according to user selection of port type; if there is no user selection, take the FW = hint" = https://github.com/freebsd/freebsd/blob/master/sys/ofed/drivers/net/mlx4/m= ain.c#L716-L718 There is no man page for mlx4, so I'm a bit stumped for how to select the port type for "user selection" as mentioned there. Any hints? :) (And sure, things could be forced in the code with something like: dev->caps.port_type[i] =3D MLX4_PORT_TYPE_IB; ... for each of the ports. But that's a temporary only solution and we probably find the proper one instead. ;D Regards and best wishes, Justin Clift -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi From owner-freebsd-infiniband@freebsd.org Sat Apr 16 22:33:54 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D78FAED859 for ; Sat, 16 Apr 2016 22:33:54 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5D6316EE for ; Sat, 16 Apr 2016 22:33:53 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1arYmi-0002S6-Oq; Sat, 16 Apr 2016 22:33:51 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: How to force IB cards to come up in IB mode? From: Justin Clift In-Reply-To: Date: Sat, 16 Apr 2016 23:33:45 +0100 Cc: "freebsd-infiniband@freebsd.org" , Hans Petter Selasky , Yuval Bason Content-Transfer-Encoding: 7bit Message-Id: <0DE8164F-FB16-40E0-A512-F233658C52A0@postgresql.org> References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> To: Meny Yossefi X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 22:33:54 -0000 On 16 Apr 2016, at 15:55, Meny Yossefi wrote: > Hi Justin, > > It seems as the pre-boot configuration didn't work. > > loader.conf doesn't work on all sysctl params AFAIK. The solution in this case turned out to be super simple... I was just using the wrong file. :D It should have been /etc/sysctl.conf for the settings. Adding this in there worked for the initial port mode setup: # Ensure the IB card is in IB mode sys.device.mlx4_core0.mlx4_port1="ib" sys.device.mlx4_core0.mlx4_port2="ib" Apologies for the noise. ;) Regards and best wishes, Justin Clift -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi