From owner-freebsd-infiniband@freebsd.org Sat Apr 16 20:59:37 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3143B118C6 for ; Sat, 16 Apr 2016 20:59:37 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A6D7019FF for ; Sat, 16 Apr 2016 20:59:37 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1arXJS-0000IH-M0; Sat, 16 Apr 2016 20:59:33 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: How to force IB cards to come up in IB mode? From: Justin Clift In-Reply-To: Date: Sat, 16 Apr 2016 21:59:27 +0100 Cc: "freebsd-infiniband@freebsd.org" , Hans Petter Selasky , Yuval Bason Content-Transfer-Encoding: quoted-printable Message-Id: <037CBF0B-10E4-47FB-97A1-B281E8BE5151@postgresql.org> References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> To: Meny Yossefi X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 20:59:38 -0000 On 16 Apr 2016, at 15:55, Meny Yossefi wrote: > Hi Justin, >=20 >=20 > It seems as the pre-boot configuration didn't work. >=20 > loader.conf doesn't work on all sysctl params AFAIK. >=20 >=20 > Is it a pre-boot only issue? >=20 > What is the current sysctl output for those ports? >=20 > Have you tried switching to IB after system was loaded? >=20 >=20 > If I remember correctly, the driver was set for Eth configuration by = default. >=20 > Of course, this is reversible.. Looking at the 10.3 code, this seems to be the port selection clause: = https://github.com/freebsd/freebsd/blob/master/sys/ofed/drivers/net/mlx4/m= ain.c#L706-L736 In the comments inside that clause, it says: "If IB and ETH are supported, we set the port type according to user selection of port type; if there is no user selection, take the FW = hint" = https://github.com/freebsd/freebsd/blob/master/sys/ofed/drivers/net/mlx4/m= ain.c#L716-L718 There is no man page for mlx4, so I'm a bit stumped for how to select the port type for "user selection" as mentioned there. Any hints? :) (And sure, things could be forced in the code with something like: dev->caps.port_type[i] =3D MLX4_PORT_TYPE_IB; ... for each of the ports. But that's a temporary only solution and we probably find the proper one instead. ;D Regards and best wishes, Justin Clift -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi