From owner-freebsd-infiniband@freebsd.org Sat Apr 16 15:22:27 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EBB2DB102D0 for ; Sat, 16 Apr 2016 15:22:27 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF8CE1B6F for ; Sat, 16 Apr 2016 15:22:27 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1arS3A-0007UF-Qf; Sat, 16 Apr 2016 15:22:24 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: How to force IB cards to come up in IB mode? From: Justin Clift In-Reply-To: Date: Sat, 16 Apr 2016 16:22:18 +0100 Cc: "freebsd-infiniband@freebsd.org" , Hans Petter Selasky , Yuval Bason Content-Transfer-Encoding: quoted-printable Message-Id: References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> To: Meny Yossefi X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2016 15:22:28 -0000 Manually setting the ports via sysctl after boot works, although in a kind of strange fashion. When using the sysctl command to set the port type, the first attempt (regardless of which port) always fails: # sysctl sys.device.mlx4_core0.mlx4_port1=3D"ib" sys.device.mlx4_core0.mlx4_port1: eth sysctl: sys.device.mlx4_core0.mlx4_port1=3Dib: Invalid argument The 2nd attempt, to the other port, then succeeds: # sysctl sys.device.mlx4_core0.mlx4_port2=3D"ib" sys.device.mlx4_core0.mlx4_port2: eth -> ib It doesn't matter which order or which ports this is done in.=20 The 1st attempt always fails, the second attempt to "the other" port will then work. After the 2nd port has been changed, and IB mode becomes active, the IP addressing set in rc.conf seems to automatically take effect. eg: ib0: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.cd inet 172.16.1.1 netmask 0xffffff00 broadcast 172.17.1.255 nd6 options=3D29 ib1: flags=3D8043 metric 0 mtu 65520 options=3D80018 lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.ce inet 172.16.2.1 netmask 0xffffff00 broadcast 172.17.1.255=20 nd6 options=3D29 While that's ok-ish for a once off thing, having some way for the cards to know which mode to start in seems like a better approach. Is the some way to get loader.conf (or anything really) to make that happen? :) Regards and best wishes, Justin Clift On 16 Apr 2016, at 15:55, Meny Yossefi wrote: > Hi Justin, >=20 > It seems as the pre-boot configuration didn't work. >=20 > loader.conf doesn't work on all sysctl params AFAIK. >=20 > Is it a pre-boot only issue? >=20 > What is the current sysctl output for those ports? >=20 > Have you tried switching to IB after system was loaded? >=20 > If I remember correctly, the driver was set for Eth configuration by = default. >=20 > Of course, this is reversible.. >=20 >=20 >=20 > Regards, > Meny Yossefi | SW Manager | FreeBSD Team > Mellanox Technologies Ltd > Work: +972-74-7129121, Cell: +972-52-8379557 >=20 >=20 >=20 > ________________________________________ >=20 > From: = owner-freebsd-infiniband@freebsd.orgOn Behalf OfJustin Clift >=20 > Sent: Saturday, April 16, 2016 11:15:11 AM (UTC) Monrovia, Reykjavik >=20 > To: = freebsd-infiniband@freebsd.org >=20 > Subject: How to force IB cards to come up in IB mode? >=20 >=20 >=20 > Hi all, >=20 >=20 >=20 > I'm trying to get a new FreeBSD 10.3-STABLE install to work well with = my MHGH28-XTC (dual port ConnectX DDR) cards. >=20 >=20 >=20 > The problem is they're coming up in eth mode, not ib mode, and I can't = seem to get that fixed. :( >=20 >=20 >=20 > =46rom boot: >=20 >=20 >=20 > # ifconfig >=20 > mlxen0: flags=3D8802 metric 0 mtu 1500 >=20 > = options=3D6d07bb >=20 > ether 00:02:c9:00:ca:cd >=20 > nd6 options=3D29 >=20 > media: Ethernet autoselect >=20 > status: no carrier >=20 > mlxen1: flags=3D8802 metric 0 mtu 1500 >=20 > = options=3D6d07bb >=20 > ether 00:02:c9:00:ca:ce >=20 > nd6 options=3D29 >=20 > media: Ethernet autoselect >=20 > status: no carrier >=20 >=20 >=20 > My loader.conf file seems like it should be correct: >=20 >=20 >=20 > # cat /boot/loader.conf >=20 > sys.device.mlx4_core0.mlx4_port1=3D"ib" >=20 > sys.device.mlx4_core0.mlx4_port2=3D"ib" >=20 >=20 >=20 > As background, this is a new install (coming from CentOS 6.x, where it = all works well), so I know there's no problem with the cards. They're = plugged into a 24 port DDR switch (Voltaire ISR 9024D-M) using active = optical cables. >=20 >=20 >=20 > Any ideas? >=20 >=20 >=20 > Regards and best wishes, >=20 >=20 >=20 > Justin Clift >=20 >=20 >=20 > -- >=20 > "My grandfather once told me that there are two kinds of people: those = who work and those who take the credit. He told me to try to be in the = first group; there was less competition there." >=20 > - Indira Gandhi >=20 >=20 >=20 > _______________________________________________ >=20 > freebsd-infiniband@freebsd.org = mailing list = https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >=20 > To unsubscribe, send any mail to = "freebsd-infiniband-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to = "freebsd-infiniband-unsubscribe@freebsd.org" -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi