Date: Sat, 16 Apr 2016 16:22:18 +0100 From: Justin Clift <justin@postgresql.org> To: Meny Yossefi <menyy@mellanox.com> Cc: "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org>, Hans Petter Selasky <hanss@mellanox.com>, Yuval Bason <yuvalba@mellanox.com> Subject: Re: How to force IB cards to come up in IB mode? Message-ID: <CBB7335E-69C7-4A13-8F44-ED262D56688B@postgresql.org> In-Reply-To: <DB5PR05MB1765AAD188E07C726CE5C005C3690@DB5PR05MB1765.eurprd05.prod.outlook.com> References: <9C08621D-AAE9-42D0-8079-716CDD9BC5E0@postgresql.org> <223ee8f3dd5240779d39bf78c383c50c@AMSPR05MB0580.eurprd05.prod.outlook.com> <DB5PR05MB1765AAD188E07C726CE5C005C3690@DB5PR05MB1765.eurprd05.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Manually setting the ports via sysctl after boot works, although
in a kind of strange fashion.
When using the sysctl command to set the port type, the first
attempt (regardless of which port) always fails:
# sysctl sys.device.mlx4_core0.mlx4_port1=3D"ib"
sys.device.mlx4_core0.mlx4_port1: eth
sysctl: sys.device.mlx4_core0.mlx4_port1=3Dib: Invalid argument
The 2nd attempt, to the other port, then succeeds:
# sysctl sys.device.mlx4_core0.mlx4_port2=3D"ib"
sys.device.mlx4_core0.mlx4_port2: eth -> ib
It doesn't matter which order or which ports this is done in.=20
The 1st attempt always fails, the second attempt to "the other"
port will then work.
After the 2nd port has been changed, and IB mode becomes active, the
IP addressing set in rc.conf seems to automatically take effect.
eg:
ib0: flags=3D8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
options=3D80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.cd
inet 172.16.1.1 netmask 0xffffff00 broadcast 172.17.1.255
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ib1: flags=3D8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
options=3D80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.ce
inet 172.16.2.1 netmask 0xffffff00 broadcast 172.17.1.255=20
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
While that's ok-ish for a once off thing, having some way for the
cards to know which mode to start in seems like a better approach.
Is the some way to get loader.conf (or anything really) to make that
happen? :)
Regards and best wishes,
Justin Clift
On 16 Apr 2016, at 15:55, Meny Yossefi <menyy@mellanox.com> wrote:
> Hi Justin,
>=20
> It seems as the pre-boot configuration didn't work.
>=20
> loader.conf doesn't work on all sysctl params AFAIK.
>=20
> Is it a pre-boot only issue?
>=20
> What is the current sysctl output for those ports?
>=20
> Have you tried switching to IB after system was loaded?
>=20
> If I remember correctly, the driver was set for Eth configuration by =
default.
>=20
> Of course, this is reversible..
>=20
>=20
>=20
> Regards,
> Meny Yossefi | SW Manager | FreeBSD Team
> Mellanox Technologies Ltd
> Work: +972-74-7129121, Cell: +972-52-8379557
>=20
>=20
>=20
> ________________________________________
>=20
> From: =
owner-freebsd-infiniband@freebsd.orgOn<mailto:owner-freebsd-infiniband@fre=
ebsd.orgOn> Behalf OfJustin Clift
>=20
> Sent: Saturday, April 16, 2016 11:15:11 AM (UTC) Monrovia, Reykjavik
>=20
> To: =
freebsd-infiniband@freebsd.org<mailto:freebsd-infiniband@freebsd.org>
>=20
> Subject: How to force IB cards to come up in IB mode?
>=20
>=20
>=20
> Hi all,
>=20
>=20
>=20
> I'm trying to get a new FreeBSD 10.3-STABLE install to work well with =
my MHGH28-XTC (dual port ConnectX DDR) cards.
>=20
>=20
>=20
> The problem is they're coming up in eth mode, not ib mode, and I can't =
seem to get that fixed. :(
>=20
>=20
>=20
> =46rom boot:
>=20
>=20
>=20
> # ifconfig
>=20
> mlxen0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>=20
> =
options=3D6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS=
UM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV=
6>
>=20
> ether 00:02:c9:00:ca:cd
>=20
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>=20
> media: Ethernet autoselect
>=20
> status: no carrier
>=20
> mlxen1: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>=20
> =
options=3D6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS=
UM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV=
6>
>=20
> ether 00:02:c9:00:ca:ce
>=20
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>=20
> media: Ethernet autoselect
>=20
> status: no carrier
>=20
>=20
>=20
> My loader.conf file seems like it should be correct:
>=20
>=20
>=20
> # cat /boot/loader.conf
>=20
> sys.device.mlx4_core0.mlx4_port1=3D"ib"
>=20
> sys.device.mlx4_core0.mlx4_port2=3D"ib"
>=20
>=20
>=20
> As background, this is a new install (coming from CentOS 6.x, where it =
all works well), so I know there's no problem with the cards. They're =
plugged into a 24 port DDR switch (Voltaire ISR 9024D-M) using active =
optical cables.
>=20
>=20
>=20
> Any ideas?
>=20
>=20
>=20
> Regards and best wishes,
>=20
>=20
>=20
> Justin Clift
>=20
>=20
>=20
> --
>=20
> "My grandfather once told me that there are two kinds of people: those =
who work and those who take the credit. He told me to try to be in the =
first group; there was less competition there."
>=20
> - Indira Gandhi
>=20
>=20
>=20
> _______________________________________________
>=20
> freebsd-infiniband@freebsd.org<mailto:freebsd-infiniband@freebsd.org> =
mailing list =
https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>=20
> To unsubscribe, send any mail to =
"freebsd-infiniband-unsubscribe@freebsd.org<mailto:freebsd-infiniband-unsu=
bscribe@freebsd.org>"
> _______________________________________________
> freebsd-infiniband@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to =
"freebsd-infiniband-unsubscribe@freebsd.org"
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CBB7335E-69C7-4A13-8F44-ED262D56688B>