From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 14 18:40:05 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 189CA724 for ; Sun, 14 Jun 2015 18:40:05 +0000 (UTC) (envelope-from unixro@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3318625 for ; Sun, 14 Jun 2015 18:40:04 +0000 (UTC) (envelope-from unixro@gmail.com) Received: by wgbhy7 with SMTP id hy7so20568858wgb.2 for ; Sun, 14 Jun 2015 11:40:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=6ToLrzYVir8CoX8x4aDEknZQRUWNOFq3x71HqOrOuV8=; b=UbOg5N+U1Z7NUs4IoI9KL4JefUrskHfMEnwfBMHbQz48ifjzE0qc3xehLAkAwn1eZn j75wHheFlUj+e1MTPNEyvZ0LuUwTCL0wHOExHna6Gw70oDvaOCKgPYGLfH0I96/Vs+cA M0Aj+QPhN7/VvvW4DpgSTg2xpVbxGq0p/CK74Fi6P8mXmSnffueY+yRIKS78AkN6n9Di 0QJuXXjFtmKPh6zIxBwR2Ttc3UqcHQ4kE9Z7QvWJ9zxX589+FJDEs/1s6B/ypg1uGPVO v4Nn0liX6Ej1nCd0qhuUB5bkzatqZocQil/fDxzghZOwgW/pG8FUClP1TjJ53IxRKi9k jbnQ== X-Received: by 10.194.236.68 with SMTP id us4mr40004324wjc.69.1434307202131; Sun, 14 Jun 2015 11:40:02 -0700 (PDT) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id ma15sm12379888wic.20.2015.06.14.11.40.01 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Jun 2015 11:40:01 -0700 (PDT) Message-ID: <557DCA80.50203@gmail.com> Date: Sun, 14 Jun 2015 21:40:00 +0300 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> In-Reply-To: <557C7D94.9020207@selasky.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 18:40:05 -0000 Another strange issue. Is that i have: /boot/loader.conf mlx4_core="YES" ibcore_load="YES" mlxen_load="YES" mlx4ib_load="YES" ipoib_load="YES" and in sysctl.conf sys.device.mlx4_core0.mlx4_port1=ib Only when it boots ib0 is not functional. From what i've noticed if after mlx4_core0 is up i remove ipoib module and add it again it's working. Bellow is a small output from dmesg in which i unload/load ipoib at boot in rc.local and interface still doesn't work and after port is up i do it manually and it starts working: ib0: Attached to mlx4_0 port 1 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 ib0: link state changed to DOWN ib0: Attached to mlx4_0 port 1 mlx4_core0: mlx4_ib: Port 1 logical link is up ib0: link state changed to DOWN ib0: Attached to mlx4_0 port 1 ib0: link state changed to UP in ifconfig only difference is a new lladdr for ib0. Best regards, Vintila Mihai Alexandru On 6/13/2015 9:59 PM, Hans Petter Selasky wrote: > On 06/13/15 20:43, Mihai Vintila wrote: >> Hi, >> I've posted in the stable mailing list but i guess this is where i >> should ask. >> Can you tell me what is the proper way to get subnet manager working on >> FreeBSD 10.1 ? >> I've used RELEASE branch to compile opensm and drivers from Mellanox but >> i get: >> ibportstate -G 0xe61d2dfffe0a2130 1 enable >> ibpanic: [868] madrpc_init: can't open UMAD port ((null):0): (m) >> >> opensm >> ------------------------------------------------- >> OpenSM 3.3.1 >> Reading Cached Option File: /etc/opensm/opensm.conf >> Command Line Arguments: >> Log File: /var/log/opensm.log >> ------------------------------------------------- >> OpenSM 3.3.1 >> >> Entering DISCOVERING state >> >> Using default GUID 0x0 >> >> Error from osm_opensm_bind (0x2A) >> Perhaps another instance of OpenSM is already running >> Exiting SM >> >> >> >> kldstat >> Id Refs Address Size Name >> 1 27 0xffffffff80200000 1756638 kernel >> 2 1 0xffffffff81957000 2681e8 zfs.ko >> 3 2 0xffffffff81bc0000 6780 opensolaris.ko >> 4 2 0xffffffff81bc7000 17d60 nvme.ko >> 5 1 0xffffffff81bdf000 4210 nvd.ko >> 6 1 0xffffffff81e11000 357f ums.ko >> 7 3 0xffffffff81e15000 37656 mlx4.ko >> 8 2 0xffffffff81e4d000 4e348 ibcore.ko >> 9 1 0xffffffff81e9c000 2f72 toecore.ko >> 10 1 0xffffffff81e9f000 238fa mlx4ib.ko >> 11 1 0xffffffff81ec3000 12482 mlxen.ko >> >> Driver that i'm using: >> mlx4_core0: mem 0xc7400000-0xc74fffff,0xc5000000-0xc57fffff >> irq 32 at device 0.0 on pci2 >> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 >> (Jun 12 2015) >> >> mlx4_core0: mlx4_ib_add: allocated counter index 1 for port 1 >> mlx4_en mlx4_core0: Activating port:1 >> mlxen0: Ethernet address: e4:1d:2d:0a:21:30 >> mlx4_en: mlx4_core0: Port 1: Using 12 TX rings >> mlxen0: link state changed to DOWN >> mlx4_en: mlx4_core0: Port 1: Using 8 RX rings >> mlx4_en: mlxen0: Using 12 TX rings >> mlx4_en: mlxen0: Using 8 RX rings >> mlx4_en: mlxen0: Initializing port >> > > Hi, > > Did you check your ports are configured for "ib" and not "eth" ? > > For example: > sysctl sys.device.mlx4_core0.mlx4_port1=ib > > --HPS > From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 14 18:56:58 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 16976D3F for ; Sun, 14 Jun 2015 18:56:58 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CFE93AB6 for ; Sun, 14 Jun 2015 18:56:57 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id C2B311FE023; Sun, 14 Jun 2015 20:56:54 +0200 (CEST) Message-ID: <557DCEA7.3030703@selasky.org> Date: Sun, 14 Jun 2015 20:57:43 +0200 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Mihai Vintila , freebsd-infiniband@freebsd.org Subject: Re: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> In-Reply-To: <557DCA80.50203@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 18:56:58 -0000 Hi, On 06/14/15 20:40, Mihai Vintila wrote: > Another strange issue. Is that i have: > /boot/loader.conf > mlx4_core="YES" > ibcore_load="YES" > mlxen_load="YES" > mlx4ib_load="YES" > ipoib_load="YES" If you remove this line from sysctl.conf and run it from the command line instead, does it work then? > > and in sysctl.conf > sys.device.mlx4_core0.mlx4_port1=ib > > Only when it boots ib0 is not functional. From what i've noticed if > after mlx4_core0 is up i remove ipoib module and add it again it's working. > > Bellow is a small output from dmesg in which i unload/load ipoib at boot > in rc.local and interface still doesn't work and after port is up i do > it manually and it starts working: > ib0: Attached to mlx4_0 port 1 > ums0: > on usbus0 > ums0: 3 buttons and [Z] coordinates ID=0 > ib0: link state changed to DOWN > ib0: Attached to mlx4_0 port 1 > mlx4_core0: mlx4_ib: Port 1 logical link is up > ib0: link state changed to DOWN > ib0: Attached to mlx4_0 port 1 > ib0: link state changed to UP > > > in ifconfig only difference is a new lladdr for ib0. > --HPS From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 14 19:12:14 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A1483559 for ; Sun, 14 Jun 2015 19:12:14 +0000 (UTC) (envelope-from unixro@gmail.com) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 362C8EDC for ; Sun, 14 Jun 2015 19:12:14 +0000 (UTC) (envelope-from unixro@gmail.com) Received: by wgbhy7 with SMTP id hy7so20869616wgb.2 for ; Sun, 14 Jun 2015 12:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=o8ZzaZjEHwD0/FhsI00TgmqvvdfrUqUFN7P+LU/L2K8=; b=RBkCp6Ee8PZB2mZ1Qu55Sjs0PiXgyxNQqTEuMikWTy+KWntJx6mHO1LkqwZzhCPD2B 8nY1rJHqHWE0oaJjuGgYeu/gM8yT7dDO3UEwqjG6s2SOtwYkJkGUDP+qH5asS4cayc2O 0sx3S+gt3zXQxHwlQoQhdURkRJzCSxgJjOmAj2fMgByWqs6t9y+n2sahnmAJMM3yvbeQ pF2OCByvt88aJrI1tPdEcXQNxGaFVEWB5nzUZ2dRkT3ArPnlxnIwoxMGaW2fwldpzIRH 4UeOtBI94L8Zu/G3Hs+cJk/vAqhK2R+ENez76ZpvHFxeVpw5VJeWWWH71pX+sZKk5iAQ ELmg== X-Received: by 10.180.7.169 with SMTP id k9mr25435318wia.84.1434309132665; Sun, 14 Jun 2015 12:12:12 -0700 (PDT) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id di9sm12487217wib.16.2015.06.14.12.12.11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Jun 2015 12:12:12 -0700 (PDT) Message-ID: <557DD20B.7020508@gmail.com> Date: Sun, 14 Jun 2015 22:12:11 +0300 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Hans Petter Selasky , freebsd-infiniband@freebsd.org Subject: Re: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> In-Reply-To: <557DCEA7.3030703@selasky.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 19:12:14 -0000 Same behavior: sysctl -w sys.device.mlx4_core0.mlx4_port1=ib sys.device.mlx4_core0.mlx4_port1: eth -> ib ifconfig ib0 ib0: flags=8043 metric 0 mtu 2044 options=8009b lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.a.21.31 inet 10.105.0.225 netmask 0xffffff00 broadcast 10.105.0.255 nd6 options=29 ping 10.105.0.15 PING 10.105.0.15 (10.105.0.15): 56 data bytes ping: sendto: No route to host ping: sendto: No route to host Might worth to mention that this host also holds opensm, and i actually need to run opensm first next unload/load the module. But i rather expect that the ib0 state changes along mlx4_core0 state. Best regards, Vintila Mihai Alexandru On 6/14/2015 9:57 PM, Hans Petter Selasky wrote: > Hi, > > On 06/14/15 20:40, Mihai Vintila wrote: >> Another strange issue. Is that i have: >> /boot/loader.conf >> mlx4_core="YES" >> ibcore_load="YES" >> mlxen_load="YES" >> mlx4ib_load="YES" >> ipoib_load="YES" > > If you remove this line from sysctl.conf and run it from the command > line instead, does it work then? > >> >> and in sysctl.conf >> sys.device.mlx4_core0.mlx4_port1=ib >> >> Only when it boots ib0 is not functional. From what i've noticed if >> after mlx4_core0 is up i remove ipoib module and add it again it's >> working. >> >> Bellow is a small output from dmesg in which i unload/load ipoib at boot >> in rc.local and interface still doesn't work and after port is up i do >> it manually and it starts working: >> ib0: Attached to mlx4_0 port 1 >> ums0: >> on usbus0 >> ums0: 3 buttons and [Z] coordinates ID=0 >> ib0: link state changed to DOWN >> ib0: Attached to mlx4_0 port 1 >> mlx4_core0: mlx4_ib: Port 1 logical link is up >> ib0: link state changed to DOWN >> ib0: Attached to mlx4_0 port 1 >> ib0: link state changed to UP >> >> >> in ifconfig only difference is a new lladdr for ib0. >> > > --HPS > From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 14 19:38:37 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 23A8BD57 for ; Sun, 14 Jun 2015 19:38:37 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D8E323F5 for ; Sun, 14 Jun 2015 19:38:36 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id AE0921FE023; Sun, 14 Jun 2015 21:38:32 +0200 (CEST) Message-ID: <557DD869.3060806@selasky.org> Date: Sun, 14 Jun 2015 21:39:21 +0200 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Mihai Vintila , freebsd-infiniband@freebsd.org Subject: Re: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> <557DD20B.7020508@gmail.com> In-Reply-To: <557DD20B.7020508@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 19:38:37 -0000 On 06/14/15 21:12, Mihai Vintila wrote: > Same behavior: > sysctl -w sys.device.mlx4_core0.mlx4_port1=ib > sys.device.mlx4_core0.mlx4_port1: eth -> ib > ifconfig ib0 > ib0: flags=8043 metric 0 mtu 2044 > options=8009b > lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.a.21.31 > inet 10.105.0.225 netmask 0xffffff00 broadcast 10.105.0.255 > nd6 options=29 > > ping 10.105.0.15 > PING 10.105.0.15 (10.105.0.15): 56 data bytes > ping: sendto: No route to host > ping: sendto: No route to host > > Might worth to mention that this host also holds opensm, and i actually > need to run opensm first next unload/load the module. > But i rather expect that the ib0 state changes along mlx4_core0 state. > If you do: ifconfig ib0 down ifconfig ib0 up Does it work then? --HPS From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 17 16:00:21 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D09EABDA for ; Wed, 17 Jun 2015 16:00:21 +0000 (UTC) (envelope-from unixro@gmail.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F57FF53 for ; Wed, 17 Jun 2015 16:00:21 +0000 (UTC) (envelope-from unixro@gmail.com) Received: by wifx6 with SMTP id x6so58520087wif.0 for ; Wed, 17 Jun 2015 09:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=PbRGy0G6txGlZSPyuLdkioadQ4KiMAKF0eWhbfLjJXs=; b=yjOmb+I5vZNgZNy61PpdfKSwyElg+LT6r4kd4/AH8DYgRJUJI4stWlqI/64P80ur6i U6mHCjI/Dx/OyjywexzBVFXCxE5DZRluuvyW48QlxBR+Q1G20xVpW6lvMNTvDD2ci/go SWxrVvl7j/n4Y7ewdjrQmu6q1T2Iq+JGvzioK75NAQbzCu/PSJn96A8XFFd0rlGvInnh zzOf7b2Pi744ChDoiw+0VVfP1mIST+JOrfwszFekAOyy8OUOB5/vGdTTEdzNJ9EeW9qL LouoAGEFwrVbDy0fwf2rNDbm2d3mXQ27LhYjpfOPwOAOsBKylk0Re+4FSAcNFMEKi4gp XrJw== X-Received: by 10.180.188.48 with SMTP id fx16mr19064686wic.35.1434556820005; Wed, 17 Jun 2015 09:00:20 -0700 (PDT) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id ei8sm7509988wjd.32.2015.06.17.09.00.18 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Jun 2015 09:00:19 -0700 (PDT) Message-ID: <55819992.2070207@gmail.com> Date: Wed, 17 Jun 2015 19:00:18 +0300 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Hans Petter Selasky , freebsd-infiniband@freebsd.org Subject: Re: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> <557DD20B.7020508@gmail.com> <557DD869.3060806@selasky.org> In-Reply-To: <557DD869.3060806@selasky.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 16:00:21 -0000 Do you guys know how to catch mlx4_core0: mlx4_ib: Port 1 logical link is up with devd ? Seems that its enough to run ifconfig ib0 down && ifconfig ib0 up to fix things, and devd should be ideal for this. Best regards, Vintila Mihai Alexandru On 6/14/2015 10:39 PM, Hans Petter Selasky wrote: > On 06/14/15 21:12, Mihai Vintila wrote: >> Same behavior: >> sysctl -w sys.device.mlx4_core0.mlx4_port1=ib >> sys.device.mlx4_core0.mlx4_port1: eth -> ib >> ifconfig ib0 >> ib0: flags=8043 metric 0 mtu 2044 >> options=8009b >> >> lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.a.21.31 >> inet 10.105.0.225 netmask 0xffffff00 broadcast 10.105.0.255 >> nd6 options=29 >> >> ping 10.105.0.15 >> PING 10.105.0.15 (10.105.0.15): 56 data bytes >> ping: sendto: No route to host >> ping: sendto: No route to host >> >> Might worth to mention that this host also holds opensm, and i actually >> need to run opensm first next unload/load the module. >> But i rather expect that the ib0 state changes along mlx4_core0 state. >> > > If you do: > > ifconfig ib0 down > ifconfig ib0 up > > Does it work then? > > --HPS > From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 17 17:38:12 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7464CAF6 for ; Wed, 17 Jun 2015 17:38:12 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 38074E23 for ; Wed, 17 Jun 2015 17:38:12 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 6CB861FE023; Wed, 17 Jun 2015 19:38:09 +0200 (CEST) Message-ID: <5581B0B7.5040406@selasky.org> Date: Wed, 17 Jun 2015 19:39:03 +0200 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Mihai Vintila , freebsd-infiniband@freebsd.org Subject: Re: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> <557DD20B.7020508@gmail.com> <557DD869.3060806@selasky.org> <55819992.2070207@gmail.com> In-Reply-To: <55819992.2070207@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 17:38:12 -0000 On 06/17/15 18:00, Mihai Vintila wrote: > Do you guys know how to catch mlx4_core0: mlx4_ib: Port 1 logical link > is up with devd ? > Seems that its enough to run ifconfig ib0 down && ifconfig ib0 up to fix > things, and devd should be ideal for this. > > Best regards, > Vintila Mihai Alexandru > Hi, man devd Run in foreground with debugging on and you'll see all events printed. If no events are printed we might need to add some. --HPS From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 17 17:56:32 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2EF24100 for ; Wed, 17 Jun 2015 17:56:32 +0000 (UTC) (envelope-from unixro@gmail.com) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8B9E3E1 for ; Wed, 17 Jun 2015 17:56:31 +0000 (UTC) (envelope-from unixro@gmail.com) Received: by wiga1 with SMTP id a1so147489670wig.0 for ; Wed, 17 Jun 2015 10:56:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UCz49UvzvuzKPlzdRT7kJbdnoqquONflD8LSS8ZYP6w=; b=xojNT920reAWcq4JM9f30WFQSnRJYLsh8t2BK4axsm3y+lrTTURcTtzDJq1bS13ml7 Lvj8olz4ugzg0Wvy9QU8cw0yWoXu2/rh5I8OK1iot/2Uc6HpEP3milRp6yYdEpX0p8Kv CiMWnRbrEU3PrcLFObtJi/wf5iCZ8gV4uUd+quLT3Ahmip03y79Ls0Jk8zEOzxHH+EIS rdajeXcJ6PgRl8/PAU7RVTPQ/iaA6t5vDPaUyq7NX0LMuMYCzblJlI7iJGKnp1Dc3GII xEM/kT1tA0+uVdOZDORxjhc13zWqveZ8EABuwqi4BaVpovgKZ/TLjNPP2EqdIEgICVaj y6Hg== X-Received: by 10.194.58.69 with SMTP id o5mr8072662wjq.22.1434563790030; Wed, 17 Jun 2015 10:56:30 -0700 (PDT) Received: from [192.168.0.140] ([178.156.136.152]) by mx.google.com with ESMTPSA id n6sm8867525wic.16.2015.06.17.10.56.28 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 17 Jun 2015 10:56:29 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: Ib0 not working unless i unload ipoib and load it again. From: Mihai Vintila X-Mailer: iPad Mail (12F69) In-Reply-To: <5581B0B7.5040406@selasky.org> Date: Wed, 17 Jun 2015 20:56:28 +0300 Cc: "freebsd-infiniband@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <73F1AE4D-820A-467A-A443-0B3F4A2A5902@gmail.com> References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> <557DD20B.7020508@gmail.com> <557DD869.3060806@selasky.org> <55819992.2070207@gmail.com> <5581B0B7.5040406@selasky.org> To: Hans Petter Selasky X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 17:56:32 -0000 I've seen them for ib0 but not for the mlx4_core0 Best regards, Mihai Vintila 4PSA - Cloud Calling Technical Support Engineer > On 17 iun. 2015, at 20:39, Hans Petter Selasky wrote: >=20 >> On 06/17/15 18:00, Mihai Vintila wrote: >> Do you guys know how to catch mlx4_core0: mlx4_ib: Port 1 logical link >> is up with devd ? >> Seems that its enough to run ifconfig ib0 down && ifconfig ib0 up to fix >> things, and devd should be ideal for this. >>=20 >> Best regards, >> Vintila Mihai Alexandru >>=20 >=20 > Hi, >=20 > man devd >=20 > Run in foreground with debugging on and you'll see all events printed. If n= o events are printed we might need to add some. >=20 > --HPS >=20 From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 17 17:58:49 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F30B9200 for ; Wed, 17 Jun 2015 17:58:49 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B1FAF5EC for ; Wed, 17 Jun 2015 17:58:49 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 108601FE023; Wed, 17 Jun 2015 19:58:46 +0200 (CEST) Message-ID: <5581B58C.40907@selasky.org> Date: Wed, 17 Jun 2015 19:59:40 +0200 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Mihai Vintila CC: "freebsd-infiniband@freebsd.org" Subject: Re: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> <557DD20B.7020508@gmail.com> <557DD869.3060806@selasky.org> <55819992.2070207@gmail.com> <5581B0B7.5040406@selasky.org> <73F1AE4D-820A-467A-A443-0B3F4A2A5902@gmail.com> In-Reply-To: <73F1AE4D-820A-467A-A443-0B3F4A2A5902@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 17:58:50 -0000 On 06/17/15 19:56, Mihai Vintila wrote: > I've seen them for ib0 but not for the mlx4_core0 > Hi, There are currently no such events from mlx4_core0. If you need them, we can probably add. Do you have a specific use-case for this? --HPS From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 17 18:20:49 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A20B755 for ; Wed, 17 Jun 2015 18:20:49 +0000 (UTC) (envelope-from unixro@gmail.com) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B33ABCDC for ; Wed, 17 Jun 2015 18:20:48 +0000 (UTC) (envelope-from unixro@gmail.com) Received: by wiga1 with SMTP id a1so148079903wig.0 for ; Wed, 17 Jun 2015 11:20:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:mime-version:subject :message-id:date:references:in-reply-to:to; bh=DXoz4PDw97sQUx9KXGzDDQksVYT+0aIvbcMyGf611AI=; b=L5wXBggPFSNPA1pdJzlyqKGp/XhhgqEaSvAeUPHDqWaGqPVZFH+lvptuyUoFEdUY9G qT394gKtKTXByqgurSHSPluXAg0V++f2fz6BiDgGtpbsXm3zb7pIBEqeQRpIgry/2/DJ BRNqiHO4bqt0uXcCMLIgd6/WVVAo3NU01Hc8z8pafqOxlsyJNUQaanTo3cK+SqYVFYAA RfHyg6PM/Fu8qkcRdCcWXBzpMeOz/a5ROdodEpBRBKDi5fg7s4hZqoBKFlrwXDCp8hjY RoTB3Hxwf7Tcp602rdzCOWaVoU+ByNuMPHXKepxN3ulV0JNbwqdiHfgIJmWDXfCW89Tf Nh1A== X-Received: by 10.180.105.38 with SMTP id gj6mr19937275wib.90.1434565247320; Wed, 17 Jun 2015 11:20:47 -0700 (PDT) Received: from [192.168.0.140] ([178.156.136.152]) by mx.google.com with ESMTPSA id di7sm8969149wib.23.2015.06.17.11.20.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 17 Jun 2015 11:20:46 -0700 (PDT) From: Mihai Vintila Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: Ib0 not working unless i unload ipoib and load it again. Message-Id: <4F0BE9EF-44A1-44FD-94A9-2BFEAE3EE3AD@gmail.com> Date: Wed, 17 Jun 2015 21:20:45 +0300 References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> <557DCA80.50203@gmail.com> <557DCEA7.3030703@selasky.org> <557DD20B.7020508@gmail.com> <557DD869.3060806@selasky.org> <55819992.2070207@gmail.com> <5581B0B7.5040406@selasky.org> <73F1AE4D-820A-467A-A443-0B3F4A2A5902@gmail.com> <5581B58C.40907@selasky.org> In-Reply-To: <5581B58C.40907@selasky.org> To: freebsd-infiniband@freebsd.org X-Mailer: iPad Mail (12F69) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 18:20:49 -0000 My only issue is that on boot if ipoib is loaded before mlx4_core0 is up the= n i'll need to do ifconfig ib0 down&&ifconfig ib0 up Best regards, Mihai Vintila 4PSA - Cloud Calling Technical Support Engineer > On 17 iun. 2015, at 20:59, Hans Petter Selasky wrote: >=20 >> On 06/17/15 19:56, Mihai Vintila wrote: >> I've seen them for ib0 but not for the mlx4_core0 >>=20 >=20 > Hi, >=20 > There are currently no such events from mlx4_core0. >=20 > If you need them, we can probably add. Do you have a specific use-case for= this? >=20 > --HPS >=20