From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 14 18:40:05 2015 Return-Path: Delivered-To: freebsd-infiniband@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 189CA724 for ; Sun, 14 Jun 2015 18:40:05 +0000 (UTC) (envelope-from unixro@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3318625 for ; Sun, 14 Jun 2015 18:40:04 +0000 (UTC) (envelope-from unixro@gmail.com) Received: by wgbhy7 with SMTP id hy7so20568858wgb.2 for ; Sun, 14 Jun 2015 11:40:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=6ToLrzYVir8CoX8x4aDEknZQRUWNOFq3x71HqOrOuV8=; b=UbOg5N+U1Z7NUs4IoI9KL4JefUrskHfMEnwfBMHbQz48ifjzE0qc3xehLAkAwn1eZn j75wHheFlUj+e1MTPNEyvZ0LuUwTCL0wHOExHna6Gw70oDvaOCKgPYGLfH0I96/Vs+cA M0Aj+QPhN7/VvvW4DpgSTg2xpVbxGq0p/CK74Fi6P8mXmSnffueY+yRIKS78AkN6n9Di 0QJuXXjFtmKPh6zIxBwR2Ttc3UqcHQ4kE9Z7QvWJ9zxX589+FJDEs/1s6B/ypg1uGPVO v4Nn0liX6Ej1nCd0qhuUB5bkzatqZocQil/fDxzghZOwgW/pG8FUClP1TjJ53IxRKi9k jbnQ== X-Received: by 10.194.236.68 with SMTP id us4mr40004324wjc.69.1434307202131; Sun, 14 Jun 2015 11:40:02 -0700 (PDT) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id ma15sm12379888wic.20.2015.06.14.11.40.01 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Jun 2015 11:40:01 -0700 (PDT) Message-ID: <557DCA80.50203@gmail.com> Date: Sun, 14 Jun 2015 21:40:00 +0300 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Ib0 not working unless i unload ipoib and load it again. References: <557C79D8.8040403@gmail.com> <557C7D94.9020207@selasky.org> In-Reply-To: <557C7D94.9020207@selasky.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 18:40:05 -0000 Another strange issue. Is that i have: /boot/loader.conf mlx4_core="YES" ibcore_load="YES" mlxen_load="YES" mlx4ib_load="YES" ipoib_load="YES" and in sysctl.conf sys.device.mlx4_core0.mlx4_port1=ib Only when it boots ib0 is not functional. From what i've noticed if after mlx4_core0 is up i remove ipoib module and add it again it's working. Bellow is a small output from dmesg in which i unload/load ipoib at boot in rc.local and interface still doesn't work and after port is up i do it manually and it starts working: ib0: Attached to mlx4_0 port 1 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 ib0: link state changed to DOWN ib0: Attached to mlx4_0 port 1 mlx4_core0: mlx4_ib: Port 1 logical link is up ib0: link state changed to DOWN ib0: Attached to mlx4_0 port 1 ib0: link state changed to UP in ifconfig only difference is a new lladdr for ib0. Best regards, Vintila Mihai Alexandru On 6/13/2015 9:59 PM, Hans Petter Selasky wrote: > On 06/13/15 20:43, Mihai Vintila wrote: >> Hi, >> I've posted in the stable mailing list but i guess this is where i >> should ask. >> Can you tell me what is the proper way to get subnet manager working on >> FreeBSD 10.1 ? >> I've used RELEASE branch to compile opensm and drivers from Mellanox but >> i get: >> ibportstate -G 0xe61d2dfffe0a2130 1 enable >> ibpanic: [868] madrpc_init: can't open UMAD port ((null):0): (m) >> >> opensm >> ------------------------------------------------- >> OpenSM 3.3.1 >> Reading Cached Option File: /etc/opensm/opensm.conf >> Command Line Arguments: >> Log File: /var/log/opensm.log >> ------------------------------------------------- >> OpenSM 3.3.1 >> >> Entering DISCOVERING state >> >> Using default GUID 0x0 >> >> Error from osm_opensm_bind (0x2A) >> Perhaps another instance of OpenSM is already running >> Exiting SM >> >> >> >> kldstat >> Id Refs Address Size Name >> 1 27 0xffffffff80200000 1756638 kernel >> 2 1 0xffffffff81957000 2681e8 zfs.ko >> 3 2 0xffffffff81bc0000 6780 opensolaris.ko >> 4 2 0xffffffff81bc7000 17d60 nvme.ko >> 5 1 0xffffffff81bdf000 4210 nvd.ko >> 6 1 0xffffffff81e11000 357f ums.ko >> 7 3 0xffffffff81e15000 37656 mlx4.ko >> 8 2 0xffffffff81e4d000 4e348 ibcore.ko >> 9 1 0xffffffff81e9c000 2f72 toecore.ko >> 10 1 0xffffffff81e9f000 238fa mlx4ib.ko >> 11 1 0xffffffff81ec3000 12482 mlxen.ko >> >> Driver that i'm using: >> mlx4_core0: mem 0xc7400000-0xc74fffff,0xc5000000-0xc57fffff >> irq 32 at device 0.0 on pci2 >> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 >> (Jun 12 2015) >> >> mlx4_core0: mlx4_ib_add: allocated counter index 1 for port 1 >> mlx4_en mlx4_core0: Activating port:1 >> mlxen0: Ethernet address: e4:1d:2d:0a:21:30 >> mlx4_en: mlx4_core0: Port 1: Using 12 TX rings >> mlxen0: link state changed to DOWN >> mlx4_en: mlx4_core0: Port 1: Using 8 RX rings >> mlx4_en: mlxen0: Using 12 TX rings >> mlx4_en: mlxen0: Using 8 RX rings >> mlx4_en: mlxen0: Initializing port >> > > Hi, > > Did you check your ports are configured for "ib" and not "eth" ? > > For example: > sysctl sys.device.mlx4_core0.mlx4_port1=ib > > --HPS >