From owner-freebsd-infiniband@freebsd.org Mon Dec 5 01:09:04 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAF56C677E0 for ; Mon, 5 Dec 2016 01:09:04 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp15-phx-sp1.mta.salesforce.com (smtp15-phx-sp1.mta.salesforce.com [136.147.46.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9719F14A0 for ; Mon, 5 Dec 2016 01:09:04 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.29] ([10.246.216.29:50048] helo=na3-app2-9-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id DA/A3-07606-DFCB4485; Mon, 05 Dec 2016 01:03:57 +0000 Date: Mon, 5 Dec 2016 01:03:57 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 0055000000366Fh X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2016 01:09:04 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Mon Dec 5 17:08:13 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3235C68424 for ; Mon, 5 Dec 2016 17:08:13 +0000 (UTC) (envelope-from ferrao@vmio.com.br) Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0115.outbound.protection.outlook.com [104.47.34.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2244DB9E for ; Mon, 5 Dec 2016 17:08:12 +0000 (UTC) (envelope-from ferrao@vmio.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmio.onmicrosoft.com; s=selector1-vmio-com-br; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=X3aUPFBXWyka4/cHkGorhmjPxK1Yk8GWaAn8wTY5w7Y=; b=0gRUU6McJvsvXMIq9YdeuKdGJQ2HwJ3Tg9kT+H7AaGZ4RfMb5gyUe/S2NsRZz/v7N/GhxrOrrvPtscMn62+nXInCO7WOIzobFNYUw6HcizQFbctmFOTJwMI26/ERMUhjOuEkUACnEHg+Bp9bWNMBTAKfhjTle941weFRSzv849Q= Received: from CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) by CP1PR80MB295.lamprd80.prod.outlook.com (10.242.254.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.761.9; Mon, 5 Dec 2016 17:08:02 +0000 Received: from CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) by CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) with mapi id 15.01.0761.015; Mon, 5 Dec 2016 17:08:02 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= To: "freebsd-infiniband@freebsd.org" Subject: How to change between Connected Mode and Datagram Mode with IPoIB? Thread-Topic: How to change between Connected Mode and Datagram Mode with IPoIB? Thread-Index: AQHSTxoiGDaBdPtGRUSXCZjnzNbcyg== Date: Mon, 5 Dec 2016 17:08:00 +0000 Message-ID: <0EC3E7A7-497B-4961-BB14-DA5B549E6750@vmio.com.br> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ferrao@vmio.com.br; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [200.204.163.103] x-ms-office365-filtering-correlation-id: c5825f33-78ff-4c61-4c45-08d41d3145c8 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CP1PR80MB295; x-microsoft-exchange-diagnostics: 1; CP1PR80MB295; 7:svPZxn8EmTLRNghNJUoRBonZhOcAoPQZvowqsYtr3c5IgtsgHS3IwD3ojSqNJIZIkGcZjakELra+96db/yE83MJvrr/bSKAzfVk4MtTHaawT0ky/58IUw3+NP+1/tQ+7XHGy4PpZ2VWCj377TzQGiqktqoRYl8oZCVvJU6qg6pWC9JHwBc8Qo++wcDNkJzJXwbw+b5vYSBCAvzE/0c1ZE4o5V+m5U7Y9isCFcivl6Guo+LtBa7r4B5JCe+dAf5pnIPZLvMGM2h1TfnM7mqsfZK06S9iFs6bNlcXh8b2EpLm+oEWc/5wmXFHB269yFsJ2PdUvUYpEgvwqAnOHmr4bcihMwRja0kqyQENNbuxeyf8utwfLHpTagqWhkpbSW5bduEo/PpWnIr4qgM5vpZbC8YVaS1kOQKGAOjlHaTa9kOLhsdYxxHCacv0DjZQ3hl9fBUivxipRpVzShaZquS4/Xg== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6041248)(20161123555025)(20161123562025)(20161123564025)(20161123560025)(2016111802025)(6043046)(6072148); SRVR:CP1PR80MB295; BCL:0; PCL:0; RULEID:; SRVR:CP1PR80MB295; x-forefront-prvs: 0147E151B5 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(199003)(51874003)(189002)(101416001)(97736004)(3280700002)(2906002)(81156014)(2351001)(122556002)(450100001)(106116001)(82746002)(8676002)(105586002)(107886002)(110136003)(2900100001)(39450400002)(102836003)(6506006)(305945005)(39830400001)(106356001)(74482002)(5640700002)(6486002)(85202003)(81166006)(83716003)(3660700001)(2501003)(85182001)(86362001)(36756003)(6512006)(77096006)(6116002)(6916009)(5660300001)(189998001)(42882006)(39410400001)(8936002)(54356999)(3846002)(50986999)(92566002)(7736002)(7846002)(66066001)(33656002)(68736007)(38730400001)(104396002); DIR:OUT; SFP:1102; SCL:1; SRVR:CP1PR80MB295; H:CP1PR80MB296.lamprd80.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:0; MX:1; LANG:en; received-spf: None (protection.outlook.com: vmio.com.br does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <74301913410F39449278BE24D4F931AD@lamprd80.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: vmio.com.br X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Dec 2016 17:08:00.8085 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3da02919-4b9d-42af-a7f7-a5975700d7c4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CP1PR80MB295 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2016 17:08:13 -0000 SGVsbG8gZ3V5cywNCg0KQ29udGludWluZyBteSBqb3VybmV5IG9uIEluZmluaWJhbmQgb24gRnJl ZUJTRCBJ4oCZdmUgZGlzY292ZXJlZCB0aGF0IHNvbWUgYnVncyBhbmQgcGVyZm9ybWFuY2UgaXNz dWVzIGFwcGVhcnMgdG8gYmUgYSBtaXNtYXRjaCBvbiBDTSBhbmQgVUQgbW9kZXMgb2YgSVBvSUIu DQoNCkkgd2FzIGFibGUgdG8gcmFpc2UgdGhlIHBlcmZvcm1hbmNlIG9mIE5GUyB0byA3NTBNQi9z IGNoYW5naW5nIHRoZSBMaW51eCBjb25uZWN0aW9uIG1vZGUgdG8gQ00uIEFmdGVyIHRoYXQgdGhl IE1UVSBjaGFuZ2VkIGZyb20gMjA0NCB0byA2NTUyMCBhbmQgSSBzYXcgc29tZSBnb29kIHZhbHVl cy4gQnV0IGFub3RoZXIgYW5ub3lhbmNlcyBhcHBlYXJlZCBvbiBMaW51eCBzaWRlLiBBIHNsdWdn aXNoIHBlcmZvcm1hbmNlIHdoZW4gcnVubmluZyBzaW1wbGUgY29tbWFuZHMgbGlrZSDigJxsc+KA nSBvbiBORlMgb3ZlciBJQiBtb3VudHMgcmVuZGVyZWQgdGhlIHNvbHV0aW9uIGludmlhYmxlLg0K DQpTbyBJ4oCZbSBjb25zaWRlcmluZyBydW5uaW5nIEZyZWVCU0QgaW4gVUQgKERhdGFncmFtKSBt b2RlIHRvIHNlZSBpZiBJIGZpbmFsbHkgY2FuIGdldCByaWQgb2YgdGhlIHByb2JsZW1zLiBCdXQs IGhvdyBjYW4gSSBjaGFuZ2UgYmV0d2VlbiBDTSBhbmQgVUQgaW4gRnJlZUJTRD8gV2UgZG9uJ3Qg aGF2ZSBhIC9zeXMgZm9sZGVyIG9uIEZyZWVCU0Qgc28gSSBjYW50IGp1c3Qgc2V0IGZyb20gMCB0 byAxIGFuZCBjaGFuZ2UgbW9kZXMuDQoNClNvbWUgb2xkIGZvcnVtIHBvc3RzIG1lbnRpb25zIGEg cmVjb21waWxlIG5lZWRlZCwgZGlzYWJsaW5nICJvcHRpb25zIElQT0lCX0NN4oCdIG9uIGtlcm5l bCBjb25maWd1cmF0aW9uIGp1c3QgdG8gY2hhbmdlIG1vZGVzLCB3aGljaCBpcyBwcmV0dHkgaW52 aWFibGUsIGFuZCBpdCBkb2VzbuKAmXQgc2VlbXMgcmlnaHQuDQoNClNvIGhvdyBjYW4gd2UgYWNo aWV2ZSBjaGFuZ2luZyBiZXR3ZWVuIENNIGFuZCBVRCBtb2Rlcz8NCg0KQW5kIG9uZSBsYXN0IHF1 ZXN0aW9uOiBob3cgaXQgd2FzIGV2ZW4gd29ya2luZyAoYmFkbHkpIGJ1dCB3b3JraW5nIHdoZW4g YSBzaWRlIGlzIHRhbGtpbmcgaW4gQ00gYW5kIHRoZSBvdGhlciBzaWRlIG9uIFVEIG1vZGU/DQoN ClRoYW5rcyBpbiBhZHZhbmNlLA0KVi4= From owner-freebsd-infiniband@freebsd.org Mon Dec 5 20:29:40 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B043C68B0E for ; Mon, 5 Dec 2016 20:29:40 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp15-phx-sp1.mta.salesforce.com (smtp15-phx-sp1.mta.salesforce.com [136.147.46.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE53E399 for ; Mon, 5 Dec 2016 20:29:39 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.27] ([10.246.216.27:47072] helo=na3-app2-10-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id D0/5B-07606-23EC5485; Mon, 05 Dec 2016 20:29:38 +0000 Date: Mon, 5 Dec 2016 20:29:38 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: <_8t42000000000000000000000000000000000000000000000OHQBLE00sXieRcZxT7u8vKjKLgpMqg@sfdc.net> Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 00550000003xo6y X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2016 20:29:40 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Mon Dec 5 22:53:15 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8679C68161 for ; Mon, 5 Dec 2016 22:53:15 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp07-phx-sp1.mta.salesforce.com (smtp07-phx-sp1.mta.salesforce.com [136.147.46.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AEC706DF for ; Mon, 5 Dec 2016 22:53:15 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.5] ([10.246.216.5:47112] helo=na3-app1-5-phx.ops.sfdc.net) by mx4-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 08/E9-07904-4DFE5485; Mon, 05 Dec 2016 22:53:08 +0000 Date: Mon, 5 Dec 2016 22:53:08 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2016 22:53:15 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Mon Dec 5 23:02:38 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B40CC684C2 for ; Mon, 5 Dec 2016 23:02:38 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp07-phx-sp1.mta.salesforce.com (smtp07-phx-sp1.mta.salesforce.com [136.147.46.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B8DBCB8 for ; Mon, 5 Dec 2016 23:02:37 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.29] ([10.246.216.29:33300] helo=na3-app2-9-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id A4/F9-07606-C02F5485; Mon, 05 Dec 2016 23:02:36 +0000 Date: Mon, 5 Dec 2016 23:02:36 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2016 23:02:38 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Wed Dec 7 21:05:24 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AB3DC6B92D for ; Wed, 7 Dec 2016 21:05:24 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp15-phx-sp1.mta.salesforce.com (smtp15-phx-sp1.mta.salesforce.com [136.147.46.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 310C21F08 for ; Wed, 7 Dec 2016 21:05:23 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.29] ([10.246.216.29:51234] helo=na3-app2-9-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 8C/41-07606-D8978485; Wed, 07 Dec 2016 21:05:17 +0000 Date: Wed, 7 Dec 2016 21:05:17 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2016 21:05:24 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Thu Dec 8 00:33:18 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DD1AC6CE82 for ; Thu, 8 Dec 2016 00:33:18 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp15-phx-sp1.mta.salesforce.com (smtp15-phx-sp1.mta.salesforce.com [136.147.46.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 23FC41217 for ; Thu, 8 Dec 2016 00:33:17 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.46] ([10.246.216.46:43152] helo=na3-app1-1-phx.ops.sfdc.net) by mx4-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id A6/E5-07904-C4AA8485; Thu, 08 Dec 2016 00:33:16 +0000 Date: Thu, 8 Dec 2016 00:33:16 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000013ipV X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2016 00:33:18 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Thu Dec 8 01:08:04 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC5D0C6B5B4 for ; Thu, 8 Dec 2016 01:08:04 +0000 (UTC) (envelope-from ferrao@vmio.com.br) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0134.outbound.protection.outlook.com [104.47.32.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 50A98118 for ; Thu, 8 Dec 2016 01:08:02 +0000 (UTC) (envelope-from ferrao@vmio.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmio.onmicrosoft.com; s=selector1-vmio-com-br; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=VA0EZDJSfVKQuuGsQQ7/uzVejZiWSzHAB0sTxmS0k7g=; b=C9aFaqw8ZUMwwA2mAOHxucJR+ibuCR4DFOC7hhkdjmk8W+6tXyaT79QT8L/2LXu2c8oMhIMm4vMahddszjE31i4EZ4jmAgmaV7EFjV0NNbf81eV2dkh2mCpEfsYicllUR2zgy0RFEFg0S0S+JQSuEM6Y2x+nTbuR2pGC0Dwalcg= Received: from CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) by CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.761.9; Thu, 8 Dec 2016 00:33:00 +0000 Received: from CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) by CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) with mapi id 15.01.0761.015; Thu, 8 Dec 2016 00:33:00 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= To: Mellanox Support Admin CC: "freebsd-infiniband@freebsd.org" Subject: Re: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] Thread-Topic: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] Thread-Index: AQHSUM2q1GiDZmvpeEmR8QlHMhs/T6D9M6QA Date: Thu, 8 Dec 2016 00:33:00 +0000 Message-ID: <7CB7B171-F685-4E00-8B9F-21CDC204F874@vmio.com.br> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ferrao@vmio.com.br; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [200.204.163.103] x-ms-office365-filtering-correlation-id: 36417f45-ddec-44c3-a7b1-08d41f01c3d8 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CP1PR80MB296; x-microsoft-exchange-diagnostics: 1; CP1PR80MB296; 7:YjMhP+fyCveAy3kqKZnwtgwTKY4t7a3Ofasq7iJsQFrB5wJfQbUALA5D8R8HQC4kVu1KMUCgNAfCcl+c1jrE9LnXa0x6T3L5V1xRGWyKwQnv2+mX0PtvAuVn/8eQiTa0OJA3HsvKaVM0Y3jF81HfKhDPyxSLf47bedWpTsWqdc9OfAySBfI/D9VOsAn2ULSyJjNwwMZwVgQ6L+Vh3R5pC4ldVORIMt0kuHBZPDaLTIPzySIn7aY9cVgoMOXlEKboWqtts45aCEiKLv4mj26ovHYT+64Z1Kteh4AlbxYHGzx96i22U9/sa4bScnjRQYX23lakcuTyVuQWXiDkp06dWMdMRtk454uQv1o6qqYfLqvK2VNnxsDnQy86Ae1rmhUW5oZwth8asr0n2d4d2kyx9y1QuDK9HTygbhsKv/3d2diFz3AcvlWQmmDjW0uo/1xwquTHDUfrWiUrXWgAi3+xOw== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(209352067349851)(241641549752487)(171992500451332)(122605397693432)(75325880899374)(21532816269658); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6041248)(2016111802025)(20161123555025)(20161123560025)(20161123562025)(20161123564025)(6043046)(6072148); SRVR:CP1PR80MB296; BCL:0; PCL:0; RULEID:; SRVR:CP1PR80MB296; x-forefront-prvs: 0150F3F97D x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(39450400003)(39830400002)(39410400002)(84964002)(50084003)(24454002)(189002)(199003)(51874003)(45074003)(377424004)(106116001)(189998001)(97736004)(7846002)(36756003)(76176999)(50986999)(7110500001)(54356999)(10710500007)(33656002)(5660300001)(3846002)(10126002)(6486002)(7736002)(2906002)(77096006)(4326007)(81156014)(2950100002)(68736007)(15650500001)(6916009)(345774005)(42882006)(2900100001)(3280700002)(6116002)(101416001)(305945005)(102836003)(16799955002)(81166006)(8676002)(19273905006)(85202003)(229853002)(106356001)(2420400007)(8936002)(122556002)(74482002)(6506006)(85182001)(92566002)(82746002)(6512006)(110136003)(38730400001)(86362001)(105586002)(83716003)(15188155005)(66066001)(3660700001)(104396002)(562404015)(563064011); DIR:OUT; SFP:1102; SCL:1; SRVR:CP1PR80MB296; H:CP1PR80MB296.lamprd80.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:0; LANG:en; received-spf: None (protection.outlook.com: vmio.com.br does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <7BD397A79097A44A8C590A48BD47A0D7@lamprd80.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: vmio.com.br X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Dec 2016 00:33:00.3353 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3da02919-4b9d-42af-a7f7-a5975700d7c4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CP1PR80MB296 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2016 01:08:05 -0000 SGVsbG8gU29waGllLA0KDQpSZWNvbXBpbGluZyB0aGUgc29mdHdhcmUgaXNu4oCZdCBhbiB2aWFi bGUgb3B0aW9uLg0KDQpJIHdhcyBhYmxlIHRvIG1pdGlnYXRlIHRoZSBpc3N1ZSBvbiB0aGlzIHNw ZWNpZmljIGNhc2UgcmFpc2luZyB0aGUgTVRVIHRvIDY1NTIwIG9uIGFsbCBtYWNoaW5lcy4gSXQg dHVybnMgb3V0IHRoYXQgdGhlIG9sZCBtYWNoaW5lcyB3ZXJlIGNvbmZpZ3VyZWQgb24gQ29ubmVj dGVkIE1vZGUgdXNpbmcgb25seSAyMDQ0IE1UVSBhbmQgYW4gTVRVIG1pc21hdGNoIGJldHdlZW4g dGhlIHNlcnZlciBhbmQgdGhlIGNsaWVudHMgd2FzIHRoZSByZWFsIGlzc3VlLg0KDQpCdXQgSeKA mW0gd29ycmllZCBhYm91dCB0aGUgdmlhYmlsaXR5IG9mIEluZmluaWJhbmQgb24gRnJlZUJTRC4g Q2hhbmdpbmcgZnJvbSBDb25uZWN0ZWQgTW9kZSB0byBVbnJlbGlhYmxlIERhdGFncmFtIHNob3Vs ZCBiZSBlYXNpZXIgdGhhbiByZWNvbXBpbGluZyB0aGUga2VybmVsIG1vZHVsZS4NCg0KVGhlcmXi gJlzIGEgd2F5IHRvIGFzayBNZWxsYW5veCB0byBjaGFuZ2UgdGhlIGRyaXZlcnMgdG8gbWFrZSB0 aGUgc3dpdGNoIGZyb20gVUQgdG8gQ00gdmlhYmxlIG9uIEZyZWVCU0Q/IEEgZmVhdHVyZSByZXF1 ZXN0IHBlcmhhcHM/DQoNClRoYW5rcyBpbiBhZHZhbmNlLA0KVi4NCg0KPiBPbiA3IERlYyAyMDE2 LCBhdCAxOTowNSwgTWVsbGFub3ggU3VwcG9ydCBBZG1pbiA8c3VwcG9ydGFkbWluQG1lbGxhbm94 LmNvbT4gd3JvdGU6DQo+IA0KPiBEZWFyIFZpbsOtY2l1cyBGZXJyw6NvLCANCj4gDQo+IFlvdXIg TWVsbGFub3ggc3VwcG9ydCBjYXNlICMgMDAzMjAwNTgvICAgaGFzIGJlZW4gdXBkYXRlZCBCeSAs IHdpdGggdGhlIGZvbGxvd2luZyBjb21tZW50OiANCj4gLy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gSGkgVmluaWNpdXMsDQo+ IA0KPiBIYXZlIHlvdSBzZWVuIG15IGxhc3Qgc3RhdGVtZW50IHBvc3RlZCB0byB0aGlzIGNhc2Ug YW5kIHdoYXQgaXMgdGhlIGN1cnJlbnQgc3RhdHVzPw0KPiBQbGVhc2UgbGV0IG1lIGtub3cuDQo+ IA0KPiBUaGFuayB5b3UsDQo+IFNvcGhpZS4NCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS8NCj4gDQo+IFRoYW5rIFlvdSwgDQo+ IA0KPiANCj4gDQo+IENsaWNrIG9uIHRoZSBsaW5rIHRvIGFjY2VzcyB0aGUgY2FzZTogDQo+IGh0 dHBzOi8vbXltZWxsYW5veC5mb3JjZS5jb20vc3VwcG9ydC81MDA1MDAwMDAwcTlOcGcNCj4gDQo+ IERvbuKAmXQgZm9yZ2V0IHRvIGNoZWNrIG91dCBvdXIgT25saW5lIEFjYWRlbXksIHJlZ2lzdGVy IG5vdyB0byBsZWFybiBhYm91dCB0aGUgcHJvZHVjdHMgYW5kIHRlY2hub2xvZ3ksIGh0dHA6Ly9h Y2FkZW15Lm1lbGxhbm94LmNvbS9sb2dpbi9pbmRleC5waHAgDQo+IA0KPiAvLS0tLS0tIFlvdXIg Q2FzZSBDb21tZW50cyBIaXN0b3J5IC0tLS0tLSANCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogU29waGllIE5hdWRpbiBQ cnVpdHQgMjAxNi0xMi0wNyAyMTowNToxNQ0KPiBIaSBWaW5pY2l1cywNCj4gDQo+IEhhdmUgeW91 IHNlZW4gbXkgbGFzdCBzdGF0ZW1lbnQgcG9zdGVkIHRvIHRoaXMgY2FzZSBhbmQgd2hhdCBpcyB0 aGUgY3VycmVudCBzdGF0dXM/DQo+IFBsZWFzZSBsZXQgbWUga25vdy4NCj4gDQo+IFRoYW5rIHlv dSwNCj4gU29waGllLg0KPiANCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogU29waGllIE5hdWRpbiBQcnVpdHQgMjAxNi0x Mi0wNSAyMzowMjozNQ0KPiBIaSBWaW5pY2l1cywNCj4gDQo+IFNlYXJjaGluZyBzb21lIGZvcnVt cywgSSBmb3VuZCB0aGUgZm9sbG93aW5nIGluZm9ybWF0aW9uOg0KPiANCj4gQnVpbGRpbmcgT0ZF RCBpbiBGcmVlQlNEDQo+IA0KPiBLZXJuZWwNCj4g4oCiQWRkIHRoZSBmb2xsb3dpbmcgb3B0aW9u cyB0byB5b3VyIEtFUk5DT05GIHRvIGNvbXBpbGUgdGhlIG5lY2Vzc2FyeSBwcm90b2NvbHMvcHJv dG9jb2wgb3B0aW9ucyBpbnRvIHlvdXIga2VybmVsOiANCj4gDQo+IA0KPiBvcHRpb25zICAgICAg ICBPRkVEICAgICAgICAgICAgIyBJbmZpbmliYW5kIHByb3RvY29sIHN0YWNrIGFuZCBzdXBwb3J0 DQo+IG9wdGlvbnMgICAgICAgIFNEUCAgICAgICAgICAgICAjIFNvY2tldHMgRGlyZWN0IFByb3Rv Y29sIGZvciBpbmZpbmliYW5kDQo+IG9wdGlvbnMgICAgICAgIElQT0lCX0NNICAgICAgICAjIFVz ZSBjb25uZWN0IG1vZGUgaXBvaWIgPj4+cmVjb21waWxlIHdpdGhvdXQgdGhhdCBvcHRpb24gYW5k IHRoZSBtb2RlIHNob3VsZCBiZSBkYXRhZ3JhbS4NCj4gDQo+IFJlZ2FyZHMsDQo+IFNvcGhpZS4N Cj4gDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0N Cj4gIENyZWF0ZWQgYnk6IFNvcGhpZSBOYXVkaW4gUHJ1aXR0IDIwMTYtMTItMDUgMjI6NTM6MDYN Cj4gSGkgVmluw61jaXVzLA0KPiANCj4gWW91IGFyZSBub3QgdXNpbmcgTWVsbGFub3ggT0ZFRCBE cml2ZXIgaWYgeW91IGFyZSBydW5uaW5nIE9GRUQgKE9wZW5GYWJyaWNzIEVudGVycHJpc2UgRGlz dHJpYnV0aW9uKSwgd2hhdCBpcyB0aGUgdmVyc2lvbiBhbmQgZGlkIHlvdSBjb25zdWx0IHRoZSBV TT8NCj4gV2l0aCBvdXIgZHJpdmVyLCB5b3UgY2FuIGNoYW5nZSB0aGUgbW9kZSB3aXRoOg0KPiAN Cj4gZWNobyBkYXRhZ3JhbSA+IC9zeXMvY2xhc3MvbmV0L2liMC9tb2RlIHNldHMgdGhlIG1vZGUg b2YgaWIwIHRvIFVEDQo+IGVjaG8gY29ubmVjdGVkID4gL3N5cy9jbGFzcy9uZXQvaWIwL21vZGUg c2V0cyB0aGUgbW9kZSBpYjAgdG8gQ00NCj4gDQo+IFRob3VnaCBGcmVlQlNEIGRvZXMgbm90IGhh dmUgYSAvc3lzLg0KPiANCj4gUmVnYXJkcywNCj4gU29waGllLg0KPiANCj4gLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogQW5k ZXJzb24gRGlhcyBTaWx2YSAyMDE2LTEyLTA1IDIwOjI5OjM1DQo+IEhpIEJlbm55LA0KPiANCj4g SSBhbSBBbmRlcnNvbiBEaWFzIFNpbHZhIChMQVRBTSBTRSkuIFBsZWFzZSwgc2VlIGJlbG93IGFu IGltcG9ydGFudCBjb25zaWRlcmF0aW9uIG9mIG91ciBjdXN0b21lci4NCj4gDQo+ICMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMNCj4gDQo+IEFmdGVyIHNvbWUgdGVz dGluZyB3ZSBkaXNjb3ZlcmVkIHRoYXQgd2FzIGEgbWlzbWF0Y2ggYmV0d2VlbiB0aGUgY2xpZW50 IGFuZCB0aGUgc2VydmVyIG9uIG9wZXJhdGluZyBtb2RlcyBvZiBJUG9JQi4gT25lIHNpZGUgd2Fz IHVzaW5nIENvbm5lY3RlZCBNb2RlICh0aGUgRnJlZUJTRCBvbmUpIGFuZCB0aGUgTGludXggY2xp ZW50IHdhcyBvbiBEYXRhZ3JhbSBtb2RlKS4NCj4gDQo+IFNvIEnigJltIGNvbnNpZGVyaW5nIHJ1 bm5pbmcgRnJlZUJTRCBpbiBVRCAoRGF0YWdyYW0pIG1vZGUgdG8gc2VlIGlmIEkgZmluYWxseSBj YW4gZ2V0IHJpZCBvZiB0aGUgcHJvYmxlbXMuIEJ1dCwgaG93IGNhbiBJIGNoYW5nZSBiZXR3ZWVu IENNIGFuZCBVRCBpbiBGcmVlQlNEPyBXZSBkb24ndCBoYXZlIGEgL3N5cyBmb2xkZXIgb24gRnJl ZUJTRCBzbyBJIGNhbnQganVzdCBzZXQgZnJvbSAwIHRvIDEgYW5kIGNoYW5nZSBtb2Rlcy4NCj4g DQo+ICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMNCj4gDQo+IFNv LCBob3cgY2FuIHdlIGNoYW5nZSBiZXR3ZWVuIENNIGFuZCBVRCBpbiBGcmVlQlNEPw0KPiANCj4g UmFnYXJkcyENCj4gDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0NCj4gIENyZWF0ZWQgYnk6IEJlbm55IEZpbGluIDIwMTYtMTItMDUgMDE6MDM6NTUN Cj4gSGVsbG8gVmluw61jaXVzIEZlcnLDo28sDQo+IA0KPiBUaGFuayB5b3UgZm9yIGNvbnRhY3Rp bmcgTWVsbGFub3ggVGVjaG5pY2FsIFN1cHBvcnQuDQo+IA0KPiAxKSBESWQgeW91IHRyeSBjaGFu Z2luZyBzd2l0Y2ggcG9ydHMgd2hlbiB0ZXN0aW5nIHBlcmZvcm1hbmNlIGJldHdlZW4gdHdvIEZy ZWVCU0QgbWFjaGluZXMgdGhyb3VnaCBzd2l0Y2g/DQo+IDIpIFdoYXQgaGFwcGVucyB3aGVuIHlv dSB0cnkgYSBiYWNrLXRvLWJhY2sgY29ubmVjdGlvbiBiZXR3ZWVuIHR3byBGcmVlQlNEIG1hY2hp bmVzPyBEbyB5b3UgZ2V0IGdvb2QgcmVzdWx0cz8NCj4gMykgRGlkIHlvdSB0ZXN0IHRoZSBwZXJm b3JtYW5jZSBiZXR3ZWVuIEZyZWVCU0QgYW5kIGEgTGludXggbWFjaGluZSB0aHJvdWdoIHN3aXRj aD8NCj4gDQo+IFBsZWFzZSwgZm9sbG93IHRoZSBwcm92aWRlZCBwcm9jZWR1cmUgaW4gb3JkZXIg dG8gcXVlcnkgeW91ciB1bm1hbmFnZWQgc3dpdGNoOiANCj4gDQo+IDEpIERvd25sb2FkIGFuZCBp bnN0YWxsIE1GVCAoTWVsbGFub3ggRmlybXdhcmUgVG9vbCk6DQo+IGh0dHA6Ly93d3cubWVsbGFu b3guY29tL2NvbnRlbnQvcGFnZXMucGhwP3BnPW1hbmFnZW1lbnRfdG9vbHMmbWVudV9zZWN0aW9u PTM0DQo+IFJlZmVyIHRvIHRoZSBVc2VyIE1hbnVhbCBmb3IgdGhlIGluc3RhbGxhdGlvbiBpbnN0 cnVjdGlvbnM6DQo+IGh0dHA6Ly93d3cubWVsbGFub3guY29tL3BkZi9NRlQvTUZUX3VzZXJfbWFu dWFsLnBkZg0KPiANCj4gMikgRnJvbSBvbmUgb2YgdGhlIG5vZGVzIGluIHRoZSBmYWJyaWMsIGlu dm9rZTogaWJzd2l0Y2hlcy4NCj4gTWF0Y2ggdGhlIHJlbGV2YW50IHN3aXRjaCBHVUlEIGZyb20g dGhlIE1lbGxhbm94IHN3aXRjaCBsYWJlbCB0byB0aGUgTElELg0KPiANCj4gMykgRnJvbSBvbmUg b2YgdGhlIG5vZGVzIGluIHRoZSBmYWJyaWMgLCBpbnZva2U6IG1zdCBzdGFydCAtPiBtc3QgaWIg YWRkIC0+IG1zdCBzdGF0dXMNCj4gDQo+IDQpIElkZW50aWZ5IHRoZSByZWxldmFudCB1bm1hbmFn ZWQgc3dpdGNoIG1zdCByZXByZXNlbnRhdGlvbiBiYXNlZCBvbiB0aGUgc3dpdGNoIExJRCAoZnJv bSBwcmV2aW91cyBzdGVwIDIpLg0KPiAqIE1hdGNoaW5nIHRoZSByZW1vdGUgc3dpdGNoIG1zdCBk ZXZpY2UgcmVwcmVzZW50YXRpb24gdG8gdGhlIHJlbGV2YW50IHN3aXRjaCBwcm9jZWR1cmU6DQo+ IExvY2F0ZSB0aGUgc3dpdGNoIGluIHRoZSAiaWJzd2l0Y2hlcyIgb3V0cHV0IGFuZCByZXRyaWV2 ZSB0aGUgc3dpdGNoIExJRC4gDQo+IENvbnZlcnQgdGhlIExJRCBmcm9tIERlY2ltYWwgdG8gSGV4 YWRlY2ltYWwgYmFzZS4NCj4gTG9jYXRlIHRoZSBkZXZpY2Ugd2l0aCB0aGlzIExJRCAoaW4gaGV4 IGJhc2UpIGZyb20gdGhlICJtc3Qgc3RhdHVzIiBvdXRwdXQuIA0KPiANCj4gNSkgSW52b2tlIHRo ZSBmb2xsb3dpbmcgY29tbWFuZCwgYW5kIHByb3ZpZGUgdGhlIG91dHB1dDoNCj4gZmxpbnQgLWQg PHJlbW90ZV9zd2l0Y2hfbXN0X2RldmljZT4gcQ0KPiANCj4gDQo+IFdlJ3JlIGxvb2tpbmcgZm9y d2FyZCB0byBoZWFyaW5nIGZyb20geW91Lg0KPiANCj4gQmVzdCByZWdhcmRzLA0KPiBNZWxsYW5v eCBUZWNobmljYWwgU3VwcG9ydA0KPiANCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogSnVicmFuIEp1YnJhbiAyMDE2LTEy LTAxIDE2OjQxOjM1DQo+IEhlbGxvIFZpbsOtY2l1cyBGZXJyw6NvLA0KPiANCj4gVGhhbmsgeW91 IGZvciBjb250YWN0aW5nIE1lbGxhbm94IFRlY2huaWNhbCBTdXBwb3J0Lg0KPiANCj4gMSkgRElk IHlvdSB0cnkgY2hhbmdpbmcgc3dpdGNoIHBvcnRzIHdoZW4gdGVzdGluZyBwZXJmb3JtYW5jZSBi ZXR3ZWVuIHR3byBGcmVlQlNEIG1hY2hpbmVzIHRocm91Z2ggc3dpdGNoPw0KPiAyKSBXaGF0IGhh cHBlbnMgd2hlbiB5b3UgdHJ5IGEgYmFjay10by1iYWNrIGNvbm5lY3Rpb24gYmV0d2VlbiB0d28g RnJlZUJTRCBtYWNoaW5lcz8gRG8geW91IGdldCBnb29kIHJlc3VsdHM/DQo+IDMpIERpZCB5b3Ug dGVzdCB0aGUgcGVyZm9ybWFuY2UgYmV0d2VlbiBGcmVlQlNEIGFuZCBhIExpbnV4IG1hY2hpbmUg dGhyb3VnaCBzd2l0Y2g/DQo+IA0KPiBQbGVhc2UsIGZvbGxvdyB0aGUgcHJvdmlkZWQgcHJvY2Vk dXJlIGluIG9yZGVyIHRvIHF1ZXJ5IHlvdXIgdW5tYW5hZ2VkIHN3aXRjaDogDQo+ICANCj4gMSkg RG93bmxvYWQgYW5kIGluc3RhbGwgTUZUIChNZWxsYW5veCBGaXJtd2FyZSBUb29sKToNCj4gaHR0 cDovL3d3dy5tZWxsYW5veC5jb20vY29udGVudC9wYWdlcy5waHA/cGc9bWFuYWdlbWVudF90b29s cyZtZW51X3NlY3Rpb249MzQNCj4gUmVmZXIgdG8gdGhlIFVzZXIgTWFudWFsIGZvciB0aGUgaW5z dGFsbGF0aW9uIGluc3RydWN0aW9uczoNCj4gaHR0cDovL3d3dy5tZWxsYW5veC5jb20vcGRmL01G VC9NRlRfdXNlcl9tYW51YWwucGRmDQo+ICANCj4gMikgRnJvbSBvbmUgb2YgdGhlIG5vZGVzIGlu IHRoZSBmYWJyaWMsIGludm9rZTogaWJzd2l0Y2hlcy4NCj4gTWF0Y2ggdGhlIHJlbGV2YW50IHN3 aXRjaCBHVUlEIGZyb20gdGhlIE1lbGxhbm94IHN3aXRjaCBsYWJlbCB0byB0aGUgTElELg0KPiAg DQo+IDMpIEZyb20gb25lIG9mIHRoZSBub2RlcyBpbiB0aGUgZmFicmljICwgaW52b2tlOiBtc3Qg c3RhcnQgLT4gbXN0IGliIGFkZCAtPiBtc3Qgc3RhdHVzDQo+ICANCj4gNCkgSWRlbnRpZnkgdGhl IHJlbGV2YW50IHVubWFuYWdlZCBzd2l0Y2ggbXN0IHJlcHJlc2VudGF0aW9uIGJhc2VkIG9uIHRo ZSBzd2l0Y2ggTElEIChmcm9tIHByZXZpb3VzIHN0ZXAgMikuDQo+ICogTWF0Y2hpbmcgdGhlIHJl bW90ZSBzd2l0Y2ggbXN0IGRldmljZSByZXByZXNlbnRhdGlvbiB0byB0aGUgcmVsZXZhbnQgc3dp dGNoIHByb2NlZHVyZToNCj4gTG9jYXRlIHRoZSBzd2l0Y2ggaW4gdGhlICJpYnN3aXRjaGVzIiBv dXRwdXQgYW5kIHJldHJpZXZlIHRoZSBzd2l0Y2ggTElELiANCj4gQ29udmVydCB0aGUgTElEIGZy b20gRGVjaW1hbCB0byBIZXhhZGVjaW1hbCBiYXNlLg0KPiBMb2NhdGUgdGhlIGRldmljZSB3aXRo IHRoaXMgTElEIChpbiBoZXggYmFzZSkgZnJvbSB0aGUgIm1zdCBzdGF0dXMiIG91dHB1dC4gDQo+ IA0KPiA1KSBJbnZva2UgdGhlIGZvbGxvd2luZyBjb21tYW5kLCBhbmQgcHJvdmlkZSB0aGUgb3V0 cHV0Og0KPiBmbGludCAtZCA8cmVtb3RlX3N3aXRjaF9tc3RfZGV2aWNlPiBxDQo+IA0KPiANCj4g V2UncmUgbG9va2luZyBmb3J3YXJkIHRvIGhlYXJpbmcgZnJvbSB5b3UuDQo+IA0KPiBCZXN0IHJl Z2FyZHMsDQo+IE1lbGxhbm94IFRlY2huaWNhbCBTdXBwb3J0IA0KPiAtLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLw0KPiBfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLWluZmluaWJhbmRA ZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQo+IGh0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFp bG1hbi9saXN0aW5mby9mcmVlYnNkLWluZmluaWJhbmQNCj4gVG8gdW5zdWJzY3JpYmUsIHNlbmQg YW55IG1haWwgdG8gImZyZWVic2QtaW5maW5pYmFuZC11bnN1YnNjcmliZUBmcmVlYnNkLm9yZyIN Cg0K From owner-freebsd-infiniband@freebsd.org Fri Dec 9 00:27:35 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D5F6C6D743 for ; Fri, 9 Dec 2016 00:27:35 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp06-phx-sp1.mta.salesforce.com (smtp06-phx-sp1.mta.salesforce.com [136.147.46.229]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33B2DE3B for ; Fri, 9 Dec 2016 00:27:34 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.6] ([10.246.216.6:45534] helo=na3-app1-10-phx.ops.sfdc.net) by mx4-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 9C/BB-07904-449F9485; Fri, 09 Dec 2016 00:22:28 +0000 Date: Fri, 9 Dec 2016 00:22:28 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 00:27:35 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Fri Dec 9 13:32:06 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45C07C6E488 for ; Fri, 9 Dec 2016 13:32:06 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp14-phx-sp1.mta.salesforce.com (smtp14-phx-sp1.mta.salesforce.com [136.147.46.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27373133 for ; Fri, 9 Dec 2016 13:32:05 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.26] ([10.246.216.26:35574] helo=na3-app2-8-phx.ops.sfdc.net) by mx4-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 76/F7-07904-321BA485; Fri, 09 Dec 2016 13:26:59 +0000 Date: Fri, 9 Dec 2016 13:26:59 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000013ipV X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 13:32:06 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- 2016-12-09 13:26:56 Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Fri Dec 9 19:22:57 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05948C6E6C5 for ; Fri, 9 Dec 2016 19:22:57 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp15-phx-sp1.mta.salesforce.com (smtp15-phx-sp1.mta.salesforce.com [136.147.46.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DFCAED8C for ; Fri, 9 Dec 2016 19:22:56 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.6] ([10.246.216.6:55188] helo=na3-app1-10-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 75/0E-07606-A840B485; Fri, 09 Dec 2016 19:22:50 +0000 Date: Fri, 9 Dec 2016 19:22:50 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 19:22:57 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Thank you. What is the driver in use for the RHEL environment? Mellanox OFED Driver or= OFED? What is the mode configured on the RHEL environment? Is this fabric will communicate between servers with FreeBSD & RHEL? Regards, Sophie. -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 19:22:48 Thank you. What is the driver in use for the RHEL environment? Mellanox OFED Driver or= OFED? What is the mode configured on the RHEL environment? Is this fabric will communicate between servers with FreeBSD & RHEL? Regards, Sophie. ------------------------------------------------- 2016-12-09 13:26:56 Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Fri Dec 9 22:04:23 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 284B3C6FFA3 for ; Fri, 9 Dec 2016 22:04:23 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp14-phx-sp1.mta.salesforce.com (smtp14-phx-sp1.mta.salesforce.com [136.147.46.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02D5B838 for ; Fri, 9 Dec 2016 22:04:22 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.59] ([10.246.216.59:37974] helo=na3-app1-8-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id C0/09-07606-56A2B485; Fri, 09 Dec 2016 22:04:21 +0000 Date: Fri, 9 Dec 2016 22:04:21 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000013ipV X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 22:04:23 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Comment By Customer: ferrao@vmio.com.br Sophie, I think we can safely close this issue. The main problem was with the UD mo= de of IPoIB. Since we=E2=80=99ve already changed all the machines on the cl= uster for IPoIB Connected Mode and with the properly MTU settings everythin= g worked as expected. It=E2=80=99s an annoyance being unable to change from CM to UD, but we will= stay with this since we cannot have a better solution, and moving the serv= er from FreeBSD to Linux is unpractical for us. But just for the sake of completeness everything was working correctly on t= he CentOS side using the Mellanox bundled OFED. Thanks for helping us out. V. > On 9 Dec 2016, at 17:22, Mellanox Support Admin -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- 2016-12-09 22:04:18 Comment By Customer: ferrao@vmio.com.br Sophie, I think we can safely close this issue. The main problem was with the UD mo= de of IPoIB. Since we=E2=80=99ve already changed all the machines on the cl= uster for IPoIB Connected Mode and with the properly MTU settings everythin= g worked as expected. It=E2=80=99s an annoyance being unable to change from CM to UD, but we will= stay with this since we cannot have a better solution, and moving the serv= er from FreeBSD to Linux is unpractical for us. But just for the sake of completeness everything was working correctly on t= he CentOS side using the Mellanox bundled OFED. Thanks for helping us out. V. > On 9 Dec 2016, at 17:22, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 19:22:48 Thank you. What is the driver in use for the RHEL environment? Mellanox OFED Driver or= OFED? What is the mode configured on the RHEL environment? Is this fabric will communicate between servers with FreeBSD & RHEL? Regards, Sophie. ------------------------------------------------- 2016-12-09 13:26:56 Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Fri Dec 9 22:36:33 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26712C6FB04 for ; Fri, 9 Dec 2016 22:36:33 +0000 (UTC) (envelope-from ferrao@vmio.com.br) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0095.outbound.protection.outlook.com [104.47.42.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A3026AF for ; Fri, 9 Dec 2016 22:36:32 +0000 (UTC) (envelope-from ferrao@vmio.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmio.onmicrosoft.com; s=selector1-vmio-com-br; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=E9nvwCfBmFgk8NcVDKMQNcncamcmT6vUkY783k+nFps=; b=HoMbQXYrH4ZHsSeCkF/w5vScbVcJf6ZhzJfMddtJlpdBi2dAQw2Kp0uZm/vODSaM+5XK8xpIIgZIKW79OdXQKMm2dDd7dZjjwaSJzeR1zGbYztv0DSIjEMLv6knzHdOePRCt5LCHMNOi64YGo2W5/4aXfgaLKo7ZolyqTba5f8E= Received: from CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) by CP1PR80MB295.lamprd80.prod.outlook.com (10.242.254.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.761.9; Fri, 9 Dec 2016 22:04:04 +0000 Received: from CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) by CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) with mapi id 15.01.0761.018; Fri, 9 Dec 2016 22:04:04 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= To: Mellanox Support Admin CC: "freebsd-infiniband@freebsd.org" Subject: Re: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] Thread-Topic: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] Thread-Index: AQHSUlGtQUabk5TiS0GlKQ0914T9u6EAK6qA Date: Fri, 9 Dec 2016 22:04:03 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ferrao@vmio.com.br; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [191.205.6.144] x-ms-office365-filtering-correlation-id: 7abdf46a-5bf7-4ae1-33d5-08d4207f4a2d x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CP1PR80MB295; x-microsoft-exchange-diagnostics: 1; CP1PR80MB295; 7:7BeL87rUMJWUkh7L/UM3pJ+XLdBWlXBhccJB2n2l4HNIlhbW9I6xA9rFXLbkb9KTK5tE7ZMpssjST2Ob4NETEk67RJhtC9qLQgxbdXGXrk8ntdrbi7nb8Bo3MgPgFdQp9Lmfh2LTMjzpk9bUzd0PJyvgBBHZlt1Np3112UMOLocbxTfQW5whLMHRvgVLy7rgxlxYVqQD1qH3BScbrPqQSLTxFRd39ZWwdlzpmOcRgg8pRT9OZX7dcv5cHRNaIEFSWKa5S0vEsToCHs8tgWKQ978Ng9EqhboQfCFtgAAuQR5J+aYn/7NgSImvd9wfoJ7IlXp9jhyku5eeBnkYCP0ez05nI6JSeJpxaNXziHPt5is1iS/kuNXr2PNTMb7OKpIuvJcGsv/es3Q5Q8nQyblKDJ34VU1Jk0tsIy8oYHutQJMgfaLVmIOhymF3A1R/RCihK4FPS3VlUMalvHzg0p4ucA== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(209352067349851)(241641549752487)(171992500451332)(122605397693432)(166494164430575)(75325880899374)(21532816269658); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123562025)(20161123564025)(20161123560025)(20161123555025)(2016111802025)(6072148)(6043046); SRVR:CP1PR80MB295; BCL:0; PCL:0; RULEID:; SRVR:CP1PR80MB295; x-forefront-prvs: 015114592F x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(39410400002)(39450400003)(39830400002)(199003)(84964002)(189002)(24454002)(45074003)(50084003)(51874003)(377424004)(345774005)(50986999)(101416001)(15395725005)(8936002)(66066001)(77096006)(105586002)(106116001)(38730400001)(2906002)(76176999)(74482002)(3846002)(6116002)(110136003)(102836003)(5660300001)(81156014)(229853002)(6486002)(42882006)(10126002)(6916009)(2950100002)(97736004)(54356999)(3660700001)(15188155005)(36756003)(189998001)(2420400007)(305945005)(7110500001)(7736002)(6512006)(19273905006)(15650500001)(83716003)(85182001)(3280700002)(122556002)(4326007)(10710500007)(33656002)(6506006)(16799955002)(8676002)(68736007)(82746002)(86362001)(92566002)(85202003)(106356001)(2900100001)(81166006)(6436002)(562404015)(104396002)(563064011); DIR:OUT; SFP:1102; SCL:1; SRVR:CP1PR80MB295; H:CP1PR80MB296.lamprd80.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:0; MX:1; LANG:en; received-spf: None (protection.outlook.com: vmio.com.br does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: vmio.com.br X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Dec 2016 22:04:03.8810 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3da02919-4b9d-42af-a7f7-a5975700d7c4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CP1PR80MB295 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 22:36:33 -0000 U29waGllLA0KDQpJIHRoaW5rIHdlIGNhbiBzYWZlbHkgY2xvc2UgdGhpcyBpc3N1ZS4gVGhlIG1h aW4gcHJvYmxlbSB3YXMgd2l0aCB0aGUgVUQgbW9kZSBvZiBJUG9JQi4gU2luY2Ugd2XigJl2ZSBh bHJlYWR5IGNoYW5nZWQgYWxsIHRoZSBtYWNoaW5lcyBvbiB0aGUgY2x1c3RlciBmb3IgSVBvSUIg Q29ubmVjdGVkIE1vZGUgYW5kIHdpdGggdGhlIHByb3Blcmx5IE1UVSBzZXR0aW5ncyBldmVyeXRo aW5nIHdvcmtlZCBhcyBleHBlY3RlZC4NCg0KSXTigJlzIGFuIGFubm95YW5jZSBiZWluZyB1bmFi bGUgdG8gY2hhbmdlIGZyb20gQ00gdG8gVUQsIGJ1dCB3ZSB3aWxsIHN0YXkgd2l0aCB0aGlzIHNp bmNlIHdlIGNhbm5vdCBoYXZlIGEgYmV0dGVyIHNvbHV0aW9uLCBhbmQgbW92aW5nIHRoZSBzZXJ2 ZXIgZnJvbSBGcmVlQlNEIHRvIExpbnV4IGlzIHVucHJhY3RpY2FsIGZvciB1cy4NCg0KQnV0IGp1 c3QgZm9yIHRoZSBzYWtlIG9mIGNvbXBsZXRlbmVzcyBldmVyeXRoaW5nIHdhcyB3b3JraW5nIGNv cnJlY3RseSBvbiB0aGUgQ2VudE9TIHNpZGUgdXNpbmcgdGhlIE1lbGxhbm94IGJ1bmRsZWQgT0ZF RC4NCg0KVGhhbmtzIGZvciBoZWxwaW5nIHVzIG91dC4NClYuDQoNCj4gT24gOSBEZWMgMjAxNiwg YXQgMTc6MjIsIE1lbGxhbm94IFN1cHBvcnQgQWRtaW4gPHN1cHBvcnRhZG1pbkBtZWxsYW5veC5j b20+IHdyb3RlOg0KPiANCj4gRGVhciBWaW7DrWNpdXMgRmVycsOjbywgDQo+IA0KPiBZb3VyIE1l bGxhbm94IHN1cHBvcnQgY2FzZSAjIDAwMzIwMDU4LyAgIGhhcyBiZWVuIHVwZGF0ZWQgQnkgLCB3 aXRoIHRoZSBmb2xsb3dpbmcgY29tbWVudDogDQo+IC8tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IFRoYW5rIHlvdS4NCj4gV2hh dCBpcyB0aGUgZHJpdmVyIGluIHVzZSBmb3IgdGhlIFJIRUwgZW52aXJvbm1lbnQ/IE1lbGxhbm94 IE9GRUQgRHJpdmVyIG9yIE9GRUQ/DQo+IFdoYXQgaXMgdGhlIG1vZGUgY29uZmlndXJlZCBvbiB0 aGUgUkhFTCBlbnZpcm9ubWVudD8NCj4gSXMgdGhpcyBmYWJyaWMgd2lsbCBjb21tdW5pY2F0ZSBi ZXR3ZWVuIHNlcnZlcnMgd2l0aCBGcmVlQlNEICYgUkhFTD8NCj4gDQo+IFJlZ2FyZHMsDQo+IFNv cGhpZS4NCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS8NCj4gDQo+IFRoYW5rIFlvdSwgDQo+IA0KPiANCj4gDQo+IENsaWNrIG9u IHRoZSBsaW5rIHRvIGFjY2VzcyB0aGUgY2FzZTogDQo+IGh0dHBzOi8vbXltZWxsYW5veC5mb3Jj ZS5jb20vc3VwcG9ydC81MDA1MDAwMDAwcTlOcGcNCj4gDQo+IERvbuKAmXQgZm9yZ2V0IHRvIGNo ZWNrIG91dCBvdXIgT25saW5lIEFjYWRlbXksIHJlZ2lzdGVyIG5vdyB0byBsZWFybiBhYm91dCB0 aGUgcHJvZHVjdHMgYW5kIHRlY2hub2xvZ3ksIGh0dHA6Ly9hY2FkZW15Lm1lbGxhbm94LmNvbS9s b2dpbi9pbmRleC5waHAgDQo+IA0KPiAvLS0tLS0tIFlvdXIgQ2FzZSBDb21tZW50cyBIaXN0b3J5 IC0tLS0tLSANCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLQ0KPiAgQ3JlYXRlZCBieTogU29waGllIE5hdWRpbiBQcnVpdHQgMjAxNi0xMi0wOSAxOToy Mjo0OA0KPiBUaGFuayB5b3UuDQo+IFdoYXQgaXMgdGhlIGRyaXZlciBpbiB1c2UgZm9yIHRoZSBS SEVMIGVudmlyb25tZW50PyBNZWxsYW5veCBPRkVEIERyaXZlciBvciBPRkVEPw0KPiBXaGF0IGlz IHRoZSBtb2RlIGNvbmZpZ3VyZWQgb24gdGhlIFJIRUwgZW52aXJvbm1lbnQ/DQo+IElzIHRoaXMg ZmFicmljIHdpbGwgY29tbXVuaWNhdGUgYmV0d2VlbiBzZXJ2ZXJzIHdpdGggRnJlZUJTRCAmIFJI RUw/DQo+IA0KPiBSZWdhcmRzLA0KPiBTb3BoaWUuDQo+IA0KPiAtLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IDIwMTYtMTItMDkgMTM6MjY6NTYNCj4g Q29tbWVudCBCeSBDdXN0b21lcjogZGVuaXNAdmVyc2F0dXNocGMuY29tLmJyDQo+IGl0J3MgRnJl ZUJTRCAxMC4zL0xpbnV4IENlbnRPUyA3LjIgRW52aXJvbm1lbnQuDQo+IA0KPiBELg0KPiANCj4g MjAxNi0xMi0wOCAyMjoyMiBHTVQtMDI6MDAgTWVsbGFub3ggU3VwcG9ydCBBZG1pbiA8c3VwcG9y dGFkbWluQG1lbGxhbm94LmNvbQ0KPj4gOg0KPiANCj4+IERlYXIgVmluw61jaXVzIEZlcnLDo28s DQo+PiANCj4+IE1lbGxhbm94IHN1cHBvcnQgY2FzZSAjIDAwMzIwMDU4IGhhcyBiZWVuIHVwZGF0 ZWQgQnkgLCB3aXRoIHRoZSBmb2xsb3dpbmcNCj4+IGNvbW1lbnQ6DQo+PiAvLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4gSGkg VmluaWNpdXMsDQo+PiANCj4+IE91ciBNZWxsYW5veCBPRkVEIERyaXZlciBjdXJyZW50bHkgb25s eSBzdXBwb3J0cyBGcmVlQlNEIDEwLjEgYW5kIHdlIGRvbid0DQo+PiBoYXZlIGFueSBpbmRpY2F0 aW9uIG9mIGZ1dHVyZSByZWxlYXNlIGF0IHRoaXMgdGltZSBzdXBwb3J0aW5nIGxhdGVyIHZlcnNp b24NCj4+IG9mIHRoaXMgT1MuDQo+PiBLZWVwIGluIG1pbmQgdGhhdCB5b3UgYXJlIHVzaW5nIHRo ZSBzb2Z0d2FyZSBzdGFjayByZWxlYXNlZCBhcyBPRkVEDQo+PiAoT3BlbkZhYnJpY3MgRW50ZXJw cmlzZSBEaXN0cmlidXRpb24pIGRyaXZlci4NCj4+IA0KPj4gRG9lcyB0aGlzIGVudmlyb25tZW50 IGhhcyBzZXJ2ZXJzIHJ1bm5pbmcgRnJlZUJTRCBvbmx5IG9yIGRvIHlvdSBoYXZlDQo+PiBzZXJ2 ZXJzIHdpdGggZGlmZmVyZW50IExpbnV4IGZsYXZvcnM/DQo+PiANCj4+IFJlZ2FyZHMsDQo+PiBT b3BoaWUuDQo+PiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLw0KPj4gDQo+PiBUaGFuayBZb3UsDQo+PiANCj4+IA0KPj4gDQo+PiBZ b3UgY2FuIGFjY2VzcyBhbGwgeW91ciBjbG9zZWQvT3BlbiBjYXNlcywganVzdCBnbyB0bw0KPj4g aHR0cDovL21lbGxhbm94LmNvbS9jb250ZW50L3BhZ2VzLnBocD9wZz1jdXN0b21lcl9zdXBwb3J0 DQo+PiANCj4+IENsaWNrIG9uIHRoZSBsaW5rIHRvIGFjY2VzcyB0aGUgY2FzZToNCj4+IA0KPj4g aHR0cHM6Ly9teW1lbGxhbm94LmZvcmNlLmNvbS9zdXBwb3J0LzUwMDUwMDAwMDBxOU5wZw0KPj4g DQo+PiANCj4+IERvbuKAmXQgZm9yZ2V0IHRvIGNoZWNrIG91dCBvdXIgT25saW5lIEFjYWRlbXks IHJlZ2lzdGVyIG5vdyB0byBsZWFybiBhYm91dA0KPj4gdGhlIHByb2R1Y3RzIGFuZCB0ZWNobm9s b2d5LCBodHRwOi8vYWNhZGVteS5tZWxsYW5veC5jb20vbG9naW4vaW5kZXgucGhwDQo+PiAvLS0t LS0tIFlvdXIgQ2FzZSBDb21tZW50cyBIaXN0b3J5ICAtLS0tLS0NCj4+IC0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+ICBDcmVhdGVkIGJ5OiBTb3Bo aWUgTmF1ZGluIFBydWl0dCAyMDE2LTEyLTA5IDAwOjIyOjI2DQo+PiBIaSBWaW5pY2l1cywNCj4+ IA0KPj4gT3VyIE1lbGxhbm94IE9GRUQgRHJpdmVyIGN1cnJlbnRseSBvbmx5IHN1cHBvcnRzIEZy ZWVCU0QgMTAuMSBhbmQgd2UgZG9uJ3QNCj4+IGhhdmUgYW55IGluZGljYXRpb24gb2YgZnV0dXJl IHJlbGVhc2UgYXQgdGhpcyB0aW1lIHN1cHBvcnRpbmcgbGF0ZXIgdmVyc2lvbg0KPj4gb2YgdGhp cyBPUy4NCj4+IEtlZXAgaW4gbWluZCB0aGF0IHlvdSBhcmUgdXNpbmcgdGhlIHNvZnR3YXJlIHN0 YWNrIHJlbGVhc2VkIGFzIE9GRUQNCj4+IChPcGVuRmFicmljcyBFbnRlcnByaXNlIERpc3RyaWJ1 dGlvbikgZHJpdmVyLg0KPj4gDQo+PiBEb2VzIHRoaXMgZW52aXJvbm1lbnQgaGFzIHNlcnZlcnMg cnVubmluZyBGcmVlQlNEIG9ubHkgb3IgZG8geW91IGhhdmUNCj4+IHNlcnZlcnMgd2l0aCBkaWZm ZXJlbnQgTGludXggZmxhdm9ycz8NCj4+IA0KPj4gUmVnYXJkcywNCj4+IFNvcGhpZS4NCj4+IA0K Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4g MjAxNi0xMi0wOCAwMDozMzoxMw0KPj4gQ29tbWVudCBCeSBDdXN0b21lcjogZmVycmFvQHZtaW8u Y29tLmJyDQo+PiBIZWxsbyBTb3BoaWUsDQo+PiANCj4+IFJlY29tcGlsaW5nIHRoZSBzb2Z0d2Fy ZSBpc27igJl0IGFuIHZpYWJsZSBvcHRpb24uDQo+PiANCj4+IEkgd2FzIGFibGUgdG8gbWl0aWdh dGUgdGhlIGlzc3VlIG9uIHRoaXMgc3BlY2lmaWMgY2FzZSByYWlzaW5nIHRoZSBNVFUgdG8NCj4+ IDY1NTIwIG9uIGFsbCBtYWNoaW5lcy4gSXQgdHVybnMgb3V0IHRoYXQgdGhlIG9sZCBtYWNoaW5l cyB3ZXJlIGNvbmZpZ3VyZWQNCj4+IG9uIENvbm5lY3RlZCBNb2RlIHVzaW5nIG9ubHkgMjA0NCBN VFUgYW5kIGFuIE1UVSBtaXNtYXRjaCBiZXR3ZWVuIHRoZQ0KPj4gc2VydmVyIGFuZCB0aGUgY2xp ZW50cyB3YXMgdGhlIHJlYWwgaXNzdWUuDQo+PiANCj4+IEJ1dCBJ4oCZbSB3b3JyaWVkIGFib3V0 IHRoZSB2aWFiaWxpdHkgb2YgSW5maW5pYmFuZCBvbiBGcmVlQlNELiBDaGFuZ2luZw0KPj4gZnJv bSBDb25uZWN0ZWQgTW9kZSB0byBVbnJlbGlhYmxlIERhdGFncmFtIHNob3VsZCBiZSBlYXNpZXIg dGhhbg0KPj4gcmVjb21waWxpbmcgdGhlIGtlcm5lbCBtb2R1bGUuDQo+PiANCj4+IFRoZXJl4oCZ cyBhIHdheSB0byBhc2sgTWVsbGFub3ggdG8gY2hhbmdlIHRoZSBkcml2ZXJzIHRvIG1ha2UgdGhl IHN3aXRjaA0KPj4gZnJvbSBVRCB0byBDTSB2aWFibGUgb24gRnJlZUJTRD8gQSBmZWF0dXJlIHJl cXVlc3QgcGVyaGFwcz8NCj4+IA0KPj4gVGhhbmtzIGluIGFkdmFuY2UsDQo+PiBWLg0KPj4gDQo+ Pj4gT24gNyBEZWMgMjAxNiwgYXQgMTk6MDUsIE1lbGxhbm94IFN1cHBvcnQgQWRtaW4gPA0KPj4g c3VwcG9ydGFkbWluQG1lbGxhbm94LmNvbT4NCj4+IA0KPj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4gIENyZWF0ZWQgYnk6IFNvcGhpZSBOYXVk aW4gUHJ1aXR0IDIwMTYtMTItMDcgMjE6MDU6MTUNCj4+IEhpIFZpbmljaXVzLA0KPj4gDQo+PiBI YXZlIHlvdSBzZWVuIG15IGxhc3Qgc3RhdGVtZW50IHBvc3RlZCB0byB0aGlzIGNhc2UgYW5kIHdo YXQgaXMgdGhlDQo+PiBjdXJyZW50IHN0YXR1cz8NCj4+IFBsZWFzZSBsZXQgbWUga25vdy4NCj4+ IA0KPj4gVGhhbmsgeW91LA0KPj4gU29waGllLg0KPj4gDQo+PiAtLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+PiAgQ3JlYXRlZCBieTogU29waGllIE5h dWRpbiBQcnVpdHQgMjAxNi0xMi0wNSAyMzowMjozNQ0KPj4gSGkgVmluaWNpdXMsDQo+PiANCj4+ IFNlYXJjaGluZyBzb21lIGZvcnVtcywgSSBmb3VuZCB0aGUgZm9sbG93aW5nIGluZm9ybWF0aW9u Og0KPj4gDQo+PiBCdWlsZGluZw0KPiANCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogU29waGllIE5hdWRpbiBQcnVpdHQg MjAxNi0xMi0wOSAwMDoyMjoyNg0KPiBIaSBWaW5pY2l1cywNCj4gDQo+IE91ciBNZWxsYW5veCBP RkVEIERyaXZlciBjdXJyZW50bHkgb25seSBzdXBwb3J0cyBGcmVlQlNEIDEwLjEgYW5kIHdlIGRv bid0IGhhdmUgYW55IGluZGljYXRpb24gb2YgZnV0dXJlIHJlbGVhc2UgYXQgdGhpcyB0aW1lIHN1 cHBvcnRpbmcgbGF0ZXIgdmVyc2lvbiBvZiB0aGlzIE9TLg0KPiBLZWVwIGluIG1pbmQgdGhhdCB5 b3UgYXJlIHVzaW5nIHRoZSBzb2Z0d2FyZSBzdGFjayByZWxlYXNlZCBhcyBPRkVEIChPcGVuRmFi cmljcyBFbnRlcnByaXNlIERpc3RyaWJ1dGlvbikgZHJpdmVyLg0KPiANCj4gRG9lcyB0aGlzIGVu dmlyb25tZW50IGhhcyBzZXJ2ZXJzIHJ1bm5pbmcgRnJlZUJTRCBvbmx5IG9yIGRvIHlvdSBoYXZl IHNlcnZlcnMgd2l0aCBkaWZmZXJlbnQgTGludXggZmxhdm9ycz8NCj4gDQo+IFJlZ2FyZHMsDQo+ IFNvcGhpZS4NCj4gDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0NCj4gMjAxNi0xMi0wOCAwMDozMzoxMw0KPiBDb21tZW50IEJ5IEN1c3RvbWVyOiBm ZXJyYW9Adm1pby5jb20uYnINCj4gSGVsbG8gU29waGllLA0KPiANCj4gUmVjb21waWxpbmcgdGhl IHNvZnR3YXJlIGlzbuKAmXQgYW4gdmlhYmxlIG9wdGlvbi4NCj4gDQo+IEkgd2FzIGFibGUgdG8g bWl0aWdhdGUgdGhlIGlzc3VlIG9uIHRoaXMgc3BlY2lmaWMgY2FzZSByYWlzaW5nIHRoZSBNVFUg dG8gNjU1MjAgb24gYWxsIG1hY2hpbmVzLiBJdCB0dXJucyBvdXQgdGhhdCB0aGUgb2xkIG1hY2hp bmVzIHdlcmUgY29uZmlndXJlZCBvbiBDb25uZWN0ZWQgTW9kZSB1c2luZyBvbmx5IDIwNDQgTVRV IGFuZCBhbiBNVFUgbWlzbWF0Y2ggYmV0d2VlbiB0aGUgc2VydmVyIGFuZCB0aGUgY2xpZW50cyB3 YXMgdGhlIHJlYWwgaXNzdWUuDQo+IA0KPiBCdXQgSeKAmW0gd29ycmllZCBhYm91dCB0aGUgdmlh YmlsaXR5IG9mIEluZmluaWJhbmQgb24gRnJlZUJTRC4gQ2hhbmdpbmcgZnJvbSBDb25uZWN0ZWQg TW9kZSB0byBVbnJlbGlhYmxlIERhdGFncmFtIHNob3VsZCBiZSBlYXNpZXIgdGhhbiByZWNvbXBp bGluZyB0aGUga2VybmVsIG1vZHVsZS4NCj4gDQo+IFRoZXJl4oCZcyBhIHdheSB0byBhc2sgTWVs bGFub3ggdG8gY2hhbmdlIHRoZSBkcml2ZXJzIHRvIG1ha2UgdGhlIHN3aXRjaCBmcm9tIFVEIHRv IENNIHZpYWJsZSBvbiBGcmVlQlNEPyBBIGZlYXR1cmUgcmVxdWVzdCBwZXJoYXBzPw0KPiANCj4g VGhhbmtzIGluIGFkdmFuY2UsDQo+IFYuDQo+IA0KPj4gT24gNyBEZWMgMjAxNiwgYXQgMTk6MDUs IE1lbGxhbm94IFN1cHBvcnQgQWRtaW4gPHN1cHBvcnRhZG1pbkBtZWxsYW5veC5jb20+DQo+IA0K PiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ICBD cmVhdGVkIGJ5OiBTb3BoaWUgTmF1ZGluIFBydWl0dCAyMDE2LTEyLTA3IDIxOjA1OjE1DQo+IEhp IFZpbmljaXVzLA0KPiANCj4gSGF2ZSB5b3Ugc2VlbiBteSBsYXN0IHN0YXRlbWVudCBwb3N0ZWQg dG8gdGhpcyBjYXNlIGFuZCB3aGF0IGlzIHRoZSBjdXJyZW50IHN0YXR1cz8NCj4gUGxlYXNlIGxl dCBtZSBrbm93Lg0KPiANCj4gVGhhbmsgeW91LA0KPiBTb3BoaWUuDQo+IA0KPiAtLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ICBDcmVhdGVkIGJ5OiBT b3BoaWUgTmF1ZGluIFBydWl0dCAyMDE2LTEyLTA1IDIzOjAyOjM1DQo+IEhpIFZpbmljaXVzLA0K PiANCj4gU2VhcmNoaW5nIHNvbWUgZm9ydW1zLCBJIGZvdW5kIHRoZSBmb2xsb3dpbmcgaW5mb3Jt YXRpb246DQo+IA0KPiBCdWlsZGluZyBPRkVEIGluIEZyZWVCU0QNCj4gDQo+IEtlcm5lbA0KPiDi gKJBZGQgdGhlIGZvbGxvd2luZyBvcHRpb25zIHRvIHlvdXIgS0VSTkNPTkYgdG8gY29tcGlsZSB0 aGUgbmVjZXNzYXJ5IHByb3RvY29scy9wcm90b2NvbCBvcHRpb25zIGludG8geW91ciBrZXJuZWw6 IA0KPiANCj4gDQo+IG9wdGlvbnMgICAgICAgIE9GRUQgICAgICAgICAgICAjIEluZmluaWJhbmQg cHJvdG9jb2wgc3RhY2sgYW5kIHN1cHBvcnQNCj4gb3B0aW9ucyAgICAgICAgU0RQICAgICAgICAg ICAgICMgU29ja2V0cyBEaXJlY3QgUHJvdG9jb2wgZm9yIGluZmluaWJhbmQNCj4gb3B0aW9ucyAg ICAgICAgSVBPSUJfQ00gICAgICAgICMgVXNlIGNvbm5lY3QgbW9kZSBpcG9pYiA+Pj5yZWNvbXBp bGUgd2l0aG91dCB0aGF0IG9wdGlvbiBhbmQgdGhlIG1vZGUgc2hvdWxkIGJlIGRhdGFncmFtLg0K PiANCj4gUmVnYXJkcywNCj4gU29waGllLg0KPiANCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogU29waGllIE5hdWRpbiBQ cnVpdHQgMjAxNi0xMi0wNSAyMjo1MzowNg0KPiBIaSBWaW7DrWNpdXMsDQo+IA0KPiBZb3UgYXJl IG5vdCB1c2luZyBNZWxsYW5veCBPRkVEIERyaXZlciBpZiB5b3UgYXJlIHJ1bm5pbmcgT0ZFRCAo T3BlbkZhYnJpY3MgRW50ZXJwcmlzZSBEaXN0cmlidXRpb24pLCB3aGF0IGlzIHRoZSB2ZXJzaW9u IGFuZCBkaWQgeW91IGNvbnN1bHQgdGhlIFVNPw0KPiBXaXRoIG91ciBkcml2ZXIsIHlvdSBjYW4g Y2hhbmdlIHRoZSBtb2RlIHdpdGg6DQo+IA0KPiBlY2hvIGRhdGFncmFtID4gL3N5cy9jbGFzcy9u ZXQvaWIwL21vZGUgc2V0cyB0aGUgbW9kZSBvZiBpYjAgdG8gVUQNCj4gZWNobyBjb25uZWN0ZWQg PiAvc3lzL2NsYXNzL25ldC9pYjAvbW9kZSBzZXRzIHRoZSBtb2RlIGliMCB0byBDTQ0KPiANCj4g VGhvdWdoIEZyZWVCU0QgZG9lcyBub3QgaGF2ZSBhIC9zeXMuDQo+IA0KPiBSZWdhcmRzLA0KPiBT b3BoaWUuDQo+IA0KPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tDQo+ICBDcmVhdGVkIGJ5OiBBbmRlcnNvbiBEaWFzIFNpbHZhIDIwMTYtMTItMDUgMjA6 Mjk6MzUNCj4gSGkgQmVubnksDQo+IA0KPiBJIGFtIEFuZGVyc29uIERpYXMgU2lsdmEgKExBVEFN IFNFKS4gUGxlYXNlLCBzZWUgYmVsb3cgYW4gaW1wb3J0YW50IGNvbnNpZGVyYXRpb24gb2Ygb3Vy IGN1c3RvbWVyLg0KPiANCj4gIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIw0KPiANCj4gQWZ0ZXIgc29tZSB0ZXN0aW5nIHdlIGRpc2NvdmVyZWQgdGhhdCB3YXMgYSBt aXNtYXRjaCBiZXR3ZWVuIHRoZSBjbGllbnQgYW5kIHRoZSBzZXJ2ZXIgb24gb3BlcmF0aW5nIG1v ZGVzIG9mIElQb0lCLiBPbmUgc2lkZSB3YXMgdXNpbmcgQ29ubmVjdGVkIE1vZGUgKHRoZSBGcmVl QlNEIG9uZSkgYW5kIHRoZSBMaW51eCBjbGllbnQgd2FzIG9uIERhdGFncmFtIG1vZGUpLg0KPiAN Cj4gU28gSeKAmW0gY29uc2lkZXJpbmcgcnVubmluZyBGcmVlQlNEIGluIFVEIChEYXRhZ3JhbSkg bW9kZSB0byBzZWUgaWYgSSBmaW5hbGx5IGNhbiBnZXQgcmlkIG9mIHRoZSBwcm9ibGVtcy4gQnV0 LCBob3cgY2FuIEkgY2hhbmdlIGJldHdlZW4gQ00gYW5kIFVEIGluIEZyZWVCU0Q/IFdlIGRvbid0 IGhhdmUgYSAvc3lzIGZvbGRlciBvbiBGcmVlQlNEIHNvIEkgY2FudCBqdXN0IHNldCBmcm9tIDAg dG8gMSBhbmQgY2hhbmdlIG1vZGVzLg0KPiANCj4gIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIw0KPiANCj4gU28sIGhvdyBjYW4gd2UgY2hhbmdlIGJldHdlZW4gQ00g YW5kIFVEIGluIEZyZWVCU0Q/DQo+IA0KPiBSYWdhcmRzIQ0KPiANCj4gLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAgQ3JlYXRlZCBieTogQmVubnkg RmlsaW4gMjAxNi0xMi0wNSAwMTowMzo1NQ0KPiBIZWxsbyBWaW7DrWNpdXMgRmVycsOjbywNCj4g DQo+IFRoYW5rIHlvdSBmb3IgY29udGFjdGluZyBNZWxsYW5veCBUZWNobmljYWwgU3VwcG9ydC4N Cj4gDQo+IDEpIERJZCB5b3UgdHJ5IGNoYW5naW5nIHN3aXRjaCBwb3J0cyB3aGVuIHRlc3Rpbmcg cGVyZm9ybWFuY2UgYmV0d2VlbiB0d28gRnJlZUJTRCBtYWNoaW5lcyB0aHJvdWdoIHN3aXRjaD8N Cj4gMikgV2hhdCBoYXBwZW5zIHdoZW4geW91IHRyeSBhIGJhY2stdG8tYmFjayBjb25uZWN0aW9u IGJldHdlZW4gdHdvIEZyZWVCU0QgbWFjaGluZXM/IERvIHlvdSBnZXQgZ29vZCByZXN1bHRzPw0K PiAzKSBEaWQgeW91IHRlc3QgdGhlIHBlcmZvcm1hbmNlIGJldHdlZW4gRnJlZUJTRCBhbmQgYSBM aW51eCBtYWNoaW5lIHRocm91Z2ggc3dpdGNoPw0KPiANCj4gUGxlYXNlLCBmb2xsb3cgdGhlIHBy b3ZpZGVkIHByb2NlZHVyZSBpbiBvcmRlciB0byBxdWVyeSB5b3VyIHVubWFuYWdlZCBzd2l0Y2g6 IA0KPiANCj4gMSkgRG93bmxvYWQgYW5kIGluc3RhbGwgTUZUIChNZWxsYW5veCBGaXJtd2FyZSBU b29sKToNCj4gaHR0cDovL3d3dy5tZWxsYW5veC5jb20vY29udGVudC9wYWdlcy5waHA/cGc9bWFu YWdlbWVudF90b29scyZtZW51X3NlY3Rpb249MzQNCj4gUmVmZXIgdG8gdGhlIFVzZXIgTWFudWFs IGZvciB0aGUgaW5zdGFsbGF0aW9uIGluc3RydWN0aW9uczoNCj4gaHR0cDovL3d3dy5tZWxsYW5v eC5jb20vcGRmL01GVC9NRlRfdXNlcl9tYW51YWwucGRmDQo+IA0KPiAyKSBGcm9tIG9uZSBvZiB0 aGUgbm9kZXMgaW4gdGhlIGZhYnJpYywgaW52b2tlOiBpYnN3aXRjaGVzLg0KPiBNYXRjaCB0aGUg cmVsZXZhbnQgc3dpdGNoIEdVSUQgZnJvbSB0aGUgTWVsbGFub3ggc3dpdGNoIGxhYmVsIHRvIHRo ZSBMSUQuDQo+IA0KPiAzKSBGcm9tIG9uZSBvZiB0aGUgbm9kZXMgaW4gdGhlIGZhYnJpYyAsIGlu dm9rZTogbXN0IHN0YXJ0IC0+IG1zdCBpYiBhZGQgLT4gbXN0IHN0YXR1cw0KPiANCj4gNCkgSWRl bnRpZnkgdGhlIHJlbGV2YW50IHVubWFuYWdlZCBzd2l0Y2ggbXN0IHJlcHJlc2VudGF0aW9uIGJh c2VkIG9uIHRoZSBzd2l0Y2ggTElEIChmcm9tIHByZXZpb3VzIHN0ZXAgMikuDQo+ICogTWF0Y2hp bmcgdGhlIHJlbW90ZSBzd2l0Y2ggbXN0IGRldmljZSByZXByZXNlbnRhdGlvbiB0byB0aGUgcmVs ZXZhbnQgc3dpdGNoIHByb2NlZHVyZToNCj4gTG9jYXRlIHRoZSBzd2l0Y2ggaW4gdGhlICJpYnN3 aXRjaGVzIiBvdXRwdXQgYW5kIHJldHJpZXZlIHRoZSBzd2l0Y2ggTElELiANCj4gQ29udmVydCB0 aGUgTElEIGZyb20gRGVjaW1hbCB0byBIZXhhZGVjaW1hbCBiYXNlLg0KPiBMb2NhdGUgdGhlIGRl dmljZSB3aXRoIHRoaXMgTElEIChpbiBoZXggYmFzZSkgZnJvbSB0aGUgIm1zdCBzdGF0dXMiIG91 dHB1dC4gDQo+IA0KPiA1KSBJbnZva2UgdGhlIGZvbGxvd2luZyBjb21tYW5kLCBhbmQgcHJvdmlk ZSB0aGUgb3V0cHV0Og0KPiBmbGludCAtZCA8cmVtb3RlX3N3aXRjaF9tc3RfZGV2aWNlPiBxDQo+ IA0KPiANCj4gV2UncmUgbG9va2luZyBmb3J3YXJkIHRvIGhlYXJpbmcgZnJvbSB5b3UuDQo+IA0K PiBCZXN0IHJlZ2FyZHMsDQo+IE1lbGxhbm94IFRlY2huaWNhbCBTdXBwb3J0DQo+IA0KPiAtLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ICBDcmVhdGVk IGJ5OiBKdWJyYW4gSnVicmFuIDIwMTYtMTItMDEgMTY6NDE6MzUNCj4gSGVsbG8gVmluw61jaXVz IEZlcnLDo28sDQo+IA0KPiBUaGFuayB5b3UgZm9yIGNvbnRhY3RpbmcgTWVsbGFub3ggVGVjaG5p Y2FsIFN1cHBvcnQuDQo+IA0KPiAxKSBESWQgeW91IHRyeSBjaGFuZ2luZyBzd2l0Y2ggcG9ydHMg d2hlbiB0ZXN0aW5nIHBlcmZvcm1hbmNlIGJldHdlZW4gdHdvIEZyZWVCU0QgbWFjaGluZXMgdGhy b3VnaCBzd2l0Y2g/DQo+IDIpIFdoYXQgaGFwcGVucyB3aGVuIHlvdSB0cnkgYSBiYWNrLXRvLWJh Y2sgY29ubmVjdGlvbiBiZXR3ZWVuIHR3byBGcmVlQlNEIG1hY2hpbmVzPyBEbyB5b3UgZ2V0IGdv b2QgcmVzdWx0cz8NCj4gMykgRGlkIHlvdSB0ZXN0IHRoZSBwZXJmb3JtYW5jZSBiZXR3ZWVuIEZy ZWVCU0QgYW5kIGEgTGludXggbWFjaGluZSB0aHJvdWdoIHN3aXRjaD8NCj4gDQo+IFBsZWFzZSwg Zm9sbG93IHRoZSBwcm92aWRlZCBwcm9jZWR1cmUgaW4gb3JkZXIgdG8gcXVlcnkgeW91ciB1bm1h bmFnZWQgc3dpdGNoOiANCj4gIA0KPiAxKSBEb3dubG9hZCBhbmQgaW5zdGFsbCBNRlQgKE1lbGxh bm94IEZpcm13YXJlIFRvb2wpOg0KPiBodHRwOi8vd3d3Lm1lbGxhbm94LmNvbS9jb250ZW50L3Bh Z2VzLnBocD9wZz1tYW5hZ2VtZW50X3Rvb2xzJm1lbnVfc2VjdGlvbj0zNA0KPiBSZWZlciB0byB0 aGUgVXNlciBNYW51YWwgZm9yIHRoZSBpbnN0YWxsYXRpb24gaW5zdHJ1Y3Rpb25zOg0KPiBodHRw Oi8vd3d3Lm1lbGxhbm94LmNvbS9wZGYvTUZUL01GVF91c2VyX21hbnVhbC5wZGYNCj4gIA0KPiAy KSBGcm9tIG9uZSBvZiB0aGUgbm9kZXMgaW4gdGhlIGZhYnJpYywgaW52b2tlOiBpYnN3aXRjaGVz Lg0KPiBNYXRjaCB0aGUgcmVsZXZhbnQgc3dpdGNoIEdVSUQgZnJvbSB0aGUgTWVsbGFub3ggc3dp dGNoIGxhYmVsIHRvIHRoZSBMSUQuDQo+ICANCj4gMykgRnJvbSBvbmUgb2YgdGhlIG5vZGVzIGlu IHRoZSBmYWJyaWMgLCBpbnZva2U6IG1zdCBzdGFydCAtPiBtc3QgaWIgYWRkIC0+IG1zdCBzdGF0 dXMNCj4gIA0KPiA0KSBJZGVudGlmeSB0aGUgcmVsZXZhbnQgdW5tYW5hZ2VkIHN3aXRjaCBtc3Qg cmVwcmVzZW50YXRpb24gYmFzZWQgb24gdGhlIHN3aXRjaCBMSUQgKGZyb20gcHJldmlvdXMgc3Rl cCAyKS4NCj4gKiBNYXRjaGluZyB0aGUgcmVtb3RlIHN3aXRjaCBtc3QgZGV2aWNlIHJlcHJlc2Vu dGF0aW9uIHRvIHRoZSByZWxldmFudCBzd2l0Y2ggcHJvY2VkdXJlOg0KPiBMb2NhdGUgdGhlIHN3 aXRjaCBpbiB0aGUgImlic3dpdGNoZXMiIG91dHB1dCBhbmQgcmV0cmlldmUgdGhlIHN3aXRjaCBM SUQuIA0KPiBDb252ZXJ0IHRoZSBMSUQgZnJvbSBEZWNpbWFsIHRvIEhleGFkZWNpbWFsIGJhc2Uu DQo+IExvY2F0ZSB0aGUgZGV2aWNlIHdpdGggdGhpcyBMSUQgKGluIGhleCBiYXNlKSBmcm9tIHRo ZSAibXN0IHN0YXR1cyIgb3V0cHV0LiANCj4gDQo+IDUpIEludm9rZSB0aGUgZm9sbG93aW5nIGNv bW1hbmQsIGFuZCBwcm92aWRlIHRoZSBvdXRwdXQ6DQo+IGZsaW50IC1kIDxyZW1vdGVfc3dpdGNo X21zdF9kZXZpY2U+IHENCj4gDQo+IA0KPiBXZSdyZSBsb29raW5nIGZvcndhcmQgdG8gaGVhcmlu ZyBmcm9tIHlvdS4NCj4gDQo+IEJlc3QgcmVnYXJkcywNCj4gTWVsbGFub3ggVGVjaG5pY2FsIFN1 cHBvcnQgDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0vDQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fDQo+IGZyZWVic2QtaW5maW5pYmFuZEBmcmVlYnNkLm9yZyBtYWlsaW5nIGxpc3QNCj4gaHR0 cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyZWVic2QtaW5maW5pYmFu ZA0KPiBUbyB1bnN1YnNjcmliZSwgc2VuZCBhbnkgbWFpbCB0byAiZnJlZWJzZC1pbmZpbmliYW5k LXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnIg0KDQo=