From owner-freebsd-infiniband@freebsd.org Sun May 29 23:49:36 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C225B545C5 for ; Sun, 29 May 2016 23:49:36 +0000 (UTC) (envelope-from menyy@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0070.outbound.protection.outlook.com [157.55.234.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D070611FA for ; Sun, 29 May 2016 23:49:35 +0000 (UTC) (envelope-from menyy@mellanox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=2JUc4TrPX/YHD2g063wxs6raCRnvnc+mdccVeNTFcSs=; b=ETM575ekQwXISHGhCkFq7A9gVCAfrMgUuEEODOX2vTNIfFYS/2s4wBk2lKn3tz7VGLAHyBzigDygPR+18WSLxX0Vt1GJG1vUUdtObSN2Gl73wi0LOxuuhXJWShAPtqGHmtUVOV6xD2QRyyKRAWD2syphUxj86T2G2lNrcHTPj/g= Received: from DB5PR05MB1765.eurprd05.prod.outlook.com (10.165.7.23) by DB5PR05MB1766.eurprd05.prod.outlook.com (10.165.7.24) with Microsoft SMTP Server (TLS) id 15.1.506.9; Sun, 29 May 2016 14:16:09 +0000 Received: from DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) by DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) with mapi id 15.01.0506.011; Sun, 29 May 2016 14:16:09 +0000 From: Meny Yossefi To: "freebsd-infiniband@freebsd.org" Subject: RE: InfiniBand supported hw Thread-Topic: InfiniBand supported hw Thread-Index: AQHRt06VcV23ltpzREepdsqiL6FOLp/Lc2yAgAAKewCAAA2LAIAAFyGAgAAAEWKABFYeQA== Date: Sun, 29 May 2016 14:16:09 +0000 Message-ID: References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com>, <7C7D1835-52DD-4F27-B0CA-A59650E59639@jnielsen.net> <52cb4d9e0d464f0180c409b33d95f501@AMSPR05MB0580.eurprd05.prod.outlook.com> In-Reply-To: <52cb4d9e0d464f0180c409b33d95f501@AMSPR05MB0580.eurprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=mellanox.com; x-originating-ip: [193.47.165.251] x-ms-office365-filtering-correlation-id: cb615d91-d044-44f5-f347-08d387cbc861 x-microsoft-exchange-diagnostics: 1; DB5PR05MB1766; 5:vJbJl7K28sfuseCMFh/GKsd4jZTCajYYYff8qTxG80OIs06tQKa4JxzWLwn8tvW5KYoDMChq0RnNrXF7hUqQVf4FZw4uCAff62meLrtQr/MyWi9wJLI27jmztwRJFSGb82EeFXR9LO3PD3tzpvFedQ==; 24:pML70G7aQeQBK8TBSzPiT3dfpyAs2SeE0nSeWnlh5ExuOHsKHym4diAgMvwFKNEOr+EnayKzcruYL+vIN6bjfwYiybzrNZGVGIv225F8QZs=; 7:hslGKEVdJfnboig6v5M29RviKw+L1bQnoyI1vI9owCydmQrzOg9CXUXJfHp5C6+qpF2Wmysomqg4WURt9uXIv9b/wVsjtnBs9HN6t2MFl0vHdGxX1ktG+ug3NFd6CslkEc8ZSwbF7XbQ8cBo/JV24RQ0jTnt0lLqbjHa9KxMoGVB/KOSGz7o6SNOBKx1gnNn x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB5PR05MB1766; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(278428928389397)(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026); SRVR:DB5PR05MB1766; BCL:0; PCL:0; RULEID:; SRVR:DB5PR05MB1766; x-forefront-prvs: 0957AD37A0 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(377454003)(24454002)(74316001)(19580395003)(87936001)(122556002)(19580405001)(10400500002)(11100500001)(66066001)(5004730100002)(8676002)(81166006)(8936002)(107886002)(1220700001)(2501003)(102836003)(3846002)(77096005)(15975445007)(106116001)(86362001)(586003)(2906002)(2900100001)(76576001)(5640700001)(3280700002)(2950100001)(189998001)(3660700001)(450100001)(5003600100002)(92566002)(9686002)(33656002)(54356999)(76176999)(5008740100001)(2351001)(50986999)(3480700004)(5002640100001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR05MB1766; H:DB5PR05MB1765.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 29 May 2016 14:16:09.7001 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1766 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 23:49:36 -0000 LAGG for IPoIB is called 'Bond' and no, It's not supported in FreeBSD. -Meny > On May 26, 2016, at 12:33 PM, Grzegorz Junka wrote: > > >> On 26/05/2016 17:44, John Nielsen wrote: >>> On May 26, 2016, at 11:07 AM, Justin Clift wrot= e: >>> >>>> On 26 May 2016, at 14:00, Grzegorz Junka wrote: >>>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version a= nd it is the same as Mellanox ConnectX, which supposed to be supported. >>> Yep, they should work. You may need to flash them with their=20 >>> Mellanox equivalent firmware, but I'm not sure. (doing that flash=20 >>> isn't very hard btw) >>> >>>> Also, most InfiniBand cards provide two ports. When connecting two com= puters together (IPoIB) is it possible to use both ports to double the spee= d? >>> Not sure, as I've not needed to personally. None of my storage (or=20 >>> other uses) even fills one DDR port. Yet. :) >>> >>> I *think* the answer to that is yes, >> It depends on what you mean by "use both ports." You can of course use b= oth ports at the same time, but if you're envisioning something like an IPo= IB version of an Ethernet LAGG I don't think that is supported. >> >> If you are using the IB for storage protocols, then active/active multip= athing is one way to utilize the full bandwidth of multiple ports. >> >>>> Lastly, is NFS RDMA properly supported on FreeBSD? >>> Again not sure. Other people here would likely know. :) > > Yes, that's exactly what I was hoping for, a sort of LAGG to double the s= peed of the link between computer by utilizing both ports together. LAGG is= independent from the network interface, if I could set up an IPoIB NIC for= each port separately what could prevent me from creating a LAGG interface = on top of them? And if that's not supported, what do people usually do with= the other interface? Just leave it unconnected? LAGG is designed to work with Ethernet. InfiniBand is a very different laye= r 2 technology with its own advantages and disadvantages. I don't want to g= et too far out of my depth so I'll just suggest you do your own research on= the subject. How you use the second port is entirely up to you. You can use it for separ= ate applications, for redundancy or higher throughput. In my experience it = usually makes sense to give it its own IP. If you want higher bandwidth tha= n a single port can provide then you should at layer 3 or higher for ways t= o aggregate bandwidth. (You should also do some research on PCI-e throughpu= t. A x8 PCI-e 3.0 slot maxes out well before 112Gbit/s.) As I mentioned previously you can use active/active multipath to get more t= hroughput for block storage applications. You could also run a routing daem= on and do ECMP for any layer 3 traffic. Or your particular application may = have other ways to intelligently use two distinct IPoIB paths. _______________________________________________ freebsd-infiniband@freebsd.org mailing list https://lists.freebsd.org/mailm= an/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g"