From owner-freebsd-infiniband@freebsd.org Wed Mar 30 19:47:05 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8270AE3C70 for ; Wed, 30 Mar 2016 19:47:05 +0000 (UTC) (envelope-from chrisr@spectralogic.com) Received: from mail1.bemta12.messagelabs.com (mail1.bemta12.messagelabs.com [216.82.251.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta12.messagelabs.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EBE81878 for ; Wed, 30 Mar 2016 19:47:05 +0000 (UTC) (envelope-from chrisr@spectralogic.com) Received: from [216.82.251.33] by server-4.bemta-12.messagelabs.com id 31/DE-08444-63D2CF65; Wed, 30 Mar 2016 19:47:02 +0000 X-Env-Sender: chrisr@spectralogic.com X-Msg-Ref: server-8.tower-130.messagelabs.com!1459367222!24263300!1 X-Originating-IP: [192.30.190.20] X-StarScan-Received: X-StarScan-Version: 8.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 47184 invoked from network); 30 Mar 2016 19:47:02 -0000 Received: from outmx2.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.20) by server-8.tower-130.messagelabs.com with AES256-SHA encrypted SMTP; 30 Mar 2016 19:47:02 -0000 From: Chris Reeves To: "freebsd-infiniband@freebsd.org" Subject: IPOIB perforance in 11.0-current Thread-Topic: IPOIB perforance in 11.0-current Thread-Index: AQHRirovZuQEBZEolUyRyem4TLL7jA== Date: Wed, 30 Mar 2016 19:47:01 +0000 Message-ID: <1459367221119.55232@spectralogic.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Mar 2016 19:47:05 -0000 Hi All, I've been trying to benchmark performance using IPOIB on FreeBSD 11.0-curre= nt, but I'm not getting the speeds I think I should. The best I'm seeing o= n a 56 Gb/s link is about 10 Gb/s. I know I won't see line speed using IPO= IB, but this performance still seems slow... I have two hosts, each with a MLX 354A card, connected to an FDR switch. T= he ports show connected at 56 Gb/s. I'm running in connected mode with an = MTU of 65520. Using iperf as a benchmark, the best performance I can get is about 10 Gb/s= one way. Using the exact same setup under Ubuntu, I've been able to get I= POIB performance of 18 Gb/s. Does anyone have any advice for improving my performance in FreeBSD? I've = tried datagram mode, but performance is slower (about 5 Gb/s) which I think= is expected. I've tried different window sizes in iperf, but haven't been= able to get any higher than 10 Gb/s. I've also turned off all the debug o= ption in my kernel config. Should I expect better performance? Are there = any other changes I can make to speed things up? Thanks very much, Chris From owner-freebsd-infiniband@freebsd.org Thu Mar 31 09:48:11 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C751AE3662 for ; Thu, 31 Mar 2016 09:48:11 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD4F41644 for ; Thu, 31 Mar 2016 09:48:10 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 2F3BA1FE023 for ; Thu, 31 Mar 2016 11:48:02 +0200 (CEST) To: "freebsd-infiniband@freebsd.org" From: Hans Petter Selasky Subject: OFED v3.2 update Message-ID: <56FCF309.1070406@selasky.org> Date: Thu, 31 Mar 2016 11:51:05 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 09:48:11 -0000 Dear users of infiniband in FreeBSD: Mellanox is working on a big infiniband update towards mellanox OFED v3.2 of the infiniband stack (ibcore), drivers (mlx4/mlx5) and libraries (librdma, libibverbs ...). Currently there are 10-patches in different areas of the FreeBSD kernel and userspace which you need to apply to 11-current as of today to get a working GENERIC kernel. Compile world and kernel like this after applying the patches below: make buildkernel WITH_OFED=YES make buildworld WITH_OFED=YES https://reviews.freebsd.org/D5790 https://reviews.freebsd.org/D5791 https://reviews.freebsd.org/D5792 https://reviews.freebsd.org/D5793 https://reviews.freebsd.org/D5794 https://reviews.freebsd.org/D5795 https://reviews.freebsd.org/D5796 https://reviews.freebsd.org/D5797 https://reviews.freebsd.org/D5798 https://reviews.freebsd.org/D5799 The patches are work in progress, though at a level where basic infiniband is working with MLX5. Known issues: - LINT build is broken (Will be fixed before committing. Waiting for solution from Navdeep) - Need to unload ipoib before mlx5ib. --HPS From owner-freebsd-infiniband@freebsd.org Thu Mar 31 14:01:28 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 484B8AE4B82 for ; Thu, 31 Mar 2016 14:01:28 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1179318B6 for ; Thu, 31 Mar 2016 14:01:27 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-io0-x232.google.com with SMTP id g185so116306571ioa.2 for ; Thu, 31 Mar 2016 07:01:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=iuiRvkoMzkd9ihb5YKNC6xrSkkmg4NWqwFMSnTL6oP0=; b=F/P6f8w8pmHV02gCR/c5A4cOVhL3X9vL175vWr6tnf/RNz2eY27R9UtvSUaQj1hskz Lp0vKn97+78pMoePKPrvbPlYvf0xHxSHQt9yX1VF3VarWGTppd25OgNat9mJ4V1lOuNq VPP1+33F0exnmLeksi9veNNmlAZAwCF550ZQmN/09CmdLqZu2qpmZqaTzbX6AcSfIGBl 7/+GPEnvY2qRcVorzuIxGBJSTzJ9drkFA+obtWSOx9Pl0SW+30m1533ZJWPKzGJqQa1f ROv29ZYqHGkN1GLP0M41e7huN5r840GaUHcPDpemFRbX1tKpZP2qsh300oYDKAVfHeX4 AnZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=iuiRvkoMzkd9ihb5YKNC6xrSkkmg4NWqwFMSnTL6oP0=; b=mFY8XDVoPdBKjUcFGyeLHRXLJL9pkryENPoUayiBzChOvqjjsi94JwScOX1ddMClA9 rDUewWPXoysZyJBecofK4MJkRv+EcQVb38pez8emwl/ENw5fvYMslS8tZFLam5Dft6eh y3BWLu7z3KIdUniaLM0nf5eEjWy51FYWQE0xC+NDhUi98Ym60F1uv/zHBWNTwl//uDEP 0oumYUnOla0i5Uu3skhxMSitY6ZCtES5Ygj8Q7LQHvb5JipqNCTIUfFdPS/ajtFybaZF ILJS6G7CvIlcup2SGAApXS8RuYz8VI2lncy20lwtk0/MzUWcObROjBZo9IJKQ7rapj8P Xp4g== X-Gm-Message-State: AD7BkJInhfyUl2MsyLNk1UuvEpJIzyV4kiPKtau31rhzkGUBfpCy07lbudH31PjUaVjwKA== X-Received: by 10.107.135.10 with SMTP id j10mr2526772iod.28.1459432887110; Thu, 31 Mar 2016 07:01:27 -0700 (PDT) Received: from imacbsd.acadix.biz ([64.83.177.9]) by smtp.gmail.com with ESMTPSA id m143sm3823040ioe.43.2016.03.31.07.01.26 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 31 Mar 2016 07:01:26 -0700 (PDT) Subject: Re: OFED v3.2 update To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <56FCF309.1070406@selasky.org> From: Jason Bacon Message-ID: <56FD2DB4.5030308@gmail.com> Date: Thu, 31 Mar 2016 09:01:24 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56FCF309.1070406@selasky.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 14:01:28 -0000 Hans, Thanks very much for your continued support for FreeBSD! One question: Is it possible to build the driver as a kernel module? Regards, Jason On 03/31/16 04:51, Hans Petter Selasky wrote: > Dear users of infiniband in FreeBSD: > > > Mellanox is working on a big infiniband update towards mellanox OFED > v3.2 of the infiniband stack (ibcore), drivers (mlx4/mlx5) and > libraries (librdma, libibverbs ...). Currently there are 10-patches in > different areas of the FreeBSD kernel and userspace which you need to > apply to 11-current as of today to get a working GENERIC kernel. > > Compile world and kernel like this after applying the patches below: > > make buildkernel WITH_OFED=YES > make buildworld WITH_OFED=YES > > > https://reviews.freebsd.org/D5790 > https://reviews.freebsd.org/D5791 > https://reviews.freebsd.org/D5792 > https://reviews.freebsd.org/D5793 > https://reviews.freebsd.org/D5794 > https://reviews.freebsd.org/D5795 > https://reviews.freebsd.org/D5796 > https://reviews.freebsd.org/D5797 > https://reviews.freebsd.org/D5798 > https://reviews.freebsd.org/D5799 > > > The patches are work in progress, though at a level where basic > infiniband is working with MLX5. > > > Known issues: > - LINT build is broken (Will be fixed before committing. Waiting for > solution from Navdeep) > - Need to unload ipoib before mlx5ib. > > --HPS > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- All wars are civil wars, because all men are brothers ... Each one owes infinitely more to the human race than to the particular country in which he was born. -- Francois Fenelon From owner-freebsd-infiniband@freebsd.org Thu Mar 31 14:10:29 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72645AE4C52 for ; Thu, 31 Mar 2016 14:10:29 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3E9171B05 for ; Thu, 31 Mar 2016 14:10:28 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 0DB361FE023; Thu, 31 Mar 2016 16:10:25 +0200 (CEST) Subject: Re: OFED v3.2 update To: Jason Bacon , "freebsd-infiniband@freebsd.org" References: <56FCF309.1070406@selasky.org> <56FD2DB4.5030308@gmail.com> From: Hans Petter Selasky Message-ID: <56FD3088.5050406@selasky.org> Date: Thu, 31 Mar 2016 16:13:28 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56FD2DB4.5030308@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 14:10:29 -0000 On 03/31/16 16:01, Jason Bacon wrote: > One question: Is it possible to build the driver as a kernel module? Yes, everything is buildable as kernel modules. --HPS From owner-freebsd-infiniband@freebsd.org Thu Mar 31 19:43:55 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54593AE454E for ; Thu, 31 Mar 2016 19:43:55 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x22f.google.com (mail-pa0-x22f.google.com [IPv6:2607:f8b0:400e:c03::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 25BE91D40 for ; Thu, 31 Mar 2016 19:43:55 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pa0-x22f.google.com with SMTP id zm5so72972820pac.0 for ; Thu, 31 Mar 2016 12:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=u7Wr58qL9G5y1tzEQCokGfV3zERmxEUgDq6CeJFGweg=; b=YlsbES0ywy2wjDm8xQnDOiqmEFRE1G6wTLfqUkrLf9otCE0C4DwDJ5zLeEpbc3tc1q hZElVSSLu12N3IN/MAY3eR6PbrvxTx94uqrseJbM/mYtouJuVqGgv4toyV5Z6kwW8BXY NxWEftZynvO3GDRTMTRhDbAKGwW9AamDxdemdGvQEtPNXfmMY+0qw8CPmu6bYjkHwLC5 ppkxwjFl2dznOdMKcCwh5EUk7PnVkOEHQKUId2Zlg7Eyrd0kGaOfBlADwR98iVHPa0++ 0G81ZRzRqQGOCOwMGUimSunCTovb2+6WEasCbFsVx0CWHax+OAcRAsDuWBKcoHWyKYms zaYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=u7Wr58qL9G5y1tzEQCokGfV3zERmxEUgDq6CeJFGweg=; b=eOgVK/nfIi6mo9DvFNN0MdvmI/YITTtC8niyFbUof7bm1LKYkMxAoF4AaQ2A70TeyN Vi3NSO8DImmMTBNu0+UPZmfS10G7M5AEzS0IrOokz+O0XizjDb4R/62bbofIZuZLWrjW 1UaRGOdP8Hh3pzrqZmeD9wpSNY9nxHTnsy+IlDWhFyhjnYfpVvUNfWUCSUR9x9HAhG+a C5NDeVgkqhMsM1QTwBiVZTBLB0JkCXMM/qMwXq99VpDmjtQz7wCIXS3vRBCVCNpcq+Mi /F5VowuK080OmxS+HyPmSMcuXgh7NSBUGeQjeXKZiUj90s6OWLBINF1GXWpcx3MDaUoF lq7g== X-Gm-Message-State: AD7BkJLe1pE1HB5XCz5EMLWU0hj0JJt7//GiqGvcOhesSHXlAwsdmLKwjw32VFSIqX6FYw== X-Received: by 10.66.65.201 with SMTP id z9mr24713561pas.143.1459453434458; Thu, 31 Mar 2016 12:43:54 -0700 (PDT) Received: from [10.192.166.0] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id b74sm15272370pfd.24.2016.03.31.12.43.53 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 31 Mar 2016 12:43:53 -0700 (PDT) Sender: Navdeep Parhar Subject: Re: OFED v3.2 update To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <56FCF309.1070406@selasky.org> From: Navdeep Parhar Message-ID: <56FD7DF8.7030702@FreeBSD.org> Date: Thu, 31 Mar 2016 12:43:52 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <56FCF309.1070406@selasky.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 19:43:55 -0000 On 03/31/2016 02:51, Hans Petter Selasky wrote: > Dear users of infiniband in FreeBSD: > > > Mellanox is working on a big infiniband update towards mellanox OFED > v3.2 of the infiniband stack (ibcore), drivers (mlx4/mlx5) and libraries > (librdma, libibverbs ...). Currently there are 10-patches in different > areas of the FreeBSD kernel and userspace which you need to apply to > 11-current as of today to get a working GENERIC kernel. Hello Hans, Can you provide the OFA/linux-rdma/k.org version that matches the verb APIs/KPIs in this import? Mellanox OFED doesn't come with iWARP or non-Mellanox drivers so we need another way to track those in FreeBSD. It would be nice to be able to say that the verbs in FreeBSD are API compatible with version of some upstream source that supports all transports and all drivers. I'm not sure if it's OFA's OFED or linux-rdma/k.org that's considered authoritative these days. > ... > Known issues: > - LINT build is broken (Will be fixed before committing. Waiting for > solution from Navdeep) This one involves taking toe_l2t_resolve out of toecore.ko and into GENERIC, because ibcore now wants to use it too (and it doesn't make sense to have ibcore depend on toecore). I'll probably rename it to l2t_resolve as part of the move. Regards, Navdeep From owner-freebsd-infiniband@freebsd.org Fri Apr 1 18:08:20 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D165BB00119 for ; Fri, 1 Apr 2016 18:08:20 +0000 (UTC) (envelope-from chrisr@spectralogic.com) Received: from mail1.bemta12.messagelabs.com (mail1.bemta12.messagelabs.com [216.82.251.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta12.messagelabs.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A69312AE for ; Fri, 1 Apr 2016 18:08:20 +0000 (UTC) (envelope-from chrisr@spectralogic.com) Received: from [216.82.250.19] by server-11.bemta-12.messagelabs.com id C9/65-16136-219BEF65; Fri, 01 Apr 2016 18:08:18 +0000 X-Env-Sender: chrisr@spectralogic.com X-Msg-Ref: server-15.tower-87.messagelabs.com!1459534097!28780792!1 X-Originating-IP: [192.30.190.20] X-StarScan-Received: X-StarScan-Version: 8.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 13548 invoked from network); 1 Apr 2016 18:08:17 -0000 Received: from outmx2.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.20) by server-15.tower-87.messagelabs.com with AES256-SHA encrypted SMTP; 1 Apr 2016 18:08:17 -0000 From: Chris Reeves To: "freebsd-infiniband@freebsd.org" Subject: Re: OFED v3.2 update Thread-Topic: OFED v3.2 update Thread-Index: AQHRizJ0DVyl9MlGEEKFmHSqP+/Yt591bHKM Date: Fri, 1 Apr 2016 18:08:16 +0000 Message-ID: <1459534095550.93717@spectralogic.com> References: <56FCF309.1070406@selasky.org> In-Reply-To: <56FCF309.1070406@selasky.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Apr 2016 18:08:20 -0000 Thank you Hans,=0A= =0A= Using these patches, I was able to get my IPOIB performance up to 25 Gb/s (= vs 10 Gb/s on 11-current without the patches).=0A= =0A= Out of curiosity, do you have any plans to MFC these to stable-10?=0A= =0A= Thanks,=0A= Chris=0A= ________________________________________=0A= From: owner-freebsd-infiniband@freebsd.org on behalf of Hans Petter Selasky =0A= Sent: Thursday, March 31, 2016 3:51 AM=0A= To: freebsd-infiniband@freebsd.org=0A= Subject: OFED v3.2 update=0A= =0A= Dear users of infiniband in FreeBSD:=0A= =0A= =0A= Mellanox is working on a big infiniband update towards mellanox OFED=0A= v3.2 of the infiniband stack (ibcore), drivers (mlx4/mlx5) and libraries=0A= (librdma, libibverbs ...). Currently there are 10-patches in different=0A= areas of the FreeBSD kernel and userspace which you need to apply to=0A= 11-current as of today to get a working GENERIC kernel.=0A= =0A= Compile world and kernel like this after applying the patches below:=0A= =0A= make buildkernel WITH_OFED=3DYES=0A= make buildworld WITH_OFED=3DYES=0A= =0A= =0A= https://reviews.freebsd.org/D5790=0A= https://reviews.freebsd.org/D5791=0A= https://reviews.freebsd.org/D5792=0A= https://reviews.freebsd.org/D5793=0A= https://reviews.freebsd.org/D5794=0A= https://reviews.freebsd.org/D5795=0A= https://reviews.freebsd.org/D5796=0A= https://reviews.freebsd.org/D5797=0A= https://reviews.freebsd.org/D5798=0A= https://reviews.freebsd.org/D5799=0A= =0A= =0A= The patches are work in progress, though at a level where basic=0A= infiniband is working with MLX5.=0A= =0A= =0A= Known issues:=0A= - LINT build is broken (Will be fixed before committing. Waiting for=0A= solution from Navdeep)=0A= - Need to unload ipoib before mlx5ib.=0A= =0A= --HPS=0A= _______________________________________________=0A= freebsd-infiniband@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband=0A= To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g"=0A= From owner-freebsd-infiniband@freebsd.org Sat Apr 2 09:09:34 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03C0AB00396 for ; Sat, 2 Apr 2016 09:09:34 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2E9D17E8 for ; Sat, 2 Apr 2016 09:09:33 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 49E4B1FE023 for ; Sat, 2 Apr 2016 11:09:25 +0200 (CEST) From: Hans Petter Selasky Subject: Re: OFED v3.2 update To: "freebsd-infiniband@freebsd.org" References: <56FCF309.1070406@selasky.org> <1459533825143.30617@spectralogic.com> Message-ID: <56FF8CFE.5000103@selasky.org> Date: Sat, 2 Apr 2016 11:12:30 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <1459533825143.30617@spectralogic.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Apr 2016 09:09:34 -0000 On 04/01/16 20:03, Chris Reeves wrote: > Thank you Hans, > > Using these patches, I was able to get my IPOIB performance up to 25 Gb/s (vs 10 Gb/s on 11-current without the patches). > > Out of curiosity, do you have any plans to MFC these to stable-10? > Hi, Not currently, MFC'ing needs to be discussed, because we are breaking some library and internal APIs, thinking about binary compatibility. If people using IB with FreeBSD think this is OK, we might just do it. --HPS