From owner-freebsd-infiniband@freebsd.org Thu Mar 4 19:11:20 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 897A0555FDA for ; Thu, 4 Mar 2021 19:11:20 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ds0pH3hd8z4jsc for ; Thu, 4 Mar 2021 19:11:19 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 49448260100; Thu, 4 Mar 2021 20:11:17 +0100 (CET) Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 To: "Pokala, Ravi" , "Foster, Greg" , "freebsd-infiniband@freebsd.org" References: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> From: Hans Petter Selasky Message-ID: <71b1d84d-917e-3f6a-cad6-794f06fefaee@selasky.org> Date: Thu, 4 Mar 2021 20:10:59 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Ds0pH3hd8z4jsc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[88.99.82.50:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[88.99.82.50:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 19:11:20 -0000 On 3/4/21 8:00 PM, Pokala, Ravi wrote: > -----Original Message----- > From: Hans Petter Selasky > Date: 2021-03-04, Thursday at 02:13 > To: "Foster, Greg" , "freebsd-infiniband@freebsd.org" > Cc: Ravi Pokala > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 > > On 3/4/21 3:49 AM, Foster, Greg wrote: > > Hello FreeBSD IB, > > > > What is the procedure to change between Infiniband Connected Mode and Datagram Mode > > at runtime on FreeBSD 12.2? > > > > I've seen different references on the WEB, but the procedure still seems unclear. > > > > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html > > refers to the Linux procedure, which is not applicable on FreeBSD. > > > > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled > > to change modes. > > > > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html > > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details... > > > > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around > > and loading the desired one? > > Hi, > > For IPoIB there is an option you need to set at compile time for this > feature to be activated: > > sys/conf/options:IPOIB_CM opt_ofed.h > > --HPS > > Hi Hans, > > It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-( > > But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes? > Hi, At the moment there is no tunable for this feature, as I'm aware of. The easiest would be to build ipoib two times, one time with CM enabled and one time without CM enabled. --HPS