From owner-svn-src-head@freebsd.org Thu Sep 17 20:21:42 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 030EB3ECDA0; Thu, 17 Sep 2020 20:21:42 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BspK15Lw8z4Ywb; Thu, 17 Sep 2020 20:21:41 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-274.local (unknown [IPv6:2601:648:8681:1cb0:a135:cfa3:dd33:fe95]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 06C0F288EA; Thu, 17 Sep 2020 20:21:40 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: svn commit: r365449 - head/sbin/rcorder To: =?UTF-8?Q?Olivier_Cochard-Labb=c3=a9?= , "Andrey V. Elsukov" Cc: src-committers , svn-src-all , svn-src-head References: <202009081036.088AaCk8085096@repo.freebsd.org> From: John Baldwin Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: <1ccf2cf7-eb01-a200-4335-4674a010f8a1@FreeBSD.org> Date: Thu, 17 Sep 2020 13:21:39 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Sep 2020 20:21:42 -0000 On 9/17/20 10:49 AM, Olivier Cochard-Labbé wrote: > On Tue, Sep 8, 2020 at 12:36 PM Andrey V. Elsukov wrote: > >> Author: ae >> Date: Tue Sep 8 10:36:11 2020 >> New Revision: 365449 >> URL: https://svnweb.freebsd.org/changeset/base/365449 >> >> Log: >> Add a few features to rcorder: >> >> >> > Hi Andrey, > > I've spent some time bisecting an IPSec gateway performance regression on > -head that points to this commit. > > So my bench uses a simple static IPSec aes-gcm-16 setup, and their results > vary a lot between those 2 revisions: > > - r365448: Estimated Equilibrium Ethernet throughput= 1413 Mb/s (maximum > value seen: 1419 Mb/s) > - r365449: Estimated Equilibrium Ethernet throughput= 469 Mb/s (maximum > value seen: 469 Mb/s) > > How could a modification to the rcoder be the source cause of that ? > My rc.conf contains this: > kld_list="aesni" > ipsec_enable="YES" > > So My first thought was that the aesni module wasn't loaded anymore: > Before upgrade: > [root@hp]~# uname -a > FreeBSD hp 13.0-CURRENT FreeBSD 13.0-CURRENT #15 r365448M: Thu Sep 17 > 18:17:58 CEST 2020 olivier@lame4.bsdrp.net:/usr/src/amd64.amd64/sys/amd64 > amd64 > [root@hp]~# kldstat > Id Refs Address Size Name > 1 11 0xffffffff80200000 1ee58b0 kernel > 2 1 0xffffffff82319000 34c8 fdescfs.ko > 3 1 0xffffffff8231d000 a240 aesni.ko > 4 1 0xffffffff82328000 8c98 ioat.ko > 5 1 0xffffffff82331000 e350 ipsec.ko > > Then after upgrade: > > [root@hp]~# uname -a > FreeBSD hp 13.0-CURRENT FreeBSD 13.0-CURRENT #14 r365449M: Thu Sep 17 > 17:01:35 CEST 2020 olivier@lame4.bsdrp.net:/usr/src/amd64.amd64/sys/amd64 > amd64 > [root@hp]~# kldstat > Id Refs Address Size Name > 1 11 0xffffffff80200000 1ee58b0 kernel > 2 1 0xffffffff82319000 34c8 fdescfs.ko > 3 1 0xffffffff8231d000 e350 ipsec.ko > 4 1 0xffffffff8232c000 a240 aesni.ko > 5 1 0xffffffff82337000 8c98 ioat.ko > > => aesni.ko is correctly loaded, so it is not the problem, but notice the > order of the kernel modules that have changed. > Could be this the source of the problem ? Let's try: > > [root@hp]~# service ipsec stop > Clearing ipsec manual keys/policies. > [root@hp]~# kld > kldconfig kldload kldstat kldunload kldxref > [root@hp]~# kldunload ioat > [root@hp]~# kldunload aesni > [root@hp]~# kldunload ipsec > [root@hp]~# kldload aesni > [root@hp]~# kldload ipsec > [root@hp]~# service ipsec start > Installing ipsec manual keys/policies. > [root@hp]~# kldstat > Id Refs Address Size Name > 1 11 0xffffffff80200000 1ee58b0 kernel > 2 1 0xffffffff82319000 34c8 fdescfs.ko > 5 1 0xffffffff82337000 8c98 ioat.ko > 6 1 0xffffffff8231d000 a240 aesni.ko > 7 1 0xffffffff82328000 e350 ipsec.ko > > And after that the IPSec bench results are back to their previous value :-) > So rcorder needs to load aesni before ipsec. I don't think the issue is with rcorder though. I think the reason the ordering matters warrants further investigation. Is aesni not getting used when ipsec is loaded first? You can use dtrace with the script at https://github.com/bsdjhb/kdbg/blob/master/dtrace/crypto_drivers.d to see which driver is being used. Hmm, the crypto driver gets selected when keys for SAs are set, so if /etc/rc.d/ipsec is configuring SAs, then having it run before aesni is loaded would indeed cause this. However, that still isn't an issue with this commit, per se, it just exposed the lack of an explicit ordering requirement of /etc/rc.d/ipsec after loading aesni. The problem though is that kld_list is pretty generic, so it's hard to know when /etc/rc.d/kld should run. Possibly it should even run multiple times where subsequent runs try to load any modules not loaded by the previous runs? I suspect btw that you could just do 'sh /etc/rc.d/ipsec restart' post-boot without unloading any modules which would also fix your benchmark. I think long term we want OCF's notions of sessions to be a bit more fluid such that "client" sessions for things like GELI and IPSec can be backed by one or more "driver" sessions (including "driver" sessions coming and going as devices come and go). That's a fair bit more work however. -- John Baldwin