Date: Fri, 30 Apr 2021 16:32:52 -0400 From: Mark Johnston <markj@freebsd.org> To: =?iso-8859-1?Q?=D6zkan?= KIRIK <ozkan.kirik@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: IPsec performace - netisr hits %100 Message-ID: <YIxpdL9b6v8%2BN%2BLg@nuc> In-Reply-To: <CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug@mail.gmail.com> References: <CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 30, 2021 at 11:11:48PM +0300, Özkan KIRIK wrote:
> Hello,
>
> I'm using FreeBSD stable/12 built world on 12 April 2021.
> my setup is:
> [freebsd host cc0] <--------> [cc1 - same freebsd, but jail]
>
> without IPsec, I can achieve easily to 20Gbps. (test was run with different
> source IPs using multiple iperf to scale across multiple queues)
> My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
> T62100-LP-CR.
>
> But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
> netisr thread hits %100 cpu.
> with aesni throughput is 1,4 Gbps
> with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
> with CCR throughput is 2,0 Gbps (t6nex0)
> But always bottleneck is netisr.
>
> Is there any way to workaround this netisr bottleneck ?
> I tried to switch net.isr.dispatch to deferred and hybrid, but performance
> drops a bit.
I can suggest a couple of things to try. First, we configure only one
netisr thread by default. You can create more by setting the
net.isr.maxthreads loader tunable. I believe netipsec selects a thread
using the SPI so adding more threads might not help much depending on
your configuration, but testing with e.g., maxthreads = ncpu could be
illuminating.
Second, netipsec unconditionally hands rx processing off to netisr
threads for some reason, that's why changing the dispatch policy doesn't
help. Maybe it's to help avoid running out of kernel stack space or to
somehow avoid packet reordering in some case that is not clear to me. I
tried a patch (see below) which eliminates this and it helped somewhat.
If anyone can provide an explanation for the current behaviour I'd
appreciate it.
Could you try both approaches and report back? It would also be
interesting to know how your results compare with 13.0, if possible.
commit 618ab87449d412a74bfee4932d84a6fc17afce6c
Author: Mark Johnston <markj@FreeBSD.org>
Date: Thu Jan 7 11:29:14 2021 -0500
netipsec: Avoid deferred dispatch on the input path
diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c
index 48acba68a1fe..98d0954c4c53 100644
--- a/sys/netipsec/ipsec_input.c
+++ b/sys/netipsec/ipsec_input.c
@@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, m);
+ error = netisr_dispatch_src(isr_prot, (uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {
IPSEC_ISTAT(sproto, qfull);
@@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot,
+ error = netisr_dispatch_src(isr_prot,
(uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YIxpdL9b6v8%2BN%2BLg>