From owner-freebsd-arch@freebsd.org Tue May 30 11:12:43 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BBABB79C16 for ; Tue, 30 May 2017 11:12:43 +0000 (UTC) (envelope-from emeric.poupon@stormshield.eu) Received: from work.stormshield.eu (gwlille.netasq.com [91.212.116.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4F2B81783 for ; Tue, 30 May 2017 11:12:41 +0000 (UTC) (envelope-from emeric.poupon@stormshield.eu) Received: from work.stormshield.eu (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTPS id 706893760B1E; Tue, 30 May 2017 12:56:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTP id 630133760A0B; Tue, 30 May 2017 12:56:57 +0200 (CEST) Received: from work.stormshield.eu ([127.0.0.1]) by localhost (work.stormshield.eu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id WuoVfhcIi03v; Tue, 30 May 2017 12:56:57 +0200 (CEST) Received: from work.stormshield.eu (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTP id 4E680376068D; Tue, 30 May 2017 12:56:57 +0200 (CEST) Date: Tue, 30 May 2017 12:56:56 +0200 (CEST) From: Emeric POUPON To: Adrian Chadd Cc: freebsd-arch Message-ID: <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> In-Reply-To: References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> Subject: Re: numa and taskqueues MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Thread-Topic: numa and taskqueues Thread-Index: 1prrS86VmRWPDnLN63KY5Yag7cknDQ== X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2017 11:12:43 -0000 Hi, > Anyway - I think it'd be nice to have domain aware and pcpu aware > taskqueues so we can eventually migrate to a taskqueue group model of > "one top level things for net processing" for devices to share, etc, > etc. But for the short term just prototype it with some thin API in > crypto that wraps the taskqueue / kproc work so it gets done, then > push that work out for review/evaluation. if it does indeed work the > way you intend, we can try to use it as a template for a higher level, > shared taskqueue thing. It looks like it is somewhat mandatory to modify the taskqueue API to pin threads to the correct CPUs. The logic to define which CPU need to be set is another story that indeed can first be implemented in crypto(9). By the way: 1/ do you have some pointers on domain enumeration and other numa related code? 2/ about https://reviews.freebsd.org/D10680, I think it would be great to have this commited as a first step. Since it seems to be stuck, maybe I can add more people on this. Any suggestion? Emeric From owner-freebsd-arch@freebsd.org Tue May 30 14:46:32 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17436B7E9CB for ; Tue, 30 May 2017 14:46:32 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A00EC64C40 for ; Tue, 30 May 2017 14:46:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id d127so100013423wmf.0 for ; Tue, 30 May 2017 07:46:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=TeVkayfMb/qeYu4rLId9n20vvn72FhSFdvDcLwlXWzs=; b=iudwF1Jes6EH9hCMdVrq13DHD0RdQSSOgHZRBwz8yYS27CRPgsgiubU469kufje44b N/Kq/FfLvtH6Q8wdNfcBjWqdA9dtQfHCDXWRnCcidMrqC03j2efpduEpRbi3i4QMRGpq 27I9traUal4e9YNJxwyFgnvnPLTexBojS1y7z3ykqx0biAI6ug+agwo6QaRkWFROSbTP 91UC87qWMVFANh1+jhhAxGebAZYb5U+YLJq08EznqZ5Qu3qufqEKosX7DrWoN02Jnxxh 06QTgIje5SF1hq0xjqPdRdz5oTStM0USsIuQd3Rbdkbs5QBPrjND/RsqKKDIkBhgy4Q7 LW9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=TeVkayfMb/qeYu4rLId9n20vvn72FhSFdvDcLwlXWzs=; b=qkMzsjZGkKgcPEB8u4DWiUp0nNmZcRX1nY1BUeNrFDwwTBR5JgmzDIpsbwrMt9HFIe qxWaWr00SzVy5+0k/TwyjkGRGgRoNH6zPi9hefPZwQ+EURwLQftmhni80nwAkybT7Kc1 QD9ijqhNJAkDsc+XFRKcWgTD1lPNuuGBnn9yK4ms4N9Qk4sGdwKoOyrqpbCUy9+b+1Ak Y0NtOnmNJKcFKdzJcelNCoOPo0RRNgUenrF4l8D+JGOONLOlyjDtrk5sjioyqbcv9HBt 4+pLEfZszXPGtZQyv2kkcxz1ppq+ILO17E60X5NYqfx07eIYphYnAGV54jCFJIb+9zie bcTA== X-Gm-Message-State: AODbwcAZYpByO7ityBxi4pKIQgVJKxfenKKG6IcizH3lbKx/VhVjdmXH 4NgvDvneewvgMjmyzwY6PQJNucToFdkw X-Received: by 10.28.50.65 with SMTP id y62mr2140495wmy.5.1496155590104; Tue, 30 May 2017 07:46:30 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.28.193.134 with HTTP; Tue, 30 May 2017 07:46:29 -0700 (PDT) In-Reply-To: <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> From: Adrian Chadd Date: Tue, 30 May 2017 07:46:29 -0700 X-Google-Sender-Auth: a9CxfDR6fkNdmQK9VgDVY63UZvs Message-ID: Subject: Re: numa and taskqueues To: Emeric POUPON Cc: freebsd-arch Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2017 14:46:32 -0000 On 30 May 2017 at 03:56, Emeric POUPON wrote: > Hi, > >> Anyway - I think it'd be nice to have domain aware and pcpu aware >> taskqueues so we can eventually migrate to a taskqueue group model of >> "one top level things for net processing" for devices to share, etc, >> etc. But for the short term just prototype it with some thin API in >> crypto that wraps the taskqueue / kproc work so it gets done, then >> push that work out for review/evaluation. if it does indeed work the >> way you intend, we can try to use it as a template for a higher level, >> shared taskqueue thing. > > It looks like it is somewhat mandatory to modify the taskqueue API to pin threads to the > correct CPUs. The logic to define which CPU need to be set is another story that indeed can first > be implemented in crypto(9). > > By the way: > 1/ do you have some pointers on domain enumeration and other numa related code? Sorry, I'm a bit too busy with other things to dive in right now :( > 2/ about https://reviews.freebsd.org/D10680, I think it would be great to have this commited as a first step. > Since it seems to be stuck, maybe I can add more people on this. Any suggestion? Well, what's with the ~ 8% performance decrease? Do you know what's going on? For a "we're parallelising IPSEC operations", seeing it get slower with more flows is a bit concerning. Thanks, -adrian From owner-freebsd-arch@freebsd.org Tue May 30 14:54:10 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 296FAB7ECCB for ; Tue, 30 May 2017 14:54:10 +0000 (UTC) (envelope-from emeric.poupon@stormshield.eu) Received: from work.stormshield.eu (gwlille.netasq.com [91.212.116.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E359F6528A; Tue, 30 May 2017 14:54:09 +0000 (UTC) (envelope-from emeric.poupon@stormshield.eu) Received: from work.stormshield.eu (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTPS id B1AB9376179E; Tue, 30 May 2017 16:46:01 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTP id 927043761794; Tue, 30 May 2017 16:46:01 +0200 (CEST) Received: from work.stormshield.eu ([127.0.0.1]) by localhost (work.stormshield.eu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id WWe7UUK49nrL; Tue, 30 May 2017 16:46:01 +0200 (CEST) Received: from work.stormshield.eu (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTP id 7A2CC376176C; Tue, 30 May 2017 16:46:01 +0200 (CEST) Date: Tue, 30 May 2017 16:46:01 +0200 (CEST) From: Emeric POUPON To: Adrian Chadd Cc: freebsd-arch Message-ID: <608664209.55736023.1496155561181.JavaMail.zimbra@stormshield.eu> In-Reply-To: References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> Subject: Re: numa and taskqueues MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Thread-Topic: numa and taskqueues Thread-Index: aT1hZYkNVS61k+uN8YKtzmebjUGFoQ== X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2017 14:54:10 -0000 Hi, > >> 2/ about https://reviews.freebsd.org/D10680, I think it would be great to have >> this commited as a first step. >> Since it seems to be stuck, maybe I can add more people on this. Any suggestion? > > Well, what's with the ~ 8% performance decrease? Do you know what's > going on? For a "we're parallelising IPSEC operations", seeing it get > slower with more flows is a bit concerning. > > Thanks, > Actually, there is a performance boost only when few flows are involved. That's why this is not activated by default and a sysctl is here to enable the feature. To sum up, the more different flows you process (both ciphered and unciphered), the more network queues are hit and the more CPU units are triggered from ipsec. In this case, we indeed notice a loss, certainly due to the extra queing/reordering performed. From owner-freebsd-arch@freebsd.org Tue May 30 18:26:28 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EFD6BBD30E9 for ; Tue, 30 May 2017 18:26:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wr0-x232.google.com (mail-wr0-x232.google.com [IPv6:2a00:1450:400c:c0c::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CA7374534 for ; Tue, 30 May 2017 18:26:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wr0-x232.google.com with SMTP id j27so5424310wre.3 for ; Tue, 30 May 2017 11:26:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=h4PLh3mB1GIPT+AfW3Q2JNhcM0+Wzp24xu/4QyrzUYk=; b=mhnvJrpVdVApuaVU/54WDvvJFhGADyqGcZ+Al0Kux++v+VWbaf623gHcyzTvQmeshD Q6f6XKW4dvVOW1/6dDiXAU6MimASvxAh5Tf+CA5NxZcMsJcwH4viB2GIdBZ/fpy/njwa i6P/Ci74KA8D2qRaYHhPjn9QTQa+QhMOhFretGywoz5Bs7Buu9lizw19HowZ1E+YYhzI EAN8ZFh1D+DM4HLEAeT8h1t4YmjEr97kp33RFaPCnSUx63iL710Nb/Dg2L3/TZ8+z21z YtHAP0pZMtvrIpMK+NPZ1G4dqbAMfHfTx0DuRQg0UzmoGFePyHYe7IT/QZG3pZg+39cf GN6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=h4PLh3mB1GIPT+AfW3Q2JNhcM0+Wzp24xu/4QyrzUYk=; b=tcVF9VY9yDs20PNfa83mGGoJdWeEV+eHK/ROgoQFb2bivr2XNgZ8t03E4vD+L57Z8b FT8o3ht0GTEuvs+OqRkJ3vW+1jmJnS0pEnUOHlXSp/b2o2LC//H4Ao2V11KsGAOYlvsr ejsSmDjqpkxTJChOltLccWWaRNDWq2i54V8jFThQ+3fK0EMuRVAbj/BnNydh5J7UNPnZ vnZXY3whwaPJcuqr8mhT3N7Lw9dE23+GoXxB/kuBELJ8kwf3evDxoKn7YYP2HVG/tk5F Huc5NUiDCDAW0lekvrXPnGnyn9d7vfA4UK1SxchFEWnjqJgb+2KAa0MwxQSQUiO7pUiO /t2Q== X-Gm-Message-State: AODbwcBa4X8Ff310Dj57+0GTjQTYHauDKCbmwpTwXCLYpsiulUS30dL8 hDAckTKNgxXdKNqjfgvAw0luj1LC3Msg X-Received: by 10.223.134.80 with SMTP id 16mr18272180wrw.62.1496168786997; Tue, 30 May 2017 11:26:26 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.28.193.134 with HTTP; Tue, 30 May 2017 11:26:26 -0700 (PDT) In-Reply-To: <608664209.55736023.1496155561181.JavaMail.zimbra@stormshield.eu> References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> <608664209.55736023.1496155561181.JavaMail.zimbra@stormshield.eu> From: Adrian Chadd Date: Tue, 30 May 2017 11:26:26 -0700 X-Google-Sender-Auth: xUCwwL6UACISWRMREXYcr77W0G4 Message-ID: Subject: Re: numa and taskqueues To: Emeric POUPON Cc: freebsd-arch Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2017 18:26:29 -0000 On 30 May 2017 at 07:46, Emeric POUPON wrote: > Hi, > >> >>> 2/ about https://reviews.freebsd.org/D10680, I think it would be great to have >>> this commited as a first step. >>> Since it seems to be stuck, maybe I can add more people on this. Any suggestion? >> >> Well, what's with the ~ 8% performance decrease? Do you know what's >> going on? For a "we're parallelising IPSEC operations", seeing it get >> slower with more flows is a bit concerning. >> >> Thanks, >> > > Actually, there is a performance boost only when few flows are involved. > That's why this is not activated by default and a sysctl is here to enable the feature. > > To sum up, the more different flows you process (both ciphered and unciphered), the more network queues are hit and the more CPU units are triggered from ipsec. > In this case, we indeed notice a loss, certainly due to the extra queing/reordering performed. Can you dig into that a bit more? Do you know exactly what's going on? eg, is it a "lock contention" problem? Is it a "stuff is context switching, thus latency" problem? etc, etc. -adrian From owner-freebsd-arch@freebsd.org Wed May 31 14:01:41 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 636D9B7AC03 for ; Wed, 31 May 2017 14:01:41 +0000 (UTC) (envelope-from emeric.poupon@stormshield.eu) Received: from work.stormshield.eu (gwlille.netasq.com [91.212.116.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27D3F7B5BF; Wed, 31 May 2017 14:01:40 +0000 (UTC) (envelope-from emeric.poupon@stormshield.eu) Received: from work.stormshield.eu (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTPS id 44AE33760FD6; Wed, 31 May 2017 15:53:22 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTP id 37D733760FC3; Wed, 31 May 2017 15:53:22 +0200 (CEST) Received: from work.stormshield.eu ([127.0.0.1]) by localhost (work.stormshield.eu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id bzkMO1xsBf6B; Wed, 31 May 2017 15:53:22 +0200 (CEST) Received: from work.stormshield.eu (localhost [127.0.0.1]) by work.stormshield.eu (Postfix) with ESMTP id 24F013760AC5; Wed, 31 May 2017 15:53:22 +0200 (CEST) Date: Wed, 31 May 2017 15:53:21 +0200 (CEST) From: Emeric POUPON To: Adrian Chadd Cc: freebsd-arch Message-ID: <1349284176.55940289.1496238801718.JavaMail.zimbra@stormshield.eu> In-Reply-To: References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> <608664209.55736023.1496155561181.JavaMail.zimbra@stormshield.eu> Subject: Re: numa and taskqueues MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Thread-Topic: numa and taskqueues Thread-Index: DCTgg8GnnVgqgfJ6UUtMgfgS/ihRrg== X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 May 2017 14:01:41 -0000 >> >> Actually, there is a performance boost only when few flows are involved. >> That's why this is not activated by default and a sysctl is here to enable the >> feature. >> >> To sum up, the more different flows you process (both ciphered and unciphered), >> the more network queues are hit and the more CPU units are triggered from >> ipsec. >> In this case, we indeed notice a loss, certainly due to the extra >> queing/reordering performed. > > Can you dig into that a bit more? Do you know exactly what's going on? > eg, is it a "lock contention" problem? Is it a "stuff is context > switching, thus latency" problem? etc, etc. > Unfortunately I cannot tell you the exact reason right now. I am sure there is no lock contention involved though (except of course when several domains are involved). Did you expect such a dev to be enabled by default? Emeric From owner-freebsd-arch@freebsd.org Wed May 31 17:30:45 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47E0AB7F0BE for ; Wed, 31 May 2017 17:30:45 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D91CB84723 for ; Wed, 31 May 2017 17:30:44 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wm0-x231.google.com with SMTP id d127so28713754wmf.0 for ; Wed, 31 May 2017 10:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=ZDaJROj7Q6vCxStdTYfntp++ScQAaRv4YO9N0zU/6Lc=; b=jJ2TabaR419/4gvk1ijSb2iDVmQrXhYMBHjCv666z+bJ2P9FoHxNu7Ugyy24IRMmAV QpiH0z1zmEmE1ZgpCRCsQzEseJxudaODkGj+EC8FeLms5VTBcXwiyJXVb0SBfT324qD3 yNOvvpQoiUM1xUmZjhiPFeISxBb/3KahFxIieyHqoCxnSZMnL+yiF5cETXMnMnxsQxFD fV+fPgGin3n9oDaGsMSCBS0PLIzLd61on3nm8aezIV0aInPK3YQ6fd05Zv7ywp5q90KV Cy2uDKyocCdB1kEk3YUI56LsFQtjBlrt9cHltWFxjRUh5b4crobSZF34cpRxo/rAA2QF k9fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=ZDaJROj7Q6vCxStdTYfntp++ScQAaRv4YO9N0zU/6Lc=; b=mosEFttkKHMV5awU6BWLipZiTzvInCJfn0sAc0b0EliFUMUN5XXdtuQVdyM0Ek7kca LsRx6MSB+lJluGmM/Xpfwt4PUi7zJ7NdQRd2do6Dz7hyAlhB39nnuS/MvLqU8LpDWmdJ q6qVwnLVvJYz1Knr6yIkEronDgq8yMYZ/RfJUGwoSVHqDyWBeEBWjkUovSva+Krl0obl 1qUCEboubzTv5Q+BxHhB2O2PnwA3StR43HuqxZh+gP7bxQa8+5oXjT+yMHNpqiKDEQUs pq9ZVyrLoFuh1oowRtppjaGPJOsvkkdGe8twui6NpcUUrXNQ0ILmqpWs2CsAUiC9rQdw 47dA== X-Gm-Message-State: AODbwcCIIQ2kL88j1jdNaJLV4eU3oHyqvMNrdX8sUj50Dw995p2EMqzm XFEV+2hBltUw4GgKfBu8q/3DyjqjUqFD X-Received: by 10.28.31.135 with SMTP id f129mr6689500wmf.78.1496251842634; Wed, 31 May 2017 10:30:42 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.28.193.134 with HTTP; Wed, 31 May 2017 10:30:41 -0700 (PDT) In-Reply-To: <1349284176.55940289.1496238801718.JavaMail.zimbra@stormshield.eu> References: <1914359731.54283525.1495178031163.JavaMail.zimbra@stormshield.eu> <816581118.55670987.1496141816904.JavaMail.zimbra@stormshield.eu> <608664209.55736023.1496155561181.JavaMail.zimbra@stormshield.eu> <1349284176.55940289.1496238801718.JavaMail.zimbra@stormshield.eu> From: Adrian Chadd Date: Wed, 31 May 2017 10:30:41 -0700 X-Google-Sender-Auth: AHapgUrBolrcdiTiUEzkjvVkC_g Message-ID: Subject: Re: numa and taskqueues To: Emeric POUPON Cc: freebsd-arch Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 May 2017 17:30:45 -0000 On 31 May 2017 at 06:53, Emeric POUPON wrote: > >>> >>> Actually, there is a performance boost only when few flows are involved. >>> That's why this is not activated by default and a sysctl is here to enable the >>> feature. >>> >>> To sum up, the more different flows you process (both ciphered and unciphered), >>> the more network queues are hit and the more CPU units are triggered from >>> ipsec. >>> In this case, we indeed notice a loss, certainly due to the extra >>> queing/reordering performed. >> >> Can you dig into that a bit more? Do you know exactly what's going on? >> eg, is it a "lock contention" problem? Is it a "stuff is context >> switching, thus latency" problem? etc, etc. >> > > Unfortunately I cannot tell you the exact reason right now. > I am sure there is no lock contention involved though (except of course when several domains are involved). > Did you expect such a dev to be enabled by default? Well, I'd really like to get to the bottom of these. :-P -adrian