Date: Sun, 21 Aug 2016 10:33:05 -0700 From: Adrian Chadd <adrian@freebsd.org> To: "Robert N. M. Watson" <rwatson@freebsd.org> Cc: "Andrey V. Elsukov" <ae@freebsd.org>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: svn commit: r304313 - head/sys/net Message-ID: <CAJ-VmokrNfoJNBoa7iUv0FhQyajAZz2BM7VzaGDu3KVUTmCWUA@mail.gmail.com> In-Reply-To: <28FA9F29-FA29-4547-875D-0734DA120636@FreeBSD.org> References: <201608172021.u7HKLXJ4001584@repo.freebsd.org> <CAJ-Vmonu3ZSKZtZZMa68FAS9nDdiriE3iv81cjoW%2Bk3mHKO2=g@mail.gmail.com> <28FA9F29-FA29-4547-875D-0734DA120636@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 21 August 2016 at 07:42, Robert N. M. Watson <rwatson@freebsd.org> wrote= : > On 20 Aug 2016, at 21:27, Adrian Chadd <adrian@freebsd.org> wrote: > >> I wonder if the right-er thing to do here is to allow the cpuid to be >> whatever it needs to be, but limit the cpuid lookups when it resolves >> to a netisr array. >> >> that'd mean the hybrid model would still return the current CPU up to >> the max CPU id, but anything trying to queue into a netisr would not >> overflow the netisr queue count. > > > I think some refinement of model is required. The original intent was tha= t =E2=80=9Cworkstreams=E2=80=9D represented ordered sets of queues being de= livered for deferred dispatch across the set of CPUs (hence =E2=80=9Cnws=E2= =80=9D being allocated using DPCPU). When performing direct dispatch, as a = matter of convenience, we also bill work performed on a CPU to its workstre= am: > > 1109 /* > 1110 * If direct dispatch is forced, then unconditionally dispa= tch > 1111 * without a formal CPU selection. Borrow the current CPU'= s stats, > 1112 * even if there's no worker on it. In this case we don't = update > 1113 * nws_flags because all netisr processing will be source o= rdered due > 1114 * to always being forced to directly dispatch. > 1115 */ > > In hybrid mode, the world is a little different, because we are willing t= o direct dispatch hybrid work *only* if the worker thread isn=E2=80=99t alr= eady processing the workstream: > > 1147 /*- > 1148 * We are willing to direct dispatch only if three conditio= ns hold: > 1149 * > 1150 * (1) The netisr worker isn't already running, > 1151 * (2) Another thread isn't already directly dispatching, a= nd > 1152 * (3) The netisr hasn't already been woken up. > 1153 */ > > This is important because if the workstream already has a backlog of work= , but the thread isn=E2=80=99t yet running, we must enqueue the work to ens= ure it remains ordered with respect to other work in the workstream. > > Assuming we wish to retain the current workstream model, then we need to = adapt the code a bit to handle the case where we still have one workstream = per CPU, but where we are only willing to perform deferred dispatch (or hyb= rid dispatch) to a CPU that has a worker thread. We would then bill to any = CPU=E2=80=99s workstream for true direct dispatch, but for hybrid dispatch,= continue to always bill a CPU that has a worker thread to which we were wi= lling to assign the work. > > I.e., as I think you=E2=80=99re suggesting, we probably need to tweak the= functions slightly to differentiate between =E2=80=9Cyou can run it here a= nd must use curcpu=E2=80=99s workstream=E2=80=9D and =E2=80=9Cyou can run i= t here but must use a specific CPU=E2=80=99s workstream=E2=80=9D =E2=80=94 = the former being true direct dispatch, and the latter being hybrid dispatch= where the netisr in question isn=E2=80=99t already running, but we bill it= to that workstream/netisr even though we run it on the current CPU. > > Does this make sense? Right. Let me go and look into it a little more. I think we may want to revert the change (which just landed to -11, so maybe revert that too) so I can test both of them out for correctness. Andrey, I'm sorry for suggesting we back it out, but I'd like to make sure we don't break networking on 11. :) Is that okay? I will look at this tonight/tomorrow. -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmokrNfoJNBoa7iUv0FhQyajAZz2BM7VzaGDu3KVUTmCWUA>