Date: Fri, 6 Jun 2014 19:02:02 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Mark Delany <n7w@delta.emu.st> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Best practice for accepting TCP connections on multicore? Message-ID: <CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw@mail.gmail.com> In-Reply-To: <20140607010151.51751.qmail@f5-external.bushwire.net> References: <CAAGHsvDhaqQbwir5P%2BoaH_Qa8VZ0aj9A2SGrn%2B2shJMQ21B6Jw@mail.gmail.com> <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com> <20140607010151.51751.qmail@f5-external.bushwire.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, In the RSS model I'm hacking on: * you query the kernel for the current RSS bucket -> CPU mapping. Right now it's one bucket -> one CPU but at some point it may be one bucket -> cpuset; * you spawn a thread for each RSS bucket; * you pin each thread to the RSS bucket current CPU id; * you create a listen socket in that thread, marked as BINDMULTI (ie, multiple things can listen on this and the kernel will load balance between them) and RSS_BUCKET (ie, please place this socket in the given RSS bucket, rather than the global/wildcard list); * then when a connection comes in, the kernel will first do a lookup for a matching wildcard socket in the per-RSS PCBGROUP, rather than the global wildcard table; * if it finds it, that socket gets the incoming connection. At some point I'll add some notification via kqueue or what not that the RSS buckets need rebalancing, and userland can then re-pin the per-bucket threads. At the moment the hacks I've done only support one listen socket per entry. My hope is that BINDMULTI will do some basic hash to load balance within a set of matching PCB entries - and it'll be combined, so if you do BINDMULTI without RSS, it'll just load balance between multiple sockets with no CPU affinity knowledge. If you do RSS, it'll distribute only CPU-local requests to a thread that's sitting in the right RSS bucket. if you enable both, you can use a thread pool for each RSS bucket CPU and (eventually, when I write it) it'll load balance among those. But for now I'm assuming one incoming thread per RSS bucket will be enough for people to experiment with. anyway, I guess I should email out the details: * http://github.com/erikarn/freebsd - the 'local/rss' branch has the RSS changes to dev/e1000/if_igb.c and netinet/ * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look at rss-http. I haven't yet tested this at > 1GE because all I have at home are igb and em NICs. If someone would like to donate ixgbe and T4 hardware, i'll gratefully take it and do up RSS patches for those drivers. -a On 6 June 2014 18:01, Mark Delany <n7w@delta.emu.st> wrote: > On 06Jun14, Adrian Chadd allegedly wrote: >> Hi, >> >> I'm working on exactly this with the RSS support that's in the tree. >> >> The vague design I have in mind is to have one bound thread per RSS >> allocated CPU, and then put a listen TCP (and later UDP) socket in >> each thread. > > The most interesting aspect of this is how you notify a specific > thread that it should call accept(). > > What would be nice is if the listen socket only returns readable (via > select/poll/kqueue) on a CPUSET mask. This allows a thread to bind to > a cpu then only accept sockets for interfaces best suited to that CPU. > > A simpler alternative might be to live with the listener-thread model > that accepts then farms out the fd, but have a syscall that asks the > kernel if there is a preferred CPU for a given socket. Then the > listener-thread could use that as a hint to farm out the fd to a > CPU-bound thread, something like: > > for (int bindCPU=0; bindCPU < maxCPU; ++bindCPU) { > pool[i] = createThreadPool(bindCPU); > } > > while (select() == 1) { > int newfd = accept(); > int preferredCPU = syscall_preferredCPU(newfd); /* New fancy syscall */ > > if (preferredCPU == -1) { /* If no preference, randomize */ > preferredCPU = randomInRange(0, maxCPU-1); > } > > pool[preferredCPU]->queueAndNotify(newfd); > } > > If it hasn't already, at the point of calling syscall_preferredCPU() > the kernel could make some decisions about binding CPU preferences to > the stack for that socket. > > It would be up to user space to distribute work to the right CPU bound > thread - it can choose not to if its pool is unbalanced. > > > Mark. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw>