Date: Mon, 9 Jun 2014 13:54:12 -0400 From: Adrian Chadd <adrian@freebsd.org> To: Hooman Fazaeli <hoomanfazaeli@gmail.com> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Mark Delany <n7w@delta.emu.st> Subject: Re: Best practice for accepting TCP connections on multicore? Message-ID: <CAJ-Vmon26CoXwj2L_MOHgTYJgLSDP_0E%2B64FG_6eow2ErVe_0w@mail.gmail.com> In-Reply-To: <5395EAF6.4020505@gmail.com> References: <CAAGHsvDhaqQbwir5P%2BoaH_Qa8VZ0aj9A2SGrn%2B2shJMQ21B6Jw@mail.gmail.com> <CAJ-Vmo=eJ4o2j=3kUOjR=HOMDOR5pD2HcXvxm=dYjkaB9bj8EQ@mail.gmail.com> <20140607010151.51751.qmail@f5-external.bushwire.net> <CAJ-Vmon9Q%2BvkRRw7y9wTg5Rvn=zU-9j%2Bv5vtmc_EY8oncacUdw@mail.gmail.com> <5395EAF6.4020505@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
freebsd-rss is not freebsd; it's just the evaluation code Ive' written. 'master' is the branch there. -a On 9 June 2014 13:12, Hooman Fazaeli <hoomanfazaeli@gmail.com> wrote: > On 6/7/2014 6:32 AM, Adrian Chadd wrote: >> >> Hi, >> >> In the RSS model I'm hacking on: >> >> * you query the kernel for the current RSS bucket -> CPU mapping. >> Right now it's one bucket -> one CPU but at some point it may be one >> bucket -> cpuset; >> * you spawn a thread for each RSS bucket; >> * you pin each thread to the RSS bucket current CPU id; >> * you create a listen socket in that thread, marked as BINDMULTI (ie, >> multiple things can listen on this and the kernel will load balance >> between them) and RSS_BUCKET (ie, please place this socket in the >> given RSS bucket, rather than the global/wildcard list); >> * then when a connection comes in, the kernel will first do a lookup >> for a matching wildcard socket in the per-RSS PCBGROUP, rather than >> the global wildcard table; >> * if it finds it, that socket gets the incoming connection. >> >> At some point I'll add some notification via kqueue or what not that >> the RSS buckets need rebalancing, and userland can then re-pin the >> per-bucket threads. >> >> At the moment the hacks I've done only support one listen socket per >> entry. My hope is that BINDMULTI will do some basic hash to load >> balance within a set of matching PCB entries - and it'll be combined, >> so if you do BINDMULTI without RSS, it'll just load balance between >> multiple sockets with no CPU affinity knowledge. If you do RSS, it'll >> distribute only CPU-local requests to a thread that's sitting in the >> right RSS bucket. if you enable both, you can use a thread pool for >> each RSS bucket CPU and (eventually, when I write it) it'll load >> balance among those. >> >> But for now I'm assuming one incoming thread per RSS bucket will be >> enough for people to experiment with. >> >> anyway, I guess I should email out the details: >> >> * http://github.com/erikarn/freebsd - the 'local/rss' branch has the >> RSS changes to dev/e1000/if_igb.c and netinet/ > > would you please point to the exact URL for the branch? > >> * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look >> at rss-http. >> >> I haven't yet tested this at > 1GE because all I have at home are igb >> and em NICs. If someone would like to donate ixgbe and T4 hardware, >> i'll gratefully take it and do up RSS patches for those drivers. >> >> > > -- > > Best regards. > Hooman Fazaeli >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmon26CoXwj2L_MOHgTYJgLSDP_0E%2B64FG_6eow2ErVe_0w>