Date: Tue, 24 Apr 2012 18:18:29 +0200 From: "K. Macy" <kmacy@freebsd.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "Li, Qing" <qing.li@bluecoat.com>, "current@freebsd.org" <current@freebsd.org>, "net@freebsd.org" <net@freebsd.org> Subject: Re: Some performance measurements on the FreeBSD network stack Message-ID: <CAHM0Q_MLRVX09_jeSXmCY0=z5D%2BpfZgwqt1UGM%2BhVRU-=-Bbbw@mail.gmail.com> In-Reply-To: <20120424163423.GA59530@onelab2.iet.unipi.it> References: <20120419133018.GA91364@onelab2.iet.unipi.it> <4F907011.9080602@freebsd.org> <20120419204622.GA94904@onelab2.iet.unipi.it> <B143A8975061C446AD5E29742C531723C7C16A@pwsvl-excmbx-05.internal.cacheflow.com> <20120424163423.GA59530@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Apr 24, 2012 at 6:34 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: > On Tue, Apr 24, 2012 at 02:16:18PM +0000, Li, Qing wrote: >> > >> >From previous tests, the difference between flowtable and >> >routing table was small with a single process (about 5% or 50ns >> >in the total packet processing time, if i remember well), >> >but there was a large gain with multiple concurrent processes. >> > >> >> Yes, that sounds about right when we did the tests a long while ago. >> >> > >> > Removing flowtable increases the cost in ip_output() >> > (obviously) but also in ether_output() (because the >> > route does not have a lle entry so you need to call >> > arpresolve on each packet). >> > >> >> Yup. >> >> > >> > So in revising the route lookup i believe it would be good >> > if we could also get at once most of the info that >> > ether_output() is computing again and again. >> > >> >> Well, the routing table no longer maintains any lle info, so there >> isn't much to copy out the rtentry at the completion of route >> lookup. >> >> If I understood you correctly, you do believe there is a lot of value >> in Flowtable caching concept, but you are not suggesting we reverting >> back to having the routing table maintain L2 entries, are you ? > > I see a lot of value in caching in general. > > Especially for a bound socket it seems pointless to lookup the > route, iface and mac address(es) on every single packet instead of > caching them. And, routes and MAC addresses are volatile anyways > so making sure that we do the lookup 1us closer to the actual use > gives no additional guarantee. > > The frequency with which these info (routes and MAC addresses) > change clearly influences the mechanism to validate the cache. > I suppose we have the following options: > > - direct notification: a failure in a direct chain of calls > =A0can be used to invalidate the info cached in the socket. > =A0Similarly, some incoming traffic (e.g. TCP RST, FIN, > =A0ICMP messages) that reach a socket can invalidate the cached values > - assume a minimum lifetime for the info (i think this is what > =A0happens in the flowtable) and flush it unconditionally > =A0every such interval (say 10ms). > - if some info changes infrequently (e.g. MAC addresses) one could > =A0put a version number in the cached value and use it to validate > =A0the cache. I have a patch that has been sitting around for a long time due to review cycle latency that caches a pointer to the rtentry (and llentry) in the the inpcb. Before each use the rtentry is checked against a generation number in the routing tree that is incremented on every routing table update.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_MLRVX09_jeSXmCY0=z5D%2BpfZgwqt1UGM%2BhVRU-=-Bbbw>