From owner-freebsd-net@FreeBSD.ORG Fri Sep 9 00:13:55 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25AE1106564A; Fri, 9 Sep 2011 00:13:55 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) by mx1.freebsd.org (Postfix) with ESMTP id DAE1D8FC16; Fri, 9 Sep 2011 00:13:51 +0000 (UTC) Received: from [50.12.52.187] (helo=[192.168.1.10]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1R1oj8-0006hQ-HU; Thu, 08 Sep 2011 20:13:50 -0400 Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: text/plain; charset=us-ascii From: George Neville-Neil In-Reply-To: <20110908184928.GA87872@hub.freebsd.org> Date: Thu, 8 Sep 2011 20:13:50 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <37419C45-4436-4738-851B-2B765BC2C60F@neville-neil.com> References: <1315221674.3092.282.camel@deadeye> <201109080834.11607.jhb@freebsd.org> <20110908184928.GA87872@hub.freebsd.org> To: Navdeep Parhar X-Mailer: Apple Mail (2.1244.3) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com Cc: Ben Hutchings , freebsd-net@freebsd.org, jfv@freebsd.org, John Baldwin , Takuya ASADA Subject: Re: Adding Flow Director sysctls to ixgbe(4) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 00:13:55 -0000 On Sep 8, 2011, at 14:49 , Navdeep Parhar wrote: > On Thu, Sep 08, 2011 at 08:34:11AM -0400, John Baldwin wrote: >> On Monday, September 05, 2011 7:21:12 am Ben Hutchings wrote: >>> On Mon, 2011-09-05 at 15:51 +0900, Takuya ASADA wrote: >>>> Hi, >>>>=20 >>>> I implemented Ethernet Flow Director sysctls to ixgbe(4), here's a = detail: >>>>=20 >>>> - Adding removing signature filter >>>> On linux version of ixgbe driver, it has ability to set/remove = perfect >>>> filter from userland using ethtool command. >>>> I implemented similar feature, but on sysctl, and not perfect = filter >>>> but signature filter(which means hash collision may occurs). >>> [...] >>>=20 >>> Linux also has a generic interface to RX filtering and hashing >>> (ethtool_rxnfc) which ixgbe supports; wouldn't it be better for = FreeBSD >>> to support something like that? >>=20 >> Some sort of shared interface might be nice. The cxgb(4) and = cxgbe(4) drivers >> both provide their own tools to manipulate filters, though they do = not >> provide explicit steering IIRC. >=20 > Both of them can filter as well as steer (and the tools let you do = that). > cxgbe(4) can do a lot more (rewrite + switch, replicate, etc.) but = those > features are perhaps too specialized to be configurable via a general > purpose tool. >=20 >>=20 >> We would need to come up with some sort of standard interface = (ioctls?) for=20 >> adding filters however. >=20 > +1 for a standard interface. >=20 > imho the kernel needs to be aware of the rx and tx queues of a NIC, = and > not just for steering. But that's a separate discussion. >=20 Well I do think this is actually all of a part. Most of us realize by = now that high speed (e.g. 10G and higher) NICs only make sense if you can steer = traffic and pin queues to cores etc. Without those pieces in place it's impossible = to get the full utilization out of the NIC because things like cache misses = just kill your performance. Having looked at a few 10G NICs in the last couple of = years the one thing I notice is that everyone wants to do things like offload and = steering but that the specifics are quite different between different cards. = Some of that has to do with the libraries that expose these things to the kernel and = user programs, but some has to do with how the device is modeled. What this means is = that we have a failure of abstraction. Abstraction has a cost, and some of the = people who want access to low level queues are not interested in paying an extra = abstraction cost. I think that some of the abstractions we need are tied up in the work = that Takuya did for SoC and some of it is in the work done by Luigi on netmap. I'd go = so far as to say that what we should do is try to combine those two pieces of code into a = set of low level APIs for programs to interact with high speed NICs. The one = thing most people do not talk about is extending our socket API to do two things = that I think would be a win for 80% of our users. If a socket, and also a kqueue, could be = pinned to a CPU as well as a NIC queue that should improve overall bandwidth = for a large number of our users. The API there is definitely an ioctl() and the = hard part is doing the tying together. To do this we need to also work out our low = level story. Best, George