From owner-freebsd-net@FreeBSD.ORG Tue Sep 18 12:34:52 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C0F210657F0; Tue, 18 Sep 2012 12:34:52 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0EEAB8FC08; Tue, 18 Sep 2012 12:34:52 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 5E3CCB96F; Tue, 18 Sep 2012 08:34:51 -0400 (EDT) From: John Baldwin To: freebsd-net@freebsd.org Date: Tue, 18 Sep 2012 08:10:52 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p17; KDE/4.5.5; amd64; ; ) References: <201209171503.12517.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201209180810.52409.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 18 Sep 2012 08:34:51 -0400 (EDT) Cc: Adrian Chadd , Ryan Stone Subject: Re: What's the latest on fixing IFF_DRV_OACTIVE/if_start/etc? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Sep 2012 12:34:52 -0000 On Monday, September 17, 2012 7:16:27 pm Adrian Chadd wrote: > There's a lot less cache in these boards. Going through the stack > trace all the way and back for each packet is actually quite > expensive. > > Then there's the overhead of having if_start() be called multiple > times, concurrently, from multiple senders. It's fine for a wifi AP > setup where the if_start() is only called once or twice in an > overlapping fashion, but sucks with lots of concurrent TCP/UDP > contexts all potentially calling if_start() and having them have to > clash with each other. > > I've not sat down and instrumented it all that much, so I'm going to > spend much time harping on about it until I have some hard numbers > either way. I'm just going by what I see people do to various network > stacks when it comes time to try and squeeze high packet rates out of > smaller platforms, especially with NAT/bridging/PPPoE in the way. And > that tended not to be "complete packet to completion on each frame." I (mostly) don't get where you are coming from at all. The "old" code would do this given a packet bound for if_vlan(4) or if_bridge(4): - queue the packet to the virtual interface's if_snd(4) including locking, etc. - call the virtual interface's if_start() which dequeues the previously queued packet (again, using locking) and then passes it down to the underlying physical interface via if_transmit(). For most NICs this entails queueing the packet to if_snd and then optionally calling the NIC's if_start(). The "new" code does: - Pass the packet down to the underlying physical interface via if_transmit(). For most NICs this entails queueing the packet to if_snd and then optionally calling the NIC's if_start(). This is _less_ work. Also, consider vlan(4). vlan(4)'s job is to serve (largely) as a protocol layer that prepends (or strips) a vlan header from the packet. We don't use interface queues for other protocols such as Ethernet or IP. We shouldn't use one for vlan(4) either. The packet should have the transform applied and then be dispatched to the actual NIC. As for direct dispatch, there is a toggle to not use direct dispatch in the network stack and to use netisr for certain tasks. However, that is generally used for receive, not transmit AFAIK. As for concurrent calls to if_start(), it certainly might be nice to provide a way under the IFQ lock to know in if_transmit() (the default one) whether or not if_start() is in progress and should be called (this would mean having if_start() be called with the IFQ lock held I think, though for drivers that use if_start() this would work nicely I think, esp. if you moved OACTIVE into the IFQ). That is orthogonal to bypassing queueing for things like vlan and bridge however. -- John Baldwin