From owner-freebsd-net@FreeBSD.ORG Wed Dec 5 03:57:55 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 66EF01F9; Wed, 5 Dec 2012 03:57:55 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com [209.85.223.179]) by mx1.freebsd.org (Postfix) with ESMTP id EE27F8FC0C; Wed, 5 Dec 2012 03:57:54 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id k14so7086016iea.10 for ; Tue, 04 Dec 2012 19:57:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=hkHoL6ot2rl5hN8jxnUoDl46qp9wOoFGrA83diK/XQ4=; b=A3jGiym1yazWI2tkDSYmEAzW9U5uMCESQYRDuqWe2iZI9KgjdFAKVcuiGubHm53uy/ dIi8oIhl/9nA72e/h0EDPu0zpeKsbNltev0wkrGIzW5Z2WT2j3YueK+GWuCLNWNVH+JB LdYGuJhYk6VuZnqMaPwBbWEG4yh9P+K5UaFyKj54YyCXgC0f3Jkdo4GyRcb9PJO1lzpF ge+WLXKHnoYabs7H/HmchahvziTSCabuCXVFMOcgkA+sdPybjtudn0nSrkNmmptyqY8J d1gYnava0T9+WN2ZTJ2bmbu8JjQSkdyuoW3j8pG9tP0baXtuHMG8eH6rc2W8rPaNCn8t +F0w== Received: by 10.50.33.173 with SMTP id s13mr582385igi.23.1354679874026; Tue, 04 Dec 2012 19:57:54 -0800 (PST) Received: from [10.0.0.130] ([24.225.136.71]) by mx.google.com with ESMTPS id uj11sm11568434igb.15.2012.12.04.19.57.51 (version=SSLv3 cipher=OTHER); Tue, 04 Dec 2012 19:57:52 -0800 (PST) Message-ID: <50BEC63B.6020801@gmail.com> Date: Tue, 04 Dec 2012 22:57:47 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Andre Oppermann Subject: Re: Latency issues with buf_ring References: <1353259441.19423.YahooMailClassic@web121605.mail.ne1.yahoo.com> <201212041108.17645.jhb@freebsd.org> <50BE56C8.1030804@networx.ch> In-Reply-To: <50BE56C8.1030804@networx.ch> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Adrian Chadd , John Baldwin , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Dec 2012 03:57:55 -0000 Hi, On 04/12/2012 3:02 PM, Andre Oppermann wrote: > On 04.12.2012 20:34, Adrian Chadd wrote: >> .. and it's important to note that buf_ring itself doesn't have the >> race condition; it's the general driver implementation that's racy. >> >> I have the same races in ath(4) with the watchdog programming. Exactly >> the same issue. > > Our IF_* stack/driver boundary handoff isn't up to the task anymore. > > Also the interactions are either poorly defined or understood in many > places. I've had a few chats with yongari@ and am experimenting with > a modernized interface in my branch. > > The reason I stumbled across it was because I'm extending the hardware > offload feature set and found out that the stack and the drivers (and > the drivers among themself) are not really in sync with regards to > behavior. > > For most if not all ethernet drivers from 100Mbit/s the TX DMA rings > are so large that buffering at the IFQ level doesn't make sense anymore > and only adds latency. So it could simply directly put everything into > the TX DMA and not even try to soft-queue. If the TX DMA ring is full > ENOBUFS is returned instead of filling yet another queue. However there > are ALTQ interactions and other mechanisms which have to be considered > too making it a bit more involved. I've also bumped into this 'internalization' of drbr for quite some time now. I have been toying with some ideas around a multi-queue capable ALTQ. Not unlike IFQ_* the whole class_queue_t code in ALTQ could use some freshening up. One avenue I am looking into is drbr queues (and its associated TX lock) as the back end queue implementation for ALTQ. ALTQ(9) has a concept of driver managed queues and the approach tries to keep the same paradigm but adapt it for buf_ring. In that context, It doesn't feel natural for me that drbr logic is handled so low inside the device drivers and makes system level modifications to ALTQ unnecessarily driver dependent. ALTQ is also using very coarse grained locking (using the IFQ_LOCK for everything) which doesn't make much sense in a SMP/multiqueue system but that's another story. > > I'm coming up with a draft and some benchmark results for an updated > stack/driver boundary in the next weeks before xmas. > Sounds great, can't wait to read it while drinking that eggnog :)