From owner-freebsd-current@FreeBSD.ORG Fri Nov 18 17:07:51 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C8BC1065670; Fri, 18 Nov 2011 17:07:51 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 4AC7E8FC15; Fri, 18 Nov 2011 17:07:51 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 73A747300A; Fri, 18 Nov 2011 18:06:15 +0100 (CET) Date: Fri, 18 Nov 2011 18:06:15 +0100 From: Luigi Rizzo To: John Baldwin Message-ID: <20111118170615.GA9762@onelab2.iet.unipi.it> References: <201111170953.58151.jhb@freebsd.org> <201111180800.06593.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201111180800.06593.jhb@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Matteo Landi , freebsd-current@freebsd.org Subject: Re: ixgbe and fast interrupts X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 17:07:51 -0000 On Fri, Nov 18, 2011 at 08:00:06AM -0500, John Baldwin wrote: > On Friday, November 18, 2011 3:46:02 am Matteo Landi wrote: > > > you probably want to be using MSI-X for a 10G NIC instead of INTx anyway. > > > > Why do you say that? Is MSI-X faster than INTx in terms of interrupt > > latency? When should I use MSI-X, instead of fast filters interrupts > > (fast interrupt?), instead of ithread interrupts? Thanks in advace. > > With MSI-X you can have more than one interrupt and those interrupts can be > distributed across CPUs. This means you can (somewhat) tie each queue on your > NIC to a different CPU. > > MSI-X vs INTx is orthogonal to fast vs filter, but in general MSI and MSI-X > interrupts are not shared, and require no interrupt masking in hardware (they > are effectively edge-triggered), so using a filter for MSI is rather pointless > and only adds needless complexity. For MSI I would just use a theraded > interrupt handler. For INTx, I would only use a fast interrupt handler if > there is a really good reason to do so (e.g. em(4) does so to work around > broken Intel Host-PCI bridges). A bit more context: Matteo is looking at the latency of RPCs across the network involving userspace processes, and possibly using the netmap API. As we understand it: if you are not using a filter, the interrupt calls a "predefined" filter (kern_intr.c::intr_event_schedule_thread() ? ) which wakes up the handler thread which in turn wakes up the user process. This means two scheduler invocations on each side. In the case of netmap, all the handler needs to do is a selwakeup() of the user thread blocked on the file descriptor, so if this can be done in the filter we can save an extra step through the scheduler. cheers luigi