From owner-freebsd-current@FreeBSD.ORG Fri Mar 13 15:41:02 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05FE01065670 for ; Fri, 13 Mar 2009 15:41:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id A97398FC13 for ; Fri, 13 Mar 2009 15:41:01 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 4ECB346B06; Fri, 13 Mar 2009 11:41:01 -0400 (EDT) Date: Fri, 13 Mar 2009 15:41:01 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Barney Cordoba In-Reply-To: <484892.45250.qm@web63902.mail.re1.yahoo.com> Message-ID: References: <484892.45250.qm@web63902.mail.re1.yahoo.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: current@freebsd.org Subject: Re: Interrupt routine usage not shown by top in 8.0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2009 15:41:02 -0000 On Fri, 13 Mar 2009, Barney Cordoba wrote: > Its difficult to have "better benchmarks" when the system being tested > doesn't have accounting that works. My test is designed to isolate the > driver receive function in a controlled way. So it doesn't much matter > whether the data is real or not, as long as the tests generate a consistent > load. Strikes me that this thread is getting a bit contentious, and I don't meaning in a locking sense :-). FreeBSD provides two interrupt execution environments: fast interrupts, and ithreads. Historically (5.x ... 7.x) device drivers have had to select one of the two models, but in 8.x a hybrid mode, called interrupt filters, allows drivers to do both for an interrupt source. The problem you're running into is that "fast" interrupts borrow the complete context of the thread they preempt, including its stack and accounting characteristics. For pure ithread drivers, this is generally not a problem, as the sole purpose of that interrupt handler is to kick the scheduler to launch the full ithread context, which will typically immediately preempt (at that same point in the stack) in order to give the interrupt handler a full context that can sleep on locks, be accounted for, etc. if_em lives in a world a bit between these models, in which it wants both a fast context, to do a bit of low level interaction with the device, and a "slow" or full context in which to execute the network stack, perform memory allocation, and so on. Because interrupt filters weren't yet around (and are presumably too experimental to use in 8.x right now), it does this by creating its own ithread-like execution context using a task queue. The result is mis-billing of what is effectively an ithread as system time instead of interrupt time. You'll notice that if_em (and other drivers employing the same trick) do elevate the priority of the task queue thread so that the scheduler treats it (almost) the same way it treats an ithread (it will immediately preempt most stuff). > The only thing obviously "bogus" is that FreeBSD is launching 16,000 tasks > per second (an interrupt plus taskqueue task 8000 times per second), plus > 2000 timer interrupts and reporting 0% cpu usage. So I'm to assume that the > system will never show 100% usage as the entire overhead of the scheduler is > not accounted for? The overhead of the scheduler is billed to a combination of the thread being switched out of, and the thread being switched to. Fast interrupt execution is billed to the thread it preempts. In the scenario you describe, you will only get mis-billing to idle if those fast interrupts preempt only the idle thread. Otherwise they will get billed to whatever is preempted. On a system where you have a network interface effectively keeping the CPU busy, it will get billed to the task queue thread (I expect) because the task queue is what will get preempted. Now, this might not strictly be true because the scheduler tries hard to keep ithreads running close to where the interrupt is delivered, but if it doesn't know the task queue thread is an ithread, it may do this less well. Presumably this is a temporary state of mind while interrupt filters are being adopted, only the interrupt filter work seems stalled (?). > Calling handle_rxtx was a timesaver to determine the overhead of forcing > 8000 context switches per second (16000 in a router setup) for apparently no > reason. Since the OS doesn't account for these, there seems no way to make > this determination. Its convenient to say it works well or better than > something else when there is no way to actually find out via measurement. I > don't see how launching 8000 tasks per second could be faster than not > launching 8000 tasks per second, but I'm also not up on the newest math. The purpose of a context switch from a fast interrupt context is to give interrupt code the ability to acquire general kernel locks, as opposed to just spin locks. If you run in a borrowed context (i.e., you have synchronously preempted a thread to run a fast interrupt), you may (will) generate deadlocks due to violating lock orders, since you don't want to (can't) release the locks already acquired by the thread, and may then acquire locks in the wrong order. If you want to acquire full sleep locks, you need a full context, which requires a context switch out of the preempted thread and into an ithread (or task queue thread or whatever). Passage into the normal input paths of the network stack will encounter normal locks, so must be done from a full context. > Since you know how things work better than any of us regular programmers, if > you could please answer these questions it would save a lot of time and may > result in better drivers for us all: > > 1) MSIX interrupt routines readily do "work" and pass packets up the IP > stack, while you claim that MSI interrupts cannot? Please explain the > locking differences between MSI and MSIX, and what locks may be encountered > by an MSI interrupt routine with "real traffic" that will not be a problem > for MSIX or taskqueue launched tasks. Its certainly not obvious from any > code or docs that I've seen. It's fine to enter the network stack from any full and dedicated thread context, which means it's OK from an ithread or a task queue thread kicked by a fast interrupt, but it's not OK from a fast interrupt. There's no difference between MSI/MSIX as far as I know from this perspective, only in how the drivers use them. > 2) the bge and fxp (and many other) drivers happily pass traffic up the IP > stack directly from their interrrupt routines, so why is it bogus for em to > do so? And why do these drivers not use the taskqueue approach that you > claim is superior? Only a few drivers use the fast interrupt approach; those that don't presumably do it because the approach of mixing "fast" and "slow" contexts hasn't been applied by their authors. If the interrupt filter model is going to become mainstream, I think we'd like to see them adopt that rather than hand-crafting fast interrupts and taskqueues, to the same effect, in every driver. On the other hand, one benefit to the task queue model is that you can deliver events to it that *aren't* interrupts, such as requests for state transitions from the software side of the stack. > 2b) Does this also imply that systems with bge or network drivers that do > the "work" in the interrupt handler will yield completely bogus cpu usage > numbers? All direct interrupt deliveries bill a small amount of CPU time to the context that they execute in. Drivers that do less work in the fast interrupt delivery context will bill less time outside of their own worker ithreads. There are two ways to measure CPU use, btw: one is a sampled approach involving timers, which works badly in fast interrupt contexts with interrupts disabled because the ticks are deferred until after interrupts are re-enabled, an the other is explicit time measurement, which is quite expensive. Currently the kernel uses the TSC, where available, and an estimator to map between CPU cycles and real time, but that also has its limitations. > 3) The em driver drops packets well before 100% cpu usage is realized. Of > course I'm relying on wrong cpu usage stats, so I may be mistaken. Is there > a way (or what is the preferred way) to increase the priority of a task > relative to other system processes (rather than relative to tasks in the > queue) so that packets can avoid being dropped while the system runs other, > non-essential tasks? When the em driver creates task queue threads, it assigns them an ithread priority. You can manually adjust that priority in the code, but I'm not sure we have an explicit management API from userspace to adjust those priorities without source code changes (but I may be wrong). > 3b) Is there a way to lock down a task such as a NIC receive task to give > absolute priority or exclusive use of a cpu? The goal is to make certain > that the task doesn't yield before it completes some minimum amount of work. You can use cpuset to force a specific thread onto a specific CPU, and to force other threads not to run on that CPU. You can also use cpuset, I believe, to direct the low-level interrupt delivery for sources to specific CPUs, but I've not done this myself. > Its my view that it would be better to just suck packets out of the ring and > queue them for upper layers, but I dont yet have a handle on the trade offs. > Currently the system drops too many packets unnecessarily at extremely high > load. This is, effectively, what fast interrupt handlers + task queues do. There's another potential dispatch point, between the link layer and the network layer, which is controlled by the net.isr.direct flag; right now we dispatch the whole stack to completion in the ithread, but you can turn that off by setting the flag to zero. In practice, the context switch avoidance associated with doing that appears to be a significant win for many, but not all, workloads. Robert N M Watson Computer Laboratory University of Cambridge