Date: Fri, 28 Apr 2006 09:02:45 -0400 From: Marcos Bedinelli <bedinelli@madhaus.cns.utoronto.ca> To: Jeremie Le Hen <jeremie@le-hen.org> Cc: freebsd-net@freebsd.org Subject: Re: [fbsd] Re: [fbsd] Network performance in a dual CPU system Message-ID: <31c0d864b5b8cd45fdc01e30e6685488@madhaus.cns.utoronto.ca> In-Reply-To: <20060427143814.GD84148@obiwan.tataz.chchile.org> References: <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> <20060427093916.GC84148@obiwan.tataz.chchile.org> <20060427145252.I75848@fledge.watson.org> <20060427143814.GD84148@obiwan.tataz.chchile.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, On 27-Apr-06, at 10:38, Jeremie Le Hen wrote: > Hi, Robert, > > On Thu, Apr 27, 2006 at 02:54:21PM +0100, Robert Watson wrote: >> >> On Thu, 27 Apr 2006, Jeremie Le Hen wrote: >> >>>> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU >>>> COMMAND >>>> 60 root 1 -44 -163 0K 8K WAIT 355.6H 72.17% >>>> swi1: >>>> net >>>> 39 root 1 -68 -187 0K 8K WAIT 52.3H 5.22% >>>> irq28: >>>> bge0 >>>> 40 root 1 -68 -187 0K 8K WAIT 28.3H 2.25% >>>> irq29: >>>> bge1 >>>> 11 root 1 171 52 0K 8K RUN 166.6H 0.00% idle >>>> 63 root 1 -16 0 0K 8K - 121:55 0.00% >>>> yarrow >>>> 61 root 1 -32 -151 0K 8K WAIT 46:21 0.00% >>>> swi4: >>>> clock sio >>>> [...] >>>> >>>> Does anyone know whether a dual CPU system can help us improve the >>>> situation? I was wondering if the software interrupt threads would >>>> be >>>> divided between the two processors. >> >> I missed the original thread, but in answer to the question: if you >> set >> net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the >> ithread >> of the network device driver. This will allow the IP forwarding and >> related paths in two threads instead of one, potentially allowing >> greater >> parallelism. Of course, you also potentially contend more locks, you >> may >> increase the time it takes for the ithread to respond to new >> interrupts, >> etc, so it's not quite cut and dry, but with a workload like the one >> shown >> above, it might make quite a difference. > > Actually you already replied in the original thread, explaining mostly > the same thing. The whole thread [1] brought up multiple valuable > network performance tuning knobs, such as polling, fastforwarding, > net.isr.direct but there is no happy end to the thread. Given this is > a real world situation, I wanted to know how Marcos revolved his > problem. Shortly after I opened the thread, most replies I received (from the list or privately) were saying that a second CPU either wouldn't help to improve the situation, or that the performance gain was unclear. We had an immediate problem to solve and the management wasn't comfortable in throwing money at new and expensive hardware that, in the end, might or might not solve the issue. The performance gain with fastforwarding enabled was almost 20% and that's what we've been running since. Note, however, that I am also using virtual interface disc0 and that fastforwarding doesn't work with that interface (at least prior to and including FreeBSD 6.0-release-p4). Gleb Smirnoff sent me a one-line patch that fixed the problem. Hope this helps. Regards, -- Marcos
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?31c0d864b5b8cd45fdc01e30e6685488>