Date: Sat, 12 Sep 2009 20:59:28 +1000 (EST) From: Bruce Evans <brde@optusnet.com.au> To: rihad <rihad@mail.ru> Cc: freebsd-net@FreeBSD.org Subject: Re: [POLLING] strange interrupt/system load Message-ID: <20090912202529.X1569@besplex.bde.org> In-Reply-To: <4AAB4D56.30207@mail.ru> References: <4AAB4D56.30207@mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 12 Sep 2009, rihad wrote: > The box experiences ~230 mbit/s traffic flow through it. I've doubled some > sysctls after reading polling(4): > kern.polling.each_burst=10 # was: 5 > kern.polling.burst_max=350 # was: 150 > > FreeBSD 7.2-RELEASE-p3 amd64 > HZ=1000 How much better does it work without POLLING? > Now for the fun part. > > With kern.polling.idle_poll = 1 top shows: > CPU: 0.0% user, 0.0% nice, 26.9% system, 3.1% interrupt, 70.0% idle > ~8000 interrupts/s total according to systat -vmstat: > 1999 cpu0: time > 2000 cpu1: time > 1999 cpu2: time > 1999 cpu3: time I think the poll thread is still single-threaded. It takes all the available time from 1 idle CPU (possibly not always the same one), and thus makes an average of 1 CPU 100% not-really-idle, and thus ensures a CPU utlization of at lease 25% with 4 CPUs. Here you have slightly more than 25%. > With kern.polling.idle_poll = 0 top shows: > CPU: 0.0% user, 0.0% nice, 0.0% system, 13.9% interrupt, 86.0% idle > Still the same ~8000 clock interrupts/s. The actual overhead seems to be 14% of 4 CPUs or 56% of 1 CPU. Polling in idle is mostly wasting 44% of 1 CPU. (The wastage is not quite complete since polling may improve latency). > Under both scenarios polling is enabled on both em0 and em1 through ifconfig. POLLING is especially useless for em since its hardware has better interrupt moderation than most NICs. In non-old versions of FreeBSD em should be configured so as to not use it, and then em will use fast interrupts and threads, a method that has a chance of working much better than both POLLING and normal interrupts (but I haven't actually seen it working much better). However, em's configuration of this is bad: you have to not configure POLLING, and this prevents use of polling on NICs that might actually benefit from it. > 1) Why is the interrupt load relatively high with polling enabled? 3.1% is already high. Probably there are a lot of software interrupts (netisrs). > 2) How come 13.9% interrupts are not also in the first scenario if their > total rate is the same (~8000)? Possibly there are even more software interrupts, due to the polling thread running into itself. Look at %CPU for each individual thread. Without polling, but with non-fast em interrupts, there will be lots of em interrupts, and the interrupt overhead will be more evenly distributed between the em interrupt thread and swi thread(s), so it is easier to see where the cycles are going (still quite hard, since lots of the cycles are for cache misses and a badly-behaved thread or hardware may cause the slowness to appear to be in a different thread due to it pushing the cache misses there...). Without polling, but with fast em interrupts and em non-interrupt threads, the interrupt overhead should be nearly 0, but the overhead will mostly just have moved, but with real or potential gains to latency and throughput from parallelism (parallellism probably increases overheads a little but improves network throughput and latency at a cost to other concurrent uses of the CPUs that don't matter iff there ar CPU cycles to spare). (Overhead for fast interrupts is not accounted for accurately -- it will be charged to the interrupted context. The non-interrupt threads will be charged as "system".) Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090912202529.X1569>