Date: Sat, 12 Sep 2015 20:20:39 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Dmitry Sivachenko <trtrmitya@gmail.com> Cc: hiren panchasara <hiren@strugglingcoder.info>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: netisr process eats 100% cpu Message-ID: <20150912172039.GP2072@kib.kiev.ua> In-Reply-To: <5599DAD0-55CD-4B36-B667-F943737C2D85@gmail.com> References: <C4039E45-3DC7-4E1A-9E46-853FE812B906@gmail.com> <EA9AD61B-E709-4387-8D5E-A33E25475D81@gmail.com> <20150911171910.GN64965@strugglingcoder.info> <5599DAD0-55CD-4B36-B667-F943737C2D85@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Sep 12, 2015 at 11:47:45AM +0300, Dmitry Sivachenko wrote: > > > On 11 сент. 2015 г., at 20:19, hiren panchasara <hiren@strugglingcoder.info> wrote: > > > > On 09/11/15 at 12:46P, Dmitry Sivachenko wrote: > >> > >>> hiren panchasara <hiren at strugglingcoder.info> wrote: > >>> > >>> Unsure at the moment if loopback is causing the trouble for you or not. > >>> See: > >> > >> (please keep me CC'ed, I am not subscribed to -net) > >> > >> > >>> > >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-February/041239.html > >>> > >>> > >> > >> Yes, this thread looks similar. > >> > >> > >>> You may want to try: > >>> 1) pmcstat and see if you can catch something > >> > >> What particular should I look for? Here is first lines of pmcstat -T -S instructions -w 1: > >> PMC: [INSTR_RETIRED_ANY] Samples: 157198 (100.0%) , 0 unresolved > >> > >> %SAMP IMAGE FUNCTION CALLERS > >> 13.2 kernel cpu_search_highest cpu_search_highest:12.0 sched_idletd:1.2 > >> 8.3 kernel ipfw_chk ipfw_check_packet > >> 3.1 myprogram memsetAVX _ZN12TLz4Compress7DoWriteEPKv > >> 2.3 kernel tcp_output tcp_usr_send:1.0 tcp_do_segment:0.9 > >> > >> > >>> 2) disable checksum on localhost > >> > >> > >> I tried, but nothing has changed. > >> > >> > >>> 3) look at netisr settings. sysctl net.isr o/p and how it looks under > >>> netstat -Q. I am not sure if adding more threads to netisr via > >> > >> > >> What should I look for? > >> > >> > >>> net.isr.numthreads would help. (Note its a loader.conf variable) > >> > >> > >> This netisr load looks parasitical to me (as I noted, moving haproxy to a separate machine does not burn CPU cycles on netisr, why is localhost special?) > >> Loopback is special since network stack input path must return the packets into the output path, and doing it by direct function call would recurse the stack. Netisr allows to queue the packet and then deliver it using the safe context. At least this is my limited understanding of the loopback+netisr coupling. Why it takes so much CPU for output processing, is beyond my knowledge of the network. But some resources have to be spent on localhost. > >> Even if adding more threads to netisr would boost network utilization, wouldn't those CPU cycles spent for netisr just be a waste of energy? I have other tasks for these CPU. > >> > > > > I am not sure what keep cpu busy with netisr when localhost is involved. > > > > You may want to post o/p of > > # sysctl net.isr > > net.isr.numthreads: 1 > net.isr.maxprot: 16 > net.isr.defaultqlimit: 256 > net.isr.maxqlimit: 10240 > net.isr.bindthreads: 0 > net.isr.maxthreads: 1 > net.isr.dispatch: direct > > > > # netstat -Q > > Configuration: > Setting Current Limit > Thread count 1 1 > Default queue limit 256 10240 > Dispatch policy direct n/a > Threads bound to CPUs disabled n/a > > Protocols: > Name Proto QLimit Policy Dispatch Flags > ip 1 4096 flow default --- > igmp 2 256 source default --- > rtsock 3 256 source default --- > arp 7 256 source default --- > ether 9 256 source direct --- > ip6 10 256 flow default --- > > Workstreams: > WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled > 0 0 ip 8 165 9862463 0 0 126594714 136424969 > 0 0 igmp 0 0 0 0 0 0 0 > 0 0 rtsock 0 1 0 0 0 10 10 > 0 0 arp 0 0 15640 0 0 0 15640 > 0 0 ether 0 0 9878107 0 0 0 9878107 > 0 0 ip6 0 2 4 0 0 12 16 > > > > # sysctl net.inet | grep queue > > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > net.inet.ip.dummynet.queue_count: 0 > > > > > > A suggestion I see at https://calomel.org/freebsd_network_tuning.html is > > to increase localhost n/w buffers. Not sure if this'll help your case. > > net.local.stream.sendspace=164240 # (default 8192) > > net.local.stream.recvspace=164240 # (default 8192) > > > I already had 65536 for that, but increased up to 164240, nothing has changed. > > > > > > Now I'll let someone else with more ideas/clues comment. > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150912172039.GP2072>