Date: Thu, 26 Mar 2009 13:08:26 +0300 From: Oleg Bulyzhin <oleg@FreeBSD.org> To: Eugene L Kovalenja <qwe@qwe.net.ua> Cc: freebsd-ipfw@freebsd.org Subject: Re: FreeBSD 7.0: dummynet 99% cpu Message-ID: <20090326100826.GA44053@lath.rinet.ru> In-Reply-To: <49CA119D.7090304@qwe.net.ua> References: <200903181022.n2IAMsWs038026@lurza.secnetix.de> <49C310A9.6020102@qwe.net.ua> <20090325082925.GA13280@lath.rinet.ru> <49CA119D.7090304@qwe.net.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 25, 2009 at 01:12:29PM +0200, Eugene L Kovalenja wrote: > Oleg Bulyzhin пишет: > > On Fri, Mar 20, 2009 at 05:42:33AM +0200, Eugene L Kovalenja wrote: > > > >> Oliver Fromme ?????: > >> > >>> Eugene L Kovalenja wrote: > >>> > FreeBSD *** 7.0-RELEASE FreeBSD 7.0-RELEASE #6: Sun Nov 23 14:32:31 EET > >>> > [...] > >>> > Time in three days traffic via ipfw doesn't go. In top: > >>> > 21 root 1 -44 - 0K 8K WAIT 7 2:15 99.02% dummynet > >>> > (this is example, not copy\paste) > >>> > >>> There are a few problems that have been fixed (or worked > >>> around) after the release of 7.0. For example, look at > >>> PR kern/113548 which has a work-around in 7.1. Your > >>> problem description sounds like it could be caused by > >>> the same problem. > >>> > >>> Therefore I recommend you update to 7.1 or 7-stable. > >>> > >>> Best regards > >>> Oliver > >>> > >>> > >>> > >> Hello. > >> > >> System updated to: > >> [root@taurus /usr/home/qwe]# uname -a > >> FreeBSD *** 7.1-RELEASE-p3 FreeBSD 7.1-RELEASE-p3 #0: Thu Mar 19 > >> 16:31:53 EET 2009 root@***:/usr/obj/usr/src/sys/QWEKRN70 i386 > >> > >> but once trouble has repeated (30 mins ago). > >> > >> > >> After that I change my sysctl variables: > >> net.inet.ip.dummynet.io_fast=1 > >> net.inet.ip.dummynet.debug=1 > >> net.inet.ip.dummynet.hash_size=16384 (from 512) > >> > >> What can I'll to do? > >> > >> Sorry for my bad English :( > >> _______________________________________________ > >> freebsd-ipfw@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > >> > > > > Please do the following (when dummynet will hang next time): > > 1) Grab the output of follwing commands: > > sysctl net.inet.ip.dummynet > > ipfw pipe show > > 2) wait a bit (30 seconds should be enough) > > 3) do 1) once again. > > > > Examining counters may help in understanding problem. > > > > > before: > > sysctl net.inet.ip.dummynet > net.inet.ip.dummynet.debug: 0 > net.inet.ip.dummynet.pipe_byte_limit: 1048576 > net.inet.ip.dummynet.pipe_slot_limit: 100 > net.inet.ip.dummynet.io_pkt_drop: 14138 > net.inet.ip.dummynet.io_pkt_fast: 610276 > net.inet.ip.dummynet.io_pkt: 1789875 > net.inet.ip.dummynet.io_fast: 1 > net.inet.ip.dummynet.tick_lost: 0 > net.inet.ip.dummynet.tick_diff: -891 > net.inet.ip.dummynet.tick_adjustment: 9797 > net.inet.ip.dummynet.tick_delta_sum: 449 > net.inet.ip.dummynet.tick_delta: 0 > net.inet.ip.dummynet.red_max_pkt_size: 1500 > net.inet.ip.dummynet.red_avg_pkt_size: 512 > net.inet.ip.dummynet.red_lookup_depth: 256 > net.inet.ip.dummynet.max_chain_len: 16 > net.inet.ip.dummynet.expire: 1 > net.inet.ip.dummynet.search_steps: 1798977 > net.inet.ip.dummynet.searches: 1788798 > net.inet.ip.dummynet.extract_heap: 0 > net.inet.ip.dummynet.ready_heap: 48 > net.inet.ip.dummynet.curr_time: 138355446 > net.inet.ip.dummynet.hash_size: 512 > > ipfw pipe show | grep -v ip > > 01241: 2.048 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01122: 4.096 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 00801: 9.000 Mbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01242: 2.048 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01212: 256.000 Kbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01121: 4.096 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01091: 512.000 Kbit/s 0 ms 50 sl. 46 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00802: 9.000 Mbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01211: 256.000 Kbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01092: 512.000 Kbit/s 0 ms 50 sl. 45 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00701: 9.000 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 00821: 9.000 Mbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00791: 9.000 Mbit/s 0 ms 50 sl. 5 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01111: 2.048 Mbit/s 0 ms 50 sl. 5 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00822: 9.000 Mbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01231: 1.024 Mbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01072: 1.024 Mbit/s 0 ms 50 sl. 3 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00702: 9.000 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01251: 1.024 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01232: 1.024 Mbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01112: 2.048 Mbit/s 0 ms 50 sl. 5 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01071: 1.024 Mbit/s 0 ms 50 sl. 3 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00811: 9.000 Mbit/s 0 ms 50 sl. 2 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00792: 9.000 Mbit/s 0 ms 50 sl. 5 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01221: 512.000 Kbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01202: 128.000 Kbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01101: 1.024 Mbit/s 0 ms 50 sl. 18 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01082: 256.000 Kbit/s 0 ms 50 sl. 18 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00812: 9.000 Mbit/s 0 ms 50 sl. 2 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00722: 9.000 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01252: 1.024 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01222: 512.000 Kbit/s 0 ms 50 sl. 1 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01201: 128.000 Kbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > 01102: 1.024 Mbit/s 0 ms 50 sl. 16 queues (512 buckets) droptail > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 01081: 256.000 Kbit/s 0 ms 50 sl. 19 queues (512 buckets) droptail > mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 00721: 9.000 Mbit/s 0 ms 50 sl. 0 queues (512 buckets) droptail > > after: > sysctl net.inet.ip.dummynet > net.inet.ip.dummynet.debug: 0 > net.inet.ip.dummynet.pipe_byte_limit: 1048576 > net.inet.ip.dummynet.pipe_slot_limit: 100 > net.inet.ip.dummynet.io_pkt_drop: 15177 > net.inet.ip.dummynet.io_pkt_fast: 634143 > net.inet.ip.dummynet.io_pkt: 1873018 > net.inet.ip.dummynet.io_fast: 1 > net.inet.ip.dummynet.tick_lost: 0 > net.inet.ip.dummynet.tick_diff: -884 > net.inet.ip.dummynet.tick_adjustment: 9803 > net.inet.ip.dummynet.tick_delta_sum: 320 > net.inet.ip.dummynet.tick_delta: -1 > net.inet.ip.dummynet.red_max_pkt_size: 1500 > net.inet.ip.dummynet.red_avg_pkt_size: 512 > net.inet.ip.dummynet.red_lookup_depth: 256 > net.inet.ip.dummynet.max_chain_len: 16 > net.inet.ip.dummynet.expire: 1 > net.inet.ip.dummynet.search_steps: 1882134 > net.inet.ip.dummynet.searches: 1871939 > net.inet.ip.dummynet.extract_heap: 0 > net.inet.ip.dummynet.ready_heap: 48 > net.inet.ip.dummynet.curr_time: 138373128 > net.inet.ip.dummynet.hash_size: 512 > > dummynet cpu load now - 56% As i can see things looks normal except high cpu usage. Though i think it's not dummynet's problem - i guess most cpu cycles wasted outside dummynet's code. Could you do some hwpmc profiling? (short how-to: http://freebsd.rambler.ru/bsdmail/freebsd-current_2006/msg01582.html) -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090326100826.GA44053>