Date: Wed, 07 Oct 2009 16:23:36 +0500 From: rihad <rihad@mail.ru> To: Oleg Bulyzhin <oleg@FreeBSD.org> Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets Message-ID: <4ACC7A38.50809@mail.ru> In-Reply-To: <4ACC7308.6070301@mail.ru> References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
rihad wrote: > Oleg Bulyzhin wrote: >> On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad wrote: >>> Oleg Bulyzhin wrote: >>>> On Wed, Oct 07, 2009 at 02:23:47PM +0500, rihad wrote: >>>> >>>> Few questions: >>>> 1) why are you not using fastforwarding? >>>> 2) search_steps/searches ratio is not that good, are you using >>>> 'buckets' >>>> keyword in your pipe configuration? >>>> 3) you have net.inet.ip.fw.one_pass = 0, is it intended? >>>> >>> 1) and 3): the box does traffic accounting and shaping, so I need >>> one_pass=0 to do both ngtee and pipes. >> Still can not see any objection for not using fastforwarding, and usually >> ipfw ruleset can be rearranged for using dummynet & netgraph with >> one_pass=1. > > You probably have some special sources of documentation ;-) According to > man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet > unless one_pass=0. Or do you mean sprinkling smart skiptos here and > there? ;-) > >> Could you show your 'ipfw show' output? (hide ip addresses if you wish >> but >> keep counters please). >> > Here it is, in its whole glory: > > 00100 10434423 1484891105 allow ip from any to any via lo0 > 00200 2 14 deny ip from any to 127.0.0.0/8 > 00300 1 4 deny ip from 127.0.0.0/8 to any > 01000 3300039938 327603104711 allow ip from any to any in > 01010 26214900 421138433 allow ip from me to any out > 01020 5453857 46806278 allow icmp from any to any out > 01030 3268289053 327224694165 ngtee 1 ip from any to any out > 01040 18681181 1089636054 skipto 1100 ip from table(127) to any out > recv bce0 xmit bce1 > 01060 777488848 76743392754 pipe tablearg ip from any to table(0) out > recv bce0 xmit bce1 > 01070 776831109 76682499457 allow ip from any to table(0) out recv > bce0 xmit bce1 > 01100 13102697 808411842 pipe tablearg ip from any to table(2) out > 65535 662648946 66711487830 allow ip from any to any > > table(127) is static in nature and is under 100 entries. > table(0) and table(2) have the same IP clients' addresses but different > pipe IDs. > >>> 2) Hm, I'm not using "buckets", but rather >>> net.inet.ip.dummynet.hash_size. It's at default, 64. I've tried >>> setting net.inet.ip.dummynet.hash_size=65536 in sysctl.conf but >>> somehow it was still 64 after reboot, so I left it at 64. Should I >>> make it 128? 256? Does it matter that much? The load is at approx. >>> 70-120 consumers per pipe, so I thought 64 bucket size was enough. >> It depends on traffic pattern, try to increase it and watch >> search_steps/searches ratio (~1.001 is good enough) >> After reconfiguring all pipes with hash_size=256 (4 times as much as before), the ratio has started decreasing slowly. Run by me every 5-100 seconds: [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10639566354978963640 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10638988711274017516 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10637649664889937145 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10636898392044547569 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10634798328730542254 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10608591323771604268 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10600110020578292697 but the number of drops is still there. Run every minute: Wed Oct 7 11:00:44 UTC 2009 34630 output packets dropped due to no bufs, etc. Wed Oct 7 11:01:44 UTC 2009 34630 output packets dropped due to no bufs, etc. Wed Oct 7 11:02:44 UTC 2009 34729 output packets dropped due to no bufs, etc. Wed Oct 7 11:03:44 UTC 2009 34729 output packets dropped due to no bufs, etc. Wed Oct 7 11:04:44 UTC 2009 34861 output packets dropped due to no bufs, etc. Wed Oct 7 11:05:44 UTC 2009 34932 output packets dropped due to no bufs, etc. Wed Oct 7 11:06:44 UTC 2009 35499 output packets dropped due to no bufs, etc. Wed Oct 7 11:07:45 UTC 2009 35780 output packets dropped due to no bufs, etc. Wed Oct 7 11:08:45 UTC 2009 35841 output packets dropped due to no bufs, etc. Wed Oct 7 11:09:45 UTC 2009 36348 output packets dropped due to no bufs, etc. Wed Oct 7 11:10:45 UTC 2009 36568 output packets dropped due to no bufs, etc. Wed Oct 7 11:11:45 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:12:45 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:13:46 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:14:46 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:15:46 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:16:46 UTC 2009 36849 output packets dropped due to no bufs, etc. Wed Oct 7 11:17:46 UTC 2009 37234 output packets dropped due to no bufs, etc. Wed Oct 7 11:18:46 UTC 2009 37949 output packets dropped due to no bufs, etc. Wed Oct 7 11:19:47 UTC 2009 38043 output packets dropped due to no bufs, etc. Wed Oct 7 11:20:47 UTC 2009 38549 output packets dropped due to no bufs, etc. 2200-2350 users online (ipfw table load). I'll wait and see if the drop rate approaches 500-1000 per second as the number of online users comes close to 3-4K. net.isr.direct=0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4ACC7A38.50809>