Date: Sun, 4 Oct 2009 16:49:09 +0200 From: Luigi Rizzo <rizzo@iet.unipi.it> To: rihad <rihad@mail.ru> Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets Message-ID: <20091004144909.GA42503@onelab2.iet.unipi.it> In-Reply-To: <4AC8A76B.3050502@mail.ru> References: <4AC8A76B.3050502@mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell > PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP > users online limited by dummynet pipes of various speeds. According to > netstat -s output around 500-1000 packets are being dropped every second > (this accounts for wasting around 7-12 mbit/s worth of traffic according > to systat -ifstat): what kind of packets are you seeing as dropped ? please give the output of 'netstat -s output | grep drop' At those speeds you might be hitting various limits with your config (e.g. 50k nmbclusters is probably way too small for 4k users -- means you have an average of 10-15 buffers per user; the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, which is quite high besides the fact that in order to scale to 4k users you would need over 1GB of kernel memory just for the buffers). I'd most likely suspect the nmbclusters argument. netstat -m might give you some useful stats. cheers luigi > # while :; do netstat -z -s 2>/dev/null | fgrep -w "output packets > dropped"; sleep 1; done > 16824 output packets dropped due to no bufs, etc. > 548 output packets dropped due to no bufs, etc. > 842 output packets dropped due to no bufs, etc. > 709 output packets dropped due to no bufs, etc. > 652 output packets dropped due to no bufs, etc. > ^C > > Pipes have been created like this: > ipfw pipe 1024 config bw 1024kbit/s mask dst-ip 0xffffffff queue 350KBytes > etc., and then assigned to users by application (ipfw tablearg). > > I've tried playing with the queue setting, from as little as 1 slot to > as much as 4096KBytes - packets are still being dropped, more or less. > Should I somehow calculate the proper queue value for the given pipe > width? The manpage says 50 slots is typical for Ethernet devices (not > mentioning whether it's 10, 100 or 1000 mbit/s), and that's it. > > sysctls: > kern.ipc.nmbclusters=50000 > net.inet.ip.dummynet.io_fast=1 > > Polling can't be enabled with bce. > > Any hints? Should I provide any further info? > > Thanks. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20091004144909.GA42503>