Date: Mon, 15 Feb 2010 16:08:50 -0800 From: Alfred Perlstein <alfred@freebsd.org> To: Maxim Sobolev <sobomax@FreeBSD.org> Cc: FreeBSD Hackers <freebsd-stable@FreeBSD.org> Subject: Re: Sudden mbuf demand increase and shortage under the load Message-ID: <20100216000850.GC96165@elvis.mu.org> In-Reply-To: <4B793D1D.1000108@FreeBSD.org> References: <4B793D1D.1000108@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
* Maxim Sobolev <sobomax@FreeBSD.org> [100215 04:49] wrote: > Hi, > > Our company have a FreeBSD based product that consists of the numerous > interconnected processes and it does some high-PPS UDP processing > (30-50K PPS is not uncommon). We are seeing some strange periodic > failures under the load in several such systems, which usually evidences > itself in IPC (even through unix domain sockets) suddenly either > breaking down or pausing and restoring only some time later (like 5-10 > minutes). The only sign of failure I managed to find was the increase of > the "requests for mbufs denied" in the netstat -m and number of total > mbuf clusters (nmbclusters) raising up to the limit. Hey Maxim Can you run a process to dump sysctl -a every second or so and mark the time when you did it? Other monitoring things would probably be helpful as well (netstat -m) in a timed log format. vmstat -i? (interrupts storm?) Perhaps ps output (showing interrupt threads, etc) would be good toknow perhaps some ithreads went off into the weeds... Any console messages of note? A few people have suggested that there may be too many packets on the outgoing interface, I think there should be a limit to the number of packets queued for outgoing and probably counters to show how many were dropped due to overflow of the outgoing queue. You should be able to check these counters to see what is going on. If the driver is broken and never drops outgoing packets when the card's queue is full, then those counters should be 0. I hope this helps. -Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100216000850.GC96165>