Date: Wed, 08 Aug 2001 03:07:49 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: boshea@ricochet.net Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Tuning the 4.1-R kernel for networking Message-ID: <3B710F75.FF20B38@mindspring.com> References: <20010807213320.D529@ricochet.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Brian O'Shea wrote: > On this machine I run a program which simulates many (~150) simultaneous > TCP clients. This is actually a multithreaded Linux binary, and one > thread per simulated TCP client is created. After a few seconds the > system runs out of mbuf clusters: > > # netstat -m > 3231/3392/6144 mbufs in use (current/peak/max): > 1641 mbufs allocated to data This is 25 connections worth of full TCP window in a single direction. For 150 connections, you will need 150*16k*2/256, or 19,200 mbufs for a 16k window size, with both transmit and receive biffers full, or 9,600 mbufs, if you only have the receive windows full. > 182 mbufs allocated to packet headers This is really allocation for tcptempl structures for use in packet keepalive; it basically indicates that you have a total of 182 sockets open. > 1408 mbufs allocated to socket names and addresses Don't know why this is, but it means you need at least another 1408 mbufs... > 1536/1536/1536 mbuf clusters in use (current/peak/max) Your cluster count is way, way too small. The fact that you are maxing out at 1536 in all categories implies that you have filled your clusters out with full send or receive windows. An mbuf cluster is 2k, and is generally only filled with 1536 (MTU) bytes, unless it has been coelesced. Assuming that these are the result of full write windows, then you need _at least_ 2400 clusters (150x16k*2/2k) for bidirectional window fill. > 3920 Kbytes allocated to network (98% in use) Means you've only got 4M assigned to network resources, which is not a lot; I can see you using 5M for window buffers, alone, ignoring cluster headers and mbufs for sockets and the 1408 mbufs you disappeared into socket names and addresses. > 96993 requests for memory denied These are the number of MGET/MCLGET failures. > 0 requests for memory delayed None of them were malloc calls, they were all allocations of objects which can only be allocated from memory which you must reserve by compiling your kernel with the correct tuning parameters (or in some cases sysctls in loader.conf, but I can't tell you what works in 4.1, and what has to be done at config time; sorry). > 0 calls to protocol drain routines > > Also, I see a steady stream of these messages on the console: > > xl0: no memory for rx list -- packet dropped! > > >From the xl(4) man page: > > xl%d: no memory for rx list The driver failed to allocate an mbuf > for the receiver ring. Yeah; you are out of mbufs and cluster headers. We knew that. When the driver can't allocate a replacement mbuf, it drops the receive packet data, since it can't really safely leave an empty receive ring slot, since those are receive interrupt driven. Basically, you have exhausted all your receive resources. If I had to guess, I would say that your program was an HTTP load program of some kind, since you have a grundle of data packed up in your receive window, which is common when you receive data, but never bother actually reading it to make it go away. You could also change your program to set the window size down to a smaller size than the default (also a sysctl, to set it globally for all programs, as an administrative limit), and that would keep the sender from taking up as much memory, since you would advertise a smaller window to the sender. > Looking at the xl_newbuf() function in the xl driver, there are two > places where this message can be generated. It looks like the problem > is with the second case where the MCLGET fails, since we are running out > of those. It could still be either one, actually; your message that you quoted didn't match either one of the printf's... > I increased maxusers to 128 (2560 mbuf clusters) and it ran out of mbuf > clusters again. Then I increased it to 256 (4608 mbuf clusters), with > the same results. I don't have any sense of what is reasonable mbuf > cluster usage for the application that I am running, but the system > never seems to recover from the condition, which would seem to point to > an mbuf cluster leak. > > Does this sound like a problem with the driver (mbuf cluster leak), or > with the way that I have tuned this system? (the kernel config file for > this system is attached) Don't tune it this way: leave it at 16 or 32 users, and specifically increase the networking resources and MAXFILES (see /sys/i386/conf/LINT). Frankly, it sounds like your application is bad; does it limit itself to 150 connections, or is it trying to make as many connections as it possible can make? If so, then no matter how you tune the system, your program will always hit its head on a resource limit eventually. > I compiled a debug kernel and panicked the system while it was in the > state described above, in case that is any use. I don't know how to > analyze the crash dump to determine where the problem is. Any > suggestions are welcome. Did it panic from being in the state, or did you break to the debugger and force a crash? If it panic'ed, that's one thing: look at the traceback from the dump, and that should tell you where it blew up. You didn't _boot_ the kernel with the debugging symbols, right? That's a sure way to run yourself out of memory. You boot the kernel with the debugging information stripped (kernel, not kernel.debug, after doing a make following a config -g), and you only use the debug version for analyzing the system dump, or doing remote debugging (see the handbook). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B710F75.FF20B38>