Date: Tue, 21 May 2002 19:27:58 -0700 From: "Wilbert deGraaf" <wilbertdg@hetnet.nl> To: "Scott Hess" <scott@avantgo.com> Cc: <freebsd-net@freebsd.org> Subject: Re: High volume proxy server configuration. Message-ID: <002201c20138$4a912de0$6601a8c0@alias> References: <Pine.LNX.4.44.0205211541210.8851-100000@river.avantgo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello Scott, > Here's my theory: When the amount of space used for user processes and > kernel usage fills all of memory, and a burst of packets are received from > the backend servers, the kernel isn't able to allocate pages and drops the > packets, with the message. The sender resends, and things cascade away. > Since this is a kernel vm issue, the console also locks up. [Well, it's > the best I have.] It sounds like the proxy doesn't implement flow control. What I mean is that if a proxy reads all it can from a server, and writes it to the client as fast it can, memory usage can easily explode if the connection to the client is slower that the server connection. Did you modify that part of the Apache proxy ? If so, you probably want to stop reading from the server until you have been able to send data to the client. That way, TCP takes care of it. I expected you to run into user level memory allocation problems, but since you ran out of NMBCLUSTERS, it could be that the proxy tries to write to the client at the speed it reads itself from the servers. In this case, I understand the problem that you describe. Hope this helps, Wilbert ----- Original Message ----- From: "Scott Hess" <scott@avantgo.com> To: <freebsd-net@freebsd.org> Sent: Tuesday, May 21, 2002 3:41 PM Subject: High volume proxy server configuration. > Background: I'm working on an intelligent Apache-based proxy server for > backend servers running a custom Apache module. The server does some > inspection of the incoming request to determine how to direct it, and > passes the reseponse directly back to the client. Thus, I'd like to be > able to set the TCP buffers fairly large, with the server merely acting as > a conduit to transfer data between the backend server and the client. > Upstream data is relatively small (a handful of kilobytes), downstream can > be large (100k-2Meg). > > Setup: 2x SMP server running FreeBSD4.5. Apache 1.3.x. 2Gig of memory. > > When stress-testing, I am able to cause the kernel messages: > > m_clalloc failed, consider increase NMBCLUSTERS value > fxp0: cluster allocation failed, packet dropped! > > The system hangs for a perhaps five minutes, and then comes back and is > able to continue operating. pings work, but the console isn't responsive > (I mean "no response until things clear a couple minutes later). I've > spent some time trying to tweak things, but I haven't been able to prevent > the problem. My /boot/loader.conf includes: > > kern.maxusers="512" > kern.ipc.nmbclusters="65536" > > The problem can happen at various points. I've seen it happen with the > mbuf cluster count <1k. Usually, the current/peak/max of netstat -m will > have peak nowhere near 65536. This usually happens when I have on the > order of 2000 processes/connections running - the machine is 80% idle at > this point, though. > > I wrote a program to specifically use up mbuf clusters (many servers write > lots of data, many clients sleep), and it didn't cause any problems until > hitting the maximum. Even then, the machine wasn't locked up at the > console. So I think the message is a symptom of something else. > > Here's my theory: When the amount of space used for user processes and > kernel usage fills all of memory, and a burst of packets are received from > the backend servers, the kernel isn't able to allocate pages and drops the > packets, with the message. The sender resends, and things cascade away. > Since this is a kernel vm issue, the console also locks up. [Well, it's > the best I have.] > > I've tried upping vm.v_free_min, vm.v_free_target, and vm.v_free_reserved. > It doesn't appear to have any impact. > > I was also getting the message: > > pmap_collect: collecting pv entries -- suggest increasing PMAP_SHPGPERPROC > > From what I can tell, this sounds like a direct results of running so many > processes forked from the same parent. Each process is small (SIZE ~4M). > I increased PMAP_SHPGPERPROC to 400, now I don't seem to get this message. > I've watched 'sysctl vm.zone', and the PV ENTRY line seems more > reasonable, now. > > The last line of vmstat output when this happens (broadly similar to > previous lines): > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id > 8 2 0 2141424 41184 8255 46 0 0 3982 0 0 0 3477 5416 1264 14 38 48 > > This is consistent with top: > > last pid: 79636; load averages: 3.51, 1.59, 0.83 up 0+22:23:16 16:37:14 > 2268 processes:9 running, 2259 sleeping > CPU states: 19.6% user, 0.0% nice, 19.6% system, 5.4% interrupt, 55.4% idle > Mem: 578M Active, 25M Inact, 361M Wired, 3528K Cache, 112M Buf, 37M Free > Swap: 2048M Total, 35M Used, 2012M Free, 1% Inuse > > [Hmm, one note - I'm replicating this on a 1Gig machine, but we've also > seen it in an extreme case on the 2Gig machine which is in production.] > > Hmm. vmstat just came back, the first two lines: > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id > 2268 2 0 2352192 62236 7308 59 0 32 3306 161397 28 0 2454 5111 1153 12 40 49 > 0 2 0 266364 46240 292845 1517 9 608 38036 6843317 1 0 334730 253302 17192 0 100 0 > > top shows increased space used in swap (42M, now), so it looks like we got > a bunch of swapping going on. [Just to be clear - when the event happens, > things don't simple get a bad response time. There's _no_ response, until > the problem clears and everything comes back. Then it's all shiny-happy, > again.] > > /etc/sysctl.conf has: > > kern.ipc.somaxconn=4192 > net.inet.ip.portrange.last=40000 > kern.ipc.maxsockbuf=2097152 > > We are definitely not using the full maxsockbuf range! Actually, we've > left things at the default (sendspace=32k, recvspace=64k). > > AFAICT, everything else is at default settings. > > Thanks for any help, > scott > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-net" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?002201c20138$4a912de0$6601a8c0>