Date: Mon, 24 Jun 2002 19:16:26 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Patrick Thomas <root@utility.clubscholarship.com> Cc: freebsd-hackers@freebsd.org Subject: Re: tunings for many httpds... Message-ID: <3D17D27A.11E82B2B@mindspring.com> References: <20020624151650.I68572-100000@utility.clubscholarship.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Patrick Thomas wrote: > As a splinter to the ongoing KVA/crash/memory discussion, I am wondering: > > - given a machine that will run 250+ httpds and another ~800 misc. > processes, what system tunings would any of you suggest other than the > ones I have done: Can't comment. Best guess: whatever actually works. > (all this SHM and SEM stuff is to run multiple postgres') You didn't say this before. You have 64M of KVA space taken up by nothing but shared memory. THis is maybe twice the amount of memory that Matt freed up with his changes that you don't have. System V shared memory is allocated out of KVA space (annoying, but true). > Anything obvious I am missing ? Terry seems to think: > > <quote> > It's obvious that you are running a large number of httpd's; the > sbwait in this case could be reasonably assumed to be waits based > on "sendfile" for a change in so->so_snd->sb_cc; if that's the > case, then it may be that you are simply running out of mbufs, > and are deadlocking. This can happen if you have enough data in > the pipe that you can not receive more data (e.g. the m_pullup() > in tcp_input() could fail before other things would fail). > </quote> > > Two things about this interested me: > > a) watching `top` output anytime of the day, i see several httpd processes > in "sbwait" - granted I can only see 40 lines of processes or so in `top`, > but usually at least two show "sbwait". Worrisome ? This makes the "ps" you did pretty much useless. I expected that it was not the whole thing, but I at least expectged that it was representative. The real question is not what "top" shows, but what the kernel debug "ps" shows. > b) As I showed him, the netstat -m output 30-60 seconds before the crash > looks very benign: > > 524/2576/34816 mbufs in use (current/peak/max): > 500 mbufs allocated to data > 24 mbufs allocated to packet headers > 273/2254/8704 mbuf clusters in use (current/peak/max) > 5152 Kbytes allocated to network (19% of mb_map in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines Yes, this looks benign. > Is it possible that within 30 seconds or so current mbufs would skyrocket > and my percentage of mb_map in use would skyrocket and I would start to > see requests for memory denied ? Yes, it's possible, if you are being attacked. We don't know whether you are or not. The "top" won't update after the system siezes up, and running netstat -m over one minute intervals will probably not even show a trend upward, if the problem were real memory. You'd have to be really lucky to catch it at just point right before it siezed. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3D17D27A.11E82B2B>