Date: Tue, 11 Nov 2008 18:23:04 -0600 From: Paul Procacci <pprocacci@datapipe.com> To: freebsd-questions@freebsd.org Subject: Re: Tomcat Debugging (OT most likely) Message-ID: <491A21E8.2080209@datapipe.com> In-Reply-To: <491A17B9.7080705@datapipe.com> References: <491A17B9.7080705@datapipe.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Paul Procacci wrote: > Hey all, > > This question is probably best suited for tomcat-users, but being this > is an a FBSD machine I'd figure I'd ask here first. I have a client > running apache w/ tomcat serving thousands of requests per second. It > works, but with one caveat, pages load extremely slowly. When I first > saw the request, I figured this client was running into some kind of > resource bottleneck, but that's where my problems started. I'm unable > to find any resource that is being starved. I turn to `the mighty > list` for guidance. > > This particular machine is running FreeBSD 6.2. It has a bce network > card that currently connected to a GigE switch with a 10G uplink. > Even with all this available bandwidth, this client's application > *currently* is not doing any more than 50Mb/sec. :: Rule out network > congestion. > > Moving along to the system ram :: 2 Gigs are in the machine, with very > little usage: > Mem: 724M Active, 874M Inact, 338M Wired, 64M Cache, 112M Buf, 4944K Free > > A dual core cpu is in the machine, again with very little usage: > CPU states: 8.6% user, 0.0% nice, 3.9% system, 2.8% interrupt, > 84.7% idle > > Disk throughput is negligable at this time (50KB/sec :: 3tps). > > I did start off mentioning this machine uses tomcat...so let me > continue. We've setup the java process to use libthr via libmap.conf: > ############# > cat /etc/libmap.conf > [java] > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > ############# > > And added the following to rc.conf:: > ############### > tomcat55_java_opts="-XX:MaxPermSize=512m -Xmx512M" > ############### > > Apache Configuration: > ################### > MaxClients 1024 > ################### > > Tomcat Configuration: > ################### > maxThreads="1200" > ################### > > ---------------------------------------------------------------- > So onto my question...the slowness being encountered. My initial > thought was the slowness was due to quite possibly the cpu being > hammered by numerous requests. After checking however that's not the > case, then I looked at the networking equipment with my networking > team, that's ok too. (I think) That leaves a) thread contention > somewhere b) apache misconfiguration and c) tomcat configuration. > > Both myself and 2 other admins have looked over the apache > configuration and tomcat configuration and we believe that side of > things is probably ok. That's leaves weird contention in the kernel > or userland mutexes or something along those lines. > > Here is our current connection count on the external interface :: > ################################ > sockstat -4cp 80 | wc -l > 997 > ################################ > > Here is our connection count on localhost (from apache to tomcat > process and vica versa) :: > ################################ > sockstat -4cp 8009 | wc -l > 1679 > ################################ > ---------------------------------------------------------------- > > And finally.....top output: > > last pid: 57747; load averages: 0.79, 1.12, > 0.87 > up 153+17:56:53 18:36:57 > <#--snip --> > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 57460 www 96 0 1385M 432M ucond 0 2:44 2.88% java > 57460 www 96 0 1385M 432M ucond 0 2:44 2.73% java > 57460 www 96 0 1385M 432M select 0 2:44 0.15% java > 57460 www 96 0 1385M 432M select 1 2:44 0.15% java > <#--snip--> > > The above lines repeat up to the number of threads we have. `states` > are in ucond and select. > ---------------------------------------------------------------- > > Hopefully I hven't bored anyone....yet, and would appreciate any > guildance. Maybe good tools for debugged kernel threads, or a simple > slap in the face will do nicely. > > Thanks in advance, > Paul > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" List, Apologies, but I sever my head in shame as I have found the problem. In one word: KeepAlives. Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?491A21E8.2080209>