From owner-freebsd-questions@FreeBSD.ORG Tue Nov 11 23:39:49 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB6151065672 for ; Tue, 11 Nov 2008 23:39:49 +0000 (UTC) (envelope-from pprocacci@datapipe.com) Received: from fmailhost04.isp.att.net (fmailhost04.isp.att.net [204.127.217.104]) by mx1.freebsd.org (Postfix) with ESMTP id B66218FC0C for ; Tue, 11 Nov 2008 23:39:49 +0000 (UTC) (envelope-from pprocacci@datapipe.com) Received: from [10.5.21.122] (adsl-241-169-92.bna.bellsouth.net[74.241.169.92]) by isp.att.net (frfwmhc04) with ESMTP id <20081111233948H04007683de>; Tue, 11 Nov 2008 23:39:49 +0000 X-Originating-IP: [74.241.169.92] Message-ID: <491A17B9.7080705@datapipe.com> Date: Tue, 11 Nov 2008 17:39:37 -0600 From: Paul Procacci User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Tomcat Debugging (OT most likely) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Nov 2008 23:39:49 -0000 Hey all, This question is probably best suited for tomcat-users, but being this is an a FBSD machine I'd figure I'd ask here first. I have a client running apache w/ tomcat serving thousands of requests per second. It works, but with one caveat, pages load extremely slowly. When I first saw the request, I figured this client was running into some kind of resource bottleneck, but that's where my problems started. I'm unable to find any resource that is being starved. I turn to `the mighty list` for guidance. This particular machine is running FreeBSD 6.2. It has a bce network card that currently connected to a GigE switch with a 10G uplink. Even with all this available bandwidth, this client's application *currently* is not doing any more than 50Mb/sec. :: Rule out network congestion. Moving along to the system ram :: 2 Gigs are in the machine, with very little usage: Mem: 724M Active, 874M Inact, 338M Wired, 64M Cache, 112M Buf, 4944K Free A dual core cpu is in the machine, again with very little usage: CPU states: 8.6% user, 0.0% nice, 3.9% system, 2.8% interrupt, 84.7% idle Disk throughput is negligable at this time (50KB/sec :: 3tps). I did start off mentioning this machine uses tomcat...so let me continue. We've setup the java process to use libthr via libmap.conf: ############# cat /etc/libmap.conf [java] libpthread.so.2 libthr.so.2 libpthread.so libthr.so ############# And added the following to rc.conf:: ############### tomcat55_java_opts="-XX:MaxPermSize=512m -Xmx512M" ############### Apache Configuration: ################### MaxClients 1024 ################### Tomcat Configuration: ################### maxThreads="1200" ################### ---------------------------------------------------------------- So onto my question...the slowness being encountered. My initial thought was the slowness was due to quite possibly the cpu being hammered by numerous requests. After checking however that's not the case, then I looked at the networking equipment with my networking team, that's ok too. (I think) That leaves a) thread contention somewhere b) apache misconfiguration and c) tomcat configuration. Both myself and 2 other admins have looked over the apache configuration and tomcat configuration and we believe that side of things is probably ok. That's leaves weird contention in the kernel or userland mutexes or something along those lines. Here is our current connection count on the external interface :: ################################ sockstat -4cp 80 | wc -l 997 ################################ Here is our connection count on localhost (from apache to tomcat process and vica versa) :: ################################ sockstat -4cp 8009 | wc -l 1679 ################################ ---------------------------------------------------------------- And finally.....top output: last pid: 57747; load averages: 0.79, 1.12, 0.87 up 153+17:56:53 18:36:57 <#--snip --> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 57460 www 96 0 1385M 432M ucond 0 2:44 2.88% java 57460 www 96 0 1385M 432M ucond 0 2:44 2.73% java 57460 www 96 0 1385M 432M select 0 2:44 0.15% java 57460 www 96 0 1385M 432M select 1 2:44 0.15% java <#--snip--> The above lines repeat up to the number of threads we have. `states` are in ucond and select. ---------------------------------------------------------------- Hopefully I hven't bored anyone....yet, and would appreciate any guildance. Maybe good tools for debugged kernel threads, or a simple slap in the face will do nicely. Thanks in advance, Paul