From owner-freebsd-performance@FreeBSD.ORG Mon Nov 29 12:41:17 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4AAE016A4CE; Mon, 29 Nov 2004 12:41:17 +0000 (GMT) Received: from server1.carmatec.com (server1.carmatec.com [66.45.229.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0950243D1D; Mon, 29 Nov 2004 12:41:17 +0000 (GMT) (envelope-from akhthar@carmatec.com) Received: from [61.95.203.89] (helo=ws13.carmatec.com) by server1.carmatec.com with esmtpa (Exim 4.43) id 1CYkqA-0001oP-UC; Mon, 29 Nov 2004 07:40:53 -0500 From: "Akhthar Parvez. K" Organization: Carmatec Solutions To: freebsd-performance@freebsd.org Date: Mon, 29 Nov 2004 18:12:20 +0530 User-Agent: KMail/1.5 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200411291812.20739.akhthar@carmatec.com> X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server1.carmatec.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - carmatec.com X-Source: X-Source-Args: X-Source-Dir: cc: freebsd-questions@freebsd.org Subject: Error message X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: akhthar@carmatec.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Nov 2004 12:41:17 -0000 Hi All, I am getting the following error message in /var/log/messages tail -f /var/log/messages Nov 29 07:24:31 speedy /kernel: pid 83876 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:31 speedy /kernel: pid 84126 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:31 speedy /kernel: pid 84208 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:33 speedy /kernel: pid 83867 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:34 speedy /kernel: pid 84207 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:35 speedy /kernel: pid 84138 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:36 speedy /kernel: pid 84064 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:37 speedy /kernel: pid 84141 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:38 speedy /kernel: pid 84255 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:39 speedy /kernel: pid 84233 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:42 speedy /kernel: pid 84258 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:43 speedy /kernel: pid 84165 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:43 speedy /kernel: pid 84200 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:45 speedy /kernel: pid 84174 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:45 speedy /kernel: pid 84199 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:49 speedy /kernel: pid 83871 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:50 speedy /kernel: pid 84164 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:52 speedy /kernel: pid 84102 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:52 speedy /kernel: pid 84167 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:52 speedy /kernel: pid 84212 (httpd), uid 65534: exited on signal 4 Nov 29 07:24:53 speedy /kernel: pid 84168 (httpd), uid 65534: exited on signal 4 Anyhave has any idea why it's coming? I am not getting any error messsages in apache error logs. -- With Regards, Akhthar Parvez.K --------------------- NOTHING IS IMPOSSIBLE Because Impossible itself says I'M POSSIBLE From owner-freebsd-performance@FreeBSD.ORG Mon Nov 29 13:44:39 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 666E516A4CE; Mon, 29 Nov 2004 13:44:39 +0000 (GMT) Received: from internet.potentialtech.com (h-66-167-251-6.phlapafg.covad.net [66.167.251.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D90643D3F; Mon, 29 Nov 2004 13:44:39 +0000 (GMT) (envelope-from wmoran@potentialtech.com) Received: from working.potentialtech.com (pa-plum-cmts1e-68-68-113-64.pittpa.adelphia.net [68.68.113.64]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by internet.potentialtech.com (Postfix) with ESMTP id 0847C69A3F; Mon, 29 Nov 2004 08:44:37 -0500 (EST) Date: Mon, 29 Nov 2004 08:44:36 -0500 From: Bill Moran To: akhthar@carmatec.com Message-Id: <20041129084436.1f564096.wmoran@potentialtech.com> In-Reply-To: <200411291812.20739.akhthar@carmatec.com> References: <200411291812.20739.akhthar@carmatec.com> Organization: Potential Technologies X-Mailer: Sylpheed version 0.9.99 (GTK+ 1.2.10; i386-portbld-freebsd4.10) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org cc: freebsd-questions@freebsd.org Subject: Re: Error message X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Nov 2004 13:44:39 -0000 "Akhthar Parvez. K" wrote: > Hi All, > > I am getting the following error message in /var/log/messages > > tail -f /var/log/messages > Nov 29 07:24:31 speedy /kernel: pid 83876 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:31 speedy /kernel: pid 84126 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:31 speedy /kernel: pid 84208 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:33 speedy /kernel: pid 83867 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:34 speedy /kernel: pid 84207 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:35 speedy /kernel: pid 84138 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:36 speedy /kernel: pid 84064 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:37 speedy /kernel: pid 84141 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:38 speedy /kernel: pid 84255 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:39 speedy /kernel: pid 84233 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:42 speedy /kernel: pid 84258 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:43 speedy /kernel: pid 84165 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:43 speedy /kernel: pid 84200 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:45 speedy /kernel: pid 84174 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:45 speedy /kernel: pid 84199 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:49 speedy /kernel: pid 83871 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:50 speedy /kernel: pid 84164 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:52 speedy /kernel: pid 84102 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:52 speedy /kernel: pid 84167 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:52 speedy /kernel: pid 84212 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:53 speedy /kernel: pid 84168 (httpd), uid 65534: exited on signal 4 > > > Anyhave has any idea why it's coming? I am not getting any error > messsages in apache error logs. Please wrap your lines around 72 chars. See http://www.lemis.com/questions.html What version of Apache are you running? What modules does it have installed. Answering those questions will narrow down the problem and you can start researching it in more detail. Also, looking through your http access and error logs to correlate times will help tell you what's causing the problem. -- Bill Moran Potential Technologies http://www.potentialtech.com From owner-freebsd-performance@FreeBSD.ORG Mon Nov 29 14:17:41 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8546216A4CE; Mon, 29 Nov 2004 14:17:41 +0000 (GMT) Received: from mail.geeks.org (jacobs.Geeks.ORG [204.153.247.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2129C43D2F; Mon, 29 Nov 2004 14:17:41 +0000 (GMT) (envelope-from drechsau@Geeks.ORG) Received: by mail.geeks.org (Postfix, from userid 400) id 774B020B51; Mon, 29 Nov 2004 08:17:40 -0600 (CST) Date: Mon, 29 Nov 2004 08:17:40 -0600 From: Mike Horwath To: "Akhthar Parvez. K" Message-ID: <20041129141740.GA16560@octanews.net> References: <200411291812.20739.akhthar@carmatec.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200411291812.20739.akhthar@carmatec.com> X-PGP-Fingerprint: D8 24 CC E6 47 5F E4 60 BF B7 6E FA BF C7 6E C5 X-GPG-Fingerprint: 6A89 E78A B8B1 69D9 8CDB E966 4A5A C3F9 A1B0 C381 User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org cc: freebsd-questions@freebsd.org Subject: Re: Error message X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Nov 2004 14:17:41 -0000 On Mon, Nov 29, 2004 at 06:12:20PM +0530, Akhthar Parvez. K wrote: > Hi All, > > I am getting the following error message in /var/log/messages > > tail -f /var/log/messages > Nov 29 07:24:31 speedy /kernel: pid 83876 (httpd), uid 65534: exited on signal 4 > Nov 29 07:24:31 speedy /kernel: pid 84126 (httpd), uid 65534: exited on signal 4 [snipper] #define SIGILL 4 /* illegal instr. (not reset when caught) */ > Anyhave has any idea why it's coming? I am not getting any error > messsages in apache error logs. Only time I have seen this kind of thing is bad hardware. -- Mike Horwath, reachable via drechsau@Geeks.ORG From owner-freebsd-performance@FreeBSD.ORG Wed Dec 1 10:51:13 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AD27116A4CE for ; Wed, 1 Dec 2004 10:51:13 +0000 (GMT) Received: from web14102.mail.yahoo.com (web14102.mail.yahoo.com [216.136.172.132]) by mx1.FreeBSD.org (Postfix) with SMTP id 848CC43D66 for ; Wed, 1 Dec 2004 10:51:13 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 79350 invoked by uid 60001); 1 Dec 2004 10:51:13 -0000 Message-ID: <20041201105113.79348.qmail@web14102.mail.yahoo.com> Received: from [194.248.174.58] by web14102.mail.yahoo.com via HTTP; Wed, 01 Dec 2004 11:51:13 CET Date: Wed, 1 Dec 2004 11:51:13 +0100 (CET) From: Claus Guttesen To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: postgresql on FreeBSD 5.3 and high load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2004 10:51:13 -0000 Hi. Our postgresql-server hits the wall when we get many hits on our web. The load-average reaches 70. It serves 11 webservers. The server is a Dell PE 2850 with 4 GB RAM, dual Nocona at 3.2 GHz, HTT is not enabled. It's running FreeBSD 5.3 stable (amd64-port). Postgresql ver. 7.4.5 from ports. The SCSI-controller is LSILogic MegaRAID with six disks and three stripes which is mirrored (RAID 0+1). Only hardware RAID. Due to problems with 4 GB RAM I added hw.physmem=4G to /boot/loader.conf, which for some reason reduced available memory to 3211 MB. Since then it has run stable. I have this custom-kernel: options SHMMAXPGS=262144 options SEMMNI=160 options SEMMNS=960 options SEMUME=160 options SEMMNU=480 options PMAP_SHPGPERPROC=1280 postgresql.conf: max_connections = 768 shared_buffers = 65536 sort_mem = 4096 vacuum_mem = 65536 checkpoint_segments = 4 commit_delay = 100000 commit_siblings = 250 effective_cache_size = 32768 Fsync is *not* enabled. I have increased shared_buffers and commit_siblings gradually, but it does not seem to lower the load on the server. /etc/sysctl.conf: kern.ipc.somaxconn=2048 kern.maxfiles=65536 net.isr.enable=1 ipcs -M: shminfo: shmmax: 1073741824 (max shared memory segment size) shmmin: 1 (min shared memory segment size) shmmni: 192 (max number of shared memory identifiers) shmseg: 128 (max shared memory segments per process) shmall: 262144 (max amount of shared memory in pages) Eventhough load average is high, it's not always 0 % idle, but between 0 and 2 percent. netstat -m 18446744073709545011 mbufs in use 4519/25600 mbuf clusters in use (current/max) 0/0/0 sfbufs in use (current/peak/max) 7386 KBytes allocated to network 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 20286 calls to protocol drain routines I can't find the bottleneck. regards Claus From owner-freebsd-performance@FreeBSD.ORG Wed Dec 1 13:21:29 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B7AE616A4CE for ; Wed, 1 Dec 2004 13:21:29 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.194]) by mx1.FreeBSD.org (Postfix) with ESMTP id 57D1743D64 for ; Wed, 1 Dec 2004 13:21:29 +0000 (GMT) (envelope-from vladgalu@gmail.com) Received: by rproxy.gmail.com with SMTP id b11so604852rne for ; Wed, 01 Dec 2004 05:21:28 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=KAwaVbJ76BjvlaD/y6q7C3FIP1MROJs6BdSdvd/3TtyhZO3EXBViyumsBLyESPJqzr3Y+3yRfa3xUmMcTE6hjUv0p2q4ifHu9rvSgB9jjwUXYtLJkMjcHGHGoVTJbbWqHqJKMRRFdFj8xRmrXs0X+5s19oxU7BvAWaTieCdRjBo= Received: by 10.39.1.9 with SMTP id d9mr87445rni; Wed, 01 Dec 2004 05:21:28 -0800 (PST) Received: by 10.38.149.75 with HTTP; Wed, 1 Dec 2004 05:21:28 -0800 (PST) Message-ID: <79722fad041201052135e6028f@mail.gmail.com> Date: Wed, 1 Dec 2004 15:21:28 +0200 From: Vlad GALU To: freebsd-performance@freebsd.org In-Reply-To: <20041201105113.79348.qmail@web14102.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20041201105113.79348.qmail@web14102.mail.yahoo.com> Subject: Re: postgresql on FreeBSD 5.3 and high load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Vlad GALU List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2004 13:21:29 -0000 On Wed, 1 Dec 2004 11:51:13 +0100 (CET), Claus Guttesen wrote: > Hi. > > Our postgresql-server hits the wall when we get many > hits on our web. The load-average reaches 70. It > serves 11 webservers. > > The server is a Dell PE 2850 with 4 GB RAM, dual > Nocona at 3.2 GHz, HTT is not enabled. It's running > FreeBSD 5.3 stable (amd64-port). Postgresql ver. 7.4.5 > from ports. The SCSI-controller is LSILogic MegaRAID > with six disks and three stripes which is mirrored > (RAID 0+1). Only hardware RAID. > > Due to problems with 4 GB RAM I added hw.physmem=4G to > /boot/loader.conf, which for some reason reduced > available memory to 3211 MB. Since then it has run > stable. > > I have this custom-kernel: > > options SHMMAXPGS=262144 > options SEMMNI=160 > options SEMMNS=960 > options SEMUME=160 > options SEMMNU=480 > options PMAP_SHPGPERPROC=1280 > > postgresql.conf: > > max_connections = 768 > shared_buffers = 65536 > sort_mem = 4096 > vacuum_mem = 65536 > checkpoint_segments = 4 > commit_delay = 100000 > commit_siblings = 250 > effective_cache_size = 32768 > > Fsync is *not* enabled. I have increased > shared_buffers and commit_siblings gradually, but it > does not seem to lower the load on the server. > > /etc/sysctl.conf: > kern.ipc.somaxconn=2048 > kern.maxfiles=65536 > net.isr.enable=1 > > ipcs -M: > shminfo: > shmmax: 1073741824 (max shared memory segment size) > shmmin: 1 (min shared memory segment size) > shmmni: 192 (max number of shared memory > identifiers) > shmseg: 128 (max shared memory segments per > process) > shmall: 262144 (max amount of shared memory in > pages) > > Eventhough load average is high, it's not always 0 % > idle, but between 0 and 2 percent. > > netstat -m > 18446744073709545011 mbufs in use ^^^^^^^^^^^^^^ ?????? > 4519/25600 mbuf clusters in use (current/max) > 0/0/0 sfbufs in use (current/peak/max) > 7386 KBytes allocated to network > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 20286 calls to protocol drain routines > > I can't find the bottleneck. > > regards > Claus > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > -- If it's there, and you can see it, it's real. If it's not there, and you can see it, it's virtual. If it's there, and you can't see it, it's transparent. If it's not there, and you can't see it, you erased it. From owner-freebsd-performance@FreeBSD.ORG Wed Dec 1 13:55:33 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2A4D316A4CE for ; Wed, 1 Dec 2004 13:55:33 +0000 (GMT) Received: from web14102.mail.yahoo.com (web14102.mail.yahoo.com [216.136.172.132]) by mx1.FreeBSD.org (Postfix) with SMTP id 182FA43D46 for ; Wed, 1 Dec 2004 13:55:33 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 508 invoked by uid 60001); 1 Dec 2004 13:55:32 -0000 Message-ID: <20041201135532.506.qmail@web14102.mail.yahoo.com> Received: from [194.248.174.58] by web14102.mail.yahoo.com via HTTP; Wed, 01 Dec 2004 14:55:32 CET Date: Wed, 1 Dec 2004 14:55:32 +0100 (CET) From: Claus Guttesen To: Vlad GALU , freebsd-performance@freebsd.org In-Reply-To: <79722fad041201052135e6028f@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: postgresql on FreeBSD 5.3 and high load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2004 13:55:33 -0000 >> idle, but between 0 and 2 percent. >> >> netstat -m > >> 18446744073709545011 mbufs in use > ^^^^^^^^^^^^^^ > ?????? > Yes, it's rather odd. But that's a cut'n'paste from the server. The high load may stem from the high mbuf value. Claus From owner-freebsd-performance@FreeBSD.ORG Wed Dec 1 17:13:59 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F274916A4CE for ; Wed, 1 Dec 2004 17:13:59 +0000 (GMT) Received: from mail.trippynames.com (mail.trippynames.com [38.113.223.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB9B743D49 for ; Wed, 1 Dec 2004 17:13:59 +0000 (GMT) (envelope-from sean@chittenden.org) Received: from localhost (localhost [127.0.0.1]) by mail.trippynames.com (Postfix) with ESMTP id 906E0A6CED; Wed, 1 Dec 2004 09:13:59 -0800 (PST) Received: from mail.trippynames.com ([127.0.0.1]) by localhost (rand.nxad.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 97852-04; Wed, 1 Dec 2004 09:13:58 -0800 (PST) Received: from [192.168.102.101] (c-67-168-93-230.client.comcast.net [67.168.93.230]) by mail.trippynames.com (Postfix) with ESMTP id C568AA2D8D; Wed, 1 Dec 2004 09:13:57 -0800 (PST) In-Reply-To: <20041201105113.79348.qmail@web14102.mail.yahoo.com> References: <20041201105113.79348.qmail@web14102.mail.yahoo.com> Mime-Version: 1.0 (Apple Message framework v619) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <6174636A-43BC-11D9-ACAF-000A95C705DC@chittenden.org> Content-Transfer-Encoding: 7bit From: Sean Chittenden Date: Wed, 1 Dec 2004 09:13:55 -0800 To: Claus Guttesen X-Mailer: Apple Mail (2.619) cc: freebsd-performance@freebsd.org Subject: Re: postgresql on FreeBSD 5.3 and high load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2004 17:14:00 -0000 > Our postgresql-server hits the wall when we get many > hits on our web. The load-average reaches 70. It > serves 11 webservers. Have you looked to see if you're using spin locks or not? Search for "amd64 x86_64 spin lock" in PostgreSQL's archives. I think spin locks were disabled for amd64. When you're at a load of 70, look in pg_catalog.pg_locks to see if you have any ungranted locks. Also, is your application primarily read? You may want to investigate using pgpool as a way of reducing the overhead for connection startup. Another option is to use memcached and remove some queries from the database all together. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Wed Dec 1 22:23:29 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A6D6116A4CE for ; Wed, 1 Dec 2004 22:23:29 +0000 (GMT) Received: from mail2.dbitech.ca (radius.wavefire.com [64.141.13.252]) by mx1.FreeBSD.org (Postfix) with SMTP id 25DC843D1D for ; Wed, 1 Dec 2004 22:23:29 +0000 (GMT) (envelope-from darcy@wavefire.com) Received: (qmail 25056 invoked from network); 1 Dec 2004 23:28:51 -0000 Received: from dbitech.wavefire.com (HELO ?64.141.15.253?) (darcy@64.141.15.253) by radius.wavefire.com with SMTP; 1 Dec 2004 23:28:51 -0000 From: Darcy Buskermolen Organization: Wavefire Technologies Corp. To: freebsd-performance@freebsd.org Date: Wed, 1 Dec 2004 14:23:27 -0800 User-Agent: KMail/1.6.2 References: <20041201105113.79348.qmail@web14102.mail.yahoo.com> <6174636A-43BC-11D9-ACAF-000A95C705DC@chittenden.org> In-Reply-To: <6174636A-43BC-11D9-ACAF-000A95C705DC@chittenden.org> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200412011423.27424.darcy@wavefire.com> cc: Claus Guttesen Subject: Re: postgresql on FreeBSD 5.3 and high load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2004 22:23:29 -0000 On December 1, 2004 09:13 am, Sean Chittenden wrote: > > Our postgresql-server hits the wall when we get many > > hits on our web. The load-average reaches 70. It > > serves 11 webservers. > > Have you looked to see if you're using spin locks or not? Search for > "amd64 x86_64 spin lock" in PostgreSQL's archives. I think spin locks > were disabled for amd64. When you're at a load of 70, look in > pg_catalog.pg_locks to see if you have any ungranted locks. > > Also, is your application primarily read? You may want to investigate > using pgpool as a way of reducing the overhead for connection startup. > > Another option is to use memcached and remove some queries from the > database all together. -sc Umm I'm not sure where the misscommunication is (it could be at my end), but if I'm not mistalen a DELL 2850 is a dual Xeon EM64T box so all of the amd64 stuff is moot on this platform. -- Darcy Buskermolen Wavefire Technologies Corp. ph: 250.717.0200 fx: 250.763.1759 http://www.wavefire.com From owner-freebsd-performance@FreeBSD.ORG Thu Dec 2 00:41:33 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 711B016A4CE; Thu, 2 Dec 2004 00:41:33 +0000 (GMT) Received: from sb.santaba.com (sb.santaba.com [207.154.84.26]) by mx1.FreeBSD.org (Postfix) with ESMTP id 505FC43D3F; Thu, 2 Dec 2004 00:41:33 +0000 (GMT) (envelope-from jbehl@fastclick.com) Received: from [192.168.3.100] (unknown [205.180.85.193]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by sb.santaba.com (Postfix) with ESMTP id ECAAF28433; Wed, 1 Dec 2004 16:41:32 -0800 (PST) Message-ID: <41AE651F.3040104@fastclick.com> Date: Wed, 01 Dec 2004 16:43:11 -0800 From: Jeff Behl User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: %cpu in system - squid performance in FreeBSD 5.3 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Dec 2004 00:41:33 -0000 howdy, I've got a dual proc AMD64 (2gHz) FreeBSD 5.3R system running two squid processes (to take advantage of both CPUs). Each process is doing around 195 req/s, and the total bandwidth is ~40Mb/s (gig nic via bge driver). Squid is being used exclusively as a reverse proxy, with all content being served out of memory (very little disk activity). Top shows: CPU states: 16.0% user, 0.0% nice, 42.7% system, 7.6% interrupt, 33.6% idle Mem: 898M Active, 569M Inact, 179M Wired, 214M Buf, 171M Free Swap: 4069M Total, 4069M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 14598 squid 108 0 463M 459M select 0 39.2H 59.96% 59.96% squid 14605 squid 105 0 421M 416M CPU0 1 38.4H 49.95% 49.95% squid but the % system time can fluctuate up to 60 at times. My question is if this is about the type of performance I could expect, or if people have seen better. I was expecting to see much better performance, seeing how everything is being served out of memory, but maybe I'm asking too much? 400 reqs/s from RAM doesn't seem like much. Is this a FreeBSD issue (anybody else with similar experience)? A majority of the cpu time being spent in system would seem to indictate such. What is all the system load? How can i tell? Any help/pointers/remarks appreciated thanks, jeff FreeBSD www1.cdn.sjc 5.3-RELEASE FreeBSD 5.3-RELEASE #1: Mon Nov 8 20:25:59 PST 2004 root@www1.cdn.sjc:/usr/obj/usr/src/sys/SMP amd64 From owner-freebsd-performance@FreeBSD.ORG Wed Dec 1 22:30:32 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4C7A16A4CE for ; Wed, 1 Dec 2004 22:30:32 +0000 (GMT) Received: from zaphod.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 312C643D2D for ; Wed, 1 Dec 2004 22:30:32 +0000 (GMT) (envelope-from simon@zaphod.nitro.dk) Received: by zaphod.nitro.dk (Postfix, from userid 3000) id 0E743119C7; Wed, 1 Dec 2004 23:30:31 +0100 (CET) Date: Wed, 1 Dec 2004 23:30:30 +0100 From: "Simon L. Nielsen" To: Darcy Buskermolen Message-ID: <20041201223030.GI757@zaphod.nitro.dk> References: <20041201105113.79348.qmail@web14102.mail.yahoo.com> <6174636A-43BC-11D9-ACAF-000A95C705DC@chittenden.org> <200412011423.27424.darcy@wavefire.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ylS2wUBXLOxYXZFQ" Content-Disposition: inline In-Reply-To: <200412011423.27424.darcy@wavefire.com> User-Agent: Mutt/1.5.6i X-Mailman-Approved-At: Thu, 02 Dec 2004 13:49:37 +0000 cc: freebsd-performance@freebsd.org cc: Claus Guttesen Subject: Re: postgresql on FreeBSD 5.3 and high load X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Dec 2004 22:30:32 -0000 --ylS2wUBXLOxYXZFQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2004.12.01 14:23:27 -0800, Darcy Buskermolen wrote: > On December 1, 2004 09:13 am, Sean Chittenden wrote: > > > Our postgresql-server hits the wall when we get many > > > hits on our web. The load-average reaches 70. It > > > serves 11 webservers. > > > > Have you looked to see if you're using spin locks or not? Search for > > "amd64 x86_64 spin lock" in PostgreSQL's archives. I think spin locks > > were disabled for amd64. When you're at a load of 70, look in > > pg_catalog.pg_locks to see if you have any ungranted locks. > > > > Also, is your application primarily read? You may want to investigate > > using pgpool as a way of reducing the overhead for connection startup. > > > > Another option is to use memcached and remove some queries from the > > database all together. -sc >=20 > Umm I'm not sure where the misscommunication is (it could be at my end), = but=20 > if I'm not mistalen a DELL 2850 is a dual Xeon EM64T box so all of the a= md64=20 > stuff is moot on this platform. EM64T is Intel's name for amd64 (with a few minor differences AFAIR), since it would be kind of embarrassing for Intel to really admit they had to copy AMD :-). --=20 Simon L. Nielsen --ylS2wUBXLOxYXZFQ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFBrkYGh9pcDSc1mlERAmqEAKCdBHHQlvIE3eVGkup2Xby4o0JvgACfSqPo lSyyqlU1CFGHYDZqgunkjlE= =WhiW -----END PGP SIGNATURE----- --ylS2wUBXLOxYXZFQ-- From owner-freebsd-performance@FreeBSD.ORG Fri Dec 3 01:51:50 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5670E16A4CE for ; Fri, 3 Dec 2004 01:51:50 +0000 (GMT) Received: from mail.trippynames.com (mail.trippynames.com [38.113.223.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4046B43D2F for ; Fri, 3 Dec 2004 01:51:50 +0000 (GMT) (envelope-from sean@chittenden.org) Received: from localhost (localhost [127.0.0.1]) by mail.trippynames.com (Postfix) with ESMTP id 82810A6DA4; Thu, 2 Dec 2004 17:51:47 -0800 (PST) Received: from mail.trippynames.com ([127.0.0.1]) by localhost (rand.nxad.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 30437-03; Thu, 2 Dec 2004 17:51:46 -0800 (PST) Received: from [192.168.123.123] (unknown [38.113.223.82]) by mail.trippynames.com (Postfix) with ESMTP id E921BA124C; Thu, 2 Dec 2004 17:51:45 -0800 (PST) In-Reply-To: <41AE651F.3040104@fastclick.com> References: <41AE651F.3040104@fastclick.com> Mime-Version: 1.0 (Apple Message framework v619) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Sean Chittenden Date: Thu, 2 Dec 2004 17:51:45 -0800 To: Jeff Behl X-Mailer: Apple Mail (2.619) cc: freebsd-performance@freebsd.org Subject: Re: %cpu in system - squid performance in FreeBSD 5.3 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Dec 2004 01:51:50 -0000 > but the % system time can fluctuate up to 60 at times. My question is > if this is about the type of performance I could expect, or if people > have seen better. I don't know about other people, but I suspect you're running into lock contention. Try using a post 5.3 snapshot (something from RELENG_5) since alc@ has set debug.mpsafevm=1, which lets many calls to the VM run without GIANT, which I suspect is your problem and why the system usage is all over the place. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Fri Dec 3 21:26:01 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 975A116A4CE; Fri, 3 Dec 2004 21:26:01 +0000 (GMT) Received: from mailout01.sul.t-online.com (mailout01.sul.t-online.com [194.25.134.80]) by mx1.FreeBSD.org (Postfix) with ESMTP id 26ED343D55; Fri, 3 Dec 2004 21:26:01 +0000 (GMT) (envelope-from Alexander@Leidinger.net) Received: from fwd06.aul.t-online.de by mailout01.sul.t-online.com with smtp id 1CaKwB-00033y-01; Fri, 03 Dec 2004 22:25:59 +0100 Received: from Andro-Beta.Leidinger.net (E27wXMZFQeOx520II16Yfu+Bi+Im4eKagMtfcDdcF22IFPtuUzTFci@[84.128.206.244]) by fmrl06.sul.t-online.com with esmtp id 1CaKw3-1r0IAS0; Fri, 3 Dec 2004 22:25:51 +0100 Received: from Magellan.Leidinger.net (Magellan.Leidinger.net [192.168.1.1]) iB3LPf0Y021164; Fri, 3 Dec 2004 22:25:41 +0100 (CET) (envelope-from Alexander@Leidinger.net) Date: Fri, 3 Dec 2004 22:26:41 +0100 From: Alexander Leidinger To: Mike Horwath Message-ID: <20041203222641.7ba106d0@Magellan.Leidinger.net> In-Reply-To: <20041129141740.GA16560@octanews.net> References: <200411291812.20739.akhthar@carmatec.com> <20041129141740.GA16560@octanews.net> X-Mailer: Sylpheed-Claws 0.9.12b (GTK+ 1.2.10; i386-portbld-freebsd6.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ID: E27wXMZFQeOx520II16Yfu+Bi+Im4eKagMtfcDdcF22IFPtuUzTFci@t-dialin.net X-TOI-MSGID: f0064986-4f33-49da-9b1e-005260df07ae cc: "Akhthar Parvez. K" cc: freebsd-performance@freebsd.org cc: freebsd-questions@freebsd.org Subject: Re: Error message X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Dec 2004 21:26:01 -0000 On Mon, 29 Nov 2004 08:17:40 -0600 Mike Horwath wrote: > On Mon, Nov 29, 2004 at 06:12:20PM +0530, Akhthar Parvez. K wrote: > > Hi All, > > > > I am getting the following error message in /var/log/messages > > > > tail -f /var/log/messages > > Nov 29 07:24:31 speedy /kernel: pid 83876 (httpd), uid 65534: exited on signal 4 > > Nov 29 07:24:31 speedy /kernel: pid 84126 (httpd), uid 65534: exited on signal 4 > > [snipper] > > #define SIGILL 4 /* illegal instr. (not reset when caught) */ > > > Anyhave has any idea why it's coming? I am not getting any error > > messsages in apache error logs. > > Only time I have seen this kind of thing is bad hardware. It may also be the case that the wrong CPUTYPE is/was specified in /etc/make.conf. Bye, Alexander. -- The best things in life are free, but the expensive ones are still worth a look. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 From owner-freebsd-performance@FreeBSD.ORG Fri Dec 3 21:52:28 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C8D8316A4CE; Fri, 3 Dec 2004 21:52:28 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7CF0F43D53; Fri, 3 Dec 2004 21:52:27 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by silver.he.iki.fi (8.13.1/8.11.4) with ESMTP id iB3LqNrg045200; Fri, 3 Dec 2004 23:52:23 +0200 (EET) (envelope-from pete@he.iki.fi) Message-ID: <41B0E018.8050606@he.iki.fi> Date: Fri, 03 Dec 2004 23:52:24 +0200 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040803 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Alexander Leidinger References: <200411291812.20739.akhthar@carmatec.com> <20041129141740.GA16560@octanews.net> <20041203222641.7ba106d0@Magellan.Leidinger.net> In-Reply-To: <20041203222641.7ba106d0@Magellan.Leidinger.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: "Akhthar Parvez. K" cc: freebsd-performance@freebsd.org cc: Mike Horwath cc: freebsd-questions@freebsd.org Subject: Re: Error message X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Dec 2004 21:52:28 -0000 Alexander Leidinger wrote: >On Mon, 29 Nov 2004 08:17:40 -0600 >Mike Horwath wrote: > > > >>On Mon, Nov 29, 2004 at 06:12:20PM +0530, Akhthar Parvez. K wrote: >> >> >>>Hi All, >>> >>>I am getting the following error message in /var/log/messages >>> >>>tail -f /var/log/messages >>>Nov 29 07:24:31 speedy /kernel: pid 83876 (httpd), uid 65534: exited on signal 4 >>>Nov 29 07:24:31 speedy /kernel: pid 84126 (httpd), uid 65534: exited on signal 4 >>> >>> >>[snipper] >> >>#define SIGILL 4 /* illegal instr. (not reset when caught) */ >> >> >> >>>Anyhave has any idea why it's coming? I am not getting any error >>>messsages in apache error logs. >>> >>> >>Only time I have seen this kind of thing is bad hardware. >> >> > >It may also be the case that the wrong CPUTYPE is/was specified in >/etc/make.conf. > > Are you running old enough version of apache to be vulnerable to remote exploits but they're trying to feed you wrong shellcode? Pete