From owner-freebsd-questions@FreeBSD.ORG Wed Jun 21 07:37:11 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 556A416A474 for ; Wed, 21 Jun 2006 07:37:11 +0000 (UTC) (envelope-from travis@fitch.id.au) Received: from kaur.zehym.com (fitch.id.au [203.89.243.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id C75C843D46 for ; Wed, 21 Jun 2006 07:37:10 +0000 (GMT) (envelope-from travis@fitch.id.au) Received: from [203.89.220.206] ([203.89.220.206]) (authenticated bits=0) by kaur.zehym.com (8.13.1/8.13.1) with ESMTP id k5L7Sfra051837; Wed, 21 Jun 2006 17:28:48 +1000 (EST) (envelope-from travis@fitch.id.au) Message-ID: <4498F719.8090802@fitch.id.au> Date: Wed, 21 Jun 2006 17:36:57 +1000 From: Travis Fitch User-Agent: Thunderbird 1.5.0.4 (X11/20060615) MIME-Version: 1.0 To: freebsd-questions@freebsd.org References: <200606190148.k5J1mfxY037157@kaur.zehym.com> In-Reply-To: <200606190148.k5J1mfxY037157@kaur.zehym.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Sendmail and high kernel CPU utilisation X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2006 07:37:11 -0000 Travis Fitch wrote: > Hello, > > I was hoping someone might be able to shed some light on an issue I am > having with sendmail on FreeBSD 5.2.1 running on a Sun V120. > > If you look at the snippet from top you can see that several sendmail > process are casing the kernel to use ~ 65% of the CPU. I am having some > issues profiling what is causing this issue. You can also see the load is > quite high. > > last pid: 72705; load averages: 12.22, 12.29, 12.00 > up 1+06:09:44 11:42:28 > 86 processes: 14 running, 72 sleeping > CPU states: 35.5% user, 0.0% nice, 64.5% system, 0.0% interrupt, 0.0% > idle > Mem: 148M Active, 104M Inact, 73M Wired, 520K Cache, 60M Buf, 131M Free > Swap: 2057M Total, 102M Used, 1954M Free, 4% Inuse > > PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND > 20958 root 122 0 23368K 5472K RUN 44:32 8.64% 8.64% sendmail > 62864 root 122 0 23432K 5528K RUN 10:45 8.54% 8.54% sendmail > 70522 root 122 0 23448K 5544K RUN 2:43 8.35% 8.35% sendmail > 91279 smmsp 122 0 12224K 3832K RUN 60:17 8.25% 8.25% sendmail > 66302 root 122 0 23392K 5472K RUN 8:36 8.25% 8.25% sendmail > 16850 root 122 0 23432K 5528K RUN 51:31 8.20% 8.20% sendmail > 66306 root 121 0 23448K 5528K RUN 8:34 8.11% 8.11% sendmail > 68330 root 121 0 23432K 5528K RUN 5:37 8.11% 8.11% sendmail > 51654 root 121 0 23392K 5472K RUN 16:40 8.06% 8.06% sendmail > 66377 root 121 0 23392K 5472K RUN 8:22 8.01% 8.01% sendmail > 69364 root 121 0 23432K 5504K RUN 3:50 8.01% 8.01% sendmail > > > The other interesting thing is that a lot of the SMTP connections hang > around for quite some time. These connection just keep building up and > slowly bring the system to a crawl. > > correo:root# ps aux| grep -v grep | grep sendmail > root 62864 8.0 1.1 23432 5528 ?? R 10:06AM 10:49.58 sendmail: > k5J06CMN062864 mx02.globalcenter.net.au [203.89.192.35]: DATA (sendmail) > root 66306 8.0 1.1 23448 5528 ?? R 10:18AM 8:38.72 sendmail: > k5J0IoxC066306 omta02ps.mx.bigpond.com [144.140.83.154]: DATA (sendmail) > root 66377 7.9 1.1 23392 5472 ?? R 10:20AM 8:26.25 sendmail: > k5J0KHGa066377 vpn02.commandhub.net [203.89.202.122]: DATA (sendmail) > root 20958 8.0 1.1 23368 5472 ?? R 5:14AM 44:36.44 sendmail: > k5IJF2bd020958 [83.173.162.41]: DATA (sendmail) > root 68330 8.0 1.1 23432 5528 ?? R 10:43AM 5:41.33 sendmail: > k5J0hOAk068330 mx01.globalcenter.net.au [203.89.192.34]: DATA (sendmail) > root 69364 8.0 1.1 23432 5504 ?? R 10:59AM 3:54.45 sendmail: > k5J0xmg6069364 omta05ps.mx.bigpond.com [144.140.83.195]: DATA (sendmail) > root 70522 8.0 1.1 23448 5544 ?? R 11:11AM 2:47.79 sendmail: > k5J1BY7Y070522 mx01.globalcenter.net.au [203.89.192.34]: DATA (sendmail) > root 66302 8.0 1.1 23392 5472 ?? R 10:18AM 8:40.20 sendmail: > k5J0If8t066302 vpn02.commandhub.net [203.89.202.122]: DATA (sendmail) > smmsp 91279 8.0 0.7 12224 3832 ?? Rs 4:01AM 60:20.92 sendmail: > Queue runner@00:10:00 for /var/spool/clientmqueue (sendmail) > root 16850 8.0 1.1 23432 5528 ?? R 4:30AM 51:34.91 sendmail: > k5IIUaWm016850 [201.150.67.51]: DATA (sendmail) > root 51654 8.0 1.1 23392 5472 ?? R 9:37AM 16:44.52 sendmail: > k5INbbYo051654 vpn02.commandhub.net [203.89.202.122]: DATA (sendmail) > root 91276 0.0 1.0 22928 5032 ?? Ss 4:01AM 0:03.05 sendmail: > rejecting connections on daemon MSA: load average: 12 (sendmail) > > I have rebuilt the world and kernel to see if the resolves my issue, but > alas no luck. > > Hopefully someone will be able to point be in the right direction. > > Regards, > > Travis > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > Further to this, when I put truss or ktrace into action on one of the sendmail process that is using a lot of kernel time, I see the following gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) poll(0x30e000,0x1,0xffffffffffffffff) = 1 (0x1) gettimeofday(0x40dfc210,0x0) = 0 (0x0) I am not running ntpd atm, however ntpdate to pool.ntp.org syncs ok. Does, this help shed some light? Regards, Travis