From owner-freebsd-threads@FreeBSD.ORG Sun Feb 25 07:06:00 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ED59116A402 for ; Sun, 25 Feb 2007 07:06:00 +0000 (UTC) (envelope-from mb@imp.ch) Received: from pop.imp.ch (mx2.imp.ch [157.161.9.17]) by mx1.freebsd.org (Postfix) with ESMTP id 80BA213C441 for ; Sun, 25 Feb 2007 07:06:00 +0000 (UTC) (envelope-from mb@imp.ch) Received: from dan.imp.ch (godot.imp.ch [157.161.4.8]) by pop.imp.ch (8.13.8/8.13.8/Submit_imp) with ESMTP id l1P75uAl018619 for ; Sun, 25 Feb 2007 08:05:58 +0100 (CET) (envelope-from mb@imp.ch) Date: Sat, 24 Feb 2007 23:24:42 +0100 (CET) From: Martin Blapp To: Daniel Eischen In-Reply-To: Message-ID: <20070224231443.S18301@godot.imp.ch> References: <20070220153632.E4139@godot.imp.ch> <20070220174221.B4139@godot.imp.ch> <20070220190347.C4139@godot.imp.ch> <20070220225303.V4139@godot.imp.ch> <20070220234734.H4139@godot.imp.ch> <20070221000830.V4139@godot.imp.ch> <20070221020335.Y4139@godot.imp.ch> <20070224103422.V18301@godot.imp.ch> <20070224185741.F18301@godot.imp.ch> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed ReSent-Date: Sun, 25 Feb 2007 08:05:52 +0100 (CET) ReSent-From: Martin Blapp ReSent-To: freebsd-threads@freebsd.org ReSent-Subject: Re: 6.2-Release and Clamd 0.90 with libpthread.so ReSent-Message-ID: <20070225080552.C18301@godot.imp.ch> Cc: Subject: Re: 6.2-Release and Clamd 0.90 with libpthread.so X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 07:06:01 -0000 Let me make a summary: With kern.threads.virtual_cpu=1 clamd works best with libpthread. It uses almost no CPU. With kern.threads.virtual_cpu=2 or kern.threads.virtual_cpu=4 and growing scanner load, the CPU load grows exponentially. The initial fork() seems not to be the problem. It happens in the foreground mode (without any fork call) too, but only if there is some amount of scanner load. Setting maxthreads in clamd.conf seems to make the situation even worse. libthr and libc_r work without any problems and ktrace here doesn't show any fork() calls at all. I wonder if something else could get libpthread spinning on kse activation stuff. I still think something is wrong here. Martin Martin Blapp, ------------------------------------------------------------------ ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133 Pratteln, CH Phone: +41 61 826 93 00 Fax: +41 61 826 93 01 PGP: PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22 D300 551E ------------------------------------------------------------------ From owner-freebsd-threads@FreeBSD.ORG Sun Feb 25 15:04:47 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7DCDB16A401; Sun, 25 Feb 2007 15:04:47 +0000 (UTC) (envelope-from mb@imp.ch) Received: from pop.imp.ch (mx2.imp.ch [157.161.9.17]) by mx1.freebsd.org (Postfix) with ESMTP id F074313C474; Sun, 25 Feb 2007 15:04:42 +0000 (UTC) (envelope-from mb@imp.ch) Received: from dan.imp.ch (godot.imp.ch [157.161.4.8]) by pop.imp.ch (8.13.8/8.13.8/Submit_imp) with ESMTP id l1PF4eaX016409; Sun, 25 Feb 2007 16:04:41 +0100 (CET) (envelope-from mb@imp.ch) Date: Sun, 25 Feb 2007 16:04:40 +0100 (CET) From: Martin Blapp To: Daniel Eischen In-Reply-To: <20070224231443.S18301@godot.imp.ch> Message-ID: <20070225151646.J18301@godot.imp.ch> References: <20070220153632.E4139@godot.imp.ch> <20070220174221.B4139@godot.imp.ch> <20070220190347.C4139@godot.imp.ch> <20070220225303.V4139@godot.imp.ch> <20070220234734.H4139@godot.imp.ch> <20070221000830.V4139@godot.imp.ch> <20070221020335.Y4139@godot.imp.ch> <20070224103422.V18301@godot.imp.ch> <20070224185741.F18301@godot.imp.ch> <20070224231443.S18301@godot.imp.ch> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: clamav-devel@lists.clamav.net, freebsd-threads@freebsd.org Subject: Re: 6.2-Release and Clamd 0.90 with libpthread.so X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 15:04:47 -0000 Hi all, I'm including now clamav-devel@lists.clamav.net too, since it's not clear if this problem is clamd or system library related. The problem is that clamd works only well with FreeBSDs libthr 1:1 kernel threading library. Using libc_r (userland threading only) seems to consume already a bit more resources. But clamd together with the N:M kernel threads library (libpthreads) is really slow. For some strange reason setting the sysctl kern.threads.virtual_cpu=1 to allow only one CPU for clamd works well and takes the CPU load significantly down. After reading http://www.gossamer-threads.com/lists/clamav/users/28795 especially this part: [...] > >We've been running 6.x our scanner boxes for a while now, but it's only >been with the more recent security/clamav-devel port installs that we >noticed a problem much like this. Most connections to the daemon (made >through clamav-milter in our case) timed out, and the only way to bring >down the daemon was with a kill -9. > >For us, the 20061029 devel snapshot was fine, but the current one >(20061217) has problems. > [...] I've now tried to go back to the development snapshot 20061129 and so I've reversed the clamd threading patches up to this date. But no luck. First it works well, then it becomes suddenly slower and more and more connections time out. Also, only 'kill -9' kills clamd. This isn't the case with lc_r and libthr btw ... Could it be that libpthread from it's kind of nature is a lot slower than libthr in that special case ? Maybe clamd expects system scope thread semantics and so libpthreads is slower ... Of course our current workload could then be too much for the poor clamd and a even more growing workload may slow it down due to resource bottlenecks. -- Martin From owner-freebsd-threads@FreeBSD.ORG Mon Feb 26 07:57:39 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 11E2B16A402; Mon, 26 Feb 2007 07:57:39 +0000 (UTC) (envelope-from mb@imp.ch) Received: from pop.imp.ch (mx2.imp.ch [157.161.9.17]) by mx1.freebsd.org (Postfix) with ESMTP id 8905513C428; Mon, 26 Feb 2007 07:57:38 +0000 (UTC) (envelope-from mb@imp.ch) Received: from dan.imp.ch (godot.imp.ch [157.161.4.8]) by pop.imp.ch (8.13.8/8.13.8/Submit_imp) with ESMTP id l1Q7vYa3010946; Mon, 26 Feb 2007 08:57:34 +0100 (CET) (envelope-from mb@imp.ch) Date: Mon, 26 Feb 2007 08:57:34 +0100 (CET) From: Martin Blapp To: John-Mark Gurney In-Reply-To: <20070226074037.GJ73385@funkthat.com> Message-ID: <20070226085541.Y18301@godot.imp.ch> References: <20070220190347.C4139@godot.imp.ch> <20070220225303.V4139@godot.imp.ch> <20070220234734.H4139@godot.imp.ch> <20070221000830.V4139@godot.imp.ch> <20070221020335.Y4139@godot.imp.ch> <20070224103422.V18301@godot.imp.ch> <20070224185741.F18301@godot.imp.ch> <20070226074037.GJ73385@funkthat.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Daniel Eischen , rob@debank.tv, freebsd-threads@freebsd.org Subject: Re: 6.2-Release and Clamd 0.90 with libpthread.so X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 07:57:39 -0000 Hi, > > Have you tried using pmc? Works at full processor speed... > Not yet since the pcm wiki 1) tells me Profiling threaded code: Currently buggy; Allows you to profile threaded applications. 1) http://wiki.freebsd.org/PmcTools/Features -- Martin From owner-freebsd-threads@FreeBSD.ORG Mon Feb 26 08:17:35 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6064B16A402; Mon, 26 Feb 2007 08:17:35 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.freebsd.org (Postfix) with ESMTP id 98B0813C4B2; Mon, 26 Feb 2007 08:17:33 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (26stdgmeoyeyoy5h@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id l1Q7eei3076865; Sun, 25 Feb 2007 23:40:40 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id l1Q7ebif076864; Sun, 25 Feb 2007 23:40:37 -0800 (PST) (envelope-from jmg) Date: Sun, 25 Feb 2007 23:40:37 -0800 From: John-Mark Gurney To: Martin Blapp Message-ID: <20070226074037.GJ73385@funkthat.com> Mail-Followup-To: Martin Blapp , Daniel Eischen , rob@debank.tv, freebsd-threads@freebsd.org References: <20070220190347.C4139@godot.imp.ch> <20070220225303.V4139@godot.imp.ch> <20070220234734.H4139@godot.imp.ch> <20070221000830.V4139@godot.imp.ch> <20070221020335.Y4139@godot.imp.ch> <20070224103422.V18301@godot.imp.ch> <20070224185741.F18301@godot.imp.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070224185741.F18301@godot.imp.ch> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: Daniel Eischen , rob@debank.tv, freebsd-threads@freebsd.org Subject: Re: 6.2-Release and Clamd 0.90 with libpthread.so X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 08:17:35 -0000 Martin Blapp wrote this message on Sat, Feb 24, 2007 at 19:06 +0100: > >>Running ktrace with libc_r or libthr I can't see a single call to fork(). > >>Is ktrace lying to me ? Why do I only see fork()s with libpthread ? > > > >The kse upcalls might look like forks to ktrace? > > > > That could be. Is it easy to fix this for ktrace ? > > Anyway, that still doesn't solve the problem for me, since > clamd with libpthreads is extremly CPU hungry and is slow. > > I really miss something to profile like strace on solaris to > see where it spends most CPU cycles. The problem is not that > it doesn't work, the problem is that it works far too slow. Have you tried using pmc? Works at full processor speed... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-threads@FreeBSD.ORG Mon Feb 26 11:16:23 2007 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A3A516A40A for ; Mon, 26 Feb 2007 11:16:23 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id D332013C4B6 for ; Mon, 26 Feb 2007 11:16:22 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l1QBGLec005589 for ; Mon, 26 Feb 2007 11:16:22 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l1QBGGIO005585 for freebsd-threads@FreeBSD.org; Mon, 26 Feb 2007 11:16:16 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 Feb 2007 11:16:16 GMT Message-Id: <200702261116.l1QBGGIO005585@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: linimon set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 11:16:23 -0000 Current FreeBSD problem reports Critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s threa/76690 threads fork hang in child for -lc_r 1 problem total. Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/20016 threads pthreads: Cannot set scheduling timer/Cannot set virtu s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s bin/32295 threads pthread dont dequeue signals s threa/34536 threads accept() blocks other threads o kern/38549 threads the procces compiled whith pthread stopped in pthread_ s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/49087 threads Signals lost in programs linked with libc_r s kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOC o threa/70975 threads unexpected and unreliable behaviour when using SYSV se o threa/72429 threads threads blocked in stdio (fgets, etc) are not cancella o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag s threa/76694 threads fork cause hang in dup()/close() function in child (-l o threa/79683 threads svctcp_create() fails if multiple threads call at the o threa/80435 threads panic on high loads o threa/83914 threads [libc] popen() doesn't work in static threaded program s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/85160 threads [libthr] [patch] libobjc + libpthread/libthr crash pro o threa/90278 threads libthr, ULE and -current produces >100% WCPU with apac o kern/91266 threads [threads] Trying sleep, but thread marked as sleeping s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc f threa/98256 threads gnome-system-monitor core dumps from pthread_testcance s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r o threa/101323 threads fork(2) in threaded programs broken. o threa/103975 threads Implicit loading/unloading of libpthread.so may crash 28 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s kern/19247 threads uthread_sigaction.c does not do anything wrt SA_NOCLDW s kern/22190 threads A threaded read(2) from a socketpair(2) fd can sometim s threa/30464 threads pthread mutex attributes -- pshared s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/69020 threads pthreads library leaks _gc_mutex o threa/79887 threads [patch] freopen() isn't thread-safe o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/81534 threads [libc_r] [patch] libc_r close() will fail on any fd ty 9 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Feb 26 19:54:44 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D19F016A401 for ; Mon, 26 Feb 2007 19:54:44 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outO.internet-mail-service.net (outO.internet-mail-service.net [216.240.47.238]) by mx1.freebsd.org (Postfix) with ESMTP id AF7C813C46B for ; Mon, 26 Feb 2007 19:54:40 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Mon, 26 Feb 2007 11:17:17 -0800 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 0609F125B23; Mon, 26 Feb 2007 11:42:15 -0800 (PST) Message-ID: <45E33815.7030703@elischer.org> Date: Mon, 26 Feb 2007 11:42:13 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Martin Blapp References: <20070220153632.E4139@godot.imp.ch> <20070220174221.B4139@godot.imp.ch> <20070220190347.C4139@godot.imp.ch> <20070220225303.V4139@godot.imp.ch> <20070220234734.H4139@godot.imp.ch> <20070221000830.V4139@godot.imp.ch> <20070221020335.Y4139@godot.imp.ch> <20070224103422.V18301@godot.imp.ch> In-Reply-To: <20070224103422.V18301@godot.imp.ch> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , rob@debank.tv, freebsd-threads@freebsd.org Subject: Re: 6.2-Release and Clamd 0.90 with libpthread.so X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 19:54:44 -0000 Martin Blapp wrote: > > Hi, > > After some busy days I found again time to work on this. I still have no > clue > what could be responsable for the problems I see. > > ktrace still shows lot of fork()s, but I can't see where whey are called. > > Running ktrace with libc_r or libthr I can't see a single call to fork(). > Is ktrace lying to me ? Why do I only see fork()s with libpthread ? 'upcalls' from the KSE threading library are erronwously reported as returns from fork they show RET fork 0 > > Martin > > Martin Blapp, > ------------------------------------------------------------------ > ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133 Pratteln, CH > Phone: +41 61 826 93 00 Fax: +41 61 826 93 01 > PGP: > PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22 D300 551E > ------------------------------------------------------------------ > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org"