From owner-freebsd-stable@FreeBSD.ORG Sun Dec 14 00:15:30 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5A9D106564A; Sun, 14 Dec 2008 00:15:30 +0000 (UTC) (envelope-from bms@incunabulum.net) Received: from out1.smtp.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.freebsd.org (Postfix) with ESMTP id 855FB8FC24; Sun, 14 Dec 2008 00:15:30 +0000 (UTC) (envelope-from bms@incunabulum.net) Received: from compute1.internal (compute1.internal [10.202.2.41]) by out1.messagingengine.com (Postfix) with ESMTP id 01EB11E6649; Sat, 13 Dec 2008 19:15:30 -0500 (EST) Received: from heartbeat2.messagingengine.com ([10.202.2.161]) by compute1.internal (MEProxy); Sat, 13 Dec 2008 19:15:30 -0500 X-Sasl-enc: rPWoR1wW7Mn2y5wGqUpbKSgJpx+RtsefLoQrQWh8JCKK 1229213729 Received: from empiric.lon.incunabulum.net (82-35-112-254.cable.ubr07.dals.blueyonder.co.uk [82.35.112.254]) by mail.messagingengine.com (Postfix) with ESMTPSA id 24D49277C2; Sat, 13 Dec 2008 19:15:28 -0500 (EST) Message-ID: <4944501E.40900@incunabulum.net> Date: Sun, 14 Dec 2008 00:15:26 +0000 From: Bruce M Simpson User-Agent: Thunderbird 2.0.0.18 (X11/20081205) MIME-Version: 1.0 To: "Paul B. Mahol" References: <8cb6106e0811241129o642dcf28re4ae177c8ccbaa25@mail.gmail.com> <20081125150342.GL2042@deviant.kiev.zoral.com.ua> <8cb6106e0812031453j6dc2f2f4i374145823c084eaa@mail.gmail.com> <200812041747.09040.gnemmi@gmail.com> <4938FE44.9090608@FreeBSD.org> <4939133E.2000701@FreeBSD.org> <493CEE90.7050104@FreeBSD.org> <3a142e750812090553l564bff84pe1f02cd1b03090ff@mail.gmail.com> <4943F43B.4060105@incunabulum.net> <3a142e750812131403p31841403ub9d5693278c74111@mail.gmail.com> In-Reply-To: <3a142e750812131403p31841403ub9d5693278c74111@mail.gmail.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, "Bruce M. Simpson" Subject: Re: ext2fuse: user-space ext2 implementation X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2008 00:15:30 -0000 Paul B. Mahol wrote: >> Can you please relay this feedback to the authors of ext2fuse? >> >> As mentioned earlier in the thread, the ext2fuse code could benefit from >> UBLIO-ization. Are you or any other volunteers happy to help out here? >> > > Well, first higher priority would be to fix existing bugs. It would be > very little > gain with user cache, because it is already too much IMHO slow and > adding user cache > will not make it faster, but that is not port problem. > I'm not aware of bugs with ext2fuse itself; my work on the port was merely to try to raise awareness that a user-space project for ext2 filesystem access existed. Can you elaborate further on your experience with ext2fuse which seems to you to be buggy, i.e. symptoms, root cause analysis etc. ? Have you reported these to the author(s)? Have you measured the performance? Is the performance sufficient for the needs of an occasional desktop user? I realise we are largely involved in content-free argument here, however the trade-off of ext2fuse vs ext2fs in the FreeBSD kernel source tree, is that of a hopefully more actively maintained implementation vs one which is not maintained at all, and any alternatives for FreeBSD users would be welcome. thanks BMS From owner-freebsd-stable@FreeBSD.ORG Sun Dec 14 09:36:47 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 478A41065700 for ; Sun, 14 Dec 2008 09:36:47 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id E0ED68FC24 for ; Sun, 14 Dec 2008 09:36:46 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [211.166.10.233]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id D0ACA28449 for ; Sun, 14 Dec 2008 17:36:45 +0800 (CST) Received: from localhost (tarsier.geekcn.org [211.166.10.233]) by tarsier.geekcn.org (Postfix) with ESMTP id 4C4F7EC4A24; Sun, 14 Dec 2008 17:36:45 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([211.166.10.233]) by localhost (mail.geekcn.org [211.166.10.233]) (amavisd-new, port 10024) with ESMTP id xP+uEQAOggEq; Sun, 14 Dec 2008 17:36:40 +0800 (CST) Received: from charlie.delphij.net (c-67-180-38-12.hsd1.ca.comcast.net [67.180.38.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id E17AAEB7780; Sun, 14 Dec 2008 17:36:38 +0800 (CST) Message-ID: <4944D3A3.8040505@delphij.net> Date: Sun, 14 Dec 2008 01:36:35 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: Vlad GALU References: <20081210160325.GA72838@mr-happy.com> In-Reply-To: X-Enigmail-Version: 0.95.7 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: multipart/mixed; boundary="------------000908050001070704080706" Cc: Mike Jakubik , freebsd-stable@freebsd.org, Jeff Blank Subject: Re: bce(4) and rx errors X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2008 09:36:47 -0000 This is a multi-part message in MIME format. --------------000908050001070704080706 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Gents, I have not yet talked this with David but it looks like this patch would make it disappear. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAklE06IACgkQi+vbBBjt66DwjgCeJ1RO+gL3mu13zAzXpPsbDi/R /BwAmQFAij74yusRq+KbcSNb3BMbLzNX =1zx3 -----END PGP SIGNATURE----- --------------000908050001070704080706 Content-Type: text/plain; name="bce-noL2Filter.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="bce-noL2Filter.diff" Index: if_bce.c =================================================================== --- if_bce.c (revision 186076) +++ if_bce.c (working copy) @@ -7408,7 +7408,6 @@ (u_long) sc->stat_IfInMBUFDiscards + (u_long) sc->stat_Dot3StatsAlignmentErrors + (u_long) sc->stat_Dot3StatsFCSErrors + - (u_long) sc->stat_IfInFramesL2FilterDiscards + (u_long) sc->stat_IfInRuleCheckerDiscards + (u_long) sc->stat_IfInFTQDiscards + (u_long) sc->com_no_buffers; --------------000908050001070704080706-- From owner-freebsd-stable@FreeBSD.ORG Sun Dec 14 09:57:53 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A98E31065678 for ; Sun, 14 Dec 2008 09:57:53 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3D0908FC22 for ; Sun, 14 Dec 2008 09:57:53 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [211.166.10.233]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id 3E5CF28448 for ; Sun, 14 Dec 2008 17:57:52 +0800 (CST) Received: from localhost (tarsier.geekcn.org [211.166.10.233]) by tarsier.geekcn.org (Postfix) with ESMTP id 9FC5DEC4A3E; Sun, 14 Dec 2008 17:57:51 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([211.166.10.233]) by localhost (mail.geekcn.org [211.166.10.233]) (amavisd-new, port 10024) with ESMTP id jMiN03E8JIiM; Sun, 14 Dec 2008 17:57:46 +0800 (CST) Received: from charlie.delphij.net (c-67-180-38-12.hsd1.ca.comcast.net [67.180.38.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id 398A9EC4A37; Sun, 14 Dec 2008 17:57:43 +0800 (CST) Message-ID: <4944D894.6070306@delphij.net> Date: Sun, 14 Dec 2008 01:57:40 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: Mike Jakubik References: <4935069A.8060209@ec-marseille.fr> <49357BD0.4000008@delphij.net> <4935944A.9090509@ec-marseille.fr> <4935C453.8070301@delphij.net> <4935D67E.4070204@delphij.net> <4936F8C4.6090006@ec-marseille.fr> <49399FA6.3060108@delphij.net> <493CE8F7.5010204@yandex-team.ru> <3a6fb7145a0a6c8af136ea1a0824e5ed.squirrel@wettoast.dyndns.org> <493D9BC6.8050902@delphij.net> <4fe0419b44a3a4c4a28b1e60fbb3a3c8.squirrel@wettoast.dyndns.org> <56272b131067237ccabd23de5f669458.squirrel@wettoast.dyndns.org> In-Reply-To: <56272b131067237ccabd23de5f669458.squirrel@wettoast.dyndns.org> X-Enigmail-Version: 0.95.7 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: multipart/mixed; boundary="------------040906020000020207050201" Cc: freebsd-stable@freebsd.org, d@delphij.net Subject: Re: RELENG_7_1: bce driver change generating too much interrupts ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2008 09:57:53 -0000 This is a multi-part message in MIME format. --------------040906020000020207050201 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mike Jakubik wrote: > On Mon, December 8, 2008 5:22 pm, Mike Jakubik wrote: >> On Mon, December 8, 2008 5:12 pm, Xin LI wrote: >> >>> Which version are you currently using? My previous commit only fixes >>> the excessive interrupt issue, I think this could be a different >>> problem, I'm taking a look at the code to see if I can have something >>> for you. >> I was running on the version just prior to the latest interrupt commit. I >> have now updated to the one with the interrupt fix. Will let you know if >> things change. >> >> Thank You. > > The interrupt rate has decreased significantly, however i am still having > having problem with applications that hold stateful connections. The rx > errors are also still showing, i suspect this is related to the problem. > How can i roll back this driver to the last known good version? Hi, Mike, I think they are different problems. Could you, please, give me feedback about whether: - The old driver does not trigger the problem? - The patched driver restore all the old driver behavior? ============= Rationale for my patch. To say it simply, it removes "Received L2 packets discarded" value from being counted from ierror. In the past, we count the following: - Undersize packets - Oversized packets - Received packets discarded due to lack of controller buffer memory - Alignment errors - Frame check sequence errors Now, it counts the following four stuff as well: - Received L2 packets discarded ** removed - Received packets discarded by rule - Received packet FTQ discards - Valid packets received but no RX buffers available I have checked the old FreeBSD driver and the Linux driver, both have the "Received L2 packets discarded" value increasing every second, so I don't believe that this is a real problem. I'll double check with David to make sure about this. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAklE2JQACgkQi+vbBBjt66Bl0gCfZ6NVNXpC2ynUZjaZButg+4jo vgYAnAzE2iFWcZMZ29j3qtpwQ5f0xh9V =3l8f -----END PGP SIGNATURE----- --------------040906020000020207050201 Content-Type: text/plain; name="bce-noL2Filter.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="bce-noL2Filter.diff" Index: if_bce.c =================================================================== --- if_bce.c (revision 186076) +++ if_bce.c (working copy) @@ -7408,7 +7408,6 @@ (u_long) sc->stat_IfInMBUFDiscards + (u_long) sc->stat_Dot3StatsAlignmentErrors + (u_long) sc->stat_Dot3StatsFCSErrors + - (u_long) sc->stat_IfInFramesL2FilterDiscards + (u_long) sc->stat_IfInRuleCheckerDiscards + (u_long) sc->stat_IfInFTQDiscards + (u_long) sc->com_no_buffers; --------------040906020000020207050201-- From owner-freebsd-stable@FreeBSD.ORG Sun Dec 14 15:47:29 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1F7E1065670; Sun, 14 Dec 2008 15:47:29 +0000 (UTC) (envelope-from onemda@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.29]) by mx1.freebsd.org (Postfix) with ESMTP id 4B7A08FC21; Sun, 14 Dec 2008 15:47:29 +0000 (UTC) (envelope-from onemda@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so942238yxb.13 for ; Sun, 14 Dec 2008 07:47:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=5FFmonQ1Zuc0CjPKY3+ALKvpy2FVIzHg5DaZQOz3iSs=; b=E9WHhKL63uzEDX6KjpzB8nVkr7M0cgFMHNSUpUUHi0EIVs9W0OMF2dmDsmD87mURQQ ifQH0mKg0QIQtELnr/bj4EYZibbn5TpStm81sYgDsXv+Z0e0NE7ATz1i1JIxDR6ZMjjE IJgIbOoVFR8d3BMSCtpayyCBLAQzCnJuvVV5s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=Owibq86s8SiTwIA+N/StEzdDFLY0QIU0e1aMt0uuhkDAvwJ1U63ex+GmN5QD8rJ7vD W49NQpa7EBB80cL/Dx6wRtTKGtzcLBXVQIrAgpJXohBEYC226q8/FzD5HZP6f9Ij/QM+ hGiOrMtvqW9XAwdJ+LZOapkPANs1BWqSM2U/8= Received: by 10.231.18.130 with SMTP id w2mr68628iba.11.1229269648344; Sun, 14 Dec 2008 07:47:28 -0800 (PST) Received: by 10.231.11.72 with HTTP; Sun, 14 Dec 2008 07:47:28 -0800 (PST) Message-ID: <3a142e750812140747r2eb5ebadp7ac2b2c8ae357bae@mail.gmail.com> Date: Sun, 14 Dec 2008 16:47:28 +0100 From: "Paul B. Mahol" To: "Bruce M Simpson" In-Reply-To: <4944501E.40900@incunabulum.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0811241129o642dcf28re4ae177c8ccbaa25@mail.gmail.com> <8cb6106e0812031453j6dc2f2f4i374145823c084eaa@mail.gmail.com> <200812041747.09040.gnemmi@gmail.com> <4938FE44.9090608@FreeBSD.org> <4939133E.2000701@FreeBSD.org> <493CEE90.7050104@FreeBSD.org> <3a142e750812090553l564bff84pe1f02cd1b03090ff@mail.gmail.com> <4943F43B.4060105@incunabulum.net> <3a142e750812131403p31841403ub9d5693278c74111@mail.gmail.com> <4944501E.40900@incunabulum.net> Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, "Bruce M. Simpson" Subject: Re: ext2fuse: user-space ext2 implementation X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2008 15:47:29 -0000 On 12/14/08, Bruce M Simpson wrote: > Paul B. Mahol wrote: >>> Can you please relay this feedback to the authors of ext2fuse? >>> >>> As mentioned earlier in the thread, the ext2fuse code could benefit from >>> UBLIO-ization. Are you or any other volunteers happy to help out here? >>> >> >> Well, first higher priority would be to fix existing bugs. It would be >> very little >> gain with user cache, because it is already too much IMHO slow and >> adding user cache >> will not make it faster, but that is not port problem. >> > > I'm not aware of bugs with ext2fuse itself; my work on the port was > merely to try to raise awareness that a user-space project for ext2 > filesystem access existed. > > Can you elaborate further on your experience with ext2fuse which seems > to you to be buggy, i.e. symptoms, root cause analysis etc. ? Have you > reported these to the author(s)? I have read TODO. > Have you measured the performance? Is the performance sufficient for the > needs of an occasional desktop user? Performance was not sufficient, and adding user cache will not improve access speed on first read. After mounting ext2fs volume (via md(4)) created with e2fsprogs port and copying data from ufs to ext2, reading was quite slow. Also ext2fuse after mount doesnt exits it is still running displaying debug data - explaining why project itselfs is in alpha state. > I realise we are largely involved in content-free argument here, however > the trade-off of ext2fuse vs ext2fs in the FreeBSD kernel source tree, > is that of a hopefully more actively maintained implementation vs one > which is not maintained at all, and any alternatives for FreeBSD users > would be welcome. Project itself doesnt look very active, but I may be wrong. It is in alpha state as reported on SF. IMHO it is better to maintain our own because it is in better shape, but I'm not intersted in ext* as developer. -- Paul From owner-freebsd-stable@FreeBSD.ORG Sun Dec 14 21:06:56 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 600A41065672 for ; Sun, 14 Dec 2008 21:06:56 +0000 (UTC) (envelope-from jfb@mr-happy.com) Received: from vexbert.mr-paradox.net (vexbert.mr-paradox.net [198.145.180.174]) by mx1.freebsd.org (Postfix) with ESMTP id 405578FC18 for ; Sun, 14 Dec 2008 21:06:56 +0000 (UTC) (envelope-from jfb@mr-happy.com) Received: from crow.mr-happy.com (crow.mr-happy.com [10.1.0.2]) by vexbert.mr-paradox.net (Postfix) with ESMTP id A4BF0844E1; Sun, 14 Dec 2008 15:47:38 -0500 (EST) Received: by crow.mr-happy.com (Postfix, from userid 16139) id 46CB145020; Sun, 14 Dec 2008 15:47:38 -0500 (EST) Date: Sun, 14 Dec 2008 15:47:38 -0500 From: Jeff Blank To: d@delphij.net Message-ID: <20081214204738.GA35372@mr-happy.com> References: <20081210160325.GA72838@mr-happy.com> <4944D3A3.8040505@delphij.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4944D3A3.8040505@delphij.net> X-Face: #0jV*~a}VtKS-&E/!EJpH('H1Va}24dxF0oT&+.R3Gu8C; xhSC+<|+H84&YLbMvphuRT4cp3.|8EN_(2Eix/6{.Up~u`a^}0Ln&b+9Fw|BPig@-{y\pL_46d&ZwA]5%_AU?}DezfE&1!>H?3E$!Yve7.O<+..Jnb4:'6Ey_]FtFzU9=*l$1p/@gA,Ze>^5<]+r(XJ+m7`/vMDc$'wy|`e X-Virus-Scanned: ClamAV 0.92.1/8756/Sun Dec 14 07:38:06 2008 on vexbert.mr-paradox.net X-Virus-Status: Clean Cc: freebsd-stable@freebsd.org Subject: Re: bce(4) and rx errors X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2008 21:06:56 -0000 On Sun, Dec 14, 2008 at 01:36:35AM -0800, Xin LI wrote: > I have not yet talked this with David but it looks like this patch would > make it disappear. Applied to 7.1-RC1, and rx errors are gone. Thank you! Jeff From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 08:24:01 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF1511065673 for ; Mon, 15 Dec 2008 08:24:01 +0000 (UTC) (envelope-from davidn04@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.24]) by mx1.freebsd.org (Postfix) with ESMTP id A78118FC0C for ; Mon, 15 Dec 2008 08:24:01 +0000 (UTC) (envelope-from davidn04@gmail.com) Received: by qw-out-2122.google.com with SMTP id 9so527655qwb.7 for ; Mon, 15 Dec 2008 00:24:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=k8NH8xKMTl9RPAYNPM12FUC80QCldk2ufpqDsipur8o=; b=XKXPg9aRQW+UBfPxQMqbvrR7jPxszuVVT1VOMV8X9VNVaN67eSbDL4IxZ3BiD4seTr k84hVt+/DMaqL30TyanZLX4gWwjmvA6OAQI4D+OEebAXP/XfC8U+DWEJ05GwHARed5iI 6aX+njzhOy56CjlQ6ShhlCJ1//xv2NAOV9RAc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=Z9g2z43der1bq3dREF6v17c68NtFQoFEfU63NKw1FKgGtfZ7rnPZzcOoIiwwFj036Y aT5Bkk1l2LMBeLkUisi6lmkRSMjBH8DYiJKAIDG8V+vlg6jSN246vZhAoQlwl/7Jz2/c xI1ECVtXtIhWmiAm7I/nDnWFAFywC+QIYQlDI= Received: by 10.214.243.9 with SMTP id q9mr7252587qah.61.1229327909665; Sun, 14 Dec 2008 23:58:29 -0800 (PST) Received: by 10.214.150.7 with HTTP; Sun, 14 Dec 2008 23:58:29 -0800 (PST) Message-ID: <4d7dd86f0812142358t714cb96erdf269573ac86ba30@mail.gmail.com> Date: Mon, 15 Dec 2008 18:58:29 +1100 From: "David N" To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Running rsnapshot via cron reboots the machine X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 08:24:02 -0000 Hi, I have a machine AMD Sepron LE-1150 ASUS M2A-VM 1GB RAM ECC 2x SATA 300GB in a RAID 1 (gmirror). 7.0-RELEASE-p2 AMD64 generic kernel it was doing backups via bacula to an external disk USB 2.0 SATA disk, and it was working well. (GLabel) /dev/ufs/BackupDisk I changed to rsnapshot recently, with the External HDD in glabel + gjournal (/dev/da0s1.journal -> /dev/ufs/BackupDisk) and it will reboot the machine roughly 30 minutes after the rsnapshot starts via CRON. If i run rsnapshot without CRON, eg. via the command line it works fine. (I'm compiling the new kernel p6 whilst doing an rsnapshot via the command line) I have changed the time it does the rsnapshot, from 3AM (reboots at 3:30-3:40AM roughly) to 4:30AM (reboots at 5AM roughly). So its nothing running on the system doing it. If i remove the rsnapshot from the CRON, the computer stays on and doesn't reboot. Its not the power supply, and the computer is on a UPS and the UPS log hasn't reported any blackouts. (There are 2 other servers that doesn't turn off so its not a blackout). I left gstat and top running L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0| acd0 0 125 125 4164 5.5 0 0 0.0 29.9| ad4 0 0 0 0 0.0 0 0 0.0 0.0| ad4s1 0 0 0 0 0.0 0 0 0.0 0.0| ad4s2 0 125 125 4164 5.6 0 0 0.0 30.5| ad4s3 34 142 71 9079 9.4 71 8697 212.0 99.3| da0 34 142 71 9079 9.6 71 8697 213.5 99.3| da0s1 0 0 0 0 0.0 0 0 0.0 0.0| da0s1c 7 135 71 9079 9.7 64 8184 33.0 85.1| da0s1.journal 0 125 125 4164 9.2 0 0 0.0 39.8| ad6 7 135 71 9079 9.7 64 8184 36.2 91.9| ufs/BackupDisk 0 0 0 0 0.0 0 0 0.0 0.0| ad6s1 0 0 0 0 0.0 0 0 0.0 0.0| ad6s2 0 125 125 4164 9.3 0 0 0.0 40.4| ad6s3 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s1 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s2 0 125 125 8328 9.9 0 0 0.0 43.9| mirror/gm0s3 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s1a 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s1b 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s1c 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s1d 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s2c 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s2d 0 0 0 0 0.0 0 0 0.0 0.0| mirror/gm0s3c 0 125 125 8328 10.2 0 0 0.0 45.2| mirror/gm0s3d last pid: 18829; load averages: 0.29, 0.45, 0.50 up 1+01:31:23 05:01:50 64 processes: 1 running, 63 sleeping CPU states: 16.5% user, 0.0% nice, 25.2% system, 3.0% interrupt, 55.3% idle Mem: 69M Active, 436M Inact, 395M Wired, 44M Cache, 108M Buf, 3944K Free Swap: 2048M Total, 308K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 18634 root 1 -4 0 15224K 2244K getblk 0:45 9.47% rsync 18633 root 1 98 0 18296K 4576K select 0:45 4.69% rsync 18375 root 1 8 0 20556K 8412K wait 7:20 0.00% perl 4447 root 1 -64 0 7656K 2036K RUN 1:21 0.00% top 4321 root 1 8 0 11784K 2008K nanslp 0:45 0.00% gstat 18627 root 1 96 0 15224K 2596K select 0:36 0.00% rsync 4219 evxadmin 1 96 0 32936K 3220K select 0:07 0.00% sshd 3800 evxadmin 1 96 0 32936K 3164K select 0:06 0.00% sshd 18629 root 1 96 0 32868K 3964K select 0:04 0.00% sshd 18628 root 1 96 0 23764K 7664K select 0:04 0.00% ssh 851 root 1 96 0 12476K 1832K select 0:04 0.00% nmbd 1788 root 1 96 0 10576K 2812K select 0:02 0.00% sendmail 1147 root 1 96 0 24456K 2404K select 0:01 0.00% nmbd 1346 root 1 96 0 22224K 2504K select 0:01 0.00% nmbd 3957 evxadmin 1 96 0 32936K 3220K select 0:01 0.00% sshd 1800 root 1 8 0 5736K 1032K nanslp 0:01 0.00% cron 1754 root 1 8 0 5736K 980K nanslp 0:00 0.00% cron 1557 root 1 8 0 5736K 980K nanslp 0:00 0.00% cron 1387 root 1 8 0 5736K 980K nanslp 0:00 0.00% cron 1192 root 1 8 0 5736K 980K nanslp 0:00 0.00% cron 4646 root 1 96 0 36968K 4892K select 0:00 0.00% smbd 775 root 1 96 0 4684K 1064K select 0:00 0.00% syslogd Nothing else seems to be running. Its driving me insane! any help would be appreciated. smart says the HDD are fine. Regards David N From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 09:02:10 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A1261065672; Mon, 15 Dec 2008 09:02:10 +0000 (UTC) (envelope-from victor@bsdes.net) Received: from alf.bsdes.net (244.Red-217-126-240.staticIP.rima-tde.net [217.126.240.244]) by mx1.freebsd.org (Postfix) with ESMTP id 0BBBF8FC08; Mon, 15 Dec 2008 09:02:09 +0000 (UTC) (envelope-from victor@bsdes.net) Received: by alf.bsdes.net (Postfix, from userid 1001) id C237D119CF7; Mon, 15 Dec 2008 10:02:07 +0100 (CET) Date: Mon, 15 Dec 2008 10:02:07 +0100 From: Victor Balada Diaz To: Pyun YongHyeon Message-ID: <20081215090207.GN1320@alf.bsdes.net> References: <20081210061226.GC37837@cdnetworks.co.kr> <20081210085934.GB1320@alf.bsdes.net> <20081210102800.GH37837@cdnetworks.co.kr> <20081210113225.GD1320@alf.bsdes.net> <20081210120719.GK37837@cdnetworks.co.kr> <20081211075707.GH1320@alf.bsdes.net> <20081211081045.GJ1320@alf.bsdes.net> <20081211090056.GH42370@cdnetworks.co.kr> <20081211095021.GL1320@alf.bsdes.net> <20081212121309.GM1320@alf.bsdes.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20081212121309.GM1320@alf.bsdes.net> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: [ATA] and re(4) stability issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 09:02:10 -0000 On Fri, Dec 12, 2008 at 01:13:09PM +0100, Victor Balada Diaz wrote: > On Thu, Dec 11, 2008 at 10:50:21AM +0100, Victor Balada Diaz wrote: > > On Thu, Dec 11, 2008 at 06:00:56PM +0900, Pyun YongHyeon wrote: > > > > > > I've reverted r185756 which caused GMII access issues on some > > > controllers. If you are brave enough to try beta code, you can > > > get latest re(4) in the following URL. Note, I don't have PCIe > > > based RealTek controllers so the code was not tested at all. > > > > > > http://people.freebsd.org/~yongari/re/if_re.c > > > http://people.freebsd.org/~yongari/re/if_rlreg.h > > > > I've recompiled the kernel with the first file in sys/dev/re/ > > and the second one in sys/pci/. I'm still testing with MSI enabled. > > > > So far tried rebooting using nextboot(8) (just in case i lost the > > network card i could boot again) and the card seems to work > > but i'll continue stress testing the machine with stress + dd + > > iperf and see if i can take it down. I'll let you know how it goes. > > After a day of stress testing the machine haven't got errors, interrupt > storms or interface up/down problems. Everything seems fine. > I'll continue stress testing the machine during the weekend, but > i would say that this time it's fixed. Stopped stress testing this morning. After all the weekend testing seems the re(4) problems were fixed. No single interface up/down error. netstat -i reports no errors and everything is fine. Thanks a lot! I'm going to deploy the patches on our production machines. I've been able to trigger interrupt storms with ATA code, though. -- La prueba más fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros. From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 09:07:12 2008 Return-Path: Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 241351065672; Mon, 15 Dec 2008 09:07:12 +0000 (UTC) (envelope-from victor@bsdes.net) Received: from alf.bsdes.net (244.Red-217-126-240.staticIP.rima-tde.net [217.126.240.244]) by mx1.freebsd.org (Postfix) with ESMTP id C17158FC17; Mon, 15 Dec 2008 09:07:11 +0000 (UTC) (envelope-from victor@bsdes.net) Received: by alf.bsdes.net (Postfix, from userid 1001) id 9841F119CF7; Mon, 15 Dec 2008 10:07:10 +0100 (CET) Date: Mon, 15 Dec 2008 10:07:10 +0100 From: Victor Balada Diaz To: =?iso-8859-1?Q?S=F8ren?= Schmidt Message-ID: <20081215090710.GO1320@alf.bsdes.net> References: <20081209185236.GA1320@alf.bsdes.net> <493F84A4.1080308@yandex.ru> <20081210091107.GC1320@alf.bsdes.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: "Andrey V. Elsukov" , freebsd-stable@FreeBSD.ORG, freebsd-amd64@FreeBSD.ORG Subject: Re: [ATA] and re(4) stability issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 09:07:12 -0000 On Wed, Dec 10, 2008 at 10:55:35AM +0100, Søren Schmidt wrote: > On 10Dec, 2008, at 10:11 , Victor Balada Diaz wrote: > > > >Thanks for explaining me what the flags do. I'm not skilled enough > >to create > >the DMA quirks but if you could give me some patches i'll test them. > >Also > >if you have any other idea on what could i test or how can i debug > >this > >it would be more than welcome. > > > Comment out the following two lines in ata_ahci_dmainit(): > > if (ATA_INL(ctlr->r_res2, ATA_AHCI_CAP) & ATA_AHCI_CAP_64BIT) > ch->dma->max_address = BUS_SPACE_MAXADDR; > > And you will not use 64bit DMA even if the chipset supports it. > However I have not seen any chipsets supporting this fail, YMMV as > usual :) > Hello Søren, I'm triggering interrupt storms with this chipset after a few days of stressing the HD calling sysutils/stress with stress -d 10 -i 10 and in other term, doing: while true; do dd if=/dev/zero of=BAH bs=1M count=1024; done; Right now, as reported by systat -vmstat i have 578k interrupts in atapci and the machine is idle. Do you have any idea on how could i debug this? any advice would be much more than welcome. Thanks a lot. Regards. -- La prueba más fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros. From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 09:30:23 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA65C1065675; Mon, 15 Dec 2008 09:30:23 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from acme.spoerlein.net (cl-43.dus-01.de.sixxs.net [IPv6:2a01:198:200:2a::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4CD118FC19; Mon, 15 Dec 2008 09:30:23 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from roadrunner.spoerlein.net (e180131180.adsl.alicedsl.de [85.180.131.180]) by acme.spoerlein.net (8.14.2/8.14.2) with ESMTP id mBF9ULI6062752 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 15 Dec 2008 10:30:21 +0100 (CET) (envelope-from uspoerlein@gmail.com) Received: from roadrunner.spoerlein.net (localhost [127.0.0.1]) by roadrunner.spoerlein.net (8.14.3/8.14.3) with ESMTP id mBF9UJOo002877 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 15 Dec 2008 10:30:20 +0100 (CET) (envelope-from uspoerlein@gmail.com) Received: (from uqs@localhost) by roadrunner.spoerlein.net (8.14.3/8.14.3/Submit) id mBF9UJtO002876; Mon, 15 Dec 2008 10:30:19 +0100 (CET) (envelope-from uspoerlein@gmail.com) Date: Mon, 15 Dec 2008 10:30:19 +0100 From: Ulrich Spoerlein To: stable@freebsd.org Message-ID: <20081215093019.GB1496@roadrunner.spoerlein.net> Mail-Followup-To: stable@freebsd.org, philip@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Cc: philip@freebsd.org Subject: moused(8) ate my umass(4) devices, it's true! X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 09:30:23 -0000 Hey all, I've observed a very weird behaviour with my USB stick for quite a while now (probably 4 months; sadly, I don't have any dates handy). Anyway, I have this weird SUN Keyboard -> USB adapter, which offers an ukbd(4) and ums(4) device to the system, although there is no mouse attached to the Sun keyboard I'm using. ukbd0: on uhub4 kbd2 at ukbd0 ums0: on uhub4 ums0: 3 buttons. This worked fine on RELENG_7 till somewhere around summer. Now, whenever there is a moused(8) listening on this fake ums(4) port, the umass(4) device will get stuck somewhere in CAM-land. It probes fine: Dec 14 10:24:49 roadrunner kernel: umass0: on uhub4 but then only BBB bulk transfer timeout messages follow every so often. The da0 device never shows up. Dec 14 10:26:59 roadrunner kernel: umass0: BBB reset failed, TIMEOUT Dec 14 10:27:04 roadrunner kernel: umass0: BBB bulk-in clear stall failed, IOERROR Dec 14 10:27:04 roadrunner kernel: umass0: BBB bulk-out clear stall failed, IOERROR Dec 14 10:28:09 roadrunner kernel: umass0: BBB reset failed, IOERROR Dec 14 10:28:09 roadrunner kernel: umass0: BBB bulk-in clear stall failed, IOERROR Dec 14 10:28:09 roadrunner kernel: umass0: BBB bulk-out clear stall failed, IOERROR Dec 14 10:29:14 roadrunner kernel: umass0: BBB reset failed, IOERROR Dec 14 10:29:14 roadrunner kernel: umass0: BBB bulk-in clear stall failed, IOERROR Dec 14 10:29:14 roadrunner kernel: umass0: BBB bulk-out clear stall failed, IOERROR Dec 14 10:30:19 roadrunner kernel: umass0: BBB reset failed, IOERROR Dec 14 10:30:19 roadrunner kernel: umass0: BBB bulk-in clear stall failed, IOERROR Dec 14 10:30:19 roadrunner kernel: umass0: BBB bulk-out clear stall failed, IOERROR Dec 14 10:31:24 roadrunner kernel: umass0: BBB reset failed, IOERROR Dec 14 10:31:24 roadrunner kernel: umass0: BBB bulk-in clear stall failed, IOERROR Dec 14 10:31:24 roadrunner kernel: umass0: BBB bulk-out clear stall failed, IOERROR I cannot unplug the USB stick (instant panic) and kldunloading umass is also bad (instant panic). I have to reboot the system and remove the device then. Today, I figured out that it depends wholly on moused(8) running on that unpopulated mouse port. Killing the moused process, which will start automatically when ums0 attaches, before plugging in the umass device and everybody is happy. I'm glad I found this workaround, but the situation sucks anyway. Other than binary searching the offending commit, what debugging could I do? Would a ktrace of the moused(8) be helpful when attaching umass? Is it perhaps polling the port too often waiting for a mouse to appear? Also, can I somehow blacklist the mouse-port of this adapter? I do not intend to use a 3 button Sun mouse, ever. Cheers, Ulrich Spoerlein -- It is better to remain silent and be thought a fool, than to speak, and remove all doubt. From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 16:47:29 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 410A21065670 for ; Mon, 15 Dec 2008 16:47:29 +0000 (UTC) (envelope-from mikej@rogers.com) Received: from smtp115.rog.mail.re2.yahoo.com (smtp115.rog.mail.re2.yahoo.com [68.142.225.231]) by mx1.freebsd.org (Postfix) with SMTP id E6F758FC1B for ; Mon, 15 Dec 2008 16:47:28 +0000 (UTC) (envelope-from mikej@rogers.com) Received: (qmail 7469 invoked from network); 15 Dec 2008 16:47:28 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:X-YMail-OSG:X-Yahoo-Newman-Property:Received:Message-ID:In-Reply-To:References:Date:Subject:From:To:Cc:User-Agent:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:Importance; b=S7u8EGcsda29dDGTpckenxlevkP5d9vs6BecNWsdUMoZ0M6Vpx863bjqeQZ0XE0CidoD7KCnNczxdLzgRY1WldeIoSGGZmrKaRCObS/84e1KYbfoBPgk6mm9TEMOQKqgahbyDR6SkFD9FWUDFoqyO3iAX19zEuPPEGfzksre41A= ; Received: from unknown (HELO wettoast.dyndns.org) (mikej@99.227.98.203 with login) by smtp115.rog.mail.re2.yahoo.com with SMTP; 15 Dec 2008 16:47:28 -0000 X-YMail-OSG: 25wsqDQVM1mGkn1x8q4SqITbnUplSvGCwHX5H0fmigIT7fSFeZgpuO4PhrkPwkl7rA-- X-Yahoo-Newman-Property: ymail-3 Received: from 38.99.187.34 (SquirrelMail authenticated user mikej) by wettoast.dyndns.org with HTTP; Mon, 15 Dec 2008 11:47:31 -0500 (EST) Message-ID: <400809c165ecedf8a0b7bc6b569e1289.squirrel@wettoast.dyndns.org> In-Reply-To: <4944D894.6070306@delphij.net> References: <4935069A.8060209@ec-marseille.fr> <49357BD0.4000008@delphij.net> <4935944A.9090509@ec-marseille.fr> <4935C453.8070301@delphij.net> <4935D67E.4070204@delphij.net> <4936F8C4.6090006@ec-marseille.fr> <49399FA6.3060108@delphij.net> <493CE8F7.5010204@yandex-team.ru> <3a6fb7145a0a6c8af136ea1a0824e5ed.squirrel@wettoast.dyndns.org> <493D9BC6.8050902@delphij.net> <4fe0419b44a3a4c4a28b1e60fbb3a3c8.squirrel@wettoast.dyndns.org> <56272b131067237ccabd23de5f669458.squirrel@wettoast.dyndns.org> <4944D894.6070306@delphij.net> Date: Mon, 15 Dec 2008 11:47:31 -0500 (EST) From: "Mike Jakubik" To: d@delphij.net User-Agent: SquirrelMail/1.4.15 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: freebsd-stable@freebsd.org Subject: Re: RELENG_7_1: bce driver change generating too much interrupts ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 16:47:29 -0000 On Sun, December 14, 2008 4:57 am, Xin LI wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Mike Jakubik wrote: >> On Mon, December 8, 2008 5:22 pm, Mike Jakubik wrote: >>> On Mon, December 8, 2008 5:12 pm, Xin LI wrote: >>> >>>> Which version are you currently using? My previous commit only fixes >>>> the excessive interrupt issue, I think this could be a different >>>> problem, I'm taking a look at the code to see if I can have something >>>> for you. >>> I was running on the version just prior to the latest interrupt commit. >>> I >>> have now updated to the one with the interrupt fix. Will let you know >>> if >>> things change. >>> >>> Thank You. >> >> The interrupt rate has decreased significantly, however i am still >> having >> having problem with applications that hold stateful connections. The rx >> errors are also still showing, i suspect this is related to the problem. >> How can i roll back this driver to the last known good version? > > Hi, Mike, > > I think they are different problems. Could you, please, give me > feedback about whether: > > - The old driver does not trigger the problem? > > - The patched driver restore all the old driver behavior? - Old driver. I have been running the system for 4 days now with this driver. My application has not stopped accepting connections, irq rate is low, and there are no rx/tx errors reported. Everything looks good. - Patched driver. Your initial import plus the IRQ fix still shows rx errors, and my application had stopped accepting connections. I have not tried the patch in your last email, and im not sure when i will be able to to as these systems are in production. Perhaps someone else could test it? As soon as i get a chance i will let you know how it goes. Thanks for the work on this. From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 17:12:11 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B90A106567B for ; Mon, 15 Dec 2008 17:12:11 +0000 (UTC) (envelope-from ghelmer@palisadesys.com) Received: from cetus.palisadesys.com (cetus.palisadesys.com [205.237.115.21]) by mx1.freebsd.org (Postfix) with ESMTP id D1FC78FC25 for ; Mon, 15 Dec 2008 17:12:10 +0000 (UTC) (envelope-from ghelmer@palisadesys.com) Received: from magellan.palisadesys.com (serverwatch [172.16.1.98]) by cetus.palisadesys.com (8.14.3/8.14.3) with ESMTP id mBFHC9Lk020046 for ; Mon, 15 Dec 2008 11:12:09 -0600 (CST) (envelope-from ghelmer@palisadesys.com) Received: from [172.16.2.242] (cetus.palisadesys.com [205.237.115.21]) (authenticated bits=0) by magellan.palisadesys.com (8.14.2/8.14.2) with ESMTP id mBFHC53V082230 for ; Mon, 15 Dec 2008 11:12:05 -0600 (CST) (envelope-from ghelmer@palisadesys.com) Message-ID: <49468FE2.4040007@palisadesys.com> Date: Mon, 15 Dec 2008 11:12:02 -0600 From: Guy Helmer User-Agent: Thunderbird 2.0.0.18 (Windows/20081105) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <49419BE0.2070502@palisadesys.com> <3a142e750812111900u1d9264bh27d39092f5dcc4bf@mail.gmail.com> <49426880.40302@palisadesys.com> <3a142e750812120729v478d850do93e31e5d8502ef68@mail.gmail.com> <4942C178.2030206@palisadesys.com> In-Reply-To: <4942C178.2030206@palisadesys.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-3.0 (magellan.palisadesys.com [205.237.115.20]); Mon, 15 Dec 2008 11:12:06 -0600 (CST) X-Palisade-MailScanner-Information: Please contact the ISP for more information X-Palisade-MailScanner: Found to be clean X-Palisade-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (not cached, score=-4.399, required 6, autolearn=not spam, ALL_TRUSTED -1.80, BAYES_00 -2.60) X-Palisade-MailScanner-From: ghelmer@palisadesys.com Subject: Re: 7.1RC1: system hang X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 17:12:11 -0000 I have a recurring hang on FreeBSD 7.1 roughly-RC1 (releng_7_1 checkout as of 2008-12-08) on a dual-CPU hyperthreaded Xeon i386. Here is the output from ps/m, show allpcpu, show locks, show alllocks, allt, show allchains, and show lockedvnods commands in the debugger. I'm suspicious of the vmstat process's locking -- it has shown up in each of the hangs I've seen so far. Any help would be appreciated. Guy db> ps /m pid ppid pgrp uid state wmesg wchan cmd 96381 96369 96364 0 S user map 0xc610d86c vmstat 96378 96368 96365 0 R sysctl 96369 96364 96364 0 S piperd 0xc4928c60 perl5.8.8 96368 96365 96365 0 S piperd 0xc75ec630 perl5.8.8 96367 96362 96362 1001 R initial thread 96365 96361 96365 0 Ss wait 0xc5cd42b8 sh 96364 96360 96364 0 Ss wait 0xc6131570 sh 96362 96359 96362 1001 Ss wait 0xc7478828 sh 96361 3851 3851 0 S piperd 0xc4986dec cron 96360 3851 3851 0 S piperd 0xc5dcf948 cron 96359 3851 3851 0 S piperd 0xc74b8000 cron 96358 1424 96358 70 Ss sbwait 0xc60a073c postgres 96215 96203 2054 1001 RL filter 96212 96209 2054 1001 R kvoop 96209 96201 2054 1001 R CPU 1 filter 96203 2055 2054 1001 S piperd 0xc5bf5948 perform_ca 96201 2055 2054 1001 S piperd 0xc61b94a4 perform_ca 92421 3821 3821 1001 S proctree 0xc0928754 httpd 92417 3821 3821 1001 R httpd 50637 1424 50637 70 Ss sbwait 0xc772c8dc postgres 50633 1441 1440 1001 S proctree 0xc0928754 ph_alert_mgr 4611 4602 4611 1001 Ss+ allproc 0xc092873c tcsh 4602 4550 4550 1001 S select 0xc097a77c sshd 4550 3836 4550 0 Ss sbwait 0xc5ebd3fc sshd 4459 4201 4459 1001 S+ allproc 0xc092873c tcsh 4202 1 4202 0 Ss sysctl l 0xc0928c34 ntpd 4201 1 4201 0 Ss+ wait 0xc48d8828 login 4200 1 4200 0 Ss+ ttyin 0xc4594010 getty 4199 1 4199 0 Ss+ ttyin 0xc4595810 getty 4198 1 4198 0 Ss+ ttyin 0xc45a3c10 getty 4197 1 4197 0 Ss+ ttyin 0xc45a3810 getty 3851 1 3851 0 Ss nanslp 0xc0928dc4 cron 3845 1 3845 25 Ss pause 0xc5cd3318 sendmail 3836 1 3836 0 Ss select 0xc097a77c sshd 3821 1 3821 0 Ss proctree 0xc0928754 initial thread 3813 1 3813 0 Ss sysctl l 0xc0928c34 sendmail 2463 2213 2463 100 Ss piperd 0xc47b1948 unlinkd 2225 1 2225 1001 Ss (threaded) proxsmtpd 100120 S accept 0xc5cdfd3a proxsmtpd 2219 1 50 0 S+ select 0xc097a77c snmpd 2213 2211 2211 100 S select 0xc097a77c squid 2211 1 2211 100 Ss wait 0xc5cd3570 squid 2189 1 2188 0 S nanslp 0xc0928dc4 smartd 2175 2167 2167 1001 S accept 0xc5c5a37a imspector 2174 2167 2167 1001 S accept 0xc48cc85a imspector 2168 2167 2167 1001 S accept 0xc5c5a6ba imspector 2167 1 2167 1001 Ss select 0xc097a77c imspector 2166 1424 2166 70 Ss sbwait 0xc4be30bc postgres 2162 2161 2161 1001 S nanslp 0xc0928dc4 initial thread 2161 1 2161 0 Ss wait 0xc5bf8570 sys-monitor 2154 2153 2153 1001 S proctree 0xc0928754 initial thread 2153 1 2153 0 Ss wait 0xc47f2ae0 sys-monitor 2138 2137 2137 1001 S proctree 0xc0928754 icapd 2137 1 2137 0 Ss wait 0xc5bf8ae0 sys-monitor 2127 2126 2126 1001 S select 0xc097a77c perl5.8.8 2126 1 2126 0 Ss wait 0xc4a70828 sys-monitor 2063 2062 2062 1001 S accept 0xc48d203a initial thread 2062 1 2062 0 Ss wait 0xc5bf72b8 sys-monitor 2055 2054 2054 1001 S proctree 0xc0928754 perform_ca 2054 1 2054 0 Ss wait 0xc5bf7000 sys-monitor 2052 1 2052 0 Ss sysctl l 0xc0928c34 sys-ifmonitor 1789 1424 1789 70 Ss sbwait 0xc4be38dc postgres 1746 1745 1745 1001 S select 0xc097a77c initial thread 1745 1 1745 0 Ss wait 0xc47f3828 sys-monitor 1738 1737 1737 1001 S select 0xc097a77c user_manager 1737 1 1737 0 Ss wait 0xc47262b8 sys-monitor 1730 1729 1729 1001 R (threaded) ph 100185 S ucond 0xc49a5440 ph 100184 S ucond 0xc49a5400 ph 100183 S ucond 0xc49a53c0 ph 100182 S ucond 0xc49a5380 ph 100082 RunQ initial thread 1729 1 1729 0 Ss wait 0xc4a6f828 sys-monitor 1441 1440 1440 1001 S accept 0xc48cd51a initial thread 1440 1 1440 0 Ss wait 0xc47f3570 sys-monitor 1429 1424 1429 70 Ss select 0xc097a77c postgres 1428 1424 1428 70 Ss select 0xc097a77c postgres 1427 1424 1427 70 Ss select 0xc097a77c postgres 1426 1424 1426 70 Ss select 0xc097a77c postgres 1424 1422 1422 70 S allproc 0xc092873c initial thread 1422 1 1422 0 Ss wait 0xc48da828 sys-monitor 1345 1 1345 0 Ss select 0xc097a77c syslogd 1316 0 0 0 SL - 0xc0926a60 [accounting] 465 1 465 0 Ss select 0xc097a77c devd 120 1 120 0 Ss pause 0xc47b55d0 adjkerntz 49 0 0 0 SL sdflush 0xc09836c8 [softdepflush] 48 0 0 0 SL vlruwt 0xc4722828 [vnlru] 47 0 0 0 SL syncer 0xc0928bec [syncer] 46 0 0 0 SL psleep 0xc097ac04 [bufdaemon] 45 0 0 0 SL pollid 0xc0928318 [idlepoll] 44 0 0 0 SL pgzero 0xc09842a0 [pagezero] 43 0 0 0 SL psleep 0xc0983eb8 [vmdaemon] 42 0 0 0 SL psleep 0xc0983e80 [pagedaemon] 41 0 0 0 SL - 0xc46c6480 [dummynet] 40 0 0 0 WL [irq7: ppbus0 ppc0] 39 0 0 0 SL - 0xc456a83c [fdc0] 38 0 0 0 WL [swi0: sio] 37 0 0 0 WL [irq1: atkbd0] 36 0 0 0 WL [irq15: ata1] 35 0 0 0 WL [irq14: ata0] 34 0 0 0 SL usbevt 0xc455d210 [usb2] 33 0 0 0 WL [irq18: uhci2] 32 0 0 0 SL usbevt 0xc453e210 [usb1] 31 0 0 0 WL [irq19: uhci1] 30 0 0 0 SL usbtsk 0xc0925bd4 [usbtask-dr] 29 0 0 0 SL usbtsk 0xc0925bc0 [usbtask-hc] 28 0 0 0 SL usbevt 0xc4544210 [usb0] 27 0 0 0 WL [irq16: uhci0] 26 0 0 0 SL - 0xc4523a00 [em1 taskq] 25 0 0 0 SL - 0xc4523c00 [em0 taskq] 24 0 0 0 WL [irq9: acpi0] 23 0 0 0 SL - 0xc446ed80 [kqueue taskq] 22 0 0 0 WL [swi2: cambio] 9 0 0 0 SL ccb_scan 0xc090c954 [xpt_thrd] 21 0 0 0 WL [swi6: task queue] 20 0 0 0 WL [swi6: Giant taskq] 8 0 0 0 SL - 0xc44b3280 [acpi_task_2] 7 0 0 0 SL - 0xc44b3280 [acpi_task_1] 6 0 0 0 SL - 0xc44b3280 [acpi_task_0] 5 0 0 0 SL - 0xc44b3300 [thread taskq] 19 0 0 0 WL [swi5: +] 18 0 0 0 SL - 0xc0928bf4 [yarrow] 4 0 0 0 SL - 0xc092634c [g_down] 3 0 0 0 SL - 0xc0926348 [g_up] 2 0 0 0 SL - 0xc0926340 [g_event] 17 0 0 0 WL [swi3: vm] 16 0 0 0 WL [swi4: clock sio] 15 0 0 0 WL [swi1: net] 14 0 0 0 RL CPU 0 [idle: cpu0] 13 0 0 0 RL [idle: cpu1] 12 0 0 0 RL CPU 2 [idle: cpu2] 11 0 0 0 RL CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc442bae0 [init] 10 0 0 0 SL audit_wo 0xc0983114 [audit] 0 0 0 0 SLs sched 0xc0926400 [swapper] 95907 1441 1440 1001 Z ph_alert_mgr 95900 1441 1440 1001 Z ph_alert_mgr 95901 1441 1440 1001 Z ph_alert_mgr 95891 1441 1440 1001 Z ph_alert_mgr 95899 1441 1440 1001 Z ph_alert_mgr 95898 1441 1440 1001 Z ph_alert_mgr 95902 1441 1440 1001 Z ph_alert_mgr db> show allpcpu Current CPU: 0 cpuid = 0 curthread = 0xc442d460: pid 14 "idle: cpu0" curpcb = 0xe300ad90 fpcurthread = none idlethread = 0xc442d460: pid 14 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 curthread = 0xc7904230: pid 96209 "filter" curpcb = 0xe72f8d90 fpcurthread = none idlethread = 0xc442d690: pid 13 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 curthread = 0xc442d8c0: pid 12 "idle: cpu2" curpcb = 0xe3004d90 fpcurthread = none idlethread = 0xc442d8c0: pid 12 "idle: cpu2" APIC ID = 6 currentldt = 0x50 spin locks held: cpuid = 3 curthread = 0xc442daf0: pid 11 "idle: cpu3" curpcb = 0xe3001d90 fpcurthread = none idlethread = 0xc442daf0: pid 11 "idle: cpu3" APIC ID = 7 currentldt = 0x50 spin locks held: db> show locks db> show alllocks Process 96381 (vmstat) thread 0xc756caf0 (100276) shared sx allproc r = 0 (0xc092873c) locked @ vm/vm_meter.c:130 exclusive sx sysctl lock r = 0 (0xc0928c34) locked @ kern/kern_sysctl.c:1396 Process 96358 (postgres) thread 0xc7551460 (100275) exclusive sx so_rcv_sx r = 0 (0xc60a070c) locked @ kern/uipc_sockbuf.c:148 Process 96215 (filter) thread 0xc613b230 (100228) exclusive sleep mutex vm object (standard object) r = 0 (0xc5de84d8) locked @ vm/vm_fault.c:886 exclusive sx user map r = 0 (0xc610d86c) locked @ vm/vm_map.c:3111 Process 50637 (postgres) thread 0xc613b8c0 (100225) exclusive sx so_rcv_sx r = 0 (0xc772c8ac) locked @ kern/uipc_sockbuf.c:148 Process 4611 (tcsh) thread 0xc47fc230 (100097) shared sx proctree r = 0 (0xc0928754) locked @ kern/kern_fork.c:300 Process 4550 (sshd) thread 0xc4a74000 (100170) exclusive sx so_rcv_sx r = 0 (0xc5ebd3cc) locked @ kern/uipc_sockbuf.c:148 Process 4459 (tcsh) thread 0xc4822af0 (100107) shared sx proctree r = 0 (0xc0928754) locked @ kern/kern_fork.c:300 Process 2166 (postgres) thread 0xc48db8c0 (100086) exclusive sx so_rcv_sx r = 0 (0xc4be308c) locked @ kern/uipc_sockbuf.c:148 Process 1789 (postgres) thread 0xc47bad20 (100051) exclusive sx so_rcv_sx r = 0 (0xc4be38ac) locked @ kern/uipc_sockbuf.c:148 Process 1424 (postgres) thread 0xc4822690 (100109) shared sx proctree r = 0 (0xc0928754) locked @ kern/kern_fork.c:300 db> show sleepchain 96381 thread 100276 (pid 96381, vmstat) blocked on sx "user map" XLOCK thread 100228 (pid 96215, filter) is on a run queue db> show sleepchain 96358 thread 100275 (pid 96358, postgres) sleeping on 0xc60a073c "sbwait" db> show sleepchain 96215 thread 100228 (pid 96215, filter) is on a run queue db> show sleepchain 50637 thread 100225 (pid 50637, postgres) sleeping on 0xc772c8dc "sbwait" db> show sleepchain 4611 thread 100097 (pid 4611, tcsh) blocked on sx "allproc" SLOCK (count 1) db> show sleepchain 4550 thread 100170 (pid 4550, sshd) sleeping on 0xc5ebd3fc "sbwait" db> show sleepchain 4459 thread 100107 (pid 4459, tcsh) blocked on sx "allproc" SLOCK (count 1) db> show sleepchain 2166 thread 100086 (pid 2166, postgres) sleeping on 0xc4be30bc "sbwait" db> show sleepchain 1789 thread 100051 (pid 1789, postgres) sleeping on 0xc4be38dc "sbwait" db> show sleepchain 1424 thread 100109 (pid 1424, initial thread) blocked on sx "allproc" SLOCK (count 1) db> allt Tracing command vmstat pid 96381 tid 100276 td 0xc756caf0 sched_switch(c756caf0,0,1,24b,d2bf8efc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c756caf0,c756caf0,c756caf0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c756caf0,0,c085c813,243,c610d86c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c610d86c,0,c086e83c,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c610d86c,c756caf0,0,c086e932,ba,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c610d86c,0,c086e932,ba,e709cb58,...) at 0xc0660b6b = _sx_xlock+0x6b _vm_map_lock_read(c610d828,c086e932,ba,b1,c47bbd98,...) at 0xc07ab7a0 = _vm_map_lock_read+0x50 vmtotal(c08fcb60,0,30,e709cba4,e709cba4,...) at 0xc07ae75b = vmtotal+0x29b sysctl_root(e709cba4,0,c085a213,574,c756caf0,...) at 0xc0662977 = sysctl_root+0x127 userland_sysctl(c756caf0,e709cc14,2,bfbfe4dc,bfbfe520,...) at 0xc0662aa5 = userland_sysctl+0x115 __sysctl(c756caf0,e709ccfc,18,c085df3a,c08e2af0,...) at 0xc0662eec = __sysctl+0xbc syscall(e709cd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x4817dc1f, esp = 0xbfbfe32c, ebp = 0xbfbfe358 --- Tracing command sysctl pid 96378 tid 100194 td 0xc5e27000 sched_switch(c5e27000,0,1,c068abe0,a9326472,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085ccbb,2e3,c09339c4,...) at 0xc06611d3 = mi_switch+0x143 turnstile_wait(c4bc61e0,c5f80af0,0,c75ec7a0,3d0,...) at 0xc068b404 = turnstile_wait+0x234 _mtx_lock_sleep(c75ec7a0,c5e27000,0,c085d7a2,3d0,...) at 0xc064d51e = _mtx_lock_sleep+0x10e _mtx_lock_flags(c75ec7a0,0,c085d7a2,3d0,8,...) at 0xc064d5a7 = _mtx_lock_flags+0x67 pipe_write(c5ebbdf4,e6f8ac60,c6c25a00,0,c5e27000,...) at 0xc0691e80 = pipe_write+0x40 dofilewrite(e6f8ac60,ffffffff,ffffffff,0,c5ebbdf4,...) at 0xc068fd65 = dofilewrite+0x95 kern_writev(c5e27000,1,e6f8ac60,bfbfd560,1,...) at 0xc068ffe8 = kern_writev+0x58 write(c5e27000,e6f8acfc,c,c084f1db,c08e1860,...) at 0xc069005f = write+0x4f syscall(e6f8ad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x4815a963, esp = 0xbfbfd3dc, ebp = 0xbfbfd3f8 --- Tracing command perl5.8.8 pid 96369 tid 100188 td 0xc5e27d20 sched_switch(c5e27d20,0,1,24b,cf951454,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5ff0840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5ff0840,0,c085c813,19e,c4928c60,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5e27d20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4928c60,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4928c60,c4928dd0,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c493689c,e6f78c60,c62ca500,0,c5e27d20,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e6f78c60,ffffffff,ffffffff,0,c493689c,...) at 0xc0690106 = dofileread+0x96 kern_readv(c5e27d20,3,e6f78c60,48305000,1000,...) at 0xc0690478 = kern_readv+0x58 read(c5e27d20,e6f78cfc,c,c5e27d20,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6f78d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x482a2983, esp = 0xbfbfca6c, ebp = 0xbfbfca88 --- Tracing command perl5.8.8 pid 96368 tid 100477 td 0xc5f80af0 sched_switch(c5f80af0,0,1,24b,a93390e4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5c642d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5c642d0,0,c085c813,19e,c75ec630,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5f80af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c75ec630,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c75ec630,c75ec7a0,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c4a00c78,e7395c60,c6c25a00,0,c5f80af0,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e7395c60,ffffffff,ffffffff,0,c4a00c78,...) at 0xc0690106 = dofileread+0x96 kern_readv(c5f80af0,4,e7395c60,48305000,1000,...) at 0xc0690478 = kern_readv+0x58 read(c5f80af0,e7395cfc,c,c082c73d,c08e1848,...) at 0xc069055f = read+0x4f syscall(e7395d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x482a2983, esp = 0xbfbfcb0c, ebp = 0xbfbfcb28 --- Tracing command php pid 96367 tid 100284 td 0xc7569d20 sched_switch(c7569d20,0,2,b4,994e18fa,...) at 0xc0676004 = sched_switch+0x324 mi_switch(2,0,c085cb08,ec,10004,...) at 0xc06611d3 = mi_switch+0x143 ast(e70b4d38) at 0xc0689c9b = ast+0x23b doreti_ast() at 0xc07e4aed = doreti_ast+0x17 Tracing command sh pid 96365 tid 100148 td 0xc4bc8af0 sched_switch(c4bc8af0,0,1,24b,7e398e8e,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5cd42d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5cd42d0,0,c085c813,19e,c5cd42b8,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c4bc8af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5cd42b8,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5cd42b8,c5cd4348,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c4bc8af0,ffffffff,e6eb8c2c,2,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c4bc8af0,e6eb8cfc,10,c4bc8af0,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6eb8d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x4814658b, esp = 0xbfbfebdc, ebp = 0xbfbfebf8 --- Tracing command sh pid 96364 tid 100231 td 0xc613aaf0 sched_switch(c613aaf0,0,1,24b,7ecc5d46,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c6131588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c6131588,0,c085c813,19e,c6131570,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c613aaf0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c6131570,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c6131570,c6131600,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c613aaf0,ffffffff,e7015c2c,2,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c613aaf0,e7015cfc,10,c613aaf0,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e7015d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x4814658b, esp = 0xbfbfebdc, ebp = 0xbfbfebf8 --- Tracing command sh pid 96362 tid 100283 td 0xc756c000 sched_switch(c756c000,0,1,24b,7e295ce4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c7478840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c7478840,0,c085c813,19e,c7478828,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c756c000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c7478828,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c7478828,c74788b8,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c756c000,ffffffff,e70b1c2c,2,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c756c000,e70b1cfc,10,c756c000,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e70b1d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x4814658b, esp = 0xbfbfebdc, ebp = 0xbfbfebf8 --- Tracing command cron pid 96361 tid 100239 td 0xc60f1690 sched_switch(c60f1690,0,1,24b,7d08d224,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c61672d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c61672d0,0,c085c813,19e,c4986dec,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c60f1690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4986dec,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4986dec,c4986f5c,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c4a000e4,e702dc60,c5cf2b00,0,c60f1690,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e702dc60,ffffffff,ffffffff,0,c4a000e4,...) at 0xc0690106 = dofileread+0x96 kern_readv(c60f1690,5,e702dc60,48221000,1000,...) at 0xc0690478 = kern_readv+0x58 read(c60f1690,e702dcfc,c,c085da67,c08e1848,...) at 0xc069055f = read+0x4f syscall(e702dd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x48174983, esp = 0xbfbfe6fc, ebp = 0xbfbfe718 --- Tracing command cron pid 96360 tid 100210 td 0xc600d460 sched_switch(c600d460,0,1,24b,7d8dcb66,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c60ed840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c60ed840,0,c085c813,19e,c5dcf948,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c600d460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5dcf948,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5dcf948,c5dcfab8,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c6108d10,e6fbac60,c5cf2b00,0,c600d460,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e6fbac60,ffffffff,ffffffff,0,c6108d10,...) at 0xc0690106 = dofileread+0x96 kern_readv(c600d460,5,e6fbac60,48221000,1000,...) at 0xc0690478 = kern_readv+0x58 read(c600d460,e6fbacfc,c,c085da67,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6fbad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x48174983, esp = 0xbfbfe6fc, ebp = 0xbfbfe718 --- Tracing command cron pid 96359 tid 100227 td 0xc613b460 sched_switch(c613b460,0,1,24b,7cf981f6,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c61322d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c61322d0,0,c085c813,19e,c74b8000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c613b460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c74b8000,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c74b8000,c74b8170,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c5ebb428,e7009c60,c5cf2b00,0,c613b460,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e7009c60,ffffffff,ffffffff,0,c5ebb428,...) at 0xc0690106 = dofileread+0x96 kern_readv(c613b460,5,e7009c60,48221000,1000,...) at 0xc0690478 = kern_readv+0x58 read(c613b460,e7009cfc,c,c085da67,c08e1848,...) at 0xc069055f = read+0x4f syscall(e7009d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x48174983, esp = 0xbfbfe6fc, ebp = 0xbfbfe718 --- Tracing command postgres pid 96358 tid 100275 td 0xc7551460 sched_switch(c7551460,0,1,24b,93a11dd4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c753a588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c753a588,0,c085c813,19e,c60a073c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c7551460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c60a073c,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c60a073c,c60a06f4,158,c08601ab,0,...) at 0xc0661707 = _sleep+0x307 sbwait(c60a06d0,0,c0860221,592,c60a06f4,...) at 0xc06a8a42 = sbwait+0x52 soreceive_generic(c60a0680,0,e7099c60,0,0,...) at 0xc06ad7ac = soreceive_generic+0x3bc soreceive(c60a0680,0,e7099c60,0,0,...) at 0xc06a98c8 = soreceive+0x38 soo_read(c5ebb6d4,e7099c60,c49b1500,0,c7551460,...) at 0xc0694bdb = soo_read+0x3b dofileread(e7099c60,ffffffff,ffffffff,0,c5ebb6d4,...) at 0xc0690106 = dofileread+0x96 kern_readv(c7551460,7,e7099c60,48970000,5,...) at 0xc0690478 = kern_readv+0x58 read(c7551460,e7099cfc,c,c084f1db,c08e1848,...) at 0xc069055f = read+0x4f syscall(e7099d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x4878a983, esp = 0xbfbfd7dc, ebp = 0xbfbfd7f8 --- Tracing command filter pid 96215 tid 100228 td 0xc613b230 sched_switch(c613b230,0,1,c068abe0,44a5e66a,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085ccbb,2e3,c09338c8,...) at 0xc06611d3 = mi_switch+0x143 turnstile_wait(c787a820,c7903af0,0,c0983e5c,377,...) at 0xc068b404 = turnstile_wait+0x234 _mtx_lock_sleep(c0983e5c,c613b230,0,c086e39f,377,...) at 0xc064d51e = _mtx_lock_sleep+0x10e _mtx_lock_flags(c0983e5c,0,c086e39f,377,0,...) at 0xc064d5a7 = _mtx_lock_flags+0x67 vm_fault(c610d828,4826a000,1,0,4826a928,...) at 0xc07a765b = vm_fault+0x1b1b trap_pfault(5,0,c0877641,c613b230,c,...) at 0xc07fbe89 = trap_pfault+0xf9 trap(e700cd38) at 0xc07fc6fd = trap+0x25d calltrap() at 0xc07e41cb = calltrap+0x6 --- trap 0xc, eip = 0x4826a928, esp = 0xbfbfda0c, ebp = 0xbfbfda38 --- Tracing command kvoop pid 96212 tid 100243 td 0xc60f0d20 sched_switch(c60f0d20,0,1,24b,2eb909b6,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c6166588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c6166588,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e7039ad8,c064d2b9,c097a764,c60f0d20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,ea61,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,ea61,3bb,2,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 poll(c60f0d20,e7039cfc,c,c085e083,c08e2b98,...) at 0xc068eec4 = poll+0x554 syscall(e7039d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x48226bb7, esp = 0x8068f3c, ebp = 0x8068f98 --- Tracing command filter pid 96209 tid 100437 td 0xc7904230 cpustop_handler(2,e72f8d2c,c07fc4d0,23,3,...) at 0xc07f2c52 = cpustop_handler+0x32 ipi_nmi_handler(23,3,c0676004,c7904230,13,...) at 0xc07f2cdf = ipi_nmi_handler+0x2f trap(e72f8d38) at 0xc07fc4d0 = trap+0x30 calltrap() at 0xc07e41cb = calltrap+0x6 --- trap 0x13, eip = 0x480a96d5, esp = 0xbfbfcf70, ebp = 0xbfbfcf78 --- Tracing command perform_ca pid 96203 tid 100203 td 0xc600e460 sched_switch(c600e460,0,1,24b,41c7c61c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c60ef018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c60ef018,0,c085c813,19e,c5bf5948,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c600e460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5bf5948,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5bf5948,c5bf5ab8,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c4b7b4c0,e6fa5c60,c4978100,0,c600e460,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e6fa5c60,ffffffff,ffffffff,0,c4b7b4c0,...) at 0xc0690106 = dofileread+0x96 kern_readv(c600e460,d,e6fa5c60,839b1db,1,...) at 0xc0690478 = kern_readv+0x58 read(c600e460,e6fa5cfc,c,c600e460,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6fa5d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x820c56f, esp = 0xbfbf998c, ebp = 0xbfbf99b8 --- Tracing command perform_ca pid 96201 tid 100481 td 0xc77d2af0 sched_switch(c77d2af0,0,1,24b,26596dcc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c78fa588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c78fa588,0,c085c813,19e,c61b94a4,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c77d2af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c61b94a4,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c61b94a4,c61b9614,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c5ebb63c,e73b7c60,c4978100,0,c77d2af0,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e73b7c60,ffffffff,ffffffff,0,c5ebb63c,...) at 0xc0690106 = dofileread+0x96 kern_readv(c77d2af0,d,e73b7c60,839b1db,1,...) at 0xc0690478 = kern_readv+0x58 read(c77d2af0,e73b7cfc,c,c77d2af0,c08e1848,...) at 0xc069055f = read+0x4f syscall(e73b7d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x820c56f, esp = 0xbfbf998c, ebp = 0xbfbf99b8 --- Tracing command httpd pid 92421 tid 100300 td 0xc754f690 sched_switch(c754f690,0,1,24b,f0f9ecd8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c754f690,c754f690,c754f690,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c754f690,0,c085c813,243,c0928754,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928754,0,c0859453,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928754,c754f690,0,c0852b41,1a0,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928754,0,c0852b41,1a0,1002,...) at 0xc0660b6b = _sx_xlock+0x6b devfs_close(e70efb30,e70efb54,c06d91c7,c08de5e0,e70efb30,...) at 0xc05f29af = devfs_close+0x3f VOP_CLOSE_APV(c08de5e0,e70efb30,c754f690,c086269f,120,...) at 0xc080ff52 = VOP_CLOSE_APV+0x42 vn_close(c63ffbdc,1,c48c2900,c754f690,c068e3e2,...) at 0xc06d91c7 = vn_close+0x87 vn_closefile(c4b7b260,c754f690,c4b7b260,0,e70efbe8,...) at 0xc06d92d7 = vn_closefile+0xe7 devfs_close_f(c4b7b260,c754f690,c0856ae5,8c6,e70efbe8,...) at 0xc05f031b = devfs_close_f+0x2b fdrop(c4b7b260,c754f690,c0856aee,c068d7b8,0,c090c2e0,c63ffbdc,c75182b8,2,e70efc1c,40,0,0,0,0,8,2,463) at 0xc062df71 = fdrop+0xf1 closef(c4b7b260,c754f690,463,448,c4b7b260,...) at 0xc062faf2 = closef+0x252 kern_close(c754f690,10,e70efd2c,c07fc233,c754f690,...) at 0xc062fe9d = kern_close+0x11d close(c754f690,e70efcfc,4,16,c08e1890,...) at 0xc062ff3a = close+0x1a syscall(e70efd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (6, FreeBSD ELF32, close), eip = 0x48501943, esp = 0xbfbfe9bc, ebp = 0xbfbfe9d8 --- Tracing command httpd pid 92417 tid 100328 td 0xc77e0000 sched_switch(c77e0000,0,2,b4,9534efa6,...) at 0xc0676004 = sched_switch+0x324 mi_switch(2,0,c085cb08,ec,10004,...) at 0xc06611d3 = mi_switch+0x143 ast(e7154d38) at 0xc0689c9b = ast+0x23b doreti_ast() at 0xc07e4aed = doreti_ast+0x17 Tracing command postgres pid 50637 tid 100225 td 0xc613b8c0 sched_switch(c613b8c0,0,1,24b,f2b10170,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c6132840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c6132840,0,c085c813,19e,c772c8dc,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c613b8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c772c8dc,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c772c8dc,c772c894,158,c08601ab,0,...) at 0xc0661707 = _sleep+0x307 sbwait(c772c870,0,c0860221,592,c772c894,...) at 0xc06a8a42 = sbwait+0x52 soreceive_generic(c772c820,e7003be8,e7003bf4,0,0,...) at 0xc06ad7ac = soreceive_generic+0x3bc soreceive(c772c820,e7003be8,e7003bf4,0,0,...) at 0xc06a98c8 = soreceive+0x38 kern_recvit(c613b8c0,7,e7003c60,0,0,...) at 0xc06af145 = kern_recvit+0x155 recvit(0,e7003c80,c06b1868,8382ea0,2000,0,0,e7003c58,1,0,4879bb98,0) at 0xc06af351 = recvit+0x31 recvfrom(c613b8c0,e7003cfc,18,c085dd08,c08e1ab8,...) at 0xc06af4c6 = recvfrom+0x76 syscall(e7003d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (29, FreeBSD ELF32, recvfrom), eip = 0x487439a7, esp = 0xbfbfd97c, ebp = 0xbfbfd9a8 --- Tracing command ph_alert_mgr pid 50633 tid 100445 td 0xc7903460 sched_switch(c7903460,0,1,24b,854fdf38,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c7903460,c7903460,c7903460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c7903460,0,c085c813,243,c0928754,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928754,0,c0859453,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928754,c7903460,0,c0856f85,2cb,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928754,0,c0856f85,2cb,c068c95e,...) at 0xc0660b6b = _sx_xlock+0x6b kern_wait(c7903460,ffffffff,e7318c2c,1,0,...) at 0xc0638838 = kern_wait+0x128 wait4(c7903460,e7318cfc,10,c085dc7c,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e7318d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x4854a58b, esp = 0xbfbfe1dc, ebp = 0xbfbfe1f8 --- Tracing command tcsh pid 4611 tid 100097 td 0xc47fc230 sched_switch(c47fc230,0,1,24b,365f0754,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c47fc230,c47fc230,c47fc230,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47fc230,0,c085c813,243,c092873c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c092873c,0,c085944b,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c092873c,c47fc230,0,c085706f,135,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c092873c,0,c085706f,135,0,...) at 0xc0660b6b = _sx_xlock+0x6b fork1(c47fc230,80000034,0,e6df3c78,c47fc230,...) at 0xc063aa6b = fork1+0x2eb vfork(c47fc230,e6df3cfc,c08775cf,c085da12,c08e1e30,...) at 0xc063bc29 = vfork+0x29 syscall(e6df3d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (66, FreeBSD ELF32, vfork), eip = 0x48164e9c, esp = 0xbfbfe760, ebp = 0xbfbfe858 --- Tracing command sshd pid 4602 tid 100094 td 0xc47fc8c0 sched_switch(c47fc8c0,0,1,24b,a9b0257c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f72d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f72d0,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6dcca84,c064d2b9,c097a764,0,c47fc8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c097a77c,c097a764,c086acda,101,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _cv_wait_sig(c097a77c,c097a764,c085d774,30f,386,...) at 0xc06262a8 = _cv_wait_sig+0x198 kern_select(c47fc8c0,b,485030c0,485030c4,0,0,0,480eaa88) at 0xc068f6dc = kern_select+0x6cc select(c47fc8c0,e6dcccfc,14,c084f1db,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6dccd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x48391903, esp = 0xbfbfe5dc, ebp = 0xbfbfe628 --- Tracing command sshd pid 4550 tid 100170 td 0xc4a74000 sched_switch(c4a74000,0,1,24b,84d1e618,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5e23018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5e23018,0,c085c813,19e,c5ebd3fc,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c4a74000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5ebd3fc,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5ebd3fc,c5ebd3b4,158,c08601ab,0,...) at 0xc0661707 = _sleep+0x307 sbwait(c5ebd390,0,c0860221,592,c5ebd3b4,...) at 0xc06a8a42 = sbwait+0x52 soreceive_generic(c5ebd340,0,e6f42c60,0,0,...) at 0xc06ad7ac = soreceive_generic+0x3bc soreceive(c5ebd340,0,e6f42c60,0,0,...) at 0xc06a98c8 = soreceive+0x38 soo_read(c4a1c558,e6f42c60,c4774c00,0,c4a74000,...) at 0xc0694bdb = soo_read+0x3b dofileread(e6f42c60,ffffffff,ffffffff,0,c4a1c558,...) at 0xc0690106 = dofileread+0x96 kern_readv(c4a74000,5,e6f42c60,bfbfe62c,4,...) at 0xc0690478 = kern_readv+0x58 read(c4a74000,e6f42cfc,c,c08691e4,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6f42d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x48391983, esp = 0xbfbfe5cc, ebp = 0xbfbfe608 --- Tracing command tcsh pid 4459 tid 100107 td 0xc4822af0 sched_switch(c4822af0,0,1,24b,861167f4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4822af0,c4822af0,c4822af0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4822af0,0,c085c813,243,c092873c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c092873c,0,c085944b,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c092873c,c4822af0,0,c085706f,135,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c092873c,0,c085706f,135,0,...) at 0xc0660b6b = _sx_xlock+0x6b fork1(c4822af0,80000034,0,e6e29c78,c4822af0,...) at 0xc063aa6b = fork1+0x2eb vfork(c4822af0,e6e29cfc,c08775cf,c085da12,c08e1e30,...) at 0xc063bc29 = vfork+0x29 syscall(e6e29d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (66, FreeBSD ELF32, vfork), eip = 0x48164e9c, esp = 0xbfbfe910, ebp = 0xbfbfea08 --- Tracing command ntpd pid 4202 tid 100114 td 0xc47fdaf0 sched_switch(c47fdaf0,0,1,24b,21106ddc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c47fdaf0,c47fdaf0,c47fdaf0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47fdaf0,0,c085c813,243,c0928c34,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928c34,0,c085a2de,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928c34,c47fdaf0,0,c085a213,574,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928c34,0,c085a213,574,c47fdaf0,...) at 0xc0660b6b = _sx_xlock+0x6b userland_sysctl(c47fdaf0,e6e3ec14,6,0,bfbfe7e8,...) at 0xc0662a81 = userland_sysctl+0xf1 __sysctl(c47fdaf0,e6e3ecfc,18,0,c08e2af0,...) at 0xc0662eec = __sysctl+0xbc syscall(e6e3ed38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x48326c1f, esp = 0xbfbfe70c, ebp = 0xbfbfe738 --- Tracing command login pid 4201 tid 100088 td 0xc48db460 sched_switch(c48db460,0,1,24b,8a494678,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d8840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d8840,0,c085c813,19e,c48d8828,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c48db460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c48d8828,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c48d8828,c48d88b8,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c48db460,116b,e6dadc2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c48db460,e6dadcfc,10,c48db460,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6dadd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x4810558b, esp = 0xbfbfed9c, ebp = 0xbfbfedb8 --- Tracing command getty pid 4200 tid 100050 td 0xc47afd20 sched_switch(c47afd20,0,1,24b,fba606b8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4722018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4722018,0,c085c813,19e,c4594010,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47afd20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4594010,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4594010,0,159,c085f2e6,0,...) at 0xc0661707 = _sleep+0x307 ttysleep(c4594000,c4594010,159,c085f2e6,0,...) at 0xc0699499 = ttysleep+0x39 ttread(c4594000,e6cfbc60,0,e6cfbb90,c05a9e4d,...) at 0xc069cce7 = ttread+0x4e7 ttyread(c4570000,e6cfbc60,0,e6cfbbb4,c0626f10,...) at 0xc06997c8 = ttyread+0x38 scread(c4570000,e6cfbc60,0,1a7,0,...) at 0xc05a9e4d = scread+0x2d giant_read(c4570000,e6cfbc60,0,0,1,...) at 0xc0626f10 = giant_read+0x60 devfs_read_f(c47ac4c0,e6cfbc60,c43f5b00,0,c47afd20,...) at 0xc05f08fd = devfs_read_f+0x7d dofileread(e6cfbc60,ffffffff,ffffffff,0,c47ac4c0,...) at 0xc0690106 = dofileread+0x96 kern_readv(c47afd20,0,e6cfbc60,bfbfee4b,1,...) at 0xc0690478 = kern_readv+0x58 read(c47afd20,e6cfbcfc,c,c087a7d6,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6cfbd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x4816a983, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 4199 tid 100084 td 0xc48dbd20 sched_switch(c48dbd20,0,1,24b,fbca346c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9588,0,c085c813,19e,c4595810,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c48dbd20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4595810,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4595810,0,159,c085f2e6,0,...) at 0xc0661707 = _sleep+0x307 ttysleep(c4595800,c4595810,159,c085f2e6,0,...) at 0xc0699499 = ttysleep+0x39 ttread(c4595800,e6da1c60,0,e6da1b90,c05a9e4d,...) at 0xc069cce7 = ttread+0x4e7 ttyread(c456fb00,e6da1c60,0,e6da1bb4,c0626f10,...) at 0xc06997c8 = ttyread+0x38 scread(c456fb00,e6da1c60,0,1a7,0,...) at 0xc05a9e4d = scread+0x2d giant_read(c456fb00,e6da1c60,0,0,1,...) at 0xc0626f10 = giant_read+0x60 devfs_read_f(c4b546d4,e6da1c60,c43f5b00,0,c48dbd20,...) at 0xc05f08fd = devfs_read_f+0x7d dofileread(e6da1c60,ffffffff,ffffffff,0,c4b546d4,...) at 0xc0690106 = dofileread+0x96 kern_readv(c48dbd20,0,e6da1c60,bfbfee4b,1,...) at 0xc0690478 = kern_readv+0x58 read(c48dbd20,e6da1cfc,c,c087a7d6,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6da1d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x4816a983, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 4198 tid 100150 td 0xc5ce5000 sched_switch(c5ce5000,0,1,24b,fbb5700c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5cd3af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5cd3af8,0,c085c813,19e,c45a3c10,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5ce5000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c45a3c10,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c45a3c10,0,159,c085f2e6,0,...) at 0xc0661707 = _sleep+0x307 ttysleep(c45a3c00,c45a3c10,159,c085f2e6,0,...) at 0xc0699499 = ttysleep+0x39 ttread(c45a3c00,e6ebec60,0,e6ebeb90,c05a9e4d,...) at 0xc069cce7 = ttread+0x4e7 ttyread(c456fc00,e6ebec60,0,e6ebebb4,c0626f10,...) at 0xc06997c8 = ttyread+0x38 scread(c456fc00,e6ebec60,0,1a7,0,...) at 0xc05a9e4d = scread+0x2d giant_read(c456fc00,e6ebec60,0,0,1,...) at 0xc0626f10 = giant_read+0x60 devfs_read_f(c49a79cc,e6ebec60,c43f5b00,0,c5ce5000,...) at 0xc05f08fd = devfs_read_f+0x7d dofileread(e6ebec60,ffffffff,ffffffff,0,c49a79cc,...) at 0xc0690106 = dofileread+0x96 kern_readv(c5ce5000,0,e6ebec60,bfbfee4b,1,...) at 0xc0690478 = kern_readv+0x58 read(c5ce5000,e6ebecfc,c,c087a7d6,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6ebed38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x4816a983, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 4197 tid 100115 td 0xc47fd8c0 sched_switch(c47fd8c0,0,1,24b,fb9ab2e0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a6eaf8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a6eaf8,0,c085c813,19e,c45a3810,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47fd8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c45a3810,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c45a3810,0,159,c085f2e6,0,...) at 0xc0661707 = _sleep+0x307 ttysleep(c45a3800,c45a3810,159,c085f2e6,0,...) at 0xc0699499 = ttysleep+0x39 ttread(c45a3800,e6e41c60,0,e6e41b90,c05a9e4d,...) at 0xc069cce7 = ttread+0x4e7 ttyread(c456fd00,e6e41c60,0,e6e41bb4,c0626f10,...) at 0xc06997c8 = ttyread+0x38 scread(c456fd00,e6e41c60,0,1a7,0,...) at 0xc05a9e4d = scread+0x2d giant_read(c456fd00,e6e41c60,0,0,1,...) at 0xc0626f10 = giant_read+0x60 devfs_read_f(c4b545f0,e6e41c60,c43f5b00,0,c47fd8c0,...) at 0xc05f08fd = devfs_read_f+0x7d dofileread(e6e41c60,ffffffff,ffffffff,0,c4b545f0,...) at 0xc0690106 = dofileread+0x96 kern_readv(c47fd8c0,0,e6e41c60,bfbfee4b,1,...) at 0xc0690478 = kern_readv+0x58 read(c47fd8c0,e6e41cfc,c,c087a7d6,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6e41d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x4816a983, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command cron pid 3851 tid 100130 td 0xc4823af0 sched_switch(c4823af0,0,1,24b,fad6651c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf9588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf9588,0,c085c813,19e,c0928dc4,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6e82bc8,c064d06d,c09310c0,8,c4823af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c0928dc4,0,c085a1a5,d8,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _sleep(c0928dc4,0,15c,c085a708,ea61,...) at 0xc06616e1 = _sleep+0x2e1 kern_nanosleep(c4823af0,e6e82c64,e6e82c6c,3c,0,...) at 0xc0667de1 = kern_nanosleep+0xc1 nanosleep(c4823af0,e6e82cfc,8,c085dc7c,c08e2e80,...) at 0xc066978f = nanosleep+0x6f syscall(e6e82d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x4815636f, esp = 0xbfbfed0c, ebp = 0xbfbfed38 --- Tracing command sendmail pid 3845 tid 100153 td 0xc5ce38c0 sched_switch(c5ce38c0,0,1,24b,fd068e0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5cd32d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5cd32d0,0,c085c813,19e,c5cd3318,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5ce38c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5cd3318,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5cd3318,c5cd3348,168,c083407d,0,...) at 0xc0661707 = _sleep+0x307 kern_sigsuspend(c5ce38c0,0,0,0,0,...) at 0xc065b8f4 = kern_sigsuspend+0xe4 sigsuspend(c5ce38c0,e6ec7cfc,4,c085da12,c08e37f8,...) at 0xc065b97d = sigsuspend+0x4d syscall(e6ec7d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x4830f5cb, esp = 0xbfbfdb1c, ebp = 0xbfbfdb48 --- Tracing command sshd pid 3836 tid 100164 td 0xc4bc7000 sched_switch(c4bc7000,0,1,24b,7f7d5490,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4bc1018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4bc1018,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6f30a84,c064d2b9,c097a764,0,c4bc7000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c097a77c,c097a764,c086acda,101,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _cv_wait_sig(c097a77c,c097a764,c085d774,30f,e6f30afc,...) at 0xc06262a8 = _cv_wait_sig+0x198 kern_select(c4bc7000,6,48509098,0,0,0,c,e6f30c98) at 0xc068f6dc = kern_select+0x6cc select(c4bc7000,e6f30cfc,14,c4bc7000,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6f30d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x48391903, esp = 0xbfbfe6ac, ebp = 0xbfbfee98 --- Tracing command httpd pid 3821 tid 100096 td 0xc47fc460 sched_switch(c47fc460,0,1,24b,87edc028,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c47fc460,c47fc460,c47fc460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47fc460,0,c085c813,243,c0928754,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928754,0,c0859453,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928754,c47fc460,0,c0856f85,2cb,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928754,0,c0856f85,2cb,c80,...) at 0xc0660b6b = _sx_xlock+0x6b kern_wait(c47fc460,ffffffff,e6dd8c2c,3,0,...) at 0xc0638838 = kern_wait+0x128 wait4(c47fc460,e6dd8cfc,10,c086acda,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6dd8d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x4848158b, esp = 0xbfbfec2c, ebp = 0xbfbfec48 --- Tracing command sendmail pid 3813 tid 100074 td 0xc47b7690 sched_switch(c47b7690,0,1,24b,edd789bc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c47b7690,c47b7690,c47b7690,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47b7690,0,c085c813,243,c0928c34,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928c34,0,c085a2de,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928c34,c47b7690,0,c085a213,574,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928c34,0,c085a213,574,c47b7690,...) at 0xc0660b6b = _sx_xlock+0x6b userland_sysctl(c47b7690,e6d5ec14,2,bfbfcf00,bfbfcf18,...) at 0xc0662a81 = userland_sysctl+0xf1 __sysctl(c47b7690,e6d5ecfc,18,c085dc7c,c08e2af0,...) at 0xc0662eec = __sysctl+0xbc syscall(e6d5ed38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x483a5c1f, esp = 0xbfbfceac, ebp = 0xbfbfced8 --- Tracing command unlinkd pid 2463 tid 100154 td 0xc5ce3690 sched_switch(c5ce3690,0,1,24b,df670e74,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5cd3018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5cd3018,0,c085c813,19e,c47b1948,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5ce3690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c47b1948,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c47b1948,c47b1ab8,14c,c085d7ef,0,...) at 0xc0661707 = _sleep+0x307 pipe_read(c47ab8e8,e6ecac60,c4760e00,0,c5ce3690,...) at 0xc06915b9 = pipe_read+0x389 dofileread(e6ecac60,ffffffff,ffffffff,0,c47ab8e8,...) at 0xc0690106 = dofileread+0x96 kern_readv(c5ce3690,0,e6ecac60,4827f7e3,1,...) at 0xc0690478 = kern_readv+0x58 read(c5ce3690,e6ecacfc,c,c5ce3690,c08e1848,...) at 0xc069055f = read+0x4f syscall(e6ecad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x4826e983, esp = 0xbfbfe87c, ebp = 0xbfbfe898 --- Tracing command proxsmtpd pid 2225 tid 100120 td 0xc4a74af0 sched_switch(c4a74af0,0,1,24b,24717858,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f32d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f32d0,0,c085c813,19e,c5cdfd3a,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c4a74af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5cdfd3a,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5cdfd3a,c097aa78,158,c085dba8,0,...) at 0xc0661707 = _sleep+0x307 kern_accept(c4a74af0,3,0,0,0,...) at 0xc06afe4b = kern_accept+0x12b accept(c4a74af0,e6e5ecfc,c,c085e815,c08e1ad0,...) at 0xc06b0241 = accept+0x41 syscall(e6e5ed38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x4811166f, esp = 0xbfbfedac, ebp = 0xbfbfedc8 --- Tracing command snmpd pid 2219 tid 100157 td 0xc5ce3000 sched_switch(c5ce3000,0,1,24b,ebf7950a,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5cd2588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5cd2588,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6ed3a84,c064d2b9,c097a764,0,c5ce3000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c097a77c,c097a764,c086acda,101,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _cv_wait_sig(c097a77c,c097a764,c085d774,30f,c0926b14,...) at 0xc06262a8 = _cv_wait_sig+0x198 kern_select(c5ce3000,7,bfbfedbc,bfbfed3c,bfbfecbc,0,4942c32f,48428b98) at 0xc068f6dc = kern_select+0x6cc select(c5ce3000,e6ed3cfc,14,c085dc7c,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6ed3d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x48417903, esp = 0xbfbfddfc, ebp = 0xbfbfee68 --- Tracing command squid pid 2213 tid 100058 td 0xc47af460 sched_switch(c47af460,0,1,24b,977af504,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f2588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f2588,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6d1fad8,c064d2b9,c097a764,c47af460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,3e8,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,3e8,3bb,46,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 poll(c47af460,e6d1fcfc,c,c085dc7c,c08e2b98,...) at 0xc068eec4 = poll+0x554 syscall(e6d1fd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x485ebbbf, esp = 0xbfbf0b7c, ebp = 0xbfbfec38 --- Tracing command squid pid 2211 tid 100152 td 0xc5ce3af0 sched_switch(c5ce3af0,0,1,24b,5a100a8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5cd3588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5cd3588,0,c085c813,19e,c5cd3570,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5ce3af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5cd3570,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5cd3570,c5cd3600,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c5ce3af0,ffffffff,e6ec4c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c5ce3af0,e6ec4cfc,10,c085e362,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6ec4d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x485c058b, esp = 0xbfbfec6c, ebp = 0xbfbfec88 --- Tracing command smartd pid 2189 tid 100143 td 0xc5bfa690 sched_switch(c5bfa690,0,1,24b,c4c958a4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4bc3af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4bc3af8,0,c085c813,19e,c0928dc4,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6ea9bc8,c064d06d,c09310c0,8,c5bfa690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c0928dc4,0,c085a1a5,d8,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _sleep(c0928dc4,0,15c,c085a708,1b7359,...) at 0xc06616e1 = _sleep+0x2e1 kern_nanosleep(c5bfa690,e6ea9c64,e6ea9c6c,707,0,...) at 0xc0667de1 = kern_nanosleep+0xc1 nanosleep(c5bfa690,e6ea9cfc,8,c085dc7c,c08e2e80,...) at 0xc066978f = nanosleep+0x6f syscall(e6ea9d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x4829036f, esp = 0xbfbfec2c, ebp = 0xbfbfec58 --- Tracing command imspector pid 2175 tid 100119 td 0xc4a74d20 sched_switch(c4a74d20,0,1,24b,581b1dac,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a6e018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a6e018,0,c085c813,19e,c5c5a37a,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c4a74d20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5c5a37a,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5c5a37a,c097aa78,158,c085dba8,0,...) at 0xc0661707 = _sleep+0x307 kern_accept(c4a74d20,6,e6e4dc70,e6e4dc6c,e6e4dc68,...) at 0xc06afe4b = kern_accept+0x12b accept(c4a74d20,e6e4dcfc,c,c4a74d20,c08e1ad0,...) at 0xc06b028f = accept+0x8f syscall(e6e4dd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x4826c66f, esp = 0xbfbfeabc, ebp = 0xbfbfeb18 --- Tracing command imspector pid 2174 tid 100140 td 0xc5bfad20 sched_switch(c5bfad20,0,1,24b,61a5fa24,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf7588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf7588,0,c085c813,19e,c48cc85a,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5bfad20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c48cc85a,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c48cc85a,c097aa78,158,c085dba8,0,...) at 0xc0661707 = _sleep+0x307 kern_accept(c5bfad20,4,e6ea0c70,e6ea0c6c,e6ea0c68,...) at 0xc06afe4b = kern_accept+0x12b accept(c5bfad20,e6ea0cfc,c,c5bfad20,c08e1ad0,...) at 0xc06b028f = accept+0x8f syscall(e6ea0d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x4826c66f, esp = 0xbfbee99c, ebp = 0xbfbee9e8 --- Tracing command imspector pid 2168 tid 100087 td 0xc48db690 sched_switch(c48db690,0,1,24b,34204670,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d8af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d8af8,0,c085c813,19e,c5c5a6ba,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c48db690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5c5a6ba,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5c5a6ba,c097aa78,158,c085dba8,0,...) at 0xc0661707 = _sleep+0x307 kern_accept(c48db690,4,e6daac70,e6daac6c,e6daac68,...) at 0xc06afe4b = kern_accept+0x12b accept(c48db690,e6daacfc,c,c085dbef,c08e1ad0,...) at 0xc06b028f = accept+0x8f syscall(e6daad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x4826c66f, esp = 0xbfbee9cc, ebp = 0xbfbeea18 --- Tracing command imspector pid 2167 tid 100089 td 0xc48db230 sched_switch(c48db230,0,1,24b,aca4d95c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d8588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d8588,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6db0a84,c064d2b9,c097a764,c48db230,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,493e1,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,493e1,30f,e6db0afc,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 kern_select(c48db230,400,bfbfeb9c,0,0,e6db0c70,12c,0) at 0xc068f6c4 = kern_select+0x6b4 select(c48db230,e6db0cfc,14,c086acda,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6db0d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x482cf903, esp = 0xbfbfeb1c, ebp = 0xbfbfee38 --- Tracing command postgres pid 2166 tid 100086 td 0xc48db8c0 sched_switch(c48db8c0,0,1,24b,9a89b5d4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9018,0,c085c813,19e,c4be30bc,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c48db8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4be30bc,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4be30bc,c4be3074,158,c08601ab,0,...) at 0xc0661707 = _sleep+0x307 sbwait(c4be3050,0,c0860221,592,c4be3074,...) at 0xc06a8a42 = sbwait+0x52 soreceive_generic(c4be3000,e6da7be8,e6da7bf4,0,0,...) at 0xc06ad7ac = soreceive_generic+0x3bc soreceive(c4be3000,e6da7be8,e6da7bf4,0,0,...) at 0xc06a98c8 = soreceive+0x38 kern_recvit(c48db8c0,7,e6da7c60,0,0,...) at 0xc06af145 = kern_recvit+0x155 recvit(0,4,0,8382ea0,2000,0,0,e6da7c58,1,0,4879bb98,0) at 0xc06af351 = recvit+0x31 recvfrom(c48db8c0,e6da7cfc,18,c085e374,c08e1ab8,...) at 0xc06af4c6 = recvfrom+0x76 syscall(e6da7d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (29, FreeBSD ELF32, recvfrom), eip = 0x487439a7, esp = 0xbfbfd97c, ebp = 0xbfbfd9a8 --- Tracing command php pid 2162 tid 100134 td 0xc5bfbaf0 sched_switch(c5bfbaf0,0,1,24b,b9b6edd4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf8840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf8840,0,c085c813,19e,c0928dc4,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6e8ebc8,c064d06d,c09310c0,8,c5bfbaf0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c0928dc4,0,c085a1a5,d8,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _sleep(c0928dc4,0,15c,c085a708,3e9,...) at 0xc06616e1 = _sleep+0x2e1 kern_nanosleep(c5bfbaf0,e6e8ec64,e6e8ec6c,1,0,...) at 0xc0667de1 = kern_nanosleep+0xc1 nanosleep(c5bfbaf0,e6e8ecfc,8,c08691e4,c08e2e80,...) at 0xc066978f = nanosleep+0x6f syscall(e6e8ed38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x4858e36f, esp = 0xbfbfcecc, ebp = 0xbfbfcef8 --- Tracing command sys-monitor pid 2161 tid 100135 td 0xc5bfb8c0 sched_switch(c5bfb8c0,0,1,24b,d6e11ac2,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf8588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf8588,0,c085c813,19e,c5bf8570,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c5bfb8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5bf8570,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5bf8570,c5bf8600,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c5bfb8c0,872,e6e91c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c5bfb8c0,e6e91cfc,10,c5bfb8c0,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e91d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed3c, ebp = 0xbfbfed58 --- Tracing command ph_citrix pid 2154 tid 100069 td 0xc4598460 sched_switch(c4598460,0,1,24b,8298e718,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4598460,c4598460,c4598460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4598460,0,c085c813,243,c0928754,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928754,0,c0859453,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928754,c4598460,0,c0856f85,2cb,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928754,0,c0856f85,2cb,c80,...) at 0xc0660b6b = _sx_xlock+0x6b kern_wait(c4598460,ffffffff,e6d4bc2c,1,0,...) at 0xc0638838 = kern_wait+0x128 wait4(c4598460,e6d4bcfc,10,c086acda,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6d4bd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x482c658b, esp = 0xbfbfd47c, ebp = 0xbfbfd498 --- Tracing command sys-monitor pid 2153 tid 100056 td 0xc47af8c0 sched_switch(c47af8c0,0,1,24b,ce3d041c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f2af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f2af8,0,c085c813,19e,c47f2ae0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47af8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c47f2ae0,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c47f2ae0,c47f2b70,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c47af8c0,86a,e6d17c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c47af8c0,e6d17cfc,10,c47af8c0,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6d17d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed4c, ebp = 0xbfbfed68 --- Tracing command icapd pid 2138 tid 100138 td 0xc5bfb230 sched_switch(c5bfb230,0,1,24b,854f92b0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c5bfb230,c5bfb230,c5bfb230,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bfb230,0,c085c813,243,c0928754,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928754,0,c0859453,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928754,c5bfb230,0,c0856f85,2cb,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928754,0,c0856f85,2cb,4,...) at 0xc0660b6b = _sx_xlock+0x6b kern_wait(c5bfb230,0,e6e9ac2c,1,0,...) at 0xc0638838 = kern_wait+0x128 wait4(c5bfb230,e6e9acfc,10,c085e083,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e9ad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x806d1ef, esp = 0xbfbfe16c, ebp = 0xbfbfe188 --- Tracing command sys-monitor pid 2137 tid 100133 td 0xc5bfbd20 sched_switch(c5bfbd20,0,1,24b,a5dda348,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf8af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf8af8,0,c085c813,19e,c5bf8ae0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c5bfbd20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5bf8ae0,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5bf8ae0,c5bf8b70,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c5bfbd20,85a,e6e8bc2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c5bfbd20,e6e8bcfc,10,c5bfbd20,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e8bd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed5c, ebp = 0xbfbfed78 --- Tracing command perl5.8.8 pid 2127 tid 100144 td 0xc5bfa460 sched_switch(c5bfa460,0,1,24b,ad899330,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4bc3840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4bc3840,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6eaca84,c064d2b9,c097a764,c5bfa460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,3e9,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,3e9,30f,3e0,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 kern_select(c5bfa460,8,81c2668,0,0,e6eacc70,1,0) at 0xc068f6c4 = kern_select+0x6b4 select(c5bfa460,e6eaccfc,14,16,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6eacd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x482a2903, esp = 0xbfbfed0c, ebp = 0xbfbfed88 --- Tracing command sys-monitor pid 2126 tid 100106 td 0xc4822d20 sched_switch(c4822d20,0,1,24b,99d965b0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a70840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a70840,0,c085c813,19e,c4a70828,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c4822d20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4a70828,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4a70828,c4a708b8,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c4822d20,84f,e6e26c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c4822d20,e6e26cfc,10,c4822d20,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e26d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed3c, ebp = 0xbfbfed58 --- Tracing command imfilterd pid 2063 tid 100137 td 0xc5bfb460 sched_switch(c5bfb460,0,1,24b,835ec9f4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf8018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf8018,0,c085c813,19e,c48d203a,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5bfb460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c48d203a,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c48d203a,c097aa78,158,c085dba8,0,...) at 0xc0661707 = _sleep+0x307 kern_accept(c5bfb460,3,e6e97c70,e6e97c6c,e6e97c68,...) at 0xc06afe4b = kern_accept+0x12b accept(c5bfb460,e6e97cfc,c,c085dbef,c08e1ad0,...) at 0xc06b028f = accept+0x8f syscall(e6e97d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x80f01a7, esp = 0xbfbfedac, ebp = 0xbfbfedc8 --- Tracing command sys-monitor pid 2062 tid 100141 td 0xc5bfaaf0 sched_switch(c5bfaaf0,0,1,24b,639cd318,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf72d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf72d0,0,c085c813,19e,c5bf72b8,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c5bfaaf0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5bf72b8,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5bf72b8,c5bf7348,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c5bfaaf0,80f,e6ea3c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c5bfaaf0,e6ea3cfc,10,c5bfaaf0,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6ea3d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed5c, ebp = 0xbfbfed78 --- Tracing command perform_ca pid 2055 tid 100139 td 0xc5bfb000 sched_switch(c5bfb000,0,1,24b,850654bc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c5bfb000,c5bfb000,c5bfb000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bfb000,0,c085c813,243,c0928754,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928754,0,c0859453,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928754,c5bfb000,0,c0856f85,2cb,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928754,0,c0856f85,2cb,c61dd400,...) at 0xc0660b6b = _sx_xlock+0x6b kern_wait(c5bfb000,ffffffff,e6e9dc2c,1,0,...) at 0xc0638838 = kern_wait+0x128 wait4(c5bfb000,e6e9dcfc,10,c085e083,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e9dd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x81d662f, esp = 0xbfbfdc5c, ebp = 0xbfbfdc78 --- Tracing command sys-monitor pid 2054 tid 100142 td 0xc5bfa8c0 sched_switch(c5bfa8c0,0,1,24b,5b133bc8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c5bf7018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c5bf7018,0,c085c813,19e,c5bf7000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c5bfa8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c5bf7000,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c5bf7000,c5bf7090,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c5bfa8c0,807,e6ea6c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c5bfa8c0,e6ea6cfc,10,c5bfa8c0,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6ea6d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed3c, ebp = 0xbfbfed58 --- Tracing command sys-ifmonitor pid 2052 tid 100066 td 0xc47fd000 sched_switch(c47fd000,0,1,24b,f46e91dc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c47fd000,c47fd000,c47fd000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47fd000,0,c085c813,243,c0928c34,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928c34,0,c085a2de,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c0928c34,c47fd000,0,c085a213,574,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c0928c34,0,c085a213,574,c47fd000,...) at 0xc0660b6b = _sx_xlock+0x6b userland_sysctl(c47fd000,e6d3fc14,6,0,bfbfe578,...) at 0xc0662a81 = userland_sysctl+0xf1 __sysctl(c47fd000,e6d3fcfc,18,c085d9c6,c08e2af0,...) at 0xc0662eec = __sysctl+0xbc syscall(e6d3fd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x4815cc1f, esp = 0xbfbfe49c, ebp = 0xbfbfe4c8 --- Tracing command postgres pid 1789 tid 100051 td 0xc47bad20 sched_switch(c47bad20,0,1,24b,9a801f78,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47b6588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47b6588,0,c085c813,19e,c4be38dc,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47bad20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4be38dc,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4be38dc,c4be3894,158,c08601ab,0,...) at 0xc0661707 = _sleep+0x307 sbwait(c4be3870,0,c0860221,592,c4be3894,...) at 0xc06a8a42 = sbwait+0x52 soreceive_generic(c4be3820,e6d00be8,e6d00bf4,0,0,...) at 0xc06ad7ac = soreceive_generic+0x3bc soreceive(c4be3820,e6d00be8,e6d00bf4,0,0,...) at 0xc06a98c8 = soreceive+0x38 kern_recvit(c47bad20,7,e6d00c60,0,0,...) at 0xc06af145 = kern_recvit+0x155 recvit(0,4,0,8382ea0,2000,0,0,e6d00c58,1,0,4879bb98,0) at 0xc06af351 = recvit+0x31 recvfrom(c47bad20,e6d00cfc,18,c085e374,c08e1ab8,...) at 0xc06af4c6 = recvfrom+0x76 syscall(e6d00d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (29, FreeBSD ELF32, recvfrom), eip = 0x487439a7, esp = 0xbfbfd97c, ebp = 0xbfbfd9a8 --- Tracing command user_ldap_mgr pid 1746 tid 100121 td 0xc4bc88c0 sched_switch(c4bc88c0,0,1,24b,b90dd744,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4bc32d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4bc32d0,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6e61a84,c064d2b9,c097a764,c4bc88c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,7e,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,7e,30f,c06752c0,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 kern_select(c4bc88c0,5,bfbfd100,bfbfd0e0,bfbfd0c0,e6e61c70,0,1e848) at 0xc068f6c4 = kern_select+0x6b4 select(c4bc88c0,e6e61cfc,14,c085dc7c,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6e61d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x481d4903, esp = 0xbfbfd084, ebp = 0xbfbfd128 --- Tracing command sys-monitor pid 1745 tid 100102 td 0xc47f9690 sched_switch(c47f9690,0,1,24b,cd039b8e,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f3840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f3840,0,c085c813,19e,c47f3828,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47f9690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c47f3828,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c47f3828,c47f38b8,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c47f9690,6d2,e6e02c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c47f9690,e6e02cfc,10,c47f9690,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e02d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed4c, ebp = 0xbfbfed68 --- Tracing command user_manager pid 1738 tid 100104 td 0xc4823230 sched_switch(c4823230,0,1,24b,bb01f09c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a71018,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a71018,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6e20ad8,c064d2b9,c097a764,c4823230,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,7e,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,7e,3bb,e6e20b44,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 poll(c4823230,e6e20cfc,c,c085dc7c,c08e2b98,...) at 0xc068eec4 = poll+0x554 syscall(e6e20d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x48123bbf, esp = 0xbfbfe1fc, ebp = 0xbfbfee78 --- Tracing command sys-monitor pid 1737 tid 100075 td 0xc47b7460 sched_switch(c47b7460,0,1,24b,c6064244,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47262d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47262d0,0,c085c813,19e,c47262b8,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47b7460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c47262b8,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c47262b8,c4726348,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c47b7460,6ca,e6d62c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c47b7460,e6d62cfc,10,c47b7460,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6d62d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed4c, ebp = 0xbfbfed68 --- Tracing command ph pid 1730 tid 100185 td 0xc4bc4460 sched_switch(c4bc4460,0,1,24b,3f6a4254,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9af8,0,c085c813,19e,c49a5440,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(0,c085c813,155,c4bc4460,c4bc4460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c49a5440,c0929e38,c085a966,100,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c49a5440,c0929e38,100,c085a966,0,...) at 0xc0661707 = _sleep+0x307 __umtx_op_cv_wait(c4bc4460,e6f6fcfc,e6f6fd2c,c07fc233,c4bc4460,...) at 0xc066e439 = __umtx_op_cv_wait+0x2d9 _umtx_op(c4bc4460,e6f6fcfc,14,c085e80c,c08e4290,...) at 0xc066aac7 = _umtx_op+0x27 syscall(e6f6fd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x48152037, esp = 0xbf6fbebc, ebp = 0xbf6fbed8 --- Tracing command ph pid 1730 tid 100184 td 0xc4bc4690 sched_switch(c4bc4690,0,1,24b,37bba2a0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9af8,0,c085c813,19e,c49a5400,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(0,c085c813,155,c4bc4690,c4bc4690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c49a5400,c0929b28,c085a966,100,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c49a5400,c0929b28,100,c085a966,0,...) at 0xc0661707 = _sleep+0x307 __umtx_op_cv_wait(c4bc4690,e6f6ccfc,e6f6cd2c,c07fc233,c4bc4690,...) at 0xc066e439 = __umtx_op_cv_wait+0x2d9 _umtx_op(c4bc4690,e6f6ccfc,14,c085e80c,c08e4290,...) at 0xc066aac7 = _umtx_op+0x27 syscall(e6f6cd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x48152037, esp = 0xbf7fcebc, ebp = 0xbf7fced8 --- Tracing command ph pid 1730 tid 100183 td 0xc4bc48c0 sched_switch(c4bc48c0,0,1,24b,39c4dc98,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9af8,0,c085c813,19e,c49a53c0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(0,c085c813,155,c4bc48c0,c4bc48c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c49a53c0,c0929850,c085a966,100,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c49a53c0,c0929850,100,c085a966,0,...) at 0xc0661707 = _sleep+0x307 __umtx_op_cv_wait(c4bc48c0,e6f69cfc,e6f69d2c,c07fc233,c4bc48c0,...) at 0xc066e439 = __umtx_op_cv_wait+0x2d9 _umtx_op(c4bc48c0,e6f69cfc,14,c085e80c,c08e4290,...) at 0xc066aac7 = _umtx_op+0x27 syscall(e6f69d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x48152037, esp = 0xbf8fdebc, ebp = 0xbf8fded8 --- Tracing command ph pid 1730 tid 100182 td 0xc4bc4af0 sched_switch(c4bc4af0,0,1,24b,3f68f566,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9af8,0,c085c813,19e,c49a5380,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(0,c085c813,155,c4bc4af0,c4bc4af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c49a5380,c0929ea8,c085a966,100,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c49a5380,c0929ea8,100,c085a966,0,...) at 0xc0661707 = _sleep+0x307 __umtx_op_cv_wait(c4bc4af0,e6f66cfc,e6f66d2c,c07fc233,c4bc4af0,...) at 0xc066e439 = __umtx_op_cv_wait+0x2d9 _umtx_op(c4bc4af0,e6f66cfc,14,c085e80c,c08e4290,...) at 0xc066aac7 = _umtx_op+0x27 syscall(e6f66d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x48152037, esp = 0xbf9feebc, ebp = 0xbf9feed8 --- Tracing command ph pid 1730 tid 100082 td 0xc48dc230 sched_switch(c48dc230,0,1,24b,3f619f36,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48d9af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48d9af8,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6d9aad8,c064d2b9,c097a764,c48dc230,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,2,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,2,3bb,92,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 poll(c48dc230,e6d9acfc,c,c085e083,c08e2b98,...) at 0xc068eec4 = poll+0x554 syscall(e6d9ad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x48233bbf, esp = 0xbfbfdf6c, ebp = 0xbfbfdf88 --- Tracing command sys-monitor pid 1729 tid 100111 td 0xc4822230 sched_switch(c4822230,0,1,24b,c0c2b368,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a6f840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a6f840,0,c085c813,19e,c4a6f828,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c4822230,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c4a6f828,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c4a6f828,c4a6f8b8,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c4822230,6c2,e6e35c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c4822230,e6e35cfc,10,c4822230,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e35d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed4c, ebp = 0xbfbfed68 --- Tracing command ph_alert_mgr pid 1441 tid 100105 td 0xc4823000 sched_switch(c4823000,0,1,24b,f3dcfad4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a70af8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a70af8,0,c085c813,19e,c48cd51a,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c4823000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c48cd51a,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c48cd51a,c097aa78,158,c085dba8,0,...) at 0xc0661707 = _sleep+0x307 kern_accept(c4823000,3,e6e23c70,e6e23c6c,e6e23c68,...) at 0xc06afe4b = kern_accept+0x12b accept(c4823000,e6e23cfc,c,c085d947,c08e1ad0,...) at 0xc06b028f = accept+0x8f syscall(e6e23d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x4856766f, esp = 0xbfbfe1fc, ebp = 0xbfbfee88 --- Tracing command sys-monitor pid 1440 tid 100103 td 0xc47f9460 sched_switch(c47f9460,0,1,24b,6364a10c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f3588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f3588,0,c085c813,19e,c47f3570,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c47f9460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c47f3570,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c47f3570,c47f3600,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c47f9460,5a1,e6e05c2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c47f9460,e6e05cfc,10,c47f9460,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6e05d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfed4c, ebp = 0xbfbfed68 --- Tracing command postgres pid 1429 tid 100116 td 0xc47fd690 sched_switch(c47fd690,0,1,24b,7b5d0310,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a6e840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a6e840,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6e44ad8,c064d2b9,c097a764,c47fd690,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,7d1,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,7d1,3bb,c0861f5f,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 poll(c47fd690,e6e44cfc,c,c085d9cb,c08e2b98,...) at 0xc068eec4 = poll+0x554 syscall(e6e44d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x48735bbf, esp = 0xbfbfd3bc, ebp = 0xbfbfd3d8 --- Tracing command postgres pid 1428 tid 100061 td 0xc47ba460 sched_switch(c47ba460,0,1,24b,91426808,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47b5840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47b5840,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6d2ba84,c064d2b9,c097a764,c47ba460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,3e9,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,3e9,30f,41c,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 kern_select(c47ba460,0,0,0,0,e6d2bc70,1,0) at 0xc068f6c4 = kern_select+0x6b4 select(c47ba460,e6d2bcfc,14,c085d9cb,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6d2bd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x4878a903, esp = 0xbfbfd76c, ebp = 0xbfbfd798 --- Tracing command postgres pid 1427 tid 100118 td 0xc47fd230 sched_switch(c47fd230,0,1,24b,b7f580a4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a6e2d0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a6e2d0,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6e4aa84,c064d2b9,c097a764,c47fd230,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,c9,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,c9,30f,0,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 kern_select(c47fd230,0,0,0,0,e6e4ac70,0,30d40) at 0xc068f6c4 = kern_select+0x6b4 select(c47fd230,e6e4acfc,14,c085d9cb,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6e4ad38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x4878a903, esp = 0xbfbfd6ec, ebp = 0xbfbfd718 --- Tracing command postgres pid 1426 tid 100065 td 0xc4598af0 sched_switch(c4598af0,0,1,24b,bd0178dc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47f1840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47f1840,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c097a77c,e6d3ba84,c064d2b9,c097a764,c4598af0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_timedwait_sig(c097a77c,c9,c086acda,101,0,...) at 0xc0687f57 = sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c097a77c,c097a764,c9,30f,e6d3bb48,...) at 0xc0625df8 = _cv_timedwait_sig+0x1a8 kern_select(c4598af0,0,0,0,0,e6d3bc70,0,30d40) at 0xc068f6c4 = kern_select+0x6b4 select(c4598af0,e6d3bcfc,14,16,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6d3bd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x4878a903, esp = 0xbfbfd6cc, ebp = 0xbfbfd6f8 --- Tracing command postgres pid 1424 tid 100109 td 0xc4822690 sched_switch(c4822690,0,1,24b,7f6ac758,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4822690,c4822690,c4822690,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4822690,0,c085c813,243,c092873c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c092873c,0,c085944b,3,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sx_xlock_hard(c092873c,c4822690,0,c085706f,135,...) at 0xc066092e = _sx_xlock_hard+0x22e _sx_xlock(c092873c,0,c085706f,135,0,...) at 0xc0660b6b = _sx_xlock+0x6b fork1(c4822690,14,0,e6e2fc78,c4822690,...) at 0xc063aa6b = fork1+0x2eb fork(c4822690,e6e2fcfc,c08775cf,c085da12,c08e1830,...) at 0xc063bc79 = fork+0x29 syscall(e6e2fd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (2, FreeBSD ELF32, fork), eip = 0x4870a5ab, esp = 0xbfbfd82c, ebp = 0xbfbfd858 --- Tracing command sys-monitor pid 1422 tid 100078 td 0xc47b7000 sched_switch(c47b7000,0,1,24b,e0b3308c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48da840,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48da840,0,c085c813,19e,c48da828,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47b7000,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c48da828,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c48da828,c48da8b8,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c47b7000,590,e6d8ec2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c47b7000,e6d8ecfc,10,c47b7000,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e6d8ed38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x480dc58b, esp = 0xbfbfecdc, ebp = 0xbfbfecf8 --- Tracing command syslogd pid 1345 tid 100110 td 0xc4822460 sched_switch(c4822460,0,1,24b,d28deacc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c4a6faf8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4a6faf8,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6e32a84,c064d2b9,c097a764,0,c4822460,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c097a77c,c097a764,c086acda,101,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _cv_wait_sig(c097a77c,c097a764,c085d774,30f,41c,...) at 0xc06262a8 = _cv_wait_sig+0x198 kern_select(c4822460,8,4821e090,0,0,0,0,4817eb98) at 0xc068f6dc = kern_select+0x6cc select(c4822460,e6e32cfc,14,c085da12,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6e32d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x4816d903, esp = 0xbfbfdf3c, ebp = 0xbfbfee88 --- Tracing command accounting pid 1316 tid 100055 td 0xc47afaf0 sched_switch(c47afaf0,0,1,24b,101c8510,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c47afaf0,c47afaf0,c47afaf0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47afaf0,0,c085c813,266,c47afaf0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0926a60,e6d13aec,e6d13ae8,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0926a60,c0926a48,0,c0853461,3a98,...) at 0xc06616f4 = _sleep+0x2f4 acct_thread(0,e6d13d38,c085706f,31c,c47f3000,...) at 0xc06252ae = acct_thread+0x32e fork_exit(c0624f80,0,e6d13d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe6d13d70, ebp = 0 --- Tracing command devd pid 465 tid 100079 td 0xc48dc8c0 sched_switch(c48dc8c0,0,1,24b,32cb3dfc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c48da588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c48da588,0,c085c813,19e,c097a77c,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(e6d91a84,c064d2b9,c097a764,0,c48dc8c0,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c097a77c,c097a764,c086acda,101,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _cv_wait_sig(c097a77c,c097a764,c085d774,30f,c0934030,...) at 0xc06262a8 = _cv_wait_sig+0x198 kern_select(c48dc8c0,5,bfbfedf8,0,0,0,3cd,8) at 0xc068f6dc = kern_select+0x6cc select(c48dc8c0,e6d91cfc,14,c082c73d,c08e20b8,...) at 0xc068f8ae = select+0x5e syscall(e6d91d38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x8086b3f, esp = 0xbfbfe9bc, ebp = 0xbfbfee98 --- Tracing command adjkerntz pid 120 tid 100062 td 0xc47ba230 sched_switch(c47ba230,0,1,24b,bce4a928,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c47b5588,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c47b5588,0,c085c813,19e,c47b55d0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c0931040,8,c085a1a5,c47ba230,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c47b55d0,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c47b55d0,c47b5600,168,c083407d,0,...) at 0xc0661707 = _sleep+0x307 kern_sigsuspend(c47ba230,0,0,0,0,...) at 0xc065b8f4 = kern_sigsuspend+0xe4 sigsuspend(c47ba230,e6d2fcfc,4,c092ac40,c08e37f8,...) at 0xc065b97d = sigsuspend+0x4d syscall(e6d2fd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x480c35cb, esp = 0xbfbfedcc, ebp = 0xbfbfee98 --- Tracing command softdepflush pid 49 tid 100048 td 0xc45598c0 sched_switch(c45598c0,0,1,24b,8ff927e8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c45598c0,c45598c0,c45598c0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c45598c0,0,c085c813,266,c45598c0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c09836c8,3e8,c086c62b,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c09836c8,c0983688,44,c086c62b,3e8,...) at 0xc06616f4 = _sleep+0x2f4 softdep_flush(0,e4b08d38,c085706f,31c,c4722570,...) at 0xc0785b02 = softdep_flush+0x2c2 fork_exit(c0785840,0,e4b08d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4b08d70, ebp = 0 --- Tracing command vnlru pid 48 tid 100047 td 0xc4559af0 sched_switch(c4559af0,0,1,24b,9cba887c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4559af0,c4559af0,c4559af0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4559af0,0,c085c813,266,c4559af0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c4722828,3e8,c0862300,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c4722828,c097aeb8,250,c0862300,3e8,...) at 0xc06616f4 = _sleep+0x2f4 vnlru_proc(0,e4b05d38,c085706f,31c,c4722828,...) at 0xc06cddd5 = vnlru_proc+0x115 fork_exit(c06cdcc0,0,e4b05d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4b05d70, ebp = 0 --- Tracing command syncer pid 47 tid 100046 td 0xc4559d20 sched_switch(c4559d20,0,1,24b,b7dc9f2c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4559d20,c4559d20,c4559d20,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4559d20,0,c085c813,243,c4559d20,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0928bec,0,c08622ba,0,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c0928bec,c097aee4,68,c08622ba,0,...) at 0xc0661716 = _sleep+0x316 sched_sync(0,e4b02d38,c085706f,31c,c4722ae0,...) at 0xc06cd4b4 = sched_sync+0x6f4 fork_exit(c06ccdc0,0,e4b02d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4b02d70, ebp = 0 --- Tracing command bufdaemon pid 46 tid 100045 td 0xc4597000 sched_switch(c4597000,0,1,24b,bcb768dc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4597000,c4597000,c4597000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4597000,0,c085c813,266,c4597000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c097ac04,3e8,c0860b21,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c097ac04,c097ac08,44,c0860b21,3e8,...) at 0xc06616f4 = _sleep+0x2f4 buf_daemon(0,e4affd38,c085706f,31c,c4725000,...) at 0xc06bab4e = buf_daemon+0x21e fork_exit(c06ba930,0,e4affd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4affd70, ebp = 0 --- Tracing command idlepoll pid 45 tid 100044 td 0xc4597230 sched_switch(c4597230,0,1,24b,a233ae50,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4597230,c4597230,c4597230,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4597230,0,c085c813,266,c4597230,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0928318,bb8,c085905c,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0928318,0,0,c085905c,bb8,...) at 0xc06616f4 = _sleep+0x2f4 poll_idle(0,e4afcd38,c085706f,31c,c47252b8,...) at 0xc064ec3a = poll_idle+0x13a fork_exit(c064eb00,0,e4afcd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4afcd70, ebp = 0 --- Tracing command pagezero pid 44 tid 100043 td 0xc4597460 sched_switch(c4597460,0,1,24b,2ab64234,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4597460,c4597460,c4597460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4597460,0,c085c813,266,c4597460,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c09842a0,493e0,c086fa26,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c09842a0,c0983e44,0,c086fa26,493e0,...) at 0xc06616f4 = _sleep+0x2f4 vm_pagezero(0,e4af9d38,c085706f,31c,c4725570,...) at 0xc07ba093 = vm_pagezero+0xb3 fork_exit(c07b9fe0,0,e4af9d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4af9d70, ebp = 0 --- Tracing command vmdaemon pid 43 tid 100042 td 0xc4597690 sched_switch(c4597690,0,1,24b,d4dd7378,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4597690,c4597690,c4597690,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4597690,0,c085c813,243,c4597690,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0983eb8,0,c085a1a5,d8,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c0983eb8,c0983ebc,68,c0860b21,0,...) at 0xc0661716 = _sleep+0x316 vm_daemon(0,e4af6d38,c085706f,31c,c4725828,...) at 0xc07b5f19 = vm_daemon+0x59 fork_exit(c07b5ec0,0,e4af6d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4af6d70, ebp = 0 --- Tracing command pagedaemon pid 42 tid 100041 td 0xc45978c0 sched_switch(c45978c0,0,1,24b,5b3a8c6c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c45978c0,c45978c0,c45978c0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c45978c0,0,c085c813,266,c45978c0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0983e80,1388,c0860b21,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0983e80,c0983e44,44,c0860b21,1388,...) at 0xc06616f4 = _sleep+0x2f4 vm_pageout(0,e4af3d38,c085706f,31c,c4725ae0,...) at 0xc07b6beb = vm_pageout+0x2bb fork_exit(c07b6930,0,e4af3d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4af3d70, ebp = 0 --- Tracing command dummynet pid 41 tid 100040 td 0xc4597af0 sched_switch(c4597af0,0,1,24b,be92c06e,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4597af0,c4597af0,c4597af0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4597af0,0,c085c813,243,c46c6480,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c46c6480,0,c085a1dd,c0853461,0,...) at 0xc06880b6 = sleepq_wait+0x36 msleep_spin(c46c6480,c46c649c,c0853461,0,c085706f,...) at 0xc0661a0d = msleep_spin+0x1bd taskqueue_thread_loop(c097bf70,e4af0d38,c085706f,31c,c4726000,...) at 0xc068929a = taskqueue_thread_loop+0x8a fork_exit(c0689210,c097bf70,e4af0d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4af0d70, ebp = 0 --- Tracing command irq7: ppbus0 ppc0 pid 40 tid 100039 td 0xc4597d20 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command fdc0 pid 39 tid 100038 td 0xc4598000 sched_switch(c4598000,0,1,24b,b93dcf8c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4598000,c4598000,c4598000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4598000,0,c085c813,266,c4598000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c456a83c,3e8,c0853461,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c456a83c,c456a8f0,4c,c0853461,3e8,...) at 0xc06616f4 = _sleep+0x2f4 fdc_thread(c456a800,e4aead38,c085706f,31c,c455a2b8,...) at 0xc07ca578 = fdc_thread+0x2b8 fork_exit(c07ca2c0,c456a800,e4aead38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4aead70, ebp = 0 --- Tracing command swi0: sio pid 38 tid 100037 td 0xc4500d20 sched_switch(c4500d20,0,1,8,1df7c480,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c4591be8,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c458b0c0,e4addd38,c085706f,31c,c455a570,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c458b0c0,e4addd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4addd70, ebp = 0 --- Tracing command irq1: atkbd0 pid 37 tid 100036 td 0xc4558000 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command irq15: ata1 pid 36 tid 100035 td 0xc4558230 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command irq14: ata0 pid 35 tid 100034 td 0xc4558460 sched_switch(c4558460,0,1,8,9a53a2d0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c4427ce8,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c456e950,e4ac1d38,c085706f,31c,c455b000,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c456e950,e4ac1d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4ac1d70, ebp = 0 --- Tracing command usb2 pid 34 tid 100033 td 0xc4558690 sched_switch(c4558690,0,1,24b,5c98da84,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4558690,c4558690,c4558690,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4558690,0,c085c813,266,c4558690,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c455d210,ea60,c0851f70,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c455d210,0,5c,c0851f70,ea60,...) at 0xc06616f4 = _sleep+0x2f4 usb_event_thread(c4532880,e4aabd38,c085706f,31c,c455b2b8,...) at 0xc05e34ea = usb_event_thread+0xca fork_exit(c05e3420,c4532880,e4aabd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4aabd70, ebp = 0 --- Tracing command irq18: uhci2 pid 33 tid 100032 td 0xc45588c0 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command usb1 pid 32 tid 100031 td 0xc4558af0 sched_switch(c4558af0,0,1,24b,5cfa515c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4558af0,c4558af0,c4558af0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4558af0,0,c085c813,266,c4558af0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c453e210,ea60,c0851f70,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c453e210,0,5c,c0851f70,ea60,...) at 0xc06616f4 = _sleep+0x2f4 usb_event_thread(c45511c0,e4aa4d38,c085706f,31c,c455b828,...) at 0xc05e34ea = usb_event_thread+0xca fork_exit(c05e3420,c45511c0,e4aa4d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4aa4d70, ebp = 0 --- Tracing command irq19: uhci1 pid 31 tid 100030 td 0xc4558d20 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command usbtask-dr pid 30 tid 100029 td 0xc4559000 sched_switch(c4559000,0,1,24b,c926f954,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4559000,c4559000,c4559000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4559000,0,c085c813,243,c4559000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0925bd4,0,c085a1a5,d8,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c0925bd4,0,5c,c0851f7e,0,...) at 0xc0661716 = _sleep+0x316 usb_task_thread(c0925bd4,e4a9dd38,c085706f,31c,c44fd2b8,...) at 0xc05e3b1e = usb_task_thread+0x5e fork_exit(c05e3ac0,c0925bd4,e4a9dd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a9dd70, ebp = 0 --- Tracing command usbtask-hc pid 29 tid 100028 td 0xc4559230 sched_switch(c4559230,0,1,24b,c926d768,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4559230,c4559230,c4559230,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4559230,0,c085c813,243,c4559230,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0925bc0,0,c085a1a5,d8,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c0925bc0,0,5c,c0851f7e,0,...) at 0xc0661716 = _sleep+0x316 usb_task_thread(c0925bc0,e4a9ad38,c085706f,31c,c44fd570,...) at 0xc05e3b1e = usb_task_thread+0x5e fork_exit(c05e3ac0,c0925bc0,e4a9ad38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a9ad70, ebp = 0 --- Tracing command usb0 pid 28 tid 100027 td 0xc4559460 sched_switch(c4559460,0,1,24b,5cb14f58,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4559460,c4559460,c4559460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4559460,0,c085c813,266,c4559460,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c4544210,ea60,c0851f70,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c4544210,0,5c,c0851f70,ea60,...) at 0xc06616f4 = _sleep+0x2f4 usb_event_thread(c4551e80,e4a97d38,c085706f,31c,c44fd828,...) at 0xc05e34ea = usb_event_thread+0xca fork_exit(c05e3420,c4551e80,e4a97d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a97d70, ebp = 0 --- Tracing command irq16: uhci0 pid 27 tid 100026 td 0xc4472690 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command em1 taskq pid 26 tid 100025 td 0xc44728c0 sched_switch(c44728c0,0,1,24b,bfb11454,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c44728c0,c44728c0,c44728c0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c44728c0,0,c085c813,243,c4523a00,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c4523a00,0,c085a1dd,c0853461,0,...) at 0xc06880b6 = sleepq_wait+0x36 msleep_spin(c4523a00,c4523a1c,c0853461,0,c085706f,...) at 0xc0661a0d = msleep_spin+0x1bd taskqueue_thread_loop(c453d35c,e4a90d38,c085706f,31c,c450f000,...) at 0xc068929a = taskqueue_thread_loop+0x8a fork_exit(c0689210,c453d35c,e4a90d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a90d70, ebp = 0 --- Tracing command em0 taskq pid 25 tid 100024 td 0xc4472af0 sched_switch(c4472af0,0,1,24b,c06d3b40,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4472af0,c4472af0,c4472af0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4472af0,0,c085c813,243,c4523c00,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c4523c00,0,c085a1dd,c0853461,0,...) at 0xc06880b6 = sleepq_wait+0x36 msleep_spin(c4523c00,c4523c1c,c0853461,0,c085706f,...) at 0xc0661a0d = msleep_spin+0x1bd taskqueue_thread_loop(c453735c,e4a6dd38,c085706f,31c,c450f2b8,...) at 0xc068929a = taskqueue_thread_loop+0x8a fork_exit(c0689210,c453735c,e4a6dd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a6dd70, ebp = 0 --- Tracing command irq9: acpi0 pid 24 tid 100023 td 0xc4472d20 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command kqueue taskq pid 23 tid 100022 td 0xc4500000 sched_switch(c4500000,0,1,24b,c9467de4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4500000,c4500000,c4500000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4500000,0,c085c813,243,c4500000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c446ed80,c446ed9c,c0853461,0,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c446ed80,c446ed9c,0,c0853461,0,...) at 0xc0661716 = _sleep+0x316 taskqueue_thread_loop(c0926b9c,e4a42d38,c085706f,31c,c450f828,...) at 0xc06892c4 = taskqueue_thread_loop+0xb4 fork_exit(c0689210,c0926b9c,e4a42d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a42d70, ebp = 0 --- Tracing command swi2: cambio pid 22 tid 100021 td 0xc4500230 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command xpt_thrd pid 9 tid 100020 td 0xc4500460 sched_switch(c4500460,0,1,24b,c92689dc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4500460,c4500460,c4500460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4500460,0,c085c813,243,c4500460,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c090c954,0,c085a1a5,d8,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c090c954,c090c96c,4c,c0829a04,0,...) at 0xc0661716 = _sleep+0x316 xpt_scanner_thread(0,e4a3cd38,c085706f,31c,c446f828,...) at 0xc0454301 = xpt_scanner_thread+0x41 fork_exit(c04542c0,0,e4a3cd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a3cd70, ebp = 0 --- Tracing command swi6: task queue pid 21 tid 100019 td 0xc4500690 sched_switch(c4500690,0,1,8,c4c7a8dc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c44b30e8,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c44ac590,e4a39d38,c085706f,31c,c446fae0,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c44ac590,e4a39d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a39d70, ebp = 0 --- Tracing command swi6: Giant taskq pid 20 tid 100018 td 0xc45008c0 sched_switch(c45008c0,0,1,8,6eb5da0c,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c44b31e8,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c44ac5a0,e4a36d38,c085706f,31c,c44fc000,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c44ac5a0,e4a36d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a36d70, ebp = 0 --- Tracing command acpi_task_2 pid 8 tid 100017 td 0xc4500af0 sched_switch(c4500af0,0,1,24b,c9465d00,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4500af0,c4500af0,c4500af0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4500af0,0,c085c813,243,c4500af0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c44b3280,c44b329c,c0853461,0,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c44b3280,c44b329c,0,c0853461,0,...) at 0xc0661716 = _sleep+0x316 taskqueue_thread_loop(c0b2bebc,e4a33d38,c085706f,31c,c44fc2b8,...) at 0xc06892c4 = taskqueue_thread_loop+0xb4 fork_exit(c0689210,c0b2bebc,e4a33d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a33d70, ebp = 0 --- Tracing command acpi_task_1 pid 7 tid 100016 td 0xc442f230 sched_switch(c442f230,0,1,24b,c9463cc8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c442f230,c442f230,c442f230,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442f230,0,c085c813,243,c442f230,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c44b3280,c44b329c,c0853461,0,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c44b3280,c44b329c,0,c0853461,0,...) at 0xc0661716 = _sleep+0x316 taskqueue_thread_loop(c0b2bebc,e4a30d38,c085706f,31c,c44fc570,...) at 0xc06892c4 = taskqueue_thread_loop+0xb4 fork_exit(c0689210,c0b2bebc,e4a30d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a30d70, ebp = 0 --- Tracing command acpi_task_0 pid 6 tid 100015 td 0xc442f460 sched_switch(c442f460,0,1,24b,c9461c40,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c442f460,c442f460,c442f460,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442f460,0,c085c813,243,c442f460,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c44b3280,c44b329c,c0853461,0,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c44b3280,c44b329c,0,c0853461,0,...) at 0xc0661716 = _sleep+0x316 taskqueue_thread_loop(c0b2bebc,e4a2dd38,c085706f,31c,c44fc828,...) at 0xc06892c4 = taskqueue_thread_loop+0xb4 fork_exit(c0689210,c0b2bebc,e4a2dd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a2dd70, ebp = 0 --- Tracing command thread taskq pid 5 tid 100014 td 0xc442f690 sched_switch(c442f690,0,1,24b,cf283398,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c442f690,c442f690,c442f690,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442f690,0,c085c813,243,c442f690,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c44b3300,c44b331c,c0853461,0,0,...) at 0xc06880b6 = sleepq_wait+0x36 _sleep(c44b3300,c44b331c,0,c0853461,0,...) at 0xc0661716 = _sleep+0x316 taskqueue_thread_loop(c093319c,e4a2ad38,c085706f,31c,c44fcae0,...) at 0xc06892c4 = taskqueue_thread_loop+0xb4 fork_exit(c0689210,c093319c,e4a2ad38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a2ad70, ebp = 0 --- Tracing command swi5: + pid 19 tid 100013 td 0xc442f8c0 sched_switch(c442f8c0,0,1,8,52e8b408,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c44b33e8,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c44ac5d0,e4a27d38,c085706f,31c,c44fd000,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c44ac5d0,e4a27d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a27d70, ebp = 0 --- Tracing command yarrow pid 18 tid 100012 td 0xc442faf0 sched_switch(c442faf0,0,1,24b,bcdbd8c8,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c442faf0,c442faf0,c442faf0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442faf0,0,c085c813,266,c442faf0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0928bf4,64,c0853461,2,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0928bf4,0,0,c0853461,64,...) at 0xc06616f4 = _sleep+0x2f4 pause(c0853461,64,c084c2fe,113,0,...) at 0xc0661810 = pause+0x30 random_kthread(0,e4a24d38,c085706f,31c,c442c2b8,...) at 0xc059a0f4 = random_kthread+0x1b4 fork_exit(c0599f40,0,e4a24d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a24d70, ebp = 0 --- Tracing command g_down pid 4 tid 100011 td 0xc442fd20 sched_switch(c442fd20,0,1,24b,bc9ef280,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c442fd20,c442fd20,c442fd20,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442fd20,0,c085c813,266,c442fd20,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c092634c,64,c0853461,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c092634c,c0926268,24c,c0853461,64,...) at 0xc06616f4 = _sleep+0x2f4 g_io_schedule_down(c442fd20,0,c08542bf,74,0,...) at 0xc060ac0b = g_io_schedule_down+0x6b g_down_procbody(0,e4a21d38,c085706f,31c,c442c570,...) at 0xc060b2e9 = g_down_procbody+0x69 fork_exit(c060b280,0,e4a21d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a21d70, ebp = 0 --- Tracing command g_up pid 3 tid 100010 td 0xc4472000 sched_switch(c4472000,0,1,24b,bcab16d4,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4472000,c4472000,c4472000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4472000,0,c085c813,266,c4472000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0926348,64,c0853461,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0926348,c09262a8,24c,c0853461,64,...) at 0xc06616f4 = _sleep+0x2f4 g_io_schedule_up(c4472000,0,c08542bf,5d,0,...) at 0xc060af12 = g_io_schedule_up+0xd2 g_up_procbody(0,e4a1ed38,c085706f,31c,c442c828,...) at 0xc060b359 = g_up_procbody+0x69 fork_exit(c060b2f0,0,e4a1ed38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a1ed70, ebp = 0 --- Tracing command g_event pid 2 tid 100009 td 0xc4472230 sched_switch(c4472230,0,1,24b,bb0d1288,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c4472230,c4472230,c4472230,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c4472230,0,c085c813,266,c4472230,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0926340,64,c0853461,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0926340,0,4c,c0853461,64,...) at 0xc06616f4 = _sleep+0x2f4 g_event_procbody(0,e4a1bd38,c085706f,31c,c442cae0,...) at 0xc060b407 = g_event_procbody+0xa7 fork_exit(c060b360,0,e4a1bd38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a1bd70, ebp = 0 --- Tracing command swi3: vm pid 17 tid 100008 td 0xc4472460 fork_trampoline() at 0xc07e4238 = fork_trampoline Tracing command swi4: clock sio pid 16 tid 100007 td 0xc442d000 sched_switch(c442d000,0,1,8,be9261ec,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c446d8e8,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c442a480,e4a15d38,c085706f,31c,c446f2b8,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c442a480,e4a15d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a15d70, ebp = 0 --- Tracing command swi1: net pid 15 tid 100006 td 0xc442d230 sched_switch(c442d230,0,1,8,417ebe60,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c085723f,4a1,c446d968,...) at 0xc06611d3 = mi_switch+0x143 ithread_loop(c442a490,e4a12d38,c085706f,31c,c446f570,...) at 0xc063cfe0 = ithread_loop+0x2a0 fork_exit(c063cd40,c442a490,e4a12d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4a12d70, ebp = 0 --- Tracing command idle: cpu0 pid 14 tid 100005 td 0xc442d460 kdb_enter_why(c082fb36,c084fd03,8,872014,c458e08c,...) at 0xc0680b4a = kdb_enter_why+0x3a siointr1(c09861cc,0,c0872014,56f,e300ac48,...) at 0xc07d5836 = siointr1+0xe6 siointr(c458e000,c092ac40,c442d460,c442d460,c4428280,...) at 0xc07d6512 = siointr+0x32 intr_execute_handlers(c4410cd0,e300ac68,e300acac,c07e4594,39,...) at 0xc07e922c = intr_execute_handlers+0xec lapic_handle_intr(39,e300ac68) at 0xc07ec81f = lapic_handle_intr+0x3f Xapic_isr1() at 0xc07e4594 = Xapic_isr1+0x34 --- interrupt, eip = 0xc07efe7b, esp = 0xe300aca8, ebp = 0xe300acac --- spinlock_exit(c0931040,8,c085adbe,32d) at 0xc07efe7b = spinlock_exit+0x2b _mtx_unlock_spin_flags(c0931040,0,c085adbe,32d,c092ac40,...) at 0xc064d06d = _mtx_unlock_spin_flags+0x4d sched_idletd(0,e300ad38,c085706f,31c,c442b000,...) at 0xc067662f = sched_idletd+0x21f fork_exit(c0676410,0,e300ad38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe300ad70, ebp = 0 --- Tracing command idle: cpu1 pid 13 tid 100004 td 0xc442d690 sched_switch(c442d690,0,6,8,2e3c1ab2,...) at 0xc0676004 = sched_switch+0x324 mi_switch(6,0,c0876302,47e,1,...) at 0xc06611d3 = mi_switch+0x143 ipi_bitmap_handler(8,c4420028,28,0,f,...) at 0xc07f1b72 = ipi_bitmap_handler+0x72 Xipi_intr_bitmap_handler() at 0xc07e48be = Xipi_intr_bitmap_handler+0x2e --- interrupt, eip = 0xc07f1562, esp = 0xe3007cbc, ebp = 0xe3007cc0 --- mp_grab_cpu_hlt(e3007cf8,c0676639,c0931040,0,c085adbe,...) at 0xc07f1562 = mp_grab_cpu_hlt+0x22 cpu_idle(c0931040,0,c085adbe,32d,c092b280,...) at 0xc07eeed8 = cpu_idle+0x8 sched_idletd(0,e3007d38,c085706f,31c,c442b2b8,...) at 0xc0676639 = sched_idletd+0x229 fork_exit(c0676410,0,e3007d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3007d70, ebp = 0 --- Tracing command idle: cpu2 pid 12 tid 100003 td 0xc442d8c0 cpustop_handler(4,e3004c68,c07fc4d0,c442b570,c09310c0,...) at 0xc07f2c52 = cpustop_handler+0x32 ipi_nmi_handler(c442b570,c09310c0,0,c442d8c0,13,...) at 0xc07f2cdf = ipi_nmi_handler+0x2f trap(e3004c74) at 0xc07fc4d0 = trap+0x30 calltrap() at 0xc07e41cb = calltrap+0x6 --- trap 0x13, eip = 0xc064d068, esp = 0xe3004cb4, ebp = 0xe3004cc8 --- _mtx_unlock_spin_flags(c09310c0,0,c085adbe,32d,c092b8c0,...) at 0xc064d068 = _mtx_unlock_spin_flags+0x48 sched_idletd(0,e3004d38,c085706f,31c,c442b570,...) at 0xc067662f = sched_idletd+0x21f fork_exit(c0676410,0,e3004d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3004d70, ebp = 0 --- Tracing command idle: cpu3 pid 11 tid 100002 td 0xc442daf0 cpustop_handler(8,e3001c70,c07fc4d0,c0675428,87,...) at 0xc07f2c52 = cpustop_handler+0x32 ipi_nmi_handler(c0675428,87,c0676004,c442daf0,13,...) at 0xc07f2cdf = ipi_nmi_handler+0x2f trap(e3001c7c) at 0xc07fc4d0 = trap+0x30 calltrap() at 0xc07e41cb = calltrap+0x6 --- trap 0x13, eip = 0xc07f1562, esp = 0xe3001cbc, ebp = 0xe3001cc0 --- mp_grab_cpu_hlt(e3001cf8,c0676639,c09310c0,0,c085adbe,...) at 0xc07f1562 = mp_grab_cpu_hlt+0x22 cpu_idle(c09310c0,0,c085adbe,32d,c092bf00,...) at 0xc07eeed8 = cpu_idle+0x8 sched_idletd(0,e3001d38,c085706f,31c,c442b828,...) at 0xc0676639 = sched_idletd+0x229 fork_exit(c0676410,0,e3001d38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3001d70, ebp = 0 --- Tracing command init pid 1 tid 100001 td 0xc442dd20 sched_switch(c442dd20,0,1,24b,b0c7b6fc,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,8,c085c813,c442baf8,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442baf8,0,c085c813,19e,c442bae0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_catch_signals(c064d06d,c09310c0,8,c085a1a5,c442dd20,...) at 0xc0687ba5 = sleepq_catch_signals+0x1e5 sleepq_wait_sig(c442bae0,0,c085a1a5,d8,0,...) at 0xc0688034 = sleepq_wait_sig+0x14 _sleep(c442bae0,c442bb70,15c,c085db38,0,...) at 0xc0661707 = _sleep+0x307 kern_wait(c442dd20,ffffffff,e2ffdc2c,0,0,...) at 0xc0639156 = kern_wait+0xa46 wait4(c442dd20,e2ffdcfc,10,c085d947,c08e18a8,...) at 0xc06391cb = wait4+0x3b syscall(e2ffdd38) at 0xc07fc233 = syscall+0x2b3 Xint0x80_syscall() at 0xc07e4230 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x805455f, esp = 0xbfbfe97c, ebp = 0xbfbfe998 --- Tracing command audit pid 10 tid 100000 td 0xc442f000 sched_switch(c442f000,0,1,24b,c9210ba0,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c442f000,c442f000,c442f000,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c442f000,0,c085c813,243,c442f000,...) at 0xc06879af = sleepq_switch+0xcf sleepq_wait(c0983114,c09830f4,c086a770,1,0,...) at 0xc06880b6 = sleepq_wait+0x36 _cv_wait(c0983114,c09830f4,c086adb0,18b,0,...) at 0xc0626708 = _cv_wait+0x198 audit_worker(0,e2ffad38,c085706f,31c,c442c000,...) at 0xc076d1a0 = audit_worker+0x50 fork_exit(c076d150,0,e2ffad38) at 0xc063a684 = fork_exit+0x94 fork_trampoline() at 0xc07e4240 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe2ffad70, ebp = 0 --- Tracing command swapper pid 0 tid 0 td 0xc09266c0 sched_switch(c09266c0,0,1,24b,1584d894,...) at 0xc0676004 = sched_switch+0x324 mi_switch(1,0,c09266c0,c09266c0,c09266c0,...) at 0xc06611d3 = mi_switch+0x143 sleepq_switch(c09266c0,0,c085c813,266,c09266c0,...) at 0xc06879af = sleepq_switch+0xcf sleepq_timedwait(c0926400,2710,c085b288,0,0,...) at 0xc0687feb = sleepq_timedwait+0x3b _sleep(c0926400,0,44,c085b288,2710,...) at 0xc06616f4 = _sleep+0x2f4 scheduler(0,c1ec00,c1ec00,c1e000,c25000,...) at 0xc07a905e = scheduler+0x25e mi_startup() at 0xc0623816 = mi_startup+0x96 begin() at 0xc0446225 = begin+0x2c db> show allchains db> ~ [EOT] capella:~ (502) exit exit Script done on Mon Dec 15 10:58:13 2008 show lockedvnods Locked vnodes 0xc63ffbdc: tag devfs, type VCHR usecount 1, writecount 0, refcount 1 mountedhere 0xc4470100 flags () lock type devfs: EXCL (count 1) by thread 0xc754f690 (pid 92421) dev random 0xc5ed5e04: tag ufs, type VREG usecount 11, writecount 0, refcount 13 mountedhere 0 flags () v_object 0xc5de84d8 ref 10 pages 130 lock type ufs: SHARED (count 1) ino 3297435, on dev ad0s1e From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 18:10:26 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C2231065679 for ; Mon, 15 Dec 2008 18:10:26 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep3.cogeco.net (smtp2.cogeco.ca [216.221.81.29]) by mx1.freebsd.org (Postfix) with ESMTP id 07DDA8FC08 for ; Mon, 15 Dec 2008 18:10:25 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [0.0.0.0] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep3.cogeco.net (Postfix) with ESMTP id C66E617DE; Mon, 15 Dec 2008 12:48:29 -0500 (EST) Message-ID: <494698E4.2070406@cogeco.ca> Date: Mon, 15 Dec 2008 12:50:28 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> In-Reply-To: <4934CB77.30906@transactionware.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Jan Mikkelsen Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 18:10:26 -0000 > Replying to my own post ... > > I have done a test on the same machine comparing 6.3-p1 to 7.1-PRE. > The performance is the expected ~6MB/s (because of the lack of cache) > on 6.3-p1, so the BIOS change doesn't seem to be at fault. > > This seems to be a regression somewhere between 6.3 to 7.1. The Areca > driver is the same in 6.3 and 7.1, so the problem seems to be elsewhere. > > I think this is more than just a "performance" problem. The > observations with gstat showing extremely high ms/w values (I have > seen them as high as 22000) makes it look like IO completion > interrupts are being lost. > > Any suggestions on where to look next? Are there obvious candidates? Hi, I too am having a terrible time with this. I have a server with 7.0 (FreeBSD 7.0-STABLE#0: Sun Jun 29 00:32:38 EDT 2008) that I do not believe the system has the same problem so the 7.1 STABLE and I am looking into downgrading to this version as a possible short term solution. To give you an idea of how slow it gets the two systems are nearly identical in hardware and a "make buildworld" on the 7.1 (FreeBSD 7.1-PRERELEASE #0: Sun Dec 14 09:25:21 EST 2008) took over 10 hours!!! On the machine without the problem and which has a much higher load it took about 2 hours. Both systems have 16gb of RAM with 2 quad core cpus (8 cpus@ 2.33GHz) on a S5000PAL (SR2500ALBRPNA) motherboard. There is a slight difference in the raid card but both use the same raid driver (arcmsr). The system with the problems has a ARC-1130 with a 1 GIG cache chip in a Raid1+0 with 4 drives. So one system took over 8 hours longer to build the world and it was visually slow on the console when building. No errors are tracked at all in the raid card or in the S5000PAL motherboard for the hardware. After weeks of working on this I now believe that anything that taxes the writing to the hard drives causes the system CPU numbers to spike through the roof (approx 80% usage) and the server grinds to a halt. And I also see wide swings in the System CPU usage. It reminds me of the QUOTA issue I had with 6.2 where the system usage was really high and it was the QUOTA code that was broken. I have Polling, Quota, and the Lagg system enabled on both of the systems and have tried to make them as similar as possible in the setup. If I can do any diagnosing on this assuming someone is interested in looking into this issue then let me know what to look for and test. Otherwise, I will have to shortly move back to 7.0 or 6.4 to try to solve the issue. I am very worried about updating to 7.1 in the future and on the other machine as I really have not seen a lot of other people with these problems being discussed. It effectively makes the system mostly unusable as a webserver when it happens as apache ends up needing to be stopped and restarted to get it going again. To this point I have shut off anything I can shut off to try to limit how often it happens. Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 19:00:23 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3205106567A for ; Mon, 15 Dec 2008 19:00:23 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 6BFBB8FC13 for ; Mon, 15 Dec 2008 19:00:23 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.3/8.14.3) with ESMTP id mBFJ0KoX052365; Mon, 15 Dec 2008 14:00:21 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id mBFJ0Jom084267 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 15 Dec 2008 14:00:19 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200812151900.mBFJ0Jom084267@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 15 Dec 2008 14:00:26 -0500 To: Paul MacKenzie , freebsd-stable@freebsd.org From: Mike Tancsa In-Reply-To: <494698E4.2070406@cogeco.ca> References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Jan Mikkelsen Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 19:00:23 -0000 At 12:50 PM 12/15/2008, Paul MacKenzie wrote: > > Any suggestions on where to look next? Are there obvious candidates? > >After weeks of working on this I now believe that anything that taxes >the writing to the hard drives causes the system CPU numbers to spike >through the roof (approx 80% usage) and the server grinds to a halt. And >I also see wide swings in the System CPU usage. It reminds me of the >QUOTA issue I had with 6.2 where the system usage was really high and it >was the QUOTA code that was broken. Hi, I dont have a box in the lab I can test a lot with right now, but I do have a couple of these cards in customer servers and write performance seems ok This is an AMD64 box, 8G RAM 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Tue Sep 23 09:25:02 EDT 2008 running a lot of postgres queries right now [ns8]# dd if=/dev/zero of=/var/tmp/test count=1000 bs=1024k 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 10.265211 secs (102148508 bytes/sec) [ns8]# which seems reasonable for raw write performance. If you bring up the little Areca web app, are there any errors about the array? When you created the raid sets, what params did you use ? The above is on Volume Capacity 320.0GB SCSI Ch/Id/Lun 0/0/0 Raid Level Raid 1+0 Stripe Size 64KBytes Block Size 512Bytes Member Disks 4 Cache Mode Write Back Tagged Queuing Enabled Volume State Normal [ns8]# vmstat -i interrupt total rate irq4: sio0 57065 0 irq17: em1 3989494045 554 irq18: arcmsr0 558098657 77 cpu0: timer 14381393929 2000 irq256: em0 22763077 3 cpu1: timer 14381384902 1999 Total 33333191675 4635 [ns8]# arcmsr0: mem 0xe8600000-0xe8600fff,0xe8000000-0xe83fffff irq 18 at device 14.0 on pci2 ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 arcmsr0: [ITHREAD] ..... Waiting 5 seconds for SCSI devices to settle (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step da0 at arcmsr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: 305175MB (624999424 512 byte sectors: 255H 63S/T 38904C) SMP: AP CPU #1 Launched! From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 20:10:50 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8193B106567D for ; Mon, 15 Dec 2008 20:10:50 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 4A5838FC12 for ; Mon, 15 Dec 2008 20:10:50 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.3/8.14.3) with ESMTP id mBFKAmRc073372; Mon, 15 Dec 2008 15:10:48 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id mBFKAlZf084580 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 15 Dec 2008 15:10:48 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200812152010.mBFKAlZf084580@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 15 Dec 2008 15:10:55 -0500 To: Paul MacKenzie , freebsd-stable@freebsd.org From: Mike Tancsa In-Reply-To: <4946B6EF.5080806@cogeco.ca> References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 20:10:50 -0000 At 02:58 PM 12/15/2008, Paul MacKenzie wrote: >This used to be on a 4.11x system with 1 cpu and only 1gb of ram and >ran flawlessly with much less resources with the same web site code >for a long time. I do not have this problem on the other 7.0 >machine. I originally thought it was just a cpu issue but it is very >closely tied to when something is trying to use the raid arrays and >this seems to be the way to reproduce it. > >I am having a hard time determining why the system load is so high. >Can you recommend the best way to identify the culprit? What does top -S show ? Most of the load is in system. Does the machine in question have a rather large master.passwd file by chance ? (http://www.freebsd.org/cgi/query-pr.cgi?pr=75855) ---Mike From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 20:16:52 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E9BB1065670 for ; Mon, 15 Dec 2008 20:16:52 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id D459E8FC25 for ; Mon, 15 Dec 2008 20:16:51 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.3/8.14.3) with ESMTP id mBFKGosf074959; Mon, 15 Dec 2008 15:16:50 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id mBFKGnN6084606 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 15 Dec 2008 15:16:49 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200812152016.mBFKGnN6084606@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 15 Dec 2008 15:16:57 -0500 To: Paul MacKenzie , freebsd-stable@freebsd.org From: Mike Tancsa In-Reply-To: <494698E4.2070406@cogeco.ca> References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 20:16:52 -0000 At 12:50 PM 12/15/2008, Paul MacKenzie wrote: >I have Polling, Quota, and the Lagg system enabled on both of the >systems and have tried to make them as similar as possible in the setup. I would also try disabling polling. Is you scheduler ULE or BSD? For an 8 core box, it should be ULE ---Mike From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 20:35:49 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 14785106564A for ; Mon, 15 Dec 2008 20:35:49 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id D05CB8FC21 for ; Mon, 15 Dec 2008 20:35:48 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.3/8.14.3) with ESMTP id mBFKZlnQ091571; Mon, 15 Dec 2008 15:35:47 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id mBFKZkj4084689 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 15 Dec 2008 15:35:46 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200812152035.mBFKZkj4084689@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 15 Dec 2008 15:35:54 -0500 To: Paul MacKenzie , freebsd-stable@freebsd.org From: Mike Tancsa In-Reply-To: <4946BDB1.8060208@cogeco.ca> References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> <4946BDB1.8060208@cogeco.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 20:35:49 -0000 At 03:27 PM 12/15/2008, Paul MacKenzie wrote: > > What does top -S show ? Most of the load is in system. Does the > > machine in question have a rather large master.passwd file by chance ? > > (http://www.freebsd.org/cgi/query-pr.cgi?pr=75855) > > ---Mike > > >Thanks for your quick reply: > >master.passwd is only 9467 (with a ls-l) I would try the change to /etc/nsswitch.conf so that group and passwd read group: files passwd: files At that file size, it sounds like you only have about 200 entries ? I doubt its the issue, but its worth a try. I know at around 9,000 files anything to do with UID lookups (e.g. ls -l) takes forever. > PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT >COMMAND > 54 root 2 0 0 7 0 7 100.00% syncer Do you have any other special tuning other than polling ? Any in /boot/loader.conf or /etc/sysctl.conf ? Does gstat show the disks busy ? ---Mike From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 20:48:09 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 261F61065676 for ; Mon, 15 Dec 2008 20:48:09 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep3.cogeco.net (smtp2.cogeco.ca [216.221.81.29]) by mx1.freebsd.org (Postfix) with ESMTP id EE29A8FC24 for ; Mon, 15 Dec 2008 20:48:08 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep3.cogeco.net (Postfix) with ESMTP id 324231433; Mon, 15 Dec 2008 15:48:08 -0500 (EST) Message-ID: <4946C2FF.3080900@cogeco.ca> Date: Mon, 15 Dec 2008 15:50:07 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> <4946BDB1.8060208@cogeco.ca> <200812152035.mBFKZkj4084689@lava.sentex.ca> In-Reply-To: <200812152035.mBFKZkj4084689@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 20:48:09 -0000 > I would try the change to /etc/nsswitch.conf so that group and passwd > read > > group: files > passwd: files > > At that file size, it sounds like you only have about 200 entries ? I > doubt its the issue, but its worth a try. I know at around 9,000 > files anything to do with UID lookups (e.g. ls -l) takes forever. > > >> PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT >> COMMAND >> 54 root 2 0 0 7 0 7 100.00% >> syncer > > Do you have any other special tuning other than polling ? Any in > /boot/loader.conf or /etc/sysctl.conf ? > > Does gstat show the disks busy ? > > ---Mike > Hi Mike, You were close. There are 109 entries. I too found the same problem on another server and I already changed this to files when I was trying to figure it all out. in boot/loader.conf I have: accf_http_load="YES". I tried turning this off but the same problems were seen so I moved on. In /etc/sysctl.conf I have: kern.maxvnodes=400000 kern.ipc.somaxconn=1024 vfs.ufs.dirhash_maxmem=10485760 I added these based on other posts of people's troubles or recommendations in other threads. For the gstat it does not really which was leading me elsewhere about a week: I stopped these at almost the same time and polling is now disabled with a -polling on both interfaces as part of the lagg0: dT: 1.110s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0| acd0 0 2 2 173 4.4 0 0 0.0 0.8| da0 0 2 2 173 15.5 0 0 0.0 2.8| da0s1 0 0 0 0 0.0 0 0 0.0 0.0| da1 0 0 0 0 0.0 0 0 0.0 0.0| da0s1a 0 0 0 0 0.0 0 0 0.0 0.0| da0s1b 0 0 0 0 0.0 0 0 0.0 0.0| da0s1c 0 0 0 0 0.0 0 0 0.0 0.0| da0s1d 0 0 0 0 0.0 0 0 0.0 0.0| da0s1e 0 2 2 173 15.9 0 0 0.0 2.9| da0s1f 0 0 0 0 0.0 0 0 0.0 0.0| da1s1 0 0 0 0 0.0 0 0 0.0 0.0| da1s1c 0 0 0 0 0.0 0 0 0.0 0.0| da1s1d top -SI last pid: 56352; load averages: 14.64, 8.57, 6.38 up 0+10:59:33 15:45:32 287 processes: 22 running, 251 sleeping, 14 waiting CPU: 19.6% user, 0.0% nice, 60.2% system, 0.5% interrupt, 19.7% idle Mem: 705M Active, 3290M Inact, 492M Wired, 6888K Cache, 214M Buf, 11G Free Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56070 www 1 104 0 193M 61040K CPU3 3 1:32 37.89% httpd 1490 mysql 97 4 -5 406M 189M sbwait 7 0:31 37.35% mysqld 14 root 1 171 ki31 0K 16K RUN 4 507:42 30.86% idle: cpu4 17 root 1 171 ki31 0K 16K RUN 1 459:32 30.76% idle: cpu1 11 root 1 171 ki31 0K 16K RUN 7 545:35 30.47% idle: cpu7 16 root 1 171 ki31 0K 16K RUN 2 459:31 28.56% idle: cpu2 13 root 1 171 ki31 0K 16K RUN 5 526:39 24.27% idle: cpu5 15 root 1 171 ki31 0K 16K RUN 3 481:30 23.39% idle: cpu3 56331 samfilter 1 -4 0 14448K 4836K ufs 2 0:05 20.90% perl5.8.8 56201 www 1 -4 0 146M 25472K ufs 3 1:11 14.45% httpd 18 root 1 171 ki31 0K 16K RUN 0 454:31 13.77% idle: cpu0 56321 www 1 -4 0 147M 25728K ufs 2 0:03 11.38% httpd 56343 www 1 -4 0 144M 22772K CPU5 1 0:02 10.79% httpd 56327 www 1 20 0 145M 24408K lockf 7 0:03 10.60% httpd 56339 www 1 54 0 144M 22732K CPU7 1 0:02 10.35% httpd 56266 www 1 -4 0 153M 30560K ufs 2 0:14 10.16% httpd 56186 www 1 -4 0 156M 32720K RUN 4 0:59 9.86% httpd 56095 www 1 -4 0 146M 25700K RUN 1 0:58 9.77% httpd 56189 www 1 -4 0 147M 25620K RUN 5 0:47 9.67% httpd 56260 www 1 -4 0 146M 24748K ufs 2 0:13 9.57% httpd 56185 www 1 -4 0 154M 31068K ufs 2 0:50 9.47% httpd 56345 www 1 -4 0 144M 22732K ufs 3 0:02 9.47% httpd 56205 www 1 -4 0 153M 30840K ufs 2 0:24 9.28% httpd 56277 www 1 -4 0 153M 30228K ufs 2 0:10 9.28% httpd 56297 www 1 4 0 145M 24372K kqread 5 0:04 9.28% httpd 56292 www 1 -4 0 145M 24400K ufs 2 0:06 9.18% httpd 56325 www 1 -4 0 144M 23140K ufs 6 0:03 9.18% httpd 56301 www 1 -4 0 146M 25208K ufs 2 0:05 9.08% httpd 56326 www 1 4 0 144M 23320K sbwait 5 0:03 8.98% httpd 56099 www 1 -4 0 147M 26460K ufs 2 1:43 8.69% httpd 56287 www 1 -4 0 146M 24404K ufs 2 0:11 8.69% httpd 56294 www 1 -4 0 147M 25952K RUN 4 0:09 8.69% httpd 56298 www 1 -4 0 144M 23340K ufs 6 0:07 8.69% httpd 56322 www 1 -4 0 147M 25600K ufs 3 0:03 8.59% httpd 56175 www 1 -4 0 151M 29140K ufs 2 1:26 8.25% httpd 56169 www 1 -4 0 155M 32016K ufs 2 0:57 8.25% httpd 56261 www 1 -4 0 149M 27048K ufs 2 0:13 8.25% httpd 56323 www 1 -4 0 144M 23296K ufs 2 0:03 8.25% httpd 56115 www 1 -4 0 155M 32244K ufs 2 0:55 7.86% httpd 56168 www 1 -4 0 147M 26368K ufs 2 0:42 7.76% httpd 56202 www 1 -4 0 154M 31096K ufs 6 0:24 7.76% httpd 56338 root 1 48 0 8112K 2592K CPU4 4 0:02 7.76% top 56263 www 1 -4 0 150M 28044K ufs 3 0:10 7.67% httpd 56254 www 1 -4 0 145M 24476K ufs 2 0:08 7.67% httpd 56094 www 1 -4 0 155M 31984K ufs 2 0:48 7.57% httpd 56328 www 1 -4 0 144M 23120K CPU0 0 0:02 7.57% httpd Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 21:33:42 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CBFA1065672 for ; Mon, 15 Dec 2008 21:33:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.terabit.net.ua (mail.terabit.net.ua [195.137.202.147]) by mx1.freebsd.org (Postfix) with ESMTP id 0C0478FC12 for ; Mon, 15 Dec 2008 21:33:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from skuns.zoral.com.ua ([91.193.166.194] helo=mail.zoral.com.ua) by mail.terabit.net.ua with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1LCL4O-000LIr-9r; Mon, 15 Dec 2008 23:33:40 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id mBFLXXI3036525 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 15 Dec 2008 23:33:33 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id mBFLXXau021428; Mon, 15 Dec 2008 23:33:33 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id mBFLXWIh021427; Mon, 15 Dec 2008 23:33:32 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 15 Dec 2008 23:33:32 +0200 From: Kostik Belousov To: Guy Helmer Message-ID: <20081215213332.GQ2038@deviant.kiev.zoral.com.ua> References: <49419BE0.2070502@palisadesys.com> <3a142e750812111900u1d9264bh27d39092f5dcc4bf@mail.gmail.com> <49426880.40302@palisadesys.com> <3a142e750812120729v478d850do93e31e5d8502ef68@mail.gmail.com> <4942C178.2030206@palisadesys.com> <49468FE2.4040007@palisadesys.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="C3csQhHPgbq3tf44" Content-Disposition: inline In-Reply-To: <49468FE2.4040007@palisadesys.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.93.3, clamav-milter version 0.93.3 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-Virus-Scanned: mail.terabit.net.ua 1LCL4O-000LIr-9r 6f39d0738b71d73bbe69cb8f2005c90e X-Terabit: YES Cc: freebsd-stable@freebsd.org Subject: Re: 7.1RC1: system hang X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 21:33:42 -0000 --C3csQhHPgbq3tf44 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Dec 15, 2008 at 11:12:02AM -0600, Guy Helmer wrote: > I have a recurring hang on FreeBSD 7.1 roughly-RC1 (releng_7_1 checkout= =20 > as of 2008-12-08) on a dual-CPU hyperthreaded Xeon i386. Here is the=20 > output from ps/m, show allpcpu, show locks, show alllocks, allt, show=20 > allchains, and show lockedvnods commands in the debugger. Can you show me the source line for vm_fault+0x1b1b ? Do the l *(vm_fault+0x1b1b) at the kgdb prompt, you do not need the core, only the same debugging kernel as was booted on deadlocked machine. --C3csQhHPgbq3tf44 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAklGzSwACgkQC3+MBN1Mb4j1YwCdHoT9o/fWeUU9ujwEXUWbyUa4 bSAAoIGBrwRSOAlAs0mfqeOCLbxIgxKk =vVqN -----END PGP SIGNATURE----- --C3csQhHPgbq3tf44-- From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 22:13:08 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36D171065672 for ; Mon, 15 Dec 2008 22:13:08 +0000 (UTC) (envelope-from ghelmer@palisadesys.com) Received: from cetus.palisadesys.com (cetus.palisadesys.com [205.237.115.21]) by mx1.freebsd.org (Postfix) with ESMTP id EB6B98FC08 for ; Mon, 15 Dec 2008 22:13:07 +0000 (UTC) (envelope-from ghelmer@palisadesys.com) Received: from magellan.palisadesys.com (serverwatch [172.16.1.98]) by cetus.palisadesys.com (8.14.3/8.14.3) with ESMTP id mBFMD7bm021796; Mon, 15 Dec 2008 16:13:07 -0600 (CST) (envelope-from ghelmer@palisadesys.com) Received: from [172.16.2.242] (cetus.palisadesys.com [205.237.115.21]) (authenticated bits=0) by magellan.palisadesys.com (8.14.2/8.14.2) with ESMTP id mBFMD5pN095195; Mon, 15 Dec 2008 16:13:05 -0600 (CST) (envelope-from ghelmer@palisadesys.com) Message-ID: <4946D670.40402@palisadesys.com> Date: Mon, 15 Dec 2008 16:13:04 -0600 From: Guy Helmer User-Agent: Thunderbird 2.0.0.18 (Windows/20081105) MIME-Version: 1.0 To: Kostik Belousov References: <49419BE0.2070502@palisadesys.com> <3a142e750812111900u1d9264bh27d39092f5dcc4bf@mail.gmail.com> <49426880.40302@palisadesys.com> <3a142e750812120729v478d850do93e31e5d8502ef68@mail.gmail.com> <4942C178.2030206@palisadesys.com> <49468FE2.4040007@palisadesys.com> <20081215213332.GQ2038@deviant.kiev.zoral.com.ua> In-Reply-To: <20081215213332.GQ2038@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-3.0 (magellan.palisadesys.com [205.237.115.20]); Mon, 15 Dec 2008 16:13:05 -0600 (CST) X-Palisade-MailScanner-Information: Please contact the ISP for more information X-Palisade-MailScanner: Found to be clean X-Palisade-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (not cached, score=-4.399, required 6, autolearn=not spam, ALL_TRUSTED -1.80, BAYES_00 -2.60) X-Palisade-MailScanner-From: ghelmer@palisadesys.com Cc: freebsd-stable@freebsd.org Subject: Re: 7.1RC1: system hang X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 22:13:08 -0000 Kostik Belousov wrote: > On Mon, Dec 15, 2008 at 11:12:02AM -0600, Guy Helmer wrote: > >> I have a recurring hang on FreeBSD 7.1 roughly-RC1 (releng_7_1 checkout >> as of 2008-12-08) on a dual-CPU hyperthreaded Xeon i386. Here is the >> output from ps/m, show allpcpu, show locks, show alllocks, allt, show >> allchains, and show lockedvnods commands in the debugger. >> > Can you show me the source line for vm_fault+0x1b1b ? > Do the l *(vm_fault+0x1b1b) at the kgdb prompt, you do not need the core, > only the same debugging kernel as was booted on deadlocked machine. > (kgdb) l *(vm_fault+0x1b1b) 0xc07a765b is in vm_fault (../../../vm/vm_fault.c:888). 883 if (((fault_flags & VM_FAULT_WIRE_MASK) == 0) && (wired == 0)) { 884 vm_fault_prefault(fs.map->pmap, vaddr, fs.entry); 885 } 886 VM_OBJECT_LOCK(fs.object); 887 vm_page_lock_queues(); 888 vm_page_flag_set(fs.m, PG_REFERENCED); 889 890 /* 891 * If the page is not wired down, then put it where the pageout daemon 892 * can find it. Thanks! Let me know if there is anything else you need, Guy From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 22:27:04 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D094F106564A for ; Mon, 15 Dec 2008 22:27:04 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep3.cogeco.net (smtp2.cogeco.ca [216.221.81.29]) by mx1.freebsd.org (Postfix) with ESMTP id A750A8FC14 for ; Mon, 15 Dec 2008 22:27:04 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [0.0.0.0] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep3.cogeco.net (Postfix) with ESMTP id E0A7AF8B; Mon, 15 Dec 2008 17:27:03 -0500 (EST) Message-ID: <4946DA2F.7090508@cogeco.ca> Date: Mon, 15 Dec 2008 17:29:03 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> <4946BDB1.8060208@cogeco.ca> <200812152035.mBFKZkj4084689@lava.sentex.ca> In-Reply-To: <200812152035.mBFKZkj4084689@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 22:27:04 -0000 > I would try the change to /etc/nsswitch.conf so that group and passwd > read > > group: files > passwd: files > > At that file size, it sounds like you only have about 200 entries ? I > doubt its the issue, but its worth a try. I know at around 9,000 > files anything to do with UID lookups (e.g. ls -l) takes forever. > > >> PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT >> COMMAND >> 54 root 2 0 0 7 0 7 100.00% >> syncer > > Do you have any other special tuning other than polling ? Any in > /boot/loader.conf or /etc/sysctl.conf ? > > Does gstat show the disks busy ? > > ---Mike > The next thing I am doing is going to be removing the QUOTA feature to see if this has any bearing on this problem. It does not appear to be even writing at a heavy load as you can see (almost nothing) but the processes are mostly in UFS when it spirals out of control. I moved the processing of amavisd-new into a memory drive to at least take that off the IO and this seems to have helped a bit. There is not a lot of mail going through the system but every little bit helps. I suspect this is one other reason that is bringing the problem to the forefront as amavisd-new can use the disks a bit to process each e-mail. Thanks for your help so far, Paul From owner-freebsd-stable@FreeBSD.ORG Mon Dec 15 23:43:58 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BF611065672 for ; Mon, 15 Dec 2008 23:43:58 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep7.cogeco.net (smtp2.cogeco.ca [216.221.81.29]) by mx1.freebsd.org (Postfix) with ESMTP id 52A0E8FC14 for ; Mon, 15 Dec 2008 23:43:57 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep7.cogeco.net (Postfix) with ESMTP id 6CF881715; Mon, 15 Dec 2008 15:30:30 -0500 (EST) Message-ID: <4946BEDE.2000709@cogeco.ca> Date: Mon, 15 Dec 2008 15:32:30 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812152016.mBFKGnN6084606@lava.sentex.ca> In-Reply-To: <200812152016.mBFKGnN6084606@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 23:43:58 -0000 > > I would also try disabling polling. Is you scheduler ULE or BSD? For > an 8 core box, it should be ULE > > ---Mike Hi Mike, Thanks I will try this now as I have not tried this yet. Here is the current custom kernel and it is using ULE: cpu HAMMER ident MYCOMPUTER makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] options COMPAT_IA32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. options STOP_NMI # Stop CPUS using NMI instead of IPI options AUDIT # Security event auditing options SMP # Symmetric MultiProcessor Kernel options IPFIREWALL # Ip Firewall options IPFIREWALL_VERBOSE # Verbose options IPFIREWALL_VERBOSE_LIMIT=5000 # limit verbosity options QUOTA # Disk Quota options DEVICE_POLLING device cpufreq device acpi device pci device fdc device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) device arcmsr # Areca SATA II RAID device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support device sc device agp # support several AGP chipsets device sio # 8250, 16[45]50 based serial ports device uart # Generic UART driver device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device device em # Intel PRO/1000 adapter Gigabit Ethernet Card device miibus # MII bus support device loop # Network loopback device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device lagg # lagg interface for shared multi homed network connection device bpf # Berkeley packet filter device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device uplcom device ucom From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 07:19:22 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64059106564A; Tue, 16 Dec 2008 07:19:22 +0000 (UTC) (envelope-from victor@bsdes.net) Received: from alf.bsdes.net (244.Red-217-126-240.staticIP.rima-tde.net [217.126.240.244]) by mx1.freebsd.org (Postfix) with ESMTP id 0700B8FC08; Tue, 16 Dec 2008 07:19:21 +0000 (UTC) (envelope-from victor@bsdes.net) Received: by alf.bsdes.net (Postfix, from userid 1001) id A3539119CF7; Tue, 16 Dec 2008 08:19:19 +0100 (CET) Date: Tue, 16 Dec 2008 08:19:19 +0100 From: Victor Balada Diaz To: Pyun YongHyeon Message-ID: <20081216071919.GR1320@alf.bsdes.net> References: <20081210085934.GB1320@alf.bsdes.net> <20081210102800.GH37837@cdnetworks.co.kr> <20081210113225.GD1320@alf.bsdes.net> <20081210120719.GK37837@cdnetworks.co.kr> <20081211075707.GH1320@alf.bsdes.net> <20081211081045.GJ1320@alf.bsdes.net> <20081211090056.GH42370@cdnetworks.co.kr> <20081211095021.GL1320@alf.bsdes.net> <20081212121309.GM1320@alf.bsdes.net> <20081215090207.GN1320@alf.bsdes.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20081215090207.GN1320@alf.bsdes.net> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: [ATA] and re(4) stability issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 07:19:22 -0000 On Mon, Dec 15, 2008 at 10:02:07AM +0100, Victor Balada Diaz wrote: > Stopped stress testing this morning. After all the weekend testing > seems the re(4) problems were fixed. No single interface up/down error. > netstat -i reports no errors and everything is fine. Thanks a lot! > > I'm going to deploy the patches on our production machines. > > I've been able to trigger interrupt storms with ATA code, though. After deploying it in various machines this night i've found in the logs messages like this one: re0: watchdog timeout (missed Tx interrupts) -- recovering I know you told me this is harmless, so this is just so you know it's happening. Regards. -- La prueba más fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros. From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 07:49:00 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 423DA1065672 for ; Tue, 16 Dec 2008 07:49:00 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.226]) by mx1.freebsd.org (Postfix) with ESMTP id 026EC8FC13 for ; Tue, 16 Dec 2008 07:48:59 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so3566862rvf.43 for ; Mon, 15 Dec 2008 23:48:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from :to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=XNpmrIAv3diOIXtYogNkIH+DSFKf25PjIeBiw05uOdY=; b=C+TQc5Z60JakvTh1zceOFr8ql104rhV3fHbOj445LEic3AQJ6XlAhcO0i/fpR79+A8 ncVIYTSv0quj4/yk6/xTixAkv6EiASQHvc5Y0qfatFjSwv+jMNPrpHtpkhsW8wzhZXQm OEDiqCTnuE2wCrPWWwuvd9D+8SjgmXHUqMUiQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=rPrCk8/aRU7G3KGIWKd7Z8lbtmf///YI85s5x/bf49/94AJXRLnG9fkkels86vqHu9 P2UFbsJEOdQDf1qiP0dQi34NuNmPZOI5h8u5Tm3/t4nwIyANBP3Iaa8CtKmK1cFCYWZ+ /R9UVLHdJHpQ9noDfQDYxaEXiZmCJUivFFl9U= Received: by 10.140.136.6 with SMTP id j6mr4192535rvd.126.1229413739716; Mon, 15 Dec 2008 23:48:59 -0800 (PST) Received: from michelle.cdnetworks.co.kr ([211.53.35.84]) by mx.google.com with ESMTPS id b8sm221167rvf.3.2008.12.15.23.48.56 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 15 Dec 2008 23:48:58 -0800 (PST) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id mBG7mqeD064164 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 16 Dec 2008 16:48:52 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id mBG7mq8e064163; Tue, 16 Dec 2008 16:48:52 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Tue, 16 Dec 2008 16:48:51 +0900 From: Pyun YongHyeon To: Victor Balada Diaz Message-ID: <20081216074851.GD62771@cdnetworks.co.kr> References: <20081210102800.GH37837@cdnetworks.co.kr> <20081210113225.GD1320@alf.bsdes.net> <20081210120719.GK37837@cdnetworks.co.kr> <20081211075707.GH1320@alf.bsdes.net> <20081211081045.GJ1320@alf.bsdes.net> <20081211090056.GH42370@cdnetworks.co.kr> <20081211095021.GL1320@alf.bsdes.net> <20081212121309.GM1320@alf.bsdes.net> <20081215090207.GN1320@alf.bsdes.net> <20081216071919.GR1320@alf.bsdes.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081216071919.GR1320@alf.bsdes.net> User-Agent: Mutt/1.4.2.1i Cc: freebsd-stable@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: [ATA] and re(4) stability issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 07:49:00 -0000 On Tue, Dec 16, 2008 at 08:19:19AM +0100, Victor Balada Diaz wrote: > On Mon, Dec 15, 2008 at 10:02:07AM +0100, Victor Balada Diaz wrote: > > Stopped stress testing this morning. After all the weekend testing > > seems the re(4) problems were fixed. No single interface up/down error. > > netstat -i reports no errors and everything is fine. Thanks a lot! > > > > I'm going to deploy the patches on our production machines. > > > > I've been able to trigger interrupt storms with ATA code, though. > > After deploying it in various machines this night i've found in the > logs messages like this one: > > re0: watchdog timeout (missed Tx interrupts) -- recovering > > I know you told me this is harmless, so this is just so you Yes, it's not real watchdog timeout as long as re(4) still works correctly. > know it's happening. > Ok. I'll update re(4) when I find spare time. -- Regards, Pyun YongHyeon From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 07:52:23 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72ECB1065676 for ; Tue, 16 Dec 2008 07:52:23 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 2B5388FC16 for ; Tue, 16 Dec 2008 07:52:22 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1LCUj7-0009rt-D5 for stable@freebsd.org; Tue, 16 Dec 2008 09:52:21 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 16 Dec 2008 09:52:21 +0200 From: Danny Braniss Message-ID: Cc: Subject: bce reporting fantom input errors? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 07:52:23 -0000 Hi, After changing cables,switches,ports, I came to the conclusion that bce is reporting input errors that are not there, or creating them. I checked this with 3 different boxes, all Dell-2950/Broadcom NetXtreme II BCM5708 1000Base-T (B2), and one of them, while running Solaris, reported 0 errors after a week, and freebsd after a few minutes its count was > 100. The errors appear under 7.-PRERELEASE, but not under 7.0 Anybody else seeing this? danny From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 08:23:26 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5354C1065670 for ; Tue, 16 Dec 2008 08:23:26 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id ED85D8FC13 for ; Tue, 16 Dec 2008 08:23:25 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [211.166.10.233]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id 273AA28448 for ; Tue, 16 Dec 2008 16:23:25 +0800 (CST) Received: from localhost (tarsier.geekcn.org [211.166.10.233]) by tarsier.geekcn.org (Postfix) with ESMTP id A3BA1EC57BB; Tue, 16 Dec 2008 16:23:24 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([211.166.10.233]) by localhost (mail.geekcn.org [211.166.10.233]) (amavisd-new, port 10024) with ESMTP id 0W2yRCRSCXaH; Tue, 16 Dec 2008 16:23:19 +0800 (CST) Received: from charlie.delphij.net (c-67-180-38-12.hsd1.ca.comcast.net [67.180.38.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id 69A7FEC5720; Tue, 16 Dec 2008 16:23:15 +0800 (CST) Message-ID: <4947656E.4050904@delphij.net> Date: Tue, 16 Dec 2008 00:23:10 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: Danny Braniss References: In-Reply-To: X-Enigmail-Version: 0.95.7 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: multipart/mixed; boundary="------------070205030901020808000803" Cc: stable@freebsd.org Subject: Re: bce reporting fantom input errors? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 08:23:26 -0000 This is a multi-part message in MIME format. --------------070205030901020808000803 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Danny Braniss wrote: > Hi, > After changing cables,switches,ports, I came to the conclusion > that bce is reporting input errors that are not there, or creating them. > I checked this with 3 different boxes, all Dell-2950/Broadcom NetXtreme II > BCM5708 1000Base-T (B2), and one of them, while running Solaris, reported > 0 errors after a week, and freebsd after a few minutes its count was > 100. > The errors appear under 7.-PRERELEASE, but not under 7.0 > > Anybody else seeing this? Please apply this patch, it was committed as revision 186169 about 3 hours ago against -HEAD. I'll MFC it after 3 days. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAklHZWsACgkQi+vbBBjt66CHxgCfQhUCadChP7mtyoOD4Wg4cP/k lAUAnj1S2vh/TtmnKZAaczJvx7V/XR4x =fdk+ -----END PGP SIGNATURE----- --------------070205030901020808000803 Content-Type: text/plain; name="bce-noL2Filter.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="bce-noL2Filter.diff" Index: if_bce.c =================================================================== --- if_bce.c (revision 186076) +++ if_bce.c (working copy) @@ -7408,7 +7408,6 @@ (u_long) sc->stat_IfInMBUFDiscards + (u_long) sc->stat_Dot3StatsAlignmentErrors + (u_long) sc->stat_Dot3StatsFCSErrors + - (u_long) sc->stat_IfInFramesL2FilterDiscards + (u_long) sc->stat_IfInRuleCheckerDiscards + (u_long) sc->stat_IfInFTQDiscards + (u_long) sc->com_no_buffers; --------------070205030901020808000803-- From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 10:25:07 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3C201065679 for ; Tue, 16 Dec 2008 10:25:07 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 6002D8FC14 for ; Tue, 16 Dec 2008 10:25:07 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1LCX6w-000BkZ-EI for stable@freebsd.org; Tue, 16 Dec 2008 12:25:06 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: stable@freebsd.org In-reply-to: <4947656E.4050904@delphij.net> References: <4947656E.4050904@delphij.net> Comments: In-reply-to Xin LI message dated "Tue, 16 Dec 2008 00:23:10 -0800." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 16 Dec 2008 12:25:06 +0200 From: Danny Braniss Message-ID: Cc: Subject: Re: bce reporting fantom input errors? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 10:25:07 -0000 > This is a multi-part message in MIME format. > --------------070205030901020808000803 > Content-Type: text/plain; charset=ISO-8859-1 > Content-Transfer-Encoding: 7bit > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Danny Braniss wrote: > > Hi, > > After changing cables,switches,ports, I came to the conclusion > > that bce is reporting input errors that are not there, or creating them. > > I checked this with 3 different boxes, all Dell-2950/Broadcom NetXtreme II > > BCM5708 1000Base-T (B2), and one of them, while running Solaris, reported > > 0 errors after a week, and freebsd after a few minutes its count was > 100. > > The errors appear under 7.-PRERELEASE, but not under 7.0 > > > > Anybody else seeing this? > > Please apply this patch, it was committed as revision 186169 about 3 > hours ago against -HEAD. I'll MFC it after 3 days. > > Cheers, > - -- > Xin LI http://www.delphij.net/ > FreeBSD - The Power to Serve! > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.9 (FreeBSD) > > iEYEARECAAYFAklHZWsACgkQi+vbBBjt66CHxgCfQhUCadChP7mtyoOD4Wg4cP/k > lAUAnj1S2vh/TtmnKZAaczJvx7V/XR4x > =fdk+ > -----END PGP SIGNATURE----- > > --------------070205030901020808000803 > Content-Type: text/plain; > name="bce-noL2Filter.diff" > Content-Transfer-Encoding: 7bit > Content-Disposition: inline; > filename="bce-noL2Filter.diff" > > Index: if_bce.c > =================================================================== > --- if_bce.c (revision 186076) > +++ if_bce.c (working copy) > @@ -7408,7 +7408,6 @@ > (u_long) sc->stat_IfInMBUFDiscards + > (u_long) sc->stat_Dot3StatsAlignmentErrors + > (u_long) sc->stat_Dot3StatsFCSErrors + > - (u_long) sc->stat_IfInFramesL2FilterDiscards + > (u_long) sc->stat_IfInRuleCheckerDiscards + > (u_long) sc->stat_IfInFTQDiscards + > (u_long) sc->com_no_buffers; > > --------------070205030901020808000803-- thanks! so actually it was counting IfInFramesL2FilterDiscards. btw, it worked, it's now 0 input errors. danny From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 15:56:13 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EFE621065679 for ; Tue, 16 Dec 2008 15:56:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id B38AA8FC26 for ; Tue, 16 Dec 2008 15:56:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.3/8.14.3) with ESMTP id mBGFuBQq010388; Tue, 16 Dec 2008 10:56:11 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id mBGFuB7n089368 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 16 Dec 2008 10:56:11 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200812161556.mBGFuB7n089368@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 16 Dec 2008 10:56:19 -0500 To: Paul MacKenzie , freebsd-stable@freebsd.org From: Mike Tancsa In-Reply-To: <4946DA2F.7090508@cogeco.ca> References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> <4946BDB1.8060208@cogeco.ca> <200812152035.mBFKZkj4084689@lava.sentex.ca> <4946DA2F.7090508@cogeco.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 15:56:14 -0000 At 05:29 PM 12/15/2008, Paul MacKenzie wrote: >The next thing I am doing is going to be removing the QUOTA feature >to see if this has any bearing >on this problem. It does not appear to be even writing at a heavy >load as you can see (almost >nothing) but the processes are mostly in UFS when it spirals out of control. Whats strange is that the output from gstat shows the disks hardly active at all.... Yet why is the syncer at 100% ? Do you have write caching disabled on the array ? What does the raw throughput look like to the disks ? e.g. if you try a simple dd if=/dev/zero of=/var/tmp bs=1024k count=1000 ? >I moved the processing of amavisd-new into a memory drive to at >least take that off the IO and this >seems to have helped a bit. There is not a lot of mail going through >the system but every little bit >helps. I suspect this is one other reason that is bringing the >problem to the forefront as >amavisd-new can use the disks a bit to process each e-mail. Is the high load average simply a function of processes blocking on network io ? On our av/spam scanners for example show a high load avg because there are many processes waiting on network io to complete (e.g. talking to RBL lists, waiting for DCC servers to complete etc) Also, is it really related to the arcmsr driver ? i.e. if you did the same tasks on a single IDE drive, is the performance profile going to be the same ? ---Mike From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 16:22:52 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 298F91065673 for ; Tue, 16 Dec 2008 16:22:52 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id D908C8FC08 for ; Tue, 16 Dec 2008 16:22:51 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1LCch8-000G0M-Q2 for stable@freebsd.org; Tue, 16 Dec 2008 18:22:50 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 16 Dec 2008 18:22:50 +0200 From: Danny Braniss Message-ID: Cc: Subject: more zfs/nfs panics X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 16:22:52 -0000 Hi, I'm trying to tar a rather big directory via nfs (some 800gb), it has many subdirectories, some of them with many files (close to 10^6 :-) just before the server panics, the tar (on the client) starts complaining about lost files, or permition denied, but not in the pathological directories. panic: kmem_malloc(-1661382656): kmem_map too small: 645009408 total allocated cpuid = 3 KDB: enter: panic [thread pid 881 tid 100112 ] Stopped at kdb_enter_why+0x3d: movq $0,0x5ef3e8(%rip) db> tr Tracing pid 881 tid 100112 td 0xffffff0004ba2000 kdb_enter_why() at kdb_enter_why+0x3d panic() at panic+0x17b kmem_malloc() at kmem_malloc+0x565 uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0xd7 nfsrv_readdir() at nfsrv_readdir+0x4e1 nfssvc() at nfssvc+0x400 syscall() at syscall+0x1bb Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006885cc, rsp = 0x7fffffffea28, rbp = 0 --- I have increased vm.kmem_size_max="1024M" vm.kmem_size="1024M" vfs.zfs.arc_max="800M" it just seems to delay the panic though, it smells like some memory leak ... the host is running amd64 quad core, 7.1-prerelease and 8GB. danny From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 18:21:46 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C0DB1065687 for ; Tue, 16 Dec 2008 18:21:46 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id A0F408FC2C for ; Tue, 16 Dec 2008 18:21:45 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1LCeY6-0003C4-QT for freebsd-stable@freebsd.org; Tue, 16 Dec 2008 18:21:38 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 16 Dec 2008 18:21:38 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 16 Dec 2008 18:21:38 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Tue, 16 Dec 2008 19:21:29 +0100 Lines: 62 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigE884A4A08C7DF35D4C4CBB6A" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.18 (X11/20081125) In-Reply-To: X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: more zfs/nfs panics X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 18:21:46 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigE884A4A08C7DF35D4C4CBB6A Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Danny Braniss wrote: > Hi, > I'm trying to tar a rather big directory via nfs (some 800gb), it > has many subdirectories, some of them with many files (close to 10^6 :-= ) >=20 > just before the server panics, the tar (on the client) starts complaini= ng=20 > about lost > files, or permition denied, but not in the pathological directories. > panic: kmem_malloc(-1661382656): kmem_map too small: 645009408 total al= located > cpuid =3D 3 > KDB: enter: panic > [thread pid 881 tid 100112 ] > Stopped at kdb_enter_why+0x3d: movq $0,0x5ef3e8(%rip) > db> tr > Tracing pid 881 tid 100112 td 0xffffff0004ba2000 > kdb_enter_why() at kdb_enter_why+0x3d > panic() at panic+0x17b > kmem_malloc() at kmem_malloc+0x565 > uma_large_malloc() at uma_large_malloc+0x4a > malloc() at malloc+0xd7 > nfsrv_readdir() at nfsrv_readdir+0x4e1 > nfssvc() at nfssvc+0x400 > syscall() at syscall+0x1bb > Xfast_syscall() at Xfast_syscall+0xab > --- syscall (155, FreeBSD ELF64, nfssvc), rip =3D 0x8006885cc, rsp =3D = > 0x7fffffffea28, rbp =3D 0 --- >=20 > I have increased=20 > vm.kmem_size_max=3D"1024M" > vm.kmem_size=3D"1024M" > vfs.zfs.arc_max=3D"800M" > it just seems to delay the panic though, it smells like some memory lea= k ... Well, the canonical fix seems be to DECREASE vfs.zfs.arc_max to something like 100M and keep decreasing until it works. --------------enigE884A4A08C7DF35D4C4CBB6A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJR/GpldnAQVacBcgRAthNAKDZEdH2ZfTlhHg7dYzBaAdhmeMV/gCfdwlt h+mkLd32q7GurKEu1r20Nxo= =7pHZ -----END PGP SIGNATURE----- --------------enigE884A4A08C7DF35D4C4CBB6A-- From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 22:23:04 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87A121065672 for ; Tue, 16 Dec 2008 22:23:04 +0000 (UTC) (envelope-from m4gicite@gmail.com) Received: from mail-bw0-f19.google.com (mail-bw0-f19.google.com [209.85.218.19]) by mx1.freebsd.org (Postfix) with ESMTP id A84CD8FC1B for ; Tue, 16 Dec 2008 22:23:03 +0000 (UTC) (envelope-from m4gicite@gmail.com) Received: by bwz12 with SMTP id 12so172025bwz.19 for ; Tue, 16 Dec 2008 14:23:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=3LZTzzduSeCT+KpLjpAr5F01drHbxRnvDWLPY2TB1LA=; b=asb26Q2kUQW2XLxkLqCZBDcZj6VG6TRKTTIt8WdIqBvIcSVXD1qEYOpdGszTPGLw8D bkbOWDtvoXvFmegIFT5kVE3tBCWN5U5pHEjr4B4ppvp5x+qT/0pSAuvbNbrcPvbEKCC1 twQTRC3TAZ+l19D+yFEsf1/TX0Lq7IC4Fpn4A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=UU9xEXq31wUIj1bey8cwk1AjWmDrnl7db4hzvgi0dcwJ1GCqLQxC4S9W/raiunMg8J sMdVQZWHps7fuUbu0ToTPL5qyRpFON4v+fm921pdg5sqNonJ51e0Iu3QWd8tyPoojEfM 97GBK173eB8D29tZ5VOUgQXy4lWONZcvbG+Rc= Received: by 10.181.145.6 with SMTP id x6mr3113188bkn.25.1229466169416; Tue, 16 Dec 2008 14:22:49 -0800 (PST) Received: by 10.180.214.7 with HTTP; Tue, 16 Dec 2008 14:22:49 -0800 (PST) Message-ID: Date: Tue, 16 Dec 2008 22:22:49 +0000 From: Ryan To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Install issues with 7.x X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 22:23:04 -0000 Ok, message didn't send entire post history, this should be it. Hopefully it's readable enough. Hello, I purchased a new Clevo M860TU on the account that it ran linux very well and was hoping it would fair the same on FreeBSD. Not so much, little help? I posted this in mobile originally but though stable would be a better choice. Don't know if it is more appropriate here or ACPI. I'm giving you as much information as I know how to get. as I cannot get sysinstall to load I am having to type all these dmesg. The boot process is hanging. This is all with 7.x, I can give 6.x if needed. Hardware: Intel P9500 4gb DDR3-1066 Nvidia 9800M GT Atheros AR5006e FreeBSD 7.1-BETA2 These snippets of dmesg happen around the end where it hangs. 1. Default ... cpu0: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6a02d40 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU0._OSC] (Node 0xc68556e0), AE_AML_INTERNAL est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6a0e300 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU1._OSC] (Node 0xc685560), AE_AML_INTERNAL est1: on cpu1 p4tcc1: on cpu1 ... cpu0: Cx states changed cpu1: Cx states changed unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config Then just stalls 2. No ACPI ... unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config Then just stalls 3. Safe Mode I can only tell you a little because console is spammed. It is the same as no ACPI, but with an interrupt storm. ... unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config When it gets to the unknowns, this is spammed. interrupt storm detected on "irq10:"; throttling interrupt source Other than the interrupt storm spam, it is halted like the others. 4. Single User Mode Same as 1, Default 5. Verbose All I can tell you is what is spammed at the end. acpi: bad write to port 0x080 (32), val hex Where hex is ever increasing and loops when it hits 0xff01. I can also see run_interrupt_driven_hooks message in all the spam. Using some googling if you add the sysctl before boot debug.acpi.block_bad_io=1 it might be of some help. This just leads to a never ending loop of acpi errors - the scroll very fast and difficult to record might I add! ... acpi: bad write to port 0x080 (32), val hex ACPI Exception (evregion-0529): AE_BAD_PARAMETER, Returned by handler for [SystemIO] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\P8XH] (Node 0xc6850a60), AE_BAD_PARAMETER ACPI Error (psparse-0626): Method parse/execution failed [\_GPE._L01] [20070320] ACPI Exception (evgpe-0687): AE_BAD_PARAMETER, while evauating GPE method [_L01] [20070320] --repeat-- ... FreeBSD 7.0-REL 7.0 is a little different than 7.1. Messages are somewhat the same but they happen near the beginning of dmesg instead of around the end. The run_interrupt_driven_hooks issue is nonexistant as well, but it still hangs. I'm guessing that's a debug tool more than an error. 1. Default ... cpu0: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6862580 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU0._OSC] (Node 0xc682d580), AE_AML_INTERNAL est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6861100 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU1._OSC] (Node 0xc682d4a0), AE_AML_INTERNAL est1: on cpu1 p4tcc1: on cpu1 ... cpu0: Cx states changed cpu1: Cx states changed unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install Hangs. 2. No ACPI .. unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ .. Hangs. 3. Safe Mode Same interrupt storm as 7.1-BETA2. ... interrupt storm detected on "irq10:"; throttling interrupt source --repeat-- 4. Single User Mode Same as 1. Default. 5. Verbose Hang like normal, cannot see the ACPI errors since they fly off the scroll lock buffer. ... cpu0: Cx states changed cpu1: Cx states changed ... unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ ... Thanks again. On Wed, 29 Oct 2008, Ryan wrote: Hello, I purchased a new Clevo M860TU on the account that it ran linux very well and was hoping it would fair the same on FreeBSD. Not so much, little help? I posted this in mobile originally but though stable would be a better choice. Don't know if it is more appropriate here or ACPI. I'm giving you as much information as I know how to get. as I cannot get sysinstall to load I am having to type all these dmesg. The boot process is hanging. This is all with 7.x, I can give 6.x if needed. xpt_config is the CAM configuration wait, so basically the system is waiting for a storage device to report back on whether it could be used as a root file system. I recently saw a similar report of problems involving a firewire controller on an nvidia motherboard following an upgrade to 7.x, and I wonder if you might try the following: see if 6.4 will install, and if so, install it. Then cvsup 7.x, and do a buildworld but not an installworld. This will let you build and experiment with 7.x kernels from a known-working environment. Make sure to keep a working 6.x kernel around -- I suggest something like "cp -r /boot/kernel /boot/kernel.good" before starting so you can always fall back to a good kernel. Now try building a 7.x kernel without USB or firewire support, and booting that? Also, it's worth checking there are no BIOS upgrades available for the motherboard... Robert N M Watson Computer Laboratory University of Cambridge Thanks for your interest Robert, unfortunately it was a no-go. I went ahead and tested 6.4RC1 and 6.3. Now all I am getting are ACPI errors which I would think I could get the installer going by disabling ACPI. And yes, I'm running on the latest - and only - bios revision for the laptop. The following were done under default boot option. No ACPI did not generate any error messages and hung, single user mode acted the same as default, and safe mode created an interrupt storm like 7.x did. 6.4-RC1 ... acpi_timer0: <24-bit timer at 3.579545 MHz> port 0x408-0x40b on acpi0 acpi_ec0: port 0x62, 0x66 on acpi0 cpu0: on acpi0 ACPI-0328: *** Error: No pointer back to NS node in buffer obj 0xc85c87c0 ACPI-1304: *** Error: Method execution failed [\_PR_.CPU0._OSC] (Node 0xc84e2580), AE_AML_INTERNAL acpi_throttle0: on cpu0 ... then this gets spammed 6 times at the end ACPI-0438: *** Error: Looking up [\_PR_.CPU0._PPC] in name space, AE_NOT_FOUND SearchNode 0xc84cdcc0 StartNode 0xc84cdcc0 ReturnNode 0 ACPI-1304: *** Error: Method execution failed [\_SB_.AC__.ADJP] (Node 0xc84cdcc0), AE_NOT_FOUND ACPI-1304: *** Error: Method execution failed [\_SB.AC__._PSR] (Node 0xc84cdd00), AE_NOT_FOUND 6.3-REL 6.3 gives the same errors but with different node addresses. ... acpi_timer0: <24-bit timer at 3.579545 MHz> port 0x408-0x40b on acpi0 acpi_ec0: port 0x62, 0x66 on acpi0 cpu0: on acpi0 ACPI-0328: *** Error: No pointer back to NS node in buffer obj 0xc85c94c0 ACPI-1304: *** Error: Method execution failed [\_PR_.CPU0._OSC] (Node 0xc84e3780), AE_AML_INTERNAL acpi_throttle0: on cpu0 ... spammed again 6 times at the end ACPI-0438: *** Error: Looking up [\_PR_.CPU0._PPC] in name space, AE_NOT_FOUND SearchNode 0xc84cdd20 StartNode 0xc84cdd20 ReturnNode 0 ACPI-1304: *** Error: Method execution failed [\_SB_.AC__.ADJP] (Node 0xc84cdd20), AE_NOT_FOUND ACPI-1304: *** Error: Method execution failed [\_SB.AC__._PSR] (Node 0xc84e3780), AE_NOT_FOUND Help at all? Ryan skrev: - Show quoted text - Hello, I purchased a new Clevo M860TU on the account that it ran linux very well and was hoping it would fair the same on FreeBSD. Not so much, little help? I posted this in mobile originally but though stable would be a better choice. Don't know if it is more appropriate here or ACPI. I'm giving you as much information as I know how to get. as I cannot get sysinstall to load I am having to type all these dmesg. The boot process is hanging. This is all with 7.x, I can give 6.x if needed. Hardware: Intel P9500 4gb DDR3-1066 Nvidia 9800M GT Atheros AR5006e FreeBSD 7.1-BETA2 These snippets of dmesg happen around the end where it hangs. 1. Default ... cpu0: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6a02d40 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU0._OSC] (Node 0xc68556e0), AE_AML_INTERNAL est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6a0e300 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU1._OSC] (Node 0xc685560), AE_AML_INTERNAL est1: on cpu1 p4tcc1: on cpu1 ... cpu0: Cx states changed cpu1: Cx states changed unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config Disabling firewire completely in BIOS might at least get the machine booting. You should try that if you haven't already. I've seen this problem on at least two different systems... -- Joel Sadly with the quality of BIOS recently, that is not an option. Not much to offer. Attached is a picture of what I have to change. Other and XP are the same, Vista unlocks AHCI. Another way of accomplishing disabling firewire is to remake the install CD with a different kernel and not quite sure how to do that. Ryan skrev: Sadly with the quality of BIOS recently, that is not an option. Not much to offer. Attached is a picture of what I have to change. Other and XP are the same, Vista unlocks AHCI. Another way of accomplishing disabling firewire is to remake the install CD with a different kernel and not quite sure how to do that. Take a look at the release(7) manpage for information about building your own customized release CD. -- Joel Finally had time to understand some how release works and made some new cds. Some new updates to the situation since as I am still having issues. I was not able to figure out how to make release with a custom kernel, would always fail that step so I had to stick with GENERIC. To simulate taking out firewire support I just deleted the corresponding kernel modules in /boot/kernel and remade the iso. Stable as of 12-07-2008 is giving the same feedback as 7.1-B2 but actually gives a panic screen. panic: run_interrupt_driven_config_hooks: waited too long cpuid = 0 KDB: enter: panic [thread pid 0 tid 100000 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> There has been a bios update recently and have applied it, but there has been no change in any of the tests by a quick glance. If it makes a difference the cds I create with cdrecord generate the added errors acd0: FAILURE - READ BIG timed out. I don't think that has anything to do with it, just saying it for full disclosure. Disregard if irrelevant. That error applies to only the stable builds I made. From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 23:03:31 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF17A1065676 for ; Tue, 16 Dec 2008 23:03:31 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep9.cogeco.net (smtp.cogeco.net [216.221.81.25]) by mx1.freebsd.org (Postfix) with ESMTP id 8E2FD8FC1F for ; Tue, 16 Dec 2008 23:03:31 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep9.cogeco.net (Postfix) with ESMTP id 8B3B5C7C; Tue, 16 Dec 2008 12:05:09 -0500 (EST) Message-ID: <4947E03F.8070408@cogeco.ca> Date: Tue, 16 Dec 2008 12:07:11 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> <4946BDB1.8060208@cogeco.ca> <200812152035.mBFKZkj4084689@lava.sentex.ca> <4946DA2F.7090508@cogeco.ca> <200812161556.mBGFuB7n089368@lava.sentex.ca> In-Reply-To: <200812161556.mBGFuB7n089368@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 23:03:31 -0000 >> The next thing I am doing is going to be removing the QUOTA feature >> to see if this has any bearing >> on this problem. It does not appear to be even writing at a heavy >> load as you can see (almost >> nothing) but the processes are mostly in UFS when it spirals out of >> control. > > > Whats strange is that the output from gstat shows the disks hardly > active at all.... Yet why is the syncer at 100% ? Do you have write > caching disabled on the array ? What does the raw throughput look > like to the disks ? e.g. if you try a simple dd if=/dev/zero > of=/var/tmp bs=1024k count=1000 ? > >> I moved the processing of amavisd-new into a memory drive to at least >> take that off the IO and this >> seems to have helped a bit. There is not a lot of mail going through >> the system but every little bit >> helps. I suspect this is one other reason that is bringing the >> problem to the forefront as >> amavisd-new can use the disks a bit to process each e-mail. > > > Is the high load average simply a function of processes blocking on > network io ? On our av/spam scanners for example show a high load avg > because there are many processes waiting on network io to complete > (e.g. talking to RBL lists, waiting for DCC servers to complete etc) > > Also, is it really related to the arcmsr driver ? i.e. if you did the > same tasks on a single IDE drive, is the performance profile going to > be the same ? > > ---Mike > Hi Mike, Well I tried to remove both the USB com ports drivers and the QUOTA out of the kernel last night and this has not solved it but it seems a bit more stable today. The HTTP only had a problem two times last night. I am not sure if it is specifically related to the arcmsr driver but unfortunately I am unable to try a single IDE setup at the moment. If I can get to the bottom of why it is locking then it might point us in the right direction.I was told that Jan downgraded to 6.4 as she could never resolve her issue and worked on it for a very long time. Write caching is enabled on the array which was the first thing I checked and I have the battery backup installed and I confirm it shows up in the areca-cli as 100% charged. Do you think it may be hardware related even though there are no errors at all? I have checked the event log in the S50000PAL which is very sensitive to errors I have found in the past as well as the event log of the areca-cli. Both are error free. With regards to the e-mail scanning waiting for RBL completion there is only usually one e-mail per minute approximately to give you an idea of the load with regards to the e-mail so this is not really a reliable test and I don't see how this is an overall contributing factor as there seem to be many ways to bring the locking forward including running a dump being one of them in my experience. What I have found is the more I take off the load of the system writing seems to help a bit so I have been doing everything I can to help with this until I can find a workable solution. I have recompiled all the ports a few times over the past month in hopes that something might get fixed if it was a port issue and all ports are as up-to-date as possible using portupgrade and tracking the port tree. The primary problem is what you said above the gstat shows very little activity but the system seems to be "stuck". The syncer is not always at 100% as it comes and goes I grabbed that at one time when watching it but it did show how "little" activity there was from the reports. Here is the output of dd on the WORKING server: dd if=/dev/zero of=/usr/test bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 17.874501 secs (58663232 bytes/sec) Here is the output of dd on the one not working right but NOT "locked" right now. I need to wait for it to "lock" again before I can test this again. dd if=/dev/zero of=/usr/test bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 34.270080 secs (30597419 bytes/sec) The numbers are pretty reasonable albeit half on one compare to the other. I did notice on the non-working server the system numbers were always much higher when I ran the dd. This could be a coincidence but the syncer was not seen in the list of the working server but it was on the list on the one with the problem when running both with top -IS which the dd was running. I also noticed the system number s are always much higher on the one with the problem. I am waiting for the system to lock again to try and see what it shows when it is locked. I suppose I will work on getting 7.0 running (downgrade) next to see if I have the same problem on this version as another clue to the problem. Thanks again for your help so far. Paul From owner-freebsd-stable@FreeBSD.ORG Tue Dec 16 23:36:52 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBF7C1065670 for ; Tue, 16 Dec 2008 23:36:52 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep9.cogeco.net (smtp.cogeco.net [216.221.81.25]) by mx1.freebsd.org (Postfix) with ESMTP id 9CFB68FC19 for ; Tue, 16 Dec 2008 23:36:52 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep9.cogeco.net (Postfix) with ESMTP id 3FF87924; Mon, 15 Dec 2008 15:25:29 -0500 (EST) Message-ID: <4946BDB1.8060208@cogeco.ca> Date: Mon, 15 Dec 2008 15:27:29 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> In-Reply-To: <200812152010.mBFKAlZf084580@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2008 23:36:52 -0000 > What does top -S show ? Most of the load is in system. Does the > machine in question have a rather large master.passwd file by chance ? > (http://www.freebsd.org/cgi/query-pr.cgi?pr=75855) > ---Mike > Thanks for your quick reply: master.passwd is only 9467 (with a ls-l) TOP -ISM at times shows syncer at the top but this ranges and is not always near the top. last pid: 55084; load averages: 17.74, 10.08, 5.58 up 0+10:19:24 15:05:23 290 processes: 50 running, 218 sleeping, 14 waiting, 8 lock CPU: 15.4% user, 0.0% nice, 68.3% system, 3.0% interrupt, 13.2% idle Mem: 795M Active, 3279M Inact, 492M Wired, 6116K Cache, 214M Buf, 11G Free Swap: 8192M Total, 8192M Free PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 54 root 2 0 0 7 0 7 100.00% syncer Here is a top with it not fully locked but high system usage. last pid: 55468; load averages: 9.93, 11.31, 8.99 up 0+10:32:58 15:18:57 259 processes: 19 running, 215 sleeping, 14 waiting, 11 lock CPU: 19.1% user, 0.0% nice, 58.2% system, 1.9% interrupt, 20.8% idle Mem: 635M Active, 3258M Inact, 481M Wired, 6856K Cache, 214M Buf, 11G Free Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 18 root 1 171 ki31 0K 16K RUN 0 439:32 31.15% idle: cpu0 55422 www 1 102 0 193M 59632K RUN 5 0:26 30.96% httpd 12 root 1 171 ki31 0K 16K RUN 6 522:14 28.37% idle: cpu6 54 root 1 20 - 0K 16K syncer 2 81:19 28.37% syncer 15 root 1 171 ki31 0K 16K RUN 3 465:15 26.56% idle: cpu3 55411 www 1 -4 0 157M 33704K *vnode 1 0:21 26.17% httpd 55388 www 1 -4 0 160M 35940K *vnode 1 0:14 26.17% httpd 13 root 1 171 ki31 0K 16K RUN 5 509:35 25.98% idle: cpu5 11 root 1 171 ki31 0K 16K RUN 7 525:53 25.88% idle: cpu7 14 root 1 171 ki31 0K 16K RUN 4 491:32 25.29% idle: cpu4 55453 www 1 101 0 157M 33608K CPU7 7 0:08 24.76% httpd 55365 www 1 -4 0 157M 33408K ufs 3 0:23 24.56% httpd 55440 www 1 69 0 154M 31180K CPU2 7 0:09 24.37% httpd 55412 www 1 -4 0 153M 30156K *vnode 3 0:07 23.97% httpd 16 root 1 171 ki31 0K 16K CPU2 2 444:38 23.88% idle: cpu2 55376 www 1 -4 0 158M 34776K *vnode 0 0:26 23.88% httpd 55459 www 1 -4 0 145M 23920K *vnode 1 0:07 23.49% httpd 55467 www 1 70 0 154M 31056K *vnode 7 0:09 22.66% httpd 17 root 1 171 ki31 0K 16K CPU1 1 443:27 20.90% idle: cpu1 55374 www 1 -4 0 146M 25312K *vnode 7 0:09 13.38% httpd 55418 www 1 -4 0 145M 24192K ufs 0 0:18 12.89% httpd 55400 www 1 58 0 146M 25460K select 5 0:20 12.79% httpd 55443 www 1 -4 0 148M 25788K *vnode 1 0:03 12.50% httpd 55410 www 1 -4 0 147M 25700K *vnode 7 0:05 12.26% httpd 55438 www 1 -4 0 145M 24148K RUN 4 0:08 11.96% httpd 21 root 1 -44 - 0K 16K WAIT 0 34:45 11.77% swi1: net 55451 www 1 -4 0 144M 22704K *vnode 7 0:02 10.99% httpd 55447 www 1 60 0 145M 24008K select 2 0:07 10.50% httpd 55406 www 1 53 0 146M 25324K select 2 0:19 9.77% httpd 55433 www 1 49 0 146M 24912K select 2 0:11 8.06% httpd 55448 www 1 52 0 144M 22972K RUN 6 0:03 8.06% httpd 55383 www 1 45 0 145M 24284K select 2 0:12 7.96% httpd 55446 www 1 44 0 146M 24988K select 3 0:09 7.96% httpd 55430 www 1 4 0 145M 24136K kqread 0 0:03 6.69% httpd 55432 www 1 20 0 146M 24324K lockf 3 0:04 6.05% httpd 55464 www 1 -4 0 145M 23464K RUN 0 0:02 5.66% httpd 55424 www 1 45 0 146M 24876K select 6 0:08 3.66% httpd 55442 www 1 47 0 145M 23852K select 3 0:03 3.56% httpd 55373 www 1 48 0 146M 25364K select 5 0:07 3.17% httpd 55375 www 1 46 0 146M 25420K select 2 0:15 3.08% httpd 19 root 1 -32 - 0K 16K *Giant 2 9:02 2.98% swi4: clock sio 48518 wusage 1 46 0 10424K 2632K select 4 2:50 2.78% wusage 1490 mysql 97 4 -5 402M 184M sbwait 4 0:29 2.78% mysqld 55372 www 1 47 0 144M 22136K CPU6 0 0:01 2.59% httpd 55437 root 1 -32 0 9136K 2940K CPU4 4 0:01 2.59% top 55387 www 1 45 0 144M 22196K CPU5 1 0:02 2.39% httpd 55468 www 1 20 0 144M 21904K lockf 4 0:00 1.56% httpd 55441 www 1 45 0 144M 22088K select 5 0:01 1.46% httpd 51563 root 1 4 0 11848K 5540K connec 5 0:09 1.37% sendmail 55458 www 1 45 0 144M 22140K select 6 0:01 1.17% httpd 55336 root 1 51 0 144M 21888K CPU0 0 0:05 1.07% httpd 55455 www 1 45 0 144M 21992K select 0 0:01 1.07% httpd 55425 www 1 20 0 146M 25304K lockf 3 0:07 0.78% httpd 55415 www 1 45 0 144M 22192K select 7 0:01 0.68% httpd 55439 www 1 44 0 144M 22308K select 1 0:01 0.49% httpd 51561 root 1 45 0 11848K 5400K select 0 0:13 0.29% sendmail 54666 root 1 4 0 10824K 4496K connec 3 0:04 0.20% sendmail 55469 root 1 51 0 144M 21888K CPU3 3 0:00 0.00% httpd From owner-freebsd-stable@FreeBSD.ORG Wed Dec 17 04:22:31 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0325D1065677 for ; Wed, 17 Dec 2008 04:22:31 +0000 (UTC) (envelope-from jcw@highperformance.net) Received: from mx1.highperformance.net (s6.stradamotorsports.com [64.81.163.124]) by mx1.freebsd.org (Postfix) with ESMTP id C663A8FC1E for ; Wed, 17 Dec 2008 04:22:30 +0000 (UTC) (envelope-from jcw@highperformance.net) Received: from [192.168.1.17] ([192.168.1.17]) by mx1.highperformance.net (8.14.3/8.14.3) with ESMTP id mBH45T70081309 for ; Tue, 16 Dec 2008 20:05:30 -0800 (PST) (envelope-from jcw@highperformance.net) Message-ID: <49487A89.80608@highperformance.net> Date: Tue, 16 Dec 2008 20:05:29 -0800 From: "Jason C. Wells" User-Agent: Mozilla-Thunderbird 2.0.0.16 (X11/20080724) MIME-Version: 1.0 To: freebsd-stable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=2.5 tests=ALL_TRUSTED autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on s4.stradamotorsports.com Subject: Heimdal Breakage X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Dec 2008 04:22:31 -0000 After installing 6.4-RELEASE on my secondary KDC I decided to test the secondary KDC. When trying kinit I get this error: jcw@w17 ~ $ kinit jcw@STRADAMOTORSPORTS.COM's Password: kinit: krb5_get_init_creds: Key size is incompatible with encryption type One post on the net says that Heimdal changed the key format to add some padding or somesuch. I haven't gone about fixing the problem yet so maybe that post is not applicable to FreeBSD. Just the same I thought I would let folks know that their key databases are probably not forward compatible with 6.4-RELEASE. This would be a pretty big deal for some users. It would be nice to see this in UPDATING. Thanks, Jason From owner-freebsd-stable@FreeBSD.ORG Wed Dec 17 07:24:20 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B287106564A for ; Wed, 17 Dec 2008 07:24:20 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from ape.monkeybrains.net (mail.monkeybrains.net [208.69.40.9]) by mx1.freebsd.org (Postfix) with ESMTP id 36C1F8FC21 for ; Wed, 17 Dec 2008 07:24:20 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from [192.168.2.105] (adsl-76-247-114-27.dsl.pltn13.sbcglobal.net [76.247.114.27]) (authenticated bits=0) by ape.monkeybrains.net (8.14.1/8.14.1) with ESMTP id mBH7NXKf033998 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 16 Dec 2008 23:23:33 -0800 (PST) (envelope-from crapsh@monkeybrains.net) Message-ID: <4948A91E.3050200@monkeybrains.net> Date: Tue, 16 Dec 2008 23:24:14 -0800 From: Rudy User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: Ivan Voras References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.94, clamav-milter version 0.94 on pita.monkeybrains.net X-Virus-Status: Clean Cc: freebsd-stable@freebsd.org Subject: Re: more zfs/nfs panics X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Dec 2008 07:24:20 -0000 >> it just seems to delay the panic though, it smells like some memory leak ... >> > > Well, the canonical fix seems be to DECREASE vfs.zfs.arc_max to > something like 100M and keep decreasing until it works. > More info here: http://wiki.freebsd.org/ZFSQuickStartGuide Once you tune, your problems will go away. The default install should not need tuning (so people stop posting this panic problem)... will that be fixed in the next stable release? Rudy From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 06:00:43 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55B0E1065673 for ; Thu, 18 Dec 2008 06:00:43 +0000 (UTC) (envelope-from nawfal@gmail.com) Received: from ti-out-0910.google.com (ti-out-0910.google.com [209.85.142.190]) by mx1.freebsd.org (Postfix) with ESMTP id E24838FC08 for ; Thu, 18 Dec 2008 06:00:42 +0000 (UTC) (envelope-from nawfal@gmail.com) Received: by ti-out-0910.google.com with SMTP id a1so126121tib.3 for ; Wed, 17 Dec 2008 22:00:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:reply-to:to :in-reply-to:references:content-type:date:message-id:mime-version :x-mailer:from; bh=YQZv4SCRLCOViRSCvRsqmOrk49sQlj4EZ2euGbCh0Uo=; b=UhdXw95Ji+ujKsmh5YMT+VIUUvHEFR3LCsdn184h3bDiGntK3jSmkCGSS70I/Z1Ukc /Fksp5MmlTeVTfs5ZEJfRDdyopba6WNac594qbCt4nCXPFTZo2RFzyqOqL/FvQezRIIN x1zBIr13zqu5jR2TQhFVopQw9IHWvs4RBhEJM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:reply-to:to:in-reply-to:references:content-type:date :message-id:mime-version:x-mailer:from; b=nugK6loQWL2ah4kfw7YMPacs4S/rd+n6gLIwe0jJmxyeIFAH++6IjasfWj/nHqblZs 75uwyuYY4o0gC4rkuFPJ8sWWsyhDzzdRM8Shui1hgWRFAYVJeMtoQbLPQAnGZ3B+zOQD AOchUsAJhoFS/IeByhP9X0Tn8B2ODCXeIwjzE= Received: by 10.110.2.2 with SMTP id 2mr2188426tib.29.1229578185844; Wed, 17 Dec 2008 21:29:45 -0800 (PST) Received: from ?10.101.8.78? ([203.106.65.19]) by mx.google.com with ESMTPS id i9sm4256009tid.39.2008.12.17.21.29.42 (version=SSLv3 cipher=RC4-MD5); Wed, 17 Dec 2008 21:29:43 -0800 (PST) To: freebsd-stable In-Reply-To: <400809c165ecedf8a0b7bc6b569e1289.squirrel@wettoast.dyndns.org> References: <4935069A.8060209@ec-marseille.fr> <49357BD0.4000008@delphij.net> <4935944A.9090509@ec-marseille.fr> <4935C453.8070301@delphij.net> <4935D67E.4070204@delphij.net> <4936F8C4.6090006@ec-marseille.fr> <49399FA6.3060108@delphij.net> <493CE8F7.5010204@yandex-team.ru> <3a6fb7145a0a6c8af136ea1a0824e5ed.squirrel@wettoast.dyndns.org> <493D9BC6.8050902@delphij.net> <4fe0419b44a3a4c4a28b1e60fbb3a3c8.squirrel@wettoast.dyndns.org> <56272b131067237ccabd23de5f669458.squirrel@wettoast.dyndns.org> <4944D894.6070306@delphij.net> <400809c165ecedf8a0b7bc6b569e1289.squirrel@wettoast.dyndns.org> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-zL3kbO00LxE6wEi3kKv6" Date: Thu, 18 Dec 2008 13:29:40 +0800 Message-Id: <1229578180.6319.21.camel@nawfal-desktop> Mime-Version: 1.0 X-Mailer: Evolution 2.24.2 From: Nawfal bin Mohmad Rouyan Subject: Re: RELENG_7_1: bce driver change generating too much interrupts ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: nawfal@googlemail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 06:00:43 -0000 --=-zL3kbO00LxE6wEi3kKv6 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Mon, 2008-12-15 at 11:47 -0500, Mike Jakubik wrote: > On Sun, December 14, 2008 4:57 am, Xin LI wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > Mike Jakubik wrote: > >> On Mon, December 8, 2008 5:22 pm, Mike Jakubik wrote: > >>> On Mon, December 8, 2008 5:12 pm, Xin LI wrote: > >>> > >>>> Which version are you currently using? My previous commit only fixe= s > >>>> the excessive interrupt issue, I think this could be a different > >>>> problem, I'm taking a look at the code to see if I can have somethin= g > >>>> for you. > >>> I was running on the version just prior to the latest interrupt commi= t. > >>> I > >>> have now updated to the one with the interrupt fix. Will let you know > >>> if > >>> things change. > >>> > >>> Thank You. > >> > >> The interrupt rate has decreased significantly, however i am still > >> having > >> having problem with applications that hold stateful connections. The r= x > >> errors are also still showing, i suspect this is related to the proble= m. > >> How can i roll back this driver to the last known good version? > > > > Hi, Mike, > > > > I think they are different problems. Could you, please, give me > > feedback about whether: > > > > - The old driver does not trigger the problem? > > > > - The patched driver restore all the old driver behavior? >=20 > - Old driver. >=20 > I have been running the system for 4 days now with this driver. My > application has not stopped accepting connections, irq rate is low, and > there are no rx/tx errors reported. Everything looks good. >=20 > - Patched driver. >=20 > Your initial import plus the IRQ fix still shows rx errors, and my > application had stopped accepting connections. I have not tried the patch > in your last email, and im not sure when i will be able to to as these > systems are in production. Perhaps someone else could test it? As soon as > i get a chance i will let you know how it goes. >=20 > Thanks for the work on this. I have been using a Dell machine with 2 bce interfaces as a bridge between my LAN and Firewall to shape the traffic. Since after the update, the machine can only run for a few minutes and after that no more connection can go through. Ping from LAN to Internet is OK but when I telnet say to www.yahoo.com at port 80 and issue "GET / HTTP/1.0" I can see the data of different application including the HTML text. For example, I can see uTorrent packets with binaries and also the HTML page being cut short. It's as if, I'm seeing packets jumbled together from different application. I'm using PF to shape the traffic. If I reboot the server, it will panic and I have about 3 different vmcores in /var/crash and not sure what to do with it :( . I've tested the patch to remove stat_IfInFramesL2FilterDiscards but the problem still occurs. As for now, I'm not using the server to shape the traffic because I suspect the driver isn't reliable. I'm going to revert back to the previous driver and hopes its going to work. Sorry if there is not much detail since I'm not sure what to provide. Just tell me what to provide and I'd be happy to do so. Thanks! --=-zL3kbO00LxE6wEi3kKv6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEABECAAYFAklJ370ACgkQbWG64UQnRrLfsQCdGQERhrxqbwcOku1nmjtJ2SIJ JOcAn3eKSw/98/UwetyraOlHMJlkRVx6 =Zeto -----END PGP SIGNATURE----- --=-zL3kbO00LxE6wEi3kKv6-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 06:52:57 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E51B106564A for ; Thu, 18 Dec 2008 06:52:57 +0000 (UTC) (envelope-from paul@elehost.com) Received: from elephants.elehost.com (elephants.elehost.com [38.99.154.67]) by mx1.freebsd.org (Postfix) with ESMTP id 4AD228FC18 for ; Thu, 18 Dec 2008 06:52:57 +0000 (UTC) (envelope-from paul@elehost.com) X-Virus-Scanned: amavisd-new at elehost.com Received: from [192.168.1.126] (greenflowers.elehost.com [209.112.16.189]) (authenticated bits=0) by elephants.elehost.com (8.14.2/8.14.2) with ESMTP id mBHKrHvb040269 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 17 Dec 2008 15:53:18 -0500 (EST) (envelope-from paul@elehost.com) Message-ID: <4949673B.2070701@elehost.com> Date: Wed, 17 Dec 2008 15:55:23 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 06:52:57 -0000 > > [ns8]# vmstat -i > > interrupt total rate > > irq4: sio0 57065 0 > > irq17: em1 3989494045 554 > > irq18: arcmsr0 558098657 77 > > cpu0: timer 14381393929 2000 > > irq256: em0 22763077 3 > > cpu1: timer 14381384902 1999 > > Total 33333191675 4635 > > [ns8]# > > > > arcmsr0: >> > > mem 0xe8600000-0xe8600fff,0xe8000000-0xe83fffff irq 18 at device >> > > 14.0 on pci2 > > ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 > > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 > > arcmsr0: [ITHREAD] > > ..... > > Waiting 5 seconds for SCSI devices to settle > > (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step > > da0 at arcmsr0 bus 0 target 0 lun 0 > > da0: Fixed Direct Access SCSI-5 device > > da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > > da0: 305175MB (624999424 512 byte sectors: 255H 63S/T 38904C) > > SMP: AP CPU #1 Launched! > Hi and thanks for your reply. I do not believe the interrupts are the problem at the moment as the stats. Here is a vmstat when the system usage is spiking and just before http needs to be killed to get going again most recently. vmstat -i interrupt total rate irq1: atkbd0 2 0 irq4: sio0 22880 0 irq14: ata0 58 0 irq22: uhci1 uhci3 18068 0 irq23: uhci0 uhci+ 1 0 irq26: arcmsr0 496094 14 cpu0: timer 61769334 1791 irq256: em0 1 0 irq258: em2 48505 1 irq259: em3 1 0 cpu1: timer 61762043 1791 cpu3: timer 61299367 1777 cpu2: timer 61299179 1777 cpu4: timer 61326132 1778 cpu7: timer 60845245 1764 cpu5: timer 61326513 1778 cpu6: timer 60845018 1764 Total 491058441 14243 There are no errors en the event console for the areca-cli. ARC-1130-VOL#00 Main Raid Array Raid1+0 1000.0GB 00/00/00 Normal Main Raid Array 4 2000.0GB 0.0GB 1234 Normal Main Processor : 500MHz CPU ICache Size : 32KB CPU DCache Size : 32KB CPU SCache Size : 0KB System Memory : 1024MB/333MHz/ECC Firmware Version : V1.44 2008-2-1 BOOT ROM Version : V1.44 2008-1-28 The buildworld taking a really long time was just an example of the problem I am seeing that is easy to quantify. If I run boxbackup, dump, clamscan or a few other IO intensive everything gets VERY slow even when reading files from the server. When the HTTP locks up (another issue that is seen and is connected to the same issue in my view) this is what it looks like. It is almost as if the http gets backed up from what I can tell and I need a plunger to clean out the blockage :) I have to kill it and then restart it to get things back to normal for a bit. last pid: 46013; load averages: 105.30, 67.67, 34.45 up 4+23:59:42 19:08:40 629 processes: 89 running, 540 sleeping CPU: 21.9% user, 0.0% nice, 74.5% system, 3.1% interrupt, 0.4% idle Mem: 1538M Active, 11G Inact, 898M Wired, 303M Cache, 214M Buf, 1346M Free Swap: 8192M Total, 1036K Used, 8191M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 46000 www 1 65 0 86728K 15008K RUN 1 0:01 12.06% httpd 45994 www 1 56 0 86728K 15008K CPU1 3 0:01 10.16% httpd 46002 www 1 -4 0 150M 20648K RUN 3 0:00 6.98% httpd 45195 www 1 68 0 121M 19748K RUN 1 0:29 6.88% httpd 45991 www 1 53 0 150M 21060K select 3 0:01 6.59% httpd 45997 www 1 -4 0 150M 20992K ufs 5 0:01 6.59% httpd 45950 www 1 57 0 153M 23388K RUN 2 0:01 6.49% httpd 45999 www 1 -4 0 150M 20640K ufs 6 0:00 5.96% httpd 45189 www 1 66 0 161M 29660K RUN 6 0:26 5.76% httpd 45974 www 1 -4 0 151M 21564K ufs 3 0:01 5.76% httpd 45984 www 1 -4 0 151M 21376K ufs 5 0:01 5.66% httpd 45998 www 1 -4 0 150M 20652K ufs 3 0:00 5.57% httpd 45780 www 1 -4 0 153M 23516K ufs 6 0:06 5.37% httpd 45972 www 1 -4 0 151M 21332K RUN 4 0:01 5.37% httpd 46001 www 1 20 0 150M 20568K lockf 4 0:00 5.37% httpd 45425 www 1 60 0 164M 31516K RUN 7 0:15 5.18% httpd 45995 www 1 63 0 150M 20820K RUN 2 0:00 5.18% httpd 45845 www 1 -4 0 151M 21692K ufs 6 0:02 4.98% httpd 45854 www 1 52 0 151M 22080K CPU6 0 0:02 4.88% httpd 45977 root 1 47 0 10160K 3260K CPU2 6 0:01 4.88% top 45509 www 1 56 0 155M 25028K RUN 0 0:14 4.79% httpd 45735 www 1 -4 0 158M 27096K RUN 3 0:07 4.79% httpd 45730 www 1 20 0 151M 21728K lockf 2 0:04 4.79% httpd 45847 www 1 -4 0 150M 21092K RUN 5 0:02 4.69% httpd 85338 root 1 46 0 150M 20560K select 7 5:03 4.59% httpd 45835 www 1 -4 0 150M 21100K ufs 0 0:02 4.59% httpd 45443 www 1 -4 0 151M 22220K ufs 6 0:12 4.49% httpd 45699 www 1 -4 0 157M 26528K RUN 0 0:07 4.39% httpd 45722 www 1 -4 0 152M 22908K RUN 0 0:05 4.39% httpd 45701 www 1 -4 0 152M 22268K RUN 2 0:07 4.30% httpd 45849 www 1 -4 0 151M 21748K ufs 6 0:02 4.30% httpd 46010 nagios 1 -4 0 7684K 1136K ufs 5 0:00 4.30% check_ping 45515 www 1 -4 0 160M 29048K ufs 5 0:13 4.20% httpd 45606 www 1 -4 0 152M 22420K ufs 0 0:09 4.20% httpd vfs.numvnodes: 355382 kern.maxvnodes: 400000 vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 3239015 vfs.ufs.dirhash_maxmem: 10485760 vfs.ufs.dirhash_minsize: 2560 kern.ipc.nsfbufs: 0 kern.openfiles: 3395 kern.maxfiles: 12328 Results from netstat -m ------------------------ 1185/3360/4545 mbufs in use (current/cache/total) 1062/2856/3918/25600 mbuf clusters in use (current/cache/total/max) 1062/1556 mbuf+clusters out of packet secondary zone in use (current/cache) 10/1550/1560/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2460K/12752K/15212K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 46262 requests for I/O initiated by sendfile 0 calls to protocol drain routines Results from vmstat -m ------------------------ Type InUse MemUse HighUse Requests Size(s) cdev 22 6K - 22 256 acd_driver 1 2K - 1 2048 sigio 1 1K - 1626 64 filedesc 684 941K - 1199696 16,32,64,128,256,512,1024,2048,4096 kenv 68 11K - 70 16,32,64 kqueue 368 414K - 1740632 256,2048,4096 proc-args 52 4K - 5389885 16,32,64,128,256 ithread 99 19K - 99 32,128,256 acpisem 13 2K - 13 128 CAM queue 12 1K - 302 16,32,64,128,256 KTRACE 100 13K - 100 128 linker 45 4K - 71 16,32,64,128,512 lockf 314 38K - 16413112 64,128,256,512,1024,2048,4096 scsi_da 0 0K - 65 16 ip6ndp 7 1K - 7 64,128 ip6opt 1 1K - 50503 256 temp 66 222K - 6704801 16,32,64,128,256,512,1024,2048,4096 devbuf 16781 35476K - 108258 16,32,64,128,256,512,1024,2048,4096 CAM SIM 2 1K - 2 256 module 204 26K - 204 128 acpidev 78 5K - 78 64 mtx_pool 1 8K - 1 subproc 1111 1606K - 1045562 512,4096 proc 2 16K - 2 session 34 5K - 20772 128 pgrp 39 5K - 158890 128 cred 24950 6238K - 11839905 256 uidinfo 13 3K - 7337 64,2048 plimit 24 6K - 226179 256 CAM periph 7 2K - 45 16,32,64,128,256 sysctltmp 0 0K - 215050 16,32,64,128,256 sysctloid 4373 216K - 4373 16,32,64,128 sysctl 0 0K - 828292 16,32,64 umtx 1692 212K - 1692 128 p1003.1b 1 1K - 1 16 SWAP 2 1097K - 2 64 CAM XPT 51 24K - 19790153 32,64,128,256,1024 bus-sc 111 101K - 1879 16,32,64,128,256,512,1024,2048,4096 bus 804 77K - 5926 16,32,64,128,256,1024 devstat 10 21K - 10 32,4096 eventhandler 57 5K - 57 64,128 kobj 125 500K - 160 4096 kbdmux 6 9K - 6 16,256,512,2048,4096 rman 168 21K - 576 16,64,128 sbuf 0 0K - 840 16,32,64,128,256,512,1024,2048,4096 pci_link 16 2K - 16 16,128 stack 0 0K - 14 256 taskqueue 19 2K - 19 16,32,128 Unitno 16 1K - 22074 32,64 iov 0 0K - 12126863 16,64,128,256,512 ioctlops 0 0K - 388714 16,32,64,128,256,512,1024,2048 msg 4 30K - 4 2048,4096 sem 4 8K - 4 512,1024,2048,4096 shm 1 16K - 1 ttys 1170 169K - 80824 128,1024 ptys 5 2K - 5 256 accf 3 1K - 301 32,64 mbuf_tag 0 0K - 520852 32,128 pcb 47 158K - 1332310 16,32,128,1024,2048,4096 soname 187 23K - 10680643 16,32,128 biobuf 1 2K - 143707 2048 vfscache 1 1024K - 1 cl_savebuf 0 0K - 154293 64,128 vfs_hash 1 512K - 1 vnodes 2 1K - 3 32,256 vnodemarker 1 1K - 124142 512 mount 111 6K - 495 16,32,64,128,256,2048 acpi_perf 8 1K - 8 64 BPF 6 1K - 6 128 ether_multi 29 2K - 32 16,32,64 ifaddr 136 48K - 136 32,64,128,256,512,4096 ifnet 7 13K - 7 256,2048 clone 6 24K - 6 4096 arpcom 5 1K - 5 16 lo 1 1K - 1 32 acpica 3057 292K - 68659 16,32,64,128,256,512,1024 routetbl 303 86K - 1027 32,64,128 ,256,512 in_multi 4 1K - 4 64 IpFw/IpAcct 60 9K - 60 64,128,2048 sctp_iter 0 0K - 65 256 sctp_ifn 4 1K - 4 128 sctp_ifa 66 9K - 66 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 65 16 hostcache 1 36K - 1 entropy 1024 64K - 1024 64 syncache 1 100K - 1 in6_multi 16 1K - 16 32,64,128 audit_evclass 150 5K - 187 32 savedino 0 0K - 406078 256 newdirblk 0 0K - 5047 64 dirrem 18 2K - 2259617 64 mkdir 1 1K - 283528 64 diradd 183 12K - 3426340 64 freefile 55 4K - 1081462 64 freeblks 26 7K - 792864 256 freefrag 2 1K - 781740 64 allocindir 5 1K - 2842332 128 indirdep 4 1K - 116594 64 allocdirect 62 16K - 4832896 256 bmsafemap 12 2K - 271759 128 newblk 1 1K - 7675229 64,512 inodedep 270 580K - 2593883 256 pagedep 12 130K - 318828 128 ufs_dirhash 2848 1230K - 42435 16,32,64,128,256,512,1024,2048,4096 ufs_quota 1 512K - 1 ufs_mount 15 241K - 51 128,256,512,2048,4096 UMAHash 5 572K - 33 512,1024,2048,4096 USBHC 0 0K - 660 16 USBdev 22 10K - 682 16,128,512 USB 761 683K - 4079 16,32,64,128,256,1024 vm_pgdata 2 129K - 2 128 DEVFS1 115 58K - 115 512 DEVFS3 250 63K - 251 256 DEVFS2 115 2K - 115 16 DEVFS_RULE 36 17K - 36 64,512 DEVFS 30 1K - 31 16,128 io_apic 2 4K - 2 2048 pfs_nodes 20 5K - 20 256 memdesc 1 4K - 2 4096 msi 4 1K - 4 128 nexusdev 4 1K - 4 16 acpitask 0 0K - 9 64 GEOM 104 20K - 882 16,32,64,128,256,512,1024,2048 atkbddev 2 1K - 2 64 isadev 7 1K - 7 128 CAM dev queue 2 1K - 2 128 ata_generic 1 1K - 1 1024 ata_dma 1 1K - 1 256 Results from systat -v ----------------------- 1 users Load 143 90.86 47.13 Nov 21 19:10 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 1754100 25964 4719924 55728 1551492 count All 1916252 113912 9413004 269144 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 179 cow 16002 total 73 133 454 2 32k 816 2520 3 29k 726 504 zfod atkbd0 1 ozfod sio1 irq3 86.8%Sys 3.5%Intr 9.2%User 0.0%Nice 0.6%Idle %ozfod sio0 irq4 | | | | | | | | | | | daefr ata0 irq14 ===========================================++>>>>> 16 prcfr uhci1 uhci 314 dtbuf 90 totfr uhci0 uhci Namei Name-cache Dir-cache 400000 desvn react 2 arcmsr0 26 Calls hits % hits % 355344 numvn pdwak 2004 cpu0: time 76763 76730 100 24902 frevn pdpgs em0 irq256 intrn 1 em2 irq258 Disks da0 da1 pass0 pass1 pass2 934624 wire em3 irq259 KB/t 9.00 0.00 0.00 0.00 0.00 1697060 act 2000 cpu1: time tps 1 0 0 0 0 12038912 inact 1996 cpu2: time MB/s 0.01 0.00 0.00 0.00 0.00 308732 cache 2000 cpu3: time %busy 0 0 0 0 0 1244784 free 2001 cpu7: time 219632 buf 1999 cpu4: time 1999 cpu6: time 2000 cpu5: time Here is a "normal" sysstat -v to compare when there are no "visible" problems: 3 users Load 1.67 1.03 1.02 Nov 25 22:12 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 797576 31388 2318500 57324 4051340 count All 952256 114916 6781828 226696 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 556 cow 16001 total 1 6 474 5463 1602 3387 1 31k 1567 853 zfod atkbd0 1 ozfod sio1 irq3 8.4%Sys 4.0%Intr 2.8%User 0.0%Nice 84.8%Idle %ozfod sio0 irq4 | | | | | | | | | | | daefr ata0 irq14 ====++>> 602 prcfr uhci1 uhci 125 dtbuf 1443 totfr uhci0 uhci Namei Name-cache Dir-cache 400000 desvn react arcmsr0 26 Calls hits % hits % 328748 numvn pdwak 2026 cpu0: time 52734 52660 100 24705 frevn pdpgs em0 irq256 intrn 1 em2 irq258 Disks da0 da1 pass0 pass1 pass2 857028 wire em3 irq259 KB/t 0.00 0.00 0.00 0.00 0.00 750716 act 2026 cpu1: time tps 0 0 0 0 0 10564316 inact 1975 cpu2: time MB/s 0.00 0.00 0.00 0.00 0.00 303468 cache 1977 cpu3: time %busy 0 0 0 0 0 3748056 free 1999 cpu7: time 219632 buf 2000 cpu4: time 1997 cpu6: time 2000 cpu5: time ---------------------------------- Here are the results of vmstat -w 1 during the problem: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 157 110 13 5544M 1111M 1141 0 0 0 1100 44 0 0 47 115 8744 7 14 79 146 34 98 5546M 1099M 4191 0 0 0 729 0 2 0 17 18583 102586 9 91 0 224 33 15 5548M 1091M 3825 0 0 0 664 0 0 0 7 14115 141707 10 90 0 165 103 11 5633M 1064M 12222 0 0 0 6745 0 2 0 42 41519 403437 14 86 0 214 73 4 5653M 1044M 4539 0 0 0 959 0 0 0 7 15698 94269 11 88 1 260 30 1 5664M 1034M 8457 0 0 0 2171 0 0 0 14 36978 248202 12 87 0 57 182 45 5667M 1029M 4761 0 0 0 2535 0 0 0 6 21004 133617 10 90 0 55 24 16 2152M 2454M 7993 0 0 0 3135 0 0 0 13 20263 173347 13 81 7 20 15 2 1972M 2537M 93820 0 0 0 465955 0 10 0 588 99274 716238 23 76 1 13 11 0 1877M 2581M 7820 0 0 0 31044 0 6 0 38 7859 76120 16 83 1 9 12 1 1816M 2599M 6889 0 0 0 14550 0 20 0 79 359198 21333 14 79 7 11 13 0 1797M 2613M 6542 0 0 0 8416 0 3 0 17 606119 15341 18 61 21 1 9 1 1740M 2636M 1744 0 0 0 6267 0 2 0 14 11617 15322 8 63 29 2 3 0 1694M 2657M 3417 0 0 0 8669 0 15 0 52 50341 12045 6 29 65 Here is another view of top at a later date with the same problem happening focusing on IO setting in Top: -------------------------------------------------------------- last pid: 17984; load averages: 39.26, 37.68, 24.75 up 8+09:25:55 04:34:53 539 processes: 59 running, 480 sleeping CPU: 9.8% user, 0.5% nice, 87.0% system, 2.3% interrupt, 0.4% idle Mem: 1146M Active, 9663M Inact, 875M Wired, 582M Cache, 214M Buf, 3577M Free Swap: 8192M Total, 1036K Used, 8191M Free PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 17587 www 446 62 0 0 0 0 0.00% httpd 17763 www 515 37 0 0 0 0 0.00% httpd 17860 www 538 47 0 0 0 0 0.00% httpd 17703 www 457 43 0 0 0 0 0.00% httpd 17701 www 485 34 0 0 0 0 0.00% httpd 17550 www 423 29 0 0 0 0 0.00% httpd 17579 www 0 0 0 0 0 0 0.00% httpd 17864 www 495 39 0 0 0 0 0.00% httpd 17836 www 520 36 0 0 0 0 0.00% httpd 17847 www 451 28 0 0 0 0 0.00% httpd 17756 www 462 29 0 0 0 0 0.00% httpd 17982 www 445 63 0 0 0 0 0.00% httpd 17581 www 451 60 0 0 0 0 0.00% httpd 17761 www 449 37 0 0 0 0 0.00% httpd 17582 www 509 30 0 0 0 0 0.00% httpd 17709 www 447 28 0 0 0 0 0.00% httpd 17705 www 515 30 0 0 0 0 0.00% httpd 17704 www 469 38 0 0 0 0 0.00% httpd 17706 www 508 53 0 0 0 0 0.00% httpd 17833 www 483 34 0 0 0 0 0.00% httpd 17834 www 499 43 0 0 0 0 0.00% httpd 17974 www 489 38 0 0 0 0 0.00% httpd 17978 www 467 45 0 0 0 0 0.00% httpd 17576 www 447 32 0 0 0 0 0.00% httpd 17570 www 443 37 0 0 0 0 0.00% httpd 17762 www 476 31 0 0 0 0 0.00% httpd 17837 www 508 44 0 0 0 0 0.00% httpd 17548 www 443 32 0 0 0 0 0.00% httpd 17783 www 390 22 0 0 0 0 0.00% httpd 17961 www 534 57 0 0 0 0 0.00% httpd 17590 www 498 50 0 0 0 0 0.00% httpd 17700 www 471 35 0 0 0 0 0.00% httpd 17580 www 438 41 0 0 0 0 0.00% httpd This used to be on a 4.11x system with 1 cpu and only 1gb of ram and ran flawlessly with much less resources with the same web site code for a long time. I do not have this problem on the other 7.0 machine. I originally thought it was just a cpu issue but it is very closely tied to when something is trying to use the raid arrays and this seems to be the way to reproduce it. I am having a hard time determining why the system load is so high. Can you recommend the best way to identify the culprit? Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 06:52:57 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E35C71065670 for ; Thu, 18 Dec 2008 06:52:57 +0000 (UTC) (envelope-from paul@elehost.com) Received: from elephants.elehost.com (elephants.elehost.com [38.99.154.67]) by mx1.freebsd.org (Postfix) with ESMTP id C45648FC14 for ; Thu, 18 Dec 2008 06:52:57 +0000 (UTC) (envelope-from paul@elehost.com) X-Virus-Scanned: amavisd-new at elehost.com Received: from [192.168.1.126] (greenflowers.elehost.com [209.112.16.189]) (authenticated bits=0) by elephants.elehost.com (8.14.2/8.14.2) with ESMTP id mBHKt7Gc040297 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 17 Dec 2008 15:55:08 -0500 (EST) (envelope-from paul@elehost.com) Message-ID: <494967A9.3000503@elehost.com> Date: Wed, 17 Dec 2008 15:57:13 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 06:52:58 -0000 > > [ns8]# vmstat -i > > interrupt total rate > > irq4: sio0 57065 0 > > irq17: em1 3989494045 554 > > irq18: arcmsr0 558098657 77 > > cpu0: timer 14381393929 2000 > > irq256: em0 22763077 3 > > cpu1: timer 14381384902 1999 > > Total 33333191675 4635 > > [ns8]# > > > > arcmsr0: >> > > mem 0xe8600000-0xe8600fff,0xe8000000-0xe83fffff irq 18 at device >> > > 14.0 on pci2 > > ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 > > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 > > arcmsr0: [ITHREAD] > > ..... > > Waiting 5 seconds for SCSI devices to settle > > (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step > > da0 at arcmsr0 bus 0 target 0 lun 0 > > da0: Fixed Direct Access SCSI-5 device > > da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > > da0: 305175MB (624999424 512 byte sectors: 255H 63S/T 38904C) > > SMP: AP CPU #1 Launched! > Hi and thanks for your reply. I do not believe the interrupts are the problem at the moment as the stats. Here is a vmstat when the system usage is spiking and just before http needs to be killed to get going again most recently. vmstat -i interrupt total rate irq1: atkbd0 2 0 irq4: sio0 22880 0 irq14: ata0 58 0 irq22: uhci1 uhci3 18068 0 irq23: uhci0 uhci+ 1 0 irq26: arcmsr0 496094 14 cpu0: timer 61769334 1791 irq256: em0 1 0 irq258: em2 48505 1 irq259: em3 1 0 cpu1: timer 61762043 1791 cpu3: timer 61299367 1777 cpu2: timer 61299179 1777 cpu4: timer 61326132 1778 cpu7: timer 60845245 1764 cpu5: timer 61326513 1778 cpu6: timer 60845018 1764 Total 491058441 14243 There are no errors en the event console for the areca-cli. ARC-1130-VOL#00 Main Raid Array Raid1+0 1000.0GB 00/00/00 Normal Main Raid Array 4 2000.0GB 0.0GB 1234 Normal Main Processor : 500MHz CPU ICache Size : 32KB CPU DCache Size : 32KB CPU SCache Size : 0KB System Memory : 1024MB/333MHz/ECC Firmware Version : V1.44 2008-2-1 BOOT ROM Version : V1.44 2008-1-28 The buildworld taking a really long time was just an example of the problem I am seeing that is easy to quantify. If I run boxbackup, dump, clamscan or a few other IO intensive everything gets VERY slow even when reading files from the server. When the HTTP locks up (another issue that is seen and is connected to the same issue in my view) this is what it looks like. It is almost as if the http gets backed up from what I can tell and I need a plunger to clean out the blockage :) I have to kill it and then restart it to get things back to normal for a bit. last pid: 46013; load averages: 105.30, 67.67, 34.45 up 4+23:59:42 19:08:40 629 processes: 89 running, 540 sleeping CPU: 21.9% user, 0.0% nice, 74.5% system, 3.1% interrupt, 0.4% idle Mem: 1538M Active, 11G Inact, 898M Wired, 303M Cache, 214M Buf, 1346M Free Swap: 8192M Total, 1036K Used, 8191M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 46000 www 1 65 0 86728K 15008K RUN 1 0:01 12.06% httpd 45994 www 1 56 0 86728K 15008K CPU1 3 0:01 10.16% httpd 46002 www 1 -4 0 150M 20648K RUN 3 0:00 6.98% httpd 45195 www 1 68 0 121M 19748K RUN 1 0:29 6.88% httpd 45991 www 1 53 0 150M 21060K select 3 0:01 6.59% httpd 45997 www 1 -4 0 150M 20992K ufs 5 0:01 6.59% httpd 45950 www 1 57 0 153M 23388K RUN 2 0:01 6.49% httpd 45999 www 1 -4 0 150M 20640K ufs 6 0:00 5.96% httpd 45189 www 1 66 0 161M 29660K RUN 6 0:26 5.76% httpd 45974 www 1 -4 0 151M 21564K ufs 3 0:01 5.76% httpd 45984 www 1 -4 0 151M 21376K ufs 5 0:01 5.66% httpd 45998 www 1 -4 0 150M 20652K ufs 3 0:00 5.57% httpd 45780 www 1 -4 0 153M 23516K ufs 6 0:06 5.37% httpd 45972 www 1 -4 0 151M 21332K RUN 4 0:01 5.37% httpd 46001 www 1 20 0 150M 20568K lockf 4 0:00 5.37% httpd 45425 www 1 60 0 164M 31516K RUN 7 0:15 5.18% httpd 45995 www 1 63 0 150M 20820K RUN 2 0:00 5.18% httpd 45845 www 1 -4 0 151M 21692K ufs 6 0:02 4.98% httpd 45854 www 1 52 0 151M 22080K CPU6 0 0:02 4.88% httpd 45977 root 1 47 0 10160K 3260K CPU2 6 0:01 4.88% top 45509 www 1 56 0 155M 25028K RUN 0 0:14 4.79% httpd 45735 www 1 -4 0 158M 27096K RUN 3 0:07 4.79% httpd 45730 www 1 20 0 151M 21728K lockf 2 0:04 4.79% httpd 45847 www 1 -4 0 150M 21092K RUN 5 0:02 4.69% httpd 85338 root 1 46 0 150M 20560K select 7 5:03 4.59% httpd 45835 www 1 -4 0 150M 21100K ufs 0 0:02 4.59% httpd 45443 www 1 -4 0 151M 22220K ufs 6 0:12 4.49% httpd 45699 www 1 -4 0 157M 26528K RUN 0 0:07 4.39% httpd 45722 www 1 -4 0 152M 22908K RUN 0 0:05 4.39% httpd 45701 www 1 -4 0 152M 22268K RUN 2 0:07 4.30% httpd 45849 www 1 -4 0 151M 21748K ufs 6 0:02 4.30% httpd 46010 nagios 1 -4 0 7684K 1136K ufs 5 0:00 4.30% check_ping 45515 www 1 -4 0 160M 29048K ufs 5 0:13 4.20% httpd 45606 www 1 -4 0 152M 22420K ufs 0 0:09 4.20% httpd vfs.numvnodes: 355382 kern.maxvnodes: 400000 vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 3239015 vfs.ufs.dirhash_maxmem: 10485760 vfs.ufs.dirhash_minsize: 2560 kern.ipc.nsfbufs: 0 kern.openfiles: 3395 kern.maxfiles: 12328 Results from netstat -m ------------------------ 1185/3360/4545 mbufs in use (current/cache/total) 1062/2856/3918/25600 mbuf clusters in use (current/cache/total/max) 1062/1556 mbuf+clusters out of packet secondary zone in use (current/cache) 10/1550/1560/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 2460K/12752K/15212K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 46262 requests for I/O initiated by sendfile 0 calls to protocol drain routines Results from vmstat -m ------------------------ Type InUse MemUse HighUse Requests Size(s) cdev 22 6K - 22 256 acd_driver 1 2K - 1 2048 sigio 1 1K - 1626 64 filedesc 684 941K - 1199696 16,32,64,128,256,512,1024,2048,4096 kenv 68 11K - 70 16,32,64 kqueue 368 414K - 1740632 256,2048,4096 proc-args 52 4K - 5389885 16,32,64,128,256 ithread 99 19K - 99 32,128,256 acpisem 13 2K - 13 128 CAM queue 12 1K - 302 16,32,64,128,256 KTRACE 100 13K - 100 128 linker 45 4K - 71 16,32,64,128,512 lockf 314 38K - 16413112 64,128,256,512,1024,2048,4096 scsi_da 0 0K - 65 16 ip6ndp 7 1K - 7 64,128 ip6opt 1 1K - 50503 256 temp 66 222K - 6704801 16,32,64,128,256,512,1024,2048,4096 devbuf 16781 35476K - 108258 16,32,64,128,256,512,1024,2048,4096 CAM SIM 2 1K - 2 256 module 204 26K - 204 128 acpidev 78 5K - 78 64 mtx_pool 1 8K - 1 subproc 1111 1606K - 1045562 512,4096 proc 2 16K - 2 session 34 5K - 20772 128 pgrp 39 5K - 158890 128 cred 24950 6238K - 11839905 256 uidinfo 13 3K - 7337 64,2048 plimit 24 6K - 226179 256 CAM periph 7 2K - 45 16,32,64,128,256 sysctltmp 0 0K - 215050 16,32,64,128,256 sysctloid 4373 216K - 4373 16,32,64,128 sysctl 0 0K - 828292 16,32,64 umtx 1692 212K - 1692 128 p1003.1b 1 1K - 1 16 SWAP 2 1097K - 2 64 CAM XPT 51 24K - 19790153 32,64,128,256,1024 bus-sc 111 101K - 1879 16,32 ,64,128,256,512,1024,2048,4096 bus 804 77K - 5926 16,32,64,128,256,1024 devstat 10 21K - 10 32,4096 eventhandler 57 5K - 57 64,128 kobj 125 500K - 160 4096 kbdmux 6 9K - 6 16,256,512,2048,4096 rman 168 21K - 576 16,64,128 sbuf 0 0K - 840 16,32,64,128,256,512,1024,2048,4096 pci_link 16 2K - 16 16,128 stack 0 0K - 14 256 taskqueue 19 2K - 19 16,32,128 Unitno 16 1K - 22074 32,64 iov 0 0K - 12126863 16,64,128,256,512 ioctlops 0 0K - 388714 16,32,64,128,256,512,1024,2048 msg 4 30K - 4 2048,4096 sem 4 8K - 4 512,1024,2048,4096 shm 1 16K - 1 ttys 1170 169K - 80824 128,1024 ptys 5 2K - 5 256 accf 3 1K - 301 32,64 mbuf_tag 0 0K - 520852 32,128 pcb 47 158K - 1332310 16,32,128,1024,2048,4096 soname 187 23K - 10680643 16,32,128 biobuf 1 2K - 143707 2048 vfscache 1 1024K - 1 cl_savebuf 0 0K - 154293 64,128 vfs_hash 1 512K - 1 vnodes 2 1K - 3 32,256 vnodemarker 1 1K - 124142 512 mount 111 6K - 495 16,32,64,128,256,2048 acpi_perf 8 1K - 8 64 BPF 6 1K - 6 128 ether_multi 29 2K - 32 16,32,64 ifaddr 136 48K - 136 32,64,128,256,512,4096 ifnet 7 13K - 7 256,2048 clone 6 24K - 6 4096 arpcom 5 1K - 5 16 lo 1 1K - 1 32 acpica 3057 292K - 68659 16,32,64,128,256,512,1024 routetbl 303 86K - 1027 32,64,128 ,256,512 in_multi 4 1K - 4 64 IpFw/IpAcct 60 9K - 60 64,128,2048 sctp_iter 0 0K - 65 256 sctp_ifn 4 1K - 4 128 sctp_ifa 66 9K - 66 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 65 16 hostcache 1 36K - 1 entropy 1024 64K - 1024 64 syncache 1 100K - 1 in6_multi 16 1K - 16 32,64,128 audit_evclass 150 5K - 187 32 savedino 0 0K - 406078 256 newdirblk 0 0K - 5047 64 dirrem 18 2K - 2259617 64 mkdir 1 1K - 283528 64 diradd 183 12K - 3426340 64 freefile 55 4K - 1081462 64 freeblks 26 7K - 792864 256 freefrag 2 1K - 781740 64 allocindir 5 1K - 2842332 128 indirdep 4 1K - 116594 64 allocdirect 62 16K - 4832896 256 bmsafemap 12 2K - 271759 128 newblk 1 1K - 7675229 64,512 inodedep 270 580K - 2593883 256 pagedep 12 130K - 318828 128 ufs_dirhash 2848 1230K - 42435 16,32,64,128,256,512,1024,2048,4096 ufs_quota 1 512K - 1 ufs_mount 15 241K - 51 128,256,512,2048,4096 UMAHash 5 572K - 33 512,1024,2048,4096 USBHC 0 0K - 660 16 USBdev 22 10K - 682 16,128,512 USB 761 683K - 4079 16,32,64,128,256,1024 vm_pgdata 2 129K - 2 128 DEVFS1 115 58K - 115 512 DEVFS3 250 63K - 251 256 DEVFS2 115 2K - 115 16 DEVFS_RULE 36 17K - 36 64,512 DEVFS 30 1K - 31 16,128 io_apic 2 4K - 2 2048 pfs_nodes 20 5K - 20 256 memdesc 1 4K - 2 4096 msi 4 1K - 4 128 nexusdev 4 1K - 4 16 acpitask 0 0K - 9 64 GEOM 104 20K - 882 16,32,64,128,256,512,1024,2048 atkbddev 2 1K - 2 64 isadev 7 1K - 7 128 CAM dev queue 2 1K - 2 128 ata_generic 1 1K - 1 1024 ata_dma 1 1K - 1 256 Results from systat -v ----------------------- 1 users Load 143 90.86 47.13 Nov 21 19:10 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 1754100 25964 4719924 55728 1551492 count All 1916252 113912 9413004 269144 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 179 cow 16002 total 73 133 454 2 32k 816 2520 3 29k 726 504 zfod atkbd0 1 ozfod sio1 irq3 86.8%Sys 3.5%Intr 9.2%User 0.0%Nice 0.6%Idle %ozfod sio0 irq4 | | | | | | | | | | | daefr ata0 irq14 ===========================================++>>>>> 16 prcfr uhci1 uhci 314 dtbuf 90 totfr uhci0 uhci Namei Name-cache Dir-cache 400000 desvn react 2 arcmsr0 26 Calls hits % hits % 355344 numvn pdwak 2004 cpu0: time 76763 76730 100 24902 frevn pdpgs em0 irq256 intrn 1 em2 irq258 Disks da0 da1 pass0 pass1 pass2 934624 wire em3 irq259 KB/t 9.00 0.00 0.00 0.00 0.00 1697060 act 2000 cpu1: time tps 1 0 0 0 0 12038912 inact 1996 cpu2: time MB/s 0.01 0.00 0.00 0.00 0.00 308732 cache 2000 cpu3: time %busy 0 0 0 0 0 1244784 free 2001 cpu7: time 219632 buf 1999 cpu4: time 1999 cpu6: time 2000 cpu5: time Here is a "normal" sysstat -v to compare when there are no "visible" problems: 3 users Load 1.67 1.03 1.02 Nov 25 22:12 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 797576 31388 2318500 57324 4051340 count All 952256 114916 6781828 226696 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 556 cow 16001 total 1 6 474 5463 1602 3387 1 31k 1567 853 zfod atkbd0 1 ozfod sio1 irq3 8.4%Sys 4.0%Intr 2.8%User 0.0%Nice 84.8%Idle %ozfod sio0 irq4 | | | | | | | | | | | daefr ata0 irq14 ====++>> 602 prcfr uhci1 uhci 125 dtbuf 1443 totfr uhci0 uhci Namei Name-cache Dir-cache 400000 desvn react arcmsr0 26 Calls hits % hits % 328748 numvn pdwak 2026 cpu0: time 52734 52660 100 24705 frevn pdpgs em0 irq256 intrn 1 em2 irq258 Disks da0 da1 pass0 pass1 pass2 857028 wire em3 irq259 KB/t 0.00 0.00 0.00 0.00 0.00 750716 act 2026 cpu1: time tps 0 0 0 0 0 10564316 inact 1975 cpu2: time MB/s 0.00 0.00 0.00 0.00 0.00 303468 cache 1977 cpu3: time %busy 0 0 0 0 0 3748056 free 1999 cpu7: time 219632 buf 2000 cpu4: time 1997 cpu6: time 2000 cpu5: time ---------------------------------- Here are the results of vmstat -w 1 during the problem: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 157 110 13 5544M 1111M 1141 0 0 0 1100 44 0 0 47 115 8744 7 14 79 146 34 98 5546M 1099M 4191 0 0 0 729 0 2 0 17 18583 102586 9 91 0 224 33 15 5548M 1091M 3825 0 0 0 664 0 0 0 7 14115 141707 10 90 0 165 103 11 5633M 1064M 12222 0 0 0 6745 0 2 0 42 41519 403437 14 86 0 214 73 4 5653M 1044M 4539 0 0 0 959 0 0 0 7 15698 94269 11 88 1 260 30 1 5664M 1034M 8457 0 0 0 2171 0 0 0 14 36978 248202 12 87 0 57 182 45 5667M 1029M 4761 0 0 0 2535 0 0 0 6 21004 133617 10 90 0 55 24 16 2152M 2454M 7993 0 0 0 3135 0 0 0 13 20263 173347 13 81 7 20 15 2 1972M 2537M 93820 0 0 0 465955 0 10 0 588 99274 716238 23 76 1 13 11 0 1877M 2581M 7820 0 0 0 31044 0 6 0 38 7859 76120 16 83 1 9 12 1 1816M 2599M 6889 0 0 0 14550 0 20 0 79 359198 21333 14 79 7 11 13 0 1797M 2613M 6542 0 0 0 8416 0 3 0 17 606119 15341 18 61 21 1 9 1 1740M 2636M 1744 0 0 0 6267 0 2 0 14 11617 15322 8 63 29 2 3 0 1694M 2657M 3417 0 0 0 8669 0 15 0 52 50341 12045 6 29 65 Here is another view of top at a later date with the same problem happening focusing on IO setting in Top: -------------------------------------------------------------- last pid: 17984; load averages: 39.26, 37.68, 24.75 up 8+09:25:55 04:34:53 539 processes: 59 running, 480 sleeping CPU: 9.8% user, 0.5% nice, 87.0% system, 2.3% interrupt, 0.4% idle Mem: 1146M Active, 9663M Inact, 875M Wired, 582M Cache, 214M Buf, 3577M Free Swap: 8192M Total, 1036K Used, 8191M Free PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 17587 www 446 62 0 0 0 0 0.00% httpd 17763 www 515 37 0 0 0 0 0.00% httpd 17860 www 538 47 0 0 0 0 0.00% httpd 17703 www 457 43 0 0 0 0 0.00% httpd 17701 www 485 34 0 0 0 0 0.00% httpd 17550 www 423 29 0 0 0 0 0.00% httpd 17579 www 0 0 0 0 0 0 0.00% httpd 17864 www 495 39 0 0 0 0 0.00% httpd 17836 www 520 36 0 0 0 0 0.00% httpd 17847 www 451 28 0 0 0 0 0.00% httpd 17756 www 462 29 0 0 0 0 0.00% httpd 17982 www 445 63 0 0 0 0 0.00% httpd 17581 www 451 60 0 0 0 0 0.00% httpd 17761 www 449 37 0 0 0 0 0.00% httpd 17582 www 509 30 0 0 0 0 0.00% httpd 17709 www 447 28 0 0 0 0 0.00% httpd 17705 www 515 30 0 0 0 0 0.00% httpd 17704 www 469 38 0 0 0 0 0.00% httpd 17706 www 508 53 0 0 0 0 0.00% httpd 17833 www 483 34 0 0 0 0 0.00% httpd 17834 www 499 43 0 0 0 0 0.00% httpd 17974 www 489 38 0 0 0 0 0.00% httpd 17978 www 467 45 0 0 0 0 0.00% httpd 17576 www 447 32 0 0 0 0 0.00% httpd 17570 www 443 37 0 0 0 0 0.00% httpd 17762 www 476 31 0 0 0 0 0.00% httpd 17837 www 508 44 0 0 0 0 0.00% httpd 17548 www 443 32 0 0 0 0 0.00% httpd 17783 www 390 22 0 0 0 0 0.00% httpd 17961 www 534 57 0 0 0 0 0.00% httpd 17590 www 498 50 0 0 0 0 0.00% httpd 17700 www 471 35 0 0 0 0 0.00% httpd 17580 www 438 41 0 0 0 0 0.00% httpd This used to be on a 4.11x system with 1 cpu and only 1gb of ram and ran flawlessly with much less resources with the same web site code for a long time. I do not have this problem on the other 7.0 machine. I originally thought it was just a cpu issue but it is very closely tied to when something is trying to use the raid arrays and this seems to be the way to reproduce it. I am having a hard time determining why the system load is so high. Can you recommend the best way to identify the culprit? Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 07:21:25 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDAAD1065673 for ; Thu, 18 Dec 2008 07:21:25 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4074E8FC08 for ; Thu, 18 Dec 2008 07:21:25 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [211.166.10.233]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id 13B3C28485 for ; Thu, 18 Dec 2008 15:21:24 +0800 (CST) Received: from localhost (tarsier.geekcn.org [211.166.10.233]) by tarsier.geekcn.org (Postfix) with ESMTP id 895E0EB1A81; Thu, 18 Dec 2008 15:21:23 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([211.166.10.233]) by localhost (mail.geekcn.org [211.166.10.233]) (amavisd-new, port 10024) with ESMTP id ljV5eg2NTj+f; Thu, 18 Dec 2008 15:21:18 +0800 (CST) Received: from charlie.delphij.net (c-67-180-38-12.hsd1.ca.comcast.net [67.180.38.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id E3AFDEB18F0; Thu, 18 Dec 2008 15:21:16 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:references:in-reply-to: x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=vEF47vPFrHWuEgVlUgGrSkkOx9BuwLA7dVoKiNXJiigWJDYKV5p/dlk4hhzXx9sVV bJJmdN7gMffLuixWkM3gg== Message-ID: <4949F9E7.6030906@delphij.net> Date: Wed, 17 Dec 2008 23:21:11 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: nawfal@googlemail.com References: <4935069A.8060209@ec-marseille.fr> <49357BD0.4000008@delphij.net> <4935944A.9090509@ec-marseille.fr> <4935C453.8070301@delphij.net> <4935D67E.4070204@delphij.net> <4936F8C4.6090006@ec-marseille.fr> <49399FA6.3060108@delphij.net> <493CE8F7.5010204@yandex-team.ru> <3a6fb7145a0a6c8af136ea1a0824e5ed.squirrel@wettoast.dyndns.org> <493D9BC6.8050902@delphij.net> <4fe0419b44a3a4c4a28b1e60fbb3a3c8.squirrel@wettoast.dyndns.org> <56272b131067237ccabd23de5f669458.squirrel@wettoast.dyndns.org> <4944D894.6070306@delphij.net> <400809c165ecedf8a0b7bc6b569e1289.squirrel@wettoast.dyndns.org> <1229578180.6319.21.camel@nawfal-desktop> In-Reply-To: <1229578180.6319.21.camel@nawfal-desktop> X-Enigmail-Version: 0.95.7 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable Subject: Re: RELENG_7_1: bce driver change generating too much interrupts ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 07:21:26 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Nawfal, Nawfal bin Mohmad Rouyan wrote: > I have been using a Dell machine with 2 bce interfaces as a bridge > between my LAN and Firewall to shape the traffic. Since after the > update, the machine can only run for a few minutes and after that no > more connection can go through. > > Ping from LAN to Internet is OK but when I telnet say to www.yahoo.com > at port 80 and issue "GET / HTTP/1.0" I can see the data of different > application including the HTML text. > > For example, I can see uTorrent packets with binaries and also the HTML > page being cut short. It's as if, I'm seeing packets jumbled together > from different application. > > I'm using PF to shape the traffic. If I reboot the server, it will panic > and I have about 3 different vmcores in /var/crash and not sure what to > do with it :( . I've tested the patch to remove > stat_IfInFramesL2FilterDiscards but the problem still occurs. The last patch is not a functional change, but a behavior change that removes the L2FilterDiscards from being counted to match previous behavior. Would you please do this: script bt.txt kgdb /boot/kernel/kernel.symbols /var/crash/vmcore.0 Then, do 'bt', press enter until all display has finished, then exit kgdb, and send me the result (bt.txt)? > As for now, I'm not using the server to shape the traffic because I > suspect the driver isn't reliable. I'm going to revert back to the > previous driver and hopes its going to work. > > Sorry if there is not much detail since I'm not sure what to provide. > Just tell me what to provide and I'd be happy to do so. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAklJ+eYACgkQi+vbBBjt66AM9wCePPUHQYhaW07lyhEzDIOqDEri da4AoKsXxzqjKZiJIAmlGaNaKmPCZboi =mdH7 -----END PGP SIGNATURE----- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 08:48:28 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6FA21065673 for ; Thu, 18 Dec 2008 08:48:27 +0000 (UTC) (envelope-from test1000@cogeco.ca) Received: from fep7.cogeco.net (smtp2.cogeco.ca [216.221.81.29]) by mx1.freebsd.org (Postfix) with ESMTP id 6D55E8FC2A for ; Thu, 18 Dec 2008 08:48:27 +0000 (UTC) (envelope-from test1000@cogeco.ca) Received: from cogeco.ca (smtp.cogeco.net [216.221.81.25]) by fep7.cogeco.net (Postfix) with SMTP id 8026F1B77 for ; Wed, 17 Dec 2008 15:39:10 -0500 (EST) Sender: test1000@cogeco.ca From: test1000@cogeco.ca To: freebsd-stable@freebsd.org X-Mailer: Cogeco Webmail - complaints to abuse@cogeco.ca ( 24.226.1.232 - test1000@cogeco.ca ) X-Originating-IP: 24.226.1.232 Date: Wed, 17 Dec 2008 15:39:09 -0500 X-Priority: 3 (Normal) Message-id: <4949636d.3da.4bc2.9176@cogeco.ca> Subject: Test X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: test1000@cogeco.ca List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 08:48:29 -0000 This is a test message from Cogeco Cable. No reply is necessary. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 09:55:46 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B6F41065673 for ; Thu, 18 Dec 2008 09:55:46 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 03DC58FC16 for ; Thu, 18 Dec 2008 09:55:45 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1LDFbW-0002Iy-VS for freebsd-stable@freebsd.org; Thu, 18 Dec 2008 09:55:39 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 18 Dec 2008 09:55:38 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 18 Dec 2008 09:55:38 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Thu, 18 Dec 2008 10:55:33 +0100 Lines: 60 Message-ID: References: <4949673B.2070701@elehost.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig2B65B221094CE6B1B6C7649D" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.18 (X11/20081125) In-Reply-To: <4949673B.2070701@elehost.com> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 09:55:46 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig2B65B221094CE6B1B6C7649D Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Paul MacKenzie wrote: > last pid: 46013; load averages: 105.30, 67.67, > 34.45 up 4+23:59:42 19:0= 8:40 > 629 processes: 89 running, 540 sleeping > CPU: 21.9% user, 0.0% nice, 74.5% system, 3.1% interrupt, 0.4% idle > Mem: 1538M Active, 11G Inact, 898M Wired, 303M Cache, 214M Buf, 1346M F= ree > Swap: 8192M Total, 1036K Used, 8191M Free >=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 46000 www 1 65 0 86728K 15008K RUN 1 0:01 12.06% = httpd > 45994 www 1 56 0 86728K 15008K CPU1 3 0:01 10.16% = httpd > 46002 www 1 -4 0 150M 20648K RUN 3 0:00 6.98% = httpd > 45195 www 1 68 0 121M 19748K RUN 1 0:29 6.88% = httpd > 45991 www 1 53 0 150M 21060K select 3 0:01 6.59% = httpd > 45997 www 1 -4 0 150M 20992K ufs 5 0:01 6.59% = httpd > 45950 www 1 57 0 153M 23388K RUN 2 0:01 6.49% = httpd > 45999 www 1 -4 0 150M 20640K ufs 6 0:00 5.96% = httpd > 45189 www 1 66 0 161M 29660K RUN 6 0:26 5.76% = httpd > 45974 www 1 -4 0 151M 21564K ufs 3 0:01 5.76% = httpd The number of httpd processes in "ufs" state is too high. Are you using PHP? And if you do, check how large is your PHP sessions' directory. Use a "sharded" layout if it's of any significant size (see php.ini for details). --------------enig2B65B221094CE6B1B6C7649D Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJSh4VldnAQVacBcgRAirRAJ4+dqpZ025LP47b09WXnH/T0GQzggCgmCAd Udfjd7uGX5jmhhPVLgDEkpA= =l4Wq -----END PGP SIGNATURE----- --------------enig2B65B221094CE6B1B6C7649D-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 10:00:33 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47EB51065673 for ; Thu, 18 Dec 2008 10:00:33 +0000 (UTC) (envelope-from prvs=1238cb8453=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id C9E508FC12 for ; Thu, 18 Dec 2008 10:00:32 +0000 (UTC) (envelope-from prvs=1238cb8453=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1229593522; x=1230198322; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=DeRB4J9dF11Vd/o6dntCN KmhAyiE5e7dYu1LDuq6kwI=; b=Dr3teNHG61InnoNJGYqn85rrYouZR5R8+vEhy sEqdabonmLn3gpLt/eF3RxOlVnwT93pklcx/2/OWxay0a5FAz2kRSM8xibF1gUkP uTf6xy9MQ8tzgCPcYR71iCdAcHCPNAQUd+lYwaNh/lvSap9MiHRGDIi2KQCzsxDj 4nciDI= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-10.0 required=6.0 tests=FORGED_MUA_OUTLOOK, USER_IN_WHITELIST,USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.6) with ESMTP id md50006735648.msg for ; Thu, 18 Dec 2008 09:45:20 +0000 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=1238cb8453=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <42A989748EC74EB3A6520F7845446360@multiplay.co.uk> From: "Steven Hartland" To: "Paul MacKenzie" , References: <494967A9.3000503@elehost.com> Date: Thu, 18 Dec 2008 09:45:16 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5512 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 18 Dec 2008 09:45:22 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 18 Dec 2008 09:45:22 +0000 Cc: Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 10:00:33 -0000 Just to confirm we see something similar on the box which runs our stats. We have updated from 5.4 -> 6.0 -> 6.2 -> 7.0 all have had no effect on the lockups which happen when the stats run. This box is also on an areca controller but it was on an Adaptec and we saw pretty much the same thing so I suspect its not related to the controller more to the way things are read from and flushed to disk. When we see this problem any ssh sessions become totally unresponsive. The stats we are running are a combination of rrdtool updates from a mysql DB and rrdtool backed mrtg for network stats. This is very reproducible here as it "stalls" the box every few mins when the stats kick off so if there needs to be more investigation we should be able to help. Regards Steve ----- Original Message ----- From: "Paul MacKenzie" >> > 14.0 on pci2 >> > ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 >> > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 >> > arcmsr0: [ITHREAD] >> > ..... >> > Waiting 5 seconds for SCSI devices to settle >> > (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step >> > da0 at arcmsr0 bus 0 target 0 lun 0 >> > da0: Fixed Direct Access SCSI-5 device >> > da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) >> > da0: 305175MB (624999424 512 byte sectors: 255H 63S/T 38904C) >> > SMP: AP CPU #1 Launched! ... ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 10:49:51 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00E801065678 for ; Thu, 18 Dec 2008 10:49:51 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id A88888FC12 for ; Thu, 18 Dec 2008 10:49:50 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1LDGRx-000H7N-H9 for freebsd-stable@freebsd.org; Thu, 18 Dec 2008 12:49:49 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: freebsd-stable In-reply-to: Your message of Wed, 17 Dec 2008 23:21:11 -0800 . Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 18 Dec 2008 12:49:49 +0200 From: Danny Braniss Message-ID: Subject: Re: RELENG_7_1: bce driver change generating too much interrupts ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 10:49:51 -0000 > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, Nawfal, > > Nawfal bin Mohmad Rouyan wrote: > > I have been using a Dell machine with 2 bce interfaces as a bridge > > between my LAN and Firewall to shape the traffic. Since after the > > update, the machine can only run for a few minutes and after that no > > more connection can go through. > > > > Ping from LAN to Internet is OK but when I telnet say to www.yahoo.com > > at port 80 and issue "GET / HTTP/1.0" I can see the data of different > > application including the HTML text. > > > > For example, I can see uTorrent packets with binaries and also the HTML > > page being cut short. It's as if, I'm seeing packets jumbled together > > from different application. > > > > I'm using PF to shape the traffic. If I reboot the server, it will panic > > and I have about 3 different vmcores in /var/crash and not sure what to > > do with it :( . I've tested the patch to remove > > stat_IfInFramesL2FilterDiscards but the problem still occurs. > > The last patch is not a functional change, but a behavior change that > removes the L2FilterDiscards from being counted to match previous behavior. > > Would you please do this: > > script bt.txt kgdb /boot/kernel/kernel.symbols /var/crash/vmcore.0 > > Then, do 'bt', press enter until all display has finished, then exit > kgdb, and send me the result (bt.txt)? > > > As for now, I'm not using the server to shape the traffic because I > > suspect the driver isn't reliable. I'm going to revert back to the > > previous driver and hopes its going to work. > > > > Sorry if there is not much detail since I'm not sure what to provide. > > Just tell me what to provide and I'd be happy to do so. I don't know if the following is related, but: - while stress testing nfs/zfs, I get many weird things on the server (dell-2950/bce) example: impossible packet length (33555456) from nfs server fr-01:/vol/system/share impossible packet length (1792323116) from nfs server fr-01:/vol/system/share ... and things get worse soon after. Now, there are no input errors, so it seems some memory starvation are not properly handled ... cheers, danny From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 10:57:07 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D883C1065674 for ; Thu, 18 Dec 2008 10:57:07 +0000 (UTC) (envelope-from utisoft@googlemail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.173]) by mx1.freebsd.org (Postfix) with ESMTP id A572A8FC1F for ; Thu, 18 Dec 2008 10:57:07 +0000 (UTC) (envelope-from utisoft@googlemail.com) Received: by wf-out-1314.google.com with SMTP id 24so495464wfg.7 for ; Thu, 18 Dec 2008 02:57:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:reply-to :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=lqVylVwN1xWt4Q8sLGryEoyAPQ/QY+H0U6kfSjmtKR8=; b=HAI8k1UXEUKx6+5h4wu0kDV7e9rpZJ/UoPu9G49BhC+M18wSwjcTIADm0YBzzKifyF 5dYjZQ3Muy7WvIrjCgSBcE6tWFJiHpaQNETMppYopNX5WPN+wJSEhRYie1qsSUPDm8VU hk7ZDWyduDBOt1WzuMD+0w+n3HvT/YlV1XLgU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=message-id:date:from:reply-to:to:subject:cc:in-reply-to :mime-version:content-type:content-transfer-encoding :content-disposition:references; b=Q+MZw1Kjb0ohEGvpIIF2a9PLEz2eT66COr16LXva/0rJkohC4mY+MWK1E5J3ywm2d6 CBnayNYdP9RK8SCFw1PMk1bVB+/rL75njwYQsan4wv1blqlqQeEe8c92DSLcdN8XMtVH SkXFBNlP0OtlBPu8UZ6CuRobT3/2gl2n8PJjY= Received: by 10.142.174.8 with SMTP id w8mr751675wfe.76.1229596276873; Thu, 18 Dec 2008 02:31:16 -0800 (PST) Received: by 10.142.199.9 with HTTP; Thu, 18 Dec 2008 02:31:16 -0800 (PST) Message-ID: Date: Thu, 18 Dec 2008 10:31:16 +0000 From: "Chris Rees" To: test1000@cogeco.ca In-Reply-To: <4949636d.3da.4bc2.9176@cogeco.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4949636d.3da.4bc2.9176@cogeco.ca> Cc: freebsd-stable@freebsd.org Subject: Re: Test X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: utisoft@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 10:57:07 -0000 2008/12/17 : > This is a test message from Cogeco Cable. No reply is necessary. What's wrong with http://lists.freebsd.org/mailman/listinfo/freebsd-test ? -- R< $&h ! > $- ! $+ $@ $2 < @ $1 .UUCP. > (sendmail.cf) From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 14:30:44 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEDEF106564A for ; Thu, 18 Dec 2008 14:30:44 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep9.cogeco.net (smtp.cogeco.net [216.221.81.25]) by mx1.freebsd.org (Postfix) with ESMTP id 8A3B88FC18 for ; Thu, 18 Dec 2008 14:30:44 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep9.cogeco.net (Postfix) with ESMTP id 45DFBD8C; Thu, 18 Dec 2008 09:30:41 -0500 (EST) Message-ID: <494A5F11.5080701@cogeco.ca> Date: Thu, 18 Dec 2008 09:32:49 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: Steven Hartland References: <494967A9.3000503@elehost.com> <42A989748EC74EB3A6520F7845446360@multiplay.co.uk> In-Reply-To: <42A989748EC74EB3A6520F7845446360@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 14:30:44 -0000 > Just to confirm we see something similar on the box which runs our stats. > > We have updated from 5.4 -> 6.0 -> 6.2 -> 7.0 all have had no effect on > the lockups which happen when the stats run. > > This box is also on an areca controller but it was on an Adaptec and we > saw pretty much the same thing so I suspect its not related to the > controller more to the way things are read from and flushed to disk. > > When we see this problem any ssh sessions become totally unresponsive. > > The stats we are running are a combination of rrdtool updates from a > mysql DB and rrdtool backed mrtg for network stats. > > This is very reproducible here as it "stalls" the box every few mins > when the stats kick off so if there needs to be more investigation > we should be able to help. > > Regards > Steve Hi Steve, Thanks for your message. Do your processes also lock in UFS state? Well I went and bought a new Areca controller to see if this would fix it. It is quicker and seems to work much better but unfortunately I am still seeing the locking. I moved from a PCI-X Areca controller w/ 1 GB of Cache to a PCI-Express controller w/ 2 GB of cache. Main Processor : 800MHz CPU ICache Size : 32KB CPU DCache Size : 32KB CPU SCache Size : 512KB System Memory : 2048MB/533MHz/ECC Firmware Version : V1.46 2008-08-06 BOOT ROM Version : V1.45 2008-08-26 Controller Name : ARC-1231 As I try to see if there is a hardware connection to this I will let you know. I guess I am going to have to replace the full chassis SR2500ALBRPNA and mainboard S5000PAL next. If no solution is found with hardware replacement then am I out of options with using Freebsd? I wonder would there be any need to also try to replace the ram and cpus given there are no errors? Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 15:43:18 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AAA21065673 for ; Thu, 18 Dec 2008 15:43:18 +0000 (UTC) (envelope-from bms@incunabulum.net) Received: from out1.smtp.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.freebsd.org (Postfix) with ESMTP id 7585C8FC1A for ; Thu, 18 Dec 2008 15:43:18 +0000 (UTC) (envelope-from bms@incunabulum.net) Received: from compute1.internal (compute1.internal [10.202.2.41]) by out1.messagingengine.com (Postfix) with ESMTP id D520A1E805F for ; Thu, 18 Dec 2008 10:43:17 -0500 (EST) Received: from heartbeat2.messagingengine.com ([10.202.2.161]) by compute1.internal (MEProxy); Thu, 18 Dec 2008 10:43:17 -0500 X-Sasl-enc: hq/Mc7dMlkADmxx7YkfZgidWyxmptUzxpueVD7/JvWMt 1229614997 Received: from anglepoise.lon.incunabulum.net (82-35-112-254.cable.ubr07.dals.blueyonder.co.uk [82.35.112.254]) by mail.messagingengine.com (Postfix) with ESMTPSA id 6F42A38A2C for ; Thu, 18 Dec 2008 10:43:17 -0500 (EST) Message-ID: <494A6F93.7040806@incunabulum.net> Date: Thu, 18 Dec 2008 15:43:15 +0000 From: Bruce Simpson User-Agent: Thunderbird 2.0.0.18 (X11/20081204) MIME-Version: 1.0 To: freebsd-stable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Q: eSATA Hot Swap supported? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 15:43:18 -0000 Hi, Does FreeBSD 7.1 support eSATA hot swappable disks? cheers BMS From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 15:44:34 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45845106567D; Thu, 18 Dec 2008 15:44:34 +0000 (UTC) (envelope-from bms@incunabulum.net) Received: from out1.smtp.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.freebsd.org (Postfix) with ESMTP id 13E948FC1B; Thu, 18 Dec 2008 15:44:34 +0000 (UTC) (envelope-from bms@incunabulum.net) Received: from compute1.internal (compute1.internal [10.202.2.41]) by out1.messagingengine.com (Postfix) with ESMTP id AE0111E7A45; Thu, 18 Dec 2008 10:44:33 -0500 (EST) Received: from heartbeat1.messagingengine.com ([10.202.2.160]) by compute1.internal (MEProxy); Thu, 18 Dec 2008 10:44:33 -0500 X-Sasl-enc: R9WdMZmryzBAEhBpnub089UsRhfpjcKr1Qfjm+Rk04oB 1229615073 Received: from anglepoise.lon.incunabulum.net (82-35-112-254.cable.ubr07.dals.blueyonder.co.uk [82.35.112.254]) by mail.messagingengine.com (Postfix) with ESMTPSA id 0BCC33297C; Thu, 18 Dec 2008 10:44:32 -0500 (EST) Message-ID: <494A6FDF.8030103@incunabulum.net> Date: Thu, 18 Dec 2008 15:44:31 +0000 From: Bruce Simpson User-Agent: Thunderbird 2.0.0.18 (X11/20081204) MIME-Version: 1.0 To: "Paul B. Mahol" References: <8cb6106e0811241129o642dcf28re4ae177c8ccbaa25@mail.gmail.com> <8cb6106e0812031453j6dc2f2f4i374145823c084eaa@mail.gmail.com> <200812041747.09040.gnemmi@gmail.com> <4938FE44.9090608@FreeBSD.org> <4939133E.2000701@FreeBSD.org> <493CEE90.7050104@FreeBSD.org> <3a142e750812090553l564bff84pe1f02cd1b03090ff@mail.gmail.com> <4943F43B.4060105@incunabulum.net> <3a142e750812131403p31841403ub9d5693278c74111@mail.gmail.com> <4944501E.40900@incunabulum.net> <3a142e750812140747r2eb5ebadp7ac2b2c8ae357bae@mail.gmail.com> In-Reply-To: <3a142e750812140747r2eb5ebadp7ac2b2c8ae357bae@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, "Bruce M. Simpson" Subject: Re: ext2fuse: user-space ext2 implementation X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 15:44:34 -0000 Paul B. Mahol wrote: > Project itself doesnt look very active, but I may be wrong. It is in alpha state > as reported on SF. > IMHO it is better to maintain our own because it is in better shape, but I'm not > intersted in ext* as developer. > Shelved due to lack of interest, then... others can feel free to pick up. thanks BMS From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 16:38:24 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCE631065676 for ; Thu, 18 Dec 2008 16:38:24 +0000 (UTC) (envelope-from aragon@phat.za.net) Received: from mail.geek.sh (decoder.geek.sh [196.36.198.81]) by mx1.freebsd.org (Postfix) with ESMTP id 810948FC25 for ; Thu, 18 Dec 2008 16:38:24 +0000 (UTC) (envelope-from aragon@phat.za.net) Received: by mail.geek.sh (Postfix, from userid 1000) id A5B3924D26; Thu, 18 Dec 2008 18:11:03 +0200 (SAST) Date: Thu, 18 Dec 2008 18:11:03 +0200 From: Aragon Gouveia To: Ken Smith Message-ID: <20081218161103.GA52173@phat.za.net> References: <1228876766.35416.23.camel@bauer.cse.buffalo.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1228876766.35416.23.camel@bauer.cse.buffalo.edu> User-Agent: Mutt/1.4i X-Operating-System: FreeBSD 4.10-RELEASE-p2 i386 Cc: freebsd-stable Subject: Re: FreeBSD 7.1-RC1 Available... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 16:38:25 -0000 Hi, | By Ken Smith | [ 2008-12-10 04:40 +0200 ] > In addition to general testing we're looking for information about > potential problems with the boot loader. There has been traffic here > about problems but the reports haven't helped narrow down the causes > yet. So far it seems to be related to USB keyboards and at least so far > systems with more than one processor in them. More data points like > cases where a USB keyboard was not involved as well as if possible > things like which motherboard it is might help us narrow down the cause. > Testing on a variety of motherboards, of varying ages and manufacturers > would help. I have been experiencing problems with loader(8) on my system. Essentially it freezes the system if "too much" keyboard input is given. Please see these posts for more background: http://lists.freebsd.org/pipermail/freebsd-stable/2008-October/046176.html http://lists.freebsd.org/pipermail/freebsd-stable/2008-October/046189.html http://lists.freebsd.org/pipermail/freebsd-stable/2008-November/046337.html I've just tested booting up with an i386 RC1 disc and the problem is still there. My motherboard is an Intel DG33BU. Regards, Aragon From owner-freebsd-stable@FreeBSD.ORG Thu Dec 18 22:58:04 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36935106567D for ; Thu, 18 Dec 2008 22:58:04 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from ape.monkeybrains.net (mail.monkeybrains.net [208.69.40.9]) by mx1.freebsd.org (Postfix) with ESMTP id 0C8498FC36 for ; Thu, 18 Dec 2008 22:58:03 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from [192.168.2.105] (adsl-76-247-114-27.dsl.pltn13.sbcglobal.net [76.247.114.27]) (authenticated bits=0) by ape.monkeybrains.net (8.14.1/8.14.1) with ESMTP id mBIMvFKv013022 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 18 Dec 2008 14:57:16 -0800 (PST) (envelope-from crapsh@monkeybrains.net) Message-ID: <494AD575.2060008@monkeybrains.net> Date: Thu, 18 Dec 2008 14:57:57 -0800 From: Rudy User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: Bruce Simpson References: <494A6F93.7040806@incunabulum.net> In-Reply-To: <494A6F93.7040806@incunabulum.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.94, clamav-milter version 0.94 on pita.monkeybrains.net X-Virus-Status: Clean Cc: freebsd-stable Subject: Re: Q: eSATA Hot Swap supported? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2008 22:58:04 -0000 Bruce Simpson wrote: > Does FreeBSD 7.1 support eSATA hot swappable disks? Answer: Yes. I have a hot-swap case and can swap my internal SATA disks. Your eSATA's show up as 'ad' devices, correct? You may need to run atacontrol to get the drives to show up after you hot-plug them. atacontrol list *find the channel* atacontrol attach ata3 (or whatever number...) "man atacontrol" for lots of useful info. Rudy From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 00:34:42 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34C6A1065676; Fri, 19 Dec 2008 00:34:42 +0000 (UTC) (envelope-from brucec@muon.cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [66.246.138.153]) by mx1.freebsd.org (Postfix) with ESMTP id 0E7288FC12; Fri, 19 Dec 2008 00:34:41 +0000 (UTC) (envelope-from brucec@muon.cran.org.uk) Received: by muon.cran.org.uk (Postfix, from userid 1000) id AFD7F245402; Thu, 18 Dec 2008 19:17:09 -0500 (EST) Date: Thu, 18 Dec 2008 19:17:09 -0500 From: Bruce Cran To: Edwin Groothuis Message-ID: <20081219001709.GA32031@muon.cran.org.uk> References: <20080928054620.GA80250@k7.mavetju> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="X1bOJ3K7DJ5YkBrT" Content-Disposition: inline In-Reply-To: <20080928054620.GA80250@k7.mavetju> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: stable@freebsd.org, current@freebsd.org Subject: Re: Request for testing - top 3.8b1 in the base system X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 00:34:42 -0000 --X1bOJ3K7DJ5YkBrT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Sep 28, 2008 at 03:46:20PM +1000, Edwin Groothuis wrote: > I have made an update for the top(1) utility in the FreeBSD base > system to get it from the 3.5b12 version to the 3.8b1 version. >=20 [...] > Please report any issues with it (compile time, run time) and a way > to reproduce it (if possible). Thanks for your help! > There's an overflow bug in format_k2 in 3.5b12 that means that top can corrupt the SIZE field of processes which allocate 2TB or more of memory; t= hat seems to be fixed in 3.8b1 but I've noticed that if a process allocates over 2TB then it doesn't show up at the top of the display when sorting by SIZE in 3.8b1; I suspect there must be another overflow somewhere. I'm sure it'll be a good few years before anyone actually hits this bug running real programs, but I don't know if it might affect other things. --=20 Bruce Cran --X1bOJ3K7DJ5YkBrT Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAklK6AUACgkQn4uvqcJsLfgJ6ACgoc4dDhvlkCIKiHk9T2sHQqNU KtQAnjzOoBU2IC8zb0rcBRQTY+Vw+Yn4 =YRP3 -----END PGP SIGNATURE----- --X1bOJ3K7DJ5YkBrT-- From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 00:38:56 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC7531065679 for ; Fri, 19 Dec 2008 00:38:56 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep9.cogeco.net (smtp.cogeco.net [216.221.81.25]) by mx1.freebsd.org (Postfix) with ESMTP id 5EF118FC19 for ; Fri, 19 Dec 2008 00:38:56 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep9.cogeco.net (Postfix) with ESMTP id 9EC0F136F; Thu, 18 Dec 2008 19:38:55 -0500 (EST) Message-ID: <494AED9E.9090900@cogeco.ca> Date: Thu, 18 Dec 2008 19:41:02 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: Ivan Voras References: <4949673B.2070701@elehost.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 00:38:56 -0000 >> last pid: 46013; load averages: 105.30, 67.67, >> 34.45 up 4+23:59:42 19:08:40 >> 629 processes: 89 running, 540 sleeping >> CPU: 21.9% user, 0.0% nice, 74.5% system, 3.1% interrupt, 0.4% idle >> Mem: 1538M Active, 11G Inact, 898M Wired, 303M Cache, 214M Buf, 1346M Free >> Swap: 8192M Total, 1036K Used, 8191M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 46000 www 1 65 0 86728K 15008K RUN 1 0:01 12.06% httpd >> 45994 www 1 56 0 86728K 15008K CPU1 3 0:01 10.16% httpd >> 46002 www 1 -4 0 150M 20648K RUN 3 0:00 6.98% httpd >> 45195 www 1 68 0 121M 19748K RUN 1 0:29 6.88% httpd >> 45991 www 1 53 0 150M 21060K select 3 0:01 6.59% httpd >> 45997 www 1 -4 0 150M 20992K ufs 5 0:01 6.59% httpd >> 45950 www 1 57 0 153M 23388K RUN 2 0:01 6.49% httpd >> 45999 www 1 -4 0 150M 20640K ufs 6 0:00 5.96% httpd >> 45189 www 1 66 0 161M 29660K RUN 6 0:26 5.76% httpd >> 45974 www 1 -4 0 151M 21564K ufs 3 0:01 5.76% httpd >> > > The number of httpd processes in "ufs" state is too high. Are you using > PHP? And if you do, check how large is your PHP sessions' directory. Use > a "sharded" layout if it's of any significant size (see php.ini for > details). > Sorry for the messed up posts and thanks for your suggestion. I apologize if any posts have been duplicated as there seems to be an issue with delivery from cogeco at times. Yes I was thinking the same thing and It was actually one of the first things I looked. I found in the temporary folder there are only about 50-200 there at any one time approximately and most of the time 5-15 files. PHP seems to be only one way to bring it forward. I increased the dirhash when I first started on this problem but the numbers of files in the folders were not nearly like some of the other people reporting a similar connection and I seem to be able to get the system into the state a number of ways. Do you think I should still do this even with the small number of files? vfs.ufs.dirhash_maxmem=10485760 I actually find that running Wusage 8.0 a few times even with nice-19 may be implicated in getting the system to spiral downwards. I hesitate to mention this as it seems to be working fine on another 7.X server. I believe that Wusage is tied to 6.X libraries and I wonder if somehow this may initiate the problem. I also have another sio/com based program running every few minutes which is also connected to the 6.X library (scom thermal application for temperature monitoring) and turning both of these off seems to help. I am going to try a 24 hour period without either of these two running after a fresh reboot and we will see if this is indeed one source to my abominable problem. Once the system spirals down into its locking then the io performance never seems to recover unless I reboot it or somehow find the process that is locked and kill it. I wondered if possibly my problem is related to this identified issue 104406? [ufs] Processes get stuck in "ufs" state under persistent CPU load http://www.freebsd.org/cgi/query-pr.cgi?pr=104406&cat= As the processes do get stuck in the ufs mode from what I can tell I thought this was an interesting connection. When the system is in this state even a simple make depend is agonizingly slow. If so I wondered if there was any way to quickly determine via programming which process is stuck and to get it unstuck as a temporary workaround? The problem reports mentions using 'kill -STOP' and continued with 'kill -CONT', which allows other processes to access the filesystem (until another such failure occurs). I have tried a number of things but so far no luck. Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 07:02:10 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42B3E1065675; Fri, 19 Dec 2008 07:02:10 +0000 (UTC) (envelope-from yohimba@mail.ru) Received: from mx44.mail.ru (mx44.mail.ru [195.239.211.10]) by mx1.freebsd.org (Postfix) with ESMTP id B811A8FC1F; Fri, 19 Dec 2008 07:02:09 +0000 (UTC) (envelope-from yohimba@mail.ru) Received: from f217.mail.ru (f217.mail.ru [194.186.55.150]) by mx44.mail.ru (mPOP.Fallback_MX) with ESMTP id CD5FB38002535; Fri, 19 Dec 2008 09:51:59 +0300 (MSK) Received: from mail by f217.mail.ru with local id 1LDZDJ-000Gxg-00; Fri, 19 Dec 2008 09:51:57 +0300 Received: from [195.208.10.252] by win.mail.ru with HTTP; Fri, 19 Dec 2008 14:51:57 +0800 From: Vyacheslav I. To: freebsd-amd64@freebsd.org Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [195.208.10.252] Date: Fri, 19 Dec 2008 14:51:57 +0800 X-Priority: 1 (High) Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: X-Spam: Not detected X-Mras: OK Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org, msmith@freebsd.org, freebsd-hardware@freebsd.org Subject: Problem with FreeBSD for AMD64 & Mylex AcceleRAID X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Vyacheslav I." List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 07:02:10 -0000 Hi! I used to use the FreeBSD on i386 architecture together with the controller Mylex AcceleRAID 170.There was RAID 0+1 configured on it out of 5 discs (Enhanced Mirroring). # pciconf -lvc ... mly0@pci0:6:1:0: class=0x010400 card=0x00521069 chip=0x00501069 rev=0x02 hdr=0x00 vendor = 'Mylex Corp' device = 'AcceleRAID Disk Array' class = mass storage subclass = RAID cap 01[80] = powerspec 2 supports D0 D3 current D0 I used this configuration on i386 architecture with the following versions FreeBSD: 4.x, 5.x, 6.x and 7.0. It always worked perfectly. But recently I got needed 8 GB RAM so I had to install the version FreeBSD for AMD64. First I tried the 7.0, and after all - RELENG_7_1 dated 12.12.2008. Both these systems work with the files system located on RAID unstablÅ! Meanwhile the same hardware on i386 architecture works with RELENG_7_1 perfectly. Hardware: MB INTEL DG33FB, CPU Intel(R) Core(TM)2 Quad CPU @ 2.40 GHz, RAM 8 GB. The base system is installed on a separate IDE disc. # df -h Filesystem Size Used Avail Capacity Mounted on /dev/ad4s1a 496M 111M 345M 24% / devfs 1.0K 1.0K 0B 100% /dev /dev/ad4s1d 3.9G 22K 3.6G 0% /tmp /dev/ad4s1f 333G 52G 255G 17% /usr /dev/ad4s1e 15G 41M 14G 0% /var /dev/da0s1d 70.7G 410K 70.1G 0% /mnt/da0/d I carried out the following test by creating 3 big sized files of 1 GB each, and then deleting them: # cd /mnt/da0/d # dd if=/dev/zero of=test1 bs=1024k count=1024 # dd if=/dev/zero of=test2 bs=1024k count=1024 # dd if=/dev/zero of=test5 bs=1024k count=1024 # rm test* After that I tried to unmount the files system, but I get a message of the kernel panic: # cd / # umount /mnt/da0/d ... bad block 123456789, ino 176 dev = da0s1d, block = 4, fs = /mnt/da0/d panic: ffs_blkfree: freeing free block cpuid = 3 Uptime: 34 min... The test was successful when using the i386 architecture with the same hardware Taking a piece of advice of my friend, I used a dirty patch: === [code] === diff -Nru src.orig/sys/cam/scsi/scsi_da.c src/sys/cam/scsi/scsi_da.c --- src.orig/sys/cam/scsi/scsi_da.c 2008-12-10 10:01:40.000000000 +0800 +++ src/sys/cam/scsi/scsi_da.c 2008-12-19 12:18:27.000000000 +0800 @@ -1187,7 +1187,7 @@ if (match != NULL) softc->quirks = ((struct da_quirk_entry *)match)->quirks; else - softc->quirks = DA_Q_NONE; + softc->quirks = DA_Q_NO_SYNC_CACHE; // Dirty hack for AMD64 /* Check if the SIM does not want 6 byte commands */ xpt_setup_ccb(&cpi.ccb_h, periph->path, /*priority*/1); === [/code] === This patch solved the problem and the systems stopped being panicking. But I assume that this solution is wrong as the hardware works on i386 architecture without this patch well. From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 09:15:16 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56E44106564A for ; Fri, 19 Dec 2008 09:15:16 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep5.cogeco.net (smtp.cogeco.net [216.221.81.25]) by mx1.freebsd.org (Postfix) with ESMTP id 28CE58FC0C for ; Fri, 19 Dec 2008 09:15:15 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep5.cogeco.net (Postfix) with ESMTP id 71EBA2D96 for ; Wed, 17 Dec 2008 14:13:38 -0500 (EST) Message-ID: <49494FE0.4040400@cogeco.ca> Date: Wed, 17 Dec 2008 14:15:44 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> <4934CB77.30906@transactionware.com> <494698E4.2070406@cogeco.ca> <200812151900.mBFJ0Jom084267@lava.sentex.ca> <4946B6EF.5080806@cogeco.ca> <200812152010.mBFKAlZf084580@lava.sentex.ca> <4946BDB1.8060208@cogeco.ca> <200812152035.mBFKZkj4084689@lava.sentex.ca> <4946DA2F.7090508@cogeco.ca> <200812161556.mBGFuB7n089368@lava.sentex.ca> In-Reply-To: <200812161556.mBGFuB7n089368@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 09:15:16 -0000 > Is the high load average simply a function of processes blocking on > network io ? On our av/spam scanners for example show a high load avg > because there are many processes waiting on network io to complete > (e.g. talking to RBL lists, waiting for DCC servers to complete etc) > > Also, is it really related to the arcmsr driver ? i.e. if you did the > same tasks on a single IDE drive, is the performance profile going to > be the same ? > > ---Mike > I wondered if possibly my problem is related to this issue 104406? [ufs] Processes get stuck in "ufs" state under persistent CPU load http://www.freebsd.org/cgi/query-pr.cgi?pr=104406&cat= As the processes do get stuck in the ufs mode from what I can tell. I am going to be replacing the RAID card tomorrow to make sure it is not the card despite the lack of errors. If so I wondered if there was any way to quickly determine via programming which process is stuck and to get it unstuck as a temporary workaround? The problem reports mentions using 'kill -STOP' and continued with 'kill -CONT', which allows other processes to access the filesystem (until another such failure occurs). I just need to be able to determine which process is the one stuck in a relatively easy manner. Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 09:15:18 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D0E81065670; Fri, 19 Dec 2008 09:15:18 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from fep5.cogeco.net (smtp.cogeco.net [216.221.81.25]) by mx1.freebsd.org (Postfix) with ESMTP id D2AAE8FC12; Fri, 19 Dec 2008 09:15:17 +0000 (UTC) (envelope-from bsdlist@cogeco.ca) Received: from [192.168.1.126] (d150-251-98.home.cgocable.net [24.150.251.98]) by fep5.cogeco.net (Postfix) with ESMTP id 45B47306A; Thu, 18 Dec 2008 10:39:34 -0500 (EST) Message-ID: <494A6F36.9040507@cogeco.ca> Date: Thu, 18 Dec 2008 10:41:42 -0500 From: Paul MacKenzie User-Agent: Thunderbird 3.0a1pre (Windows/2008022014) MIME-Version: 1.0 To: Ivan Voras References: <4949673B.2070701@elehost.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 09:15:18 -0000 >> last pid: 46013; load averages: 105.30, 67.67, >> 34.45 up 4+23:59:42 19:08:40 >> 629 processes: 89 running, 540 sleeping >> CPU: 21.9% user, 0.0% nice, 74.5% system, 3.1% interrupt, 0.4% idle >> Mem: 1538M Active, 11G Inact, 898M Wired, 303M Cache, 214M Buf, 1346M Free >> Swap: 8192M Total, 1036K Used, 8191M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 46000 www 1 65 0 86728K 15008K RUN 1 0:01 12.06% httpd >> 45994 www 1 56 0 86728K 15008K CPU1 3 0:01 10.16% httpd >> 46002 www 1 -4 0 150M 20648K RUN 3 0:00 6.98% httpd >> 45195 www 1 68 0 121M 19748K RUN 1 0:29 6.88% httpd >> 45991 www 1 53 0 150M 21060K select 3 0:01 6.59% httpd >> 45997 www 1 -4 0 150M 20992K ufs 5 0:01 6.59% httpd >> 45950 www 1 57 0 153M 23388K RUN 2 0:01 6.49% httpd >> 45999 www 1 -4 0 150M 20640K ufs 6 0:00 5.96% httpd >> 45189 www 1 66 0 161M 29660K RUN 6 0:26 5.76% httpd >> 45974 www 1 -4 0 151M 21564K ufs 3 0:01 5.76% httpd >> > > The number of httpd processes in "ufs" state is too high. Are you using > PHP? And if you do, check how large is your PHP sessions' directory. Use > a "sharded" layout if it's of any significant size (see php.ini for > details). > Sorry for the messed up posts and thanks for your suggestion. I apologize if any posts have been duplicated as there seems to be an issue with delivery from cogeco at times. Yes I was thinking the same thing and It was actually one of the first things I looked. I found in the temporary folder there are only about 50-200 there at any one time approximately and most of the time 5-15 files. PHP seems to be only one way to bring it forward. I increased the dirhash when I first started on this problem but the numbers of files in the folders were not nearly like some of the other people reporting a similar connection and I seem to be able to get the system into the state a number of ways. Do you think I should still do this even with the small number of files? vfs.ufs.dirhash_maxmem=10485760 I actually find that running Wusage 8.0 a few times even with nice-19 may be implicated in getting the system to spiral downwards. I hesitate to mention this as it seems to be working fine on another 7.X server. I believe that Wusage is tied to 6.X libraries and I wonder if somehow this may initiate the problem. I also have another sio/com based program running every few minutes which is also connected to the 6.X library (scom thermal application for temperature monitoring) and turning both of these off seems to help. I am going to try a 24 hour period without either of these two running after a fresh reboot and we will see if this is indeed one source to my abominable problem. Once the system spirals down into its locking then the io performance never seems to recover unless I reboot it or somehow find the process that is locked and kill it. I wondered if possibly my problem is related to this identified issue 104406? [ufs] Processes get stuck in "ufs" state under persistent CPU load http://www.freebsd.org/cgi/query-pr.cgi?pr=104406&cat= As the processes do get stuck in the ufs mode from what I can tell I thought this was an interesting connection. When the system is in this state even a simple make depend is agonizingly slow. If so I wondered if there was any way to quickly determine via programming which process is stuck and to get it unstuck as a temporary workaround? The problem reports mentions using 'kill -STOP' and continued with 'kill -CONT', which allows other processes to access the filesystem (until another such failure occurs). I have tried a number of things but so far no luck. Thanks, Paul From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 10:19:10 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5333C1065670 for ; Fri, 19 Dec 2008 10:19:10 +0000 (UTC) (envelope-from marius@nuenneri.ch) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.30]) by mx1.freebsd.org (Postfix) with ESMTP id 1C9928FC18 for ; Fri, 19 Dec 2008 10:19:09 +0000 (UTC) (envelope-from marius@nuenneri.ch) Received: by yw-out-2324.google.com with SMTP id 9so1021045ywe.13 for ; Fri, 19 Dec 2008 02:19:09 -0800 (PST) Received: by 10.90.94.2 with SMTP id r2mr1684816agb.62.1229681949084; Fri, 19 Dec 2008 02:19:09 -0800 (PST) Received: by 10.90.73.15 with HTTP; Fri, 19 Dec 2008 02:19:09 -0800 (PST) Message-ID: Date: Fri, 19 Dec 2008 11:19:09 +0100 From: "=?ISO-8859-1?Q?Marius_N=FCnnerich?=" To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Kernel not mounting root X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 10:19:10 -0000 Hi all, I did cvsup to 7-STABLE yesterday, build, installed and booted a new kernel then buildworld. Then I noticed that my PS/2 keyboard is not working at the loader menu so I booted multiuser and did the make installworld part from there. Then I tried to reboot but now it hangs after the line Trying to mount root from ufs:/dev/ad4s1a It sits there for at least 10min. I tried so far: - Installing 7.1-beta2 on another hd and booting it, it works. - Copied that kernel to the old hd - Moved kernel.old to kernel - fsck everything - mounted everything from the old hd under /mnt, chroot into that, installworld Anyone else seeing this or has an idea what to do next? Kind regards Marius From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 15:23:24 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C87381065673; Fri, 19 Dec 2008 15:23:24 +0000 (UTC) (envelope-from me@janh.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.9]) by mx1.freebsd.org (Postfix) with ESMTP id 3A42E8FC19; Fri, 19 Dec 2008 15:23:24 +0000 (UTC) (envelope-from me@janh.de) Received: from janh.freebsd (e177254157.adsl.alicedsl.de [85.177.254.157]) by mrelayeu.kundenserver.de (node=mrelayeu4) with ESMTP (Nemesis) id 0ML21M-1LDh021ASN-0003f7; Fri, 19 Dec 2008 16:10:46 +0100 Message-ID: <494BB96F.2040702@janh.de> Date: Fri, 19 Dec 2008 16:10:39 +0100 From: Jan Henrik Sylvester User-Agent: Thunderbird 2.0.0.18 (X11/20081125) MIME-Version: 1.0 To: stable-list freebsd Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V01U2FsdGVkX19vKqrBlP3pIX77T5mjHUUsa/IuM8Y1MIoJcGh t8ipjSrAunyUhrrxuEdJkX431ueyF+U7dGLcmaWaGY0yR/6xYg dH5cw6nnXfJfbl4pEyvpQ== Cc: Gavin Atkinson Subject: iwn on 7.1-RC1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 15:23:24 -0000 Short version: Is there any chance to get iwn working on 7.1-RC1 reliably? I have got one problem with the initial perforce version and the backport from gavin always crashes. Long version: I have been using the initial perforce version of iwn on 7-STABLE for a few month now, since the next version is already "vap'ify iwn". Usually, the connection to my WPA2 ap is established on boot, but pretty often I get an error instead: iwn0: error, INTR=82000000 STATUS=0x10000 iwn0: iwn_config: could not set power mode, error 35 Doing one or two "kldunload if_iwn" fixes the problem. Rarely, I had crashes (on boot). Today, I found that gavin did a backport of iwn in September: http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045234.html As it had some changes compared to the initial perforce version, I tried that version instead. I always get a crash on boot: [...] iwn0: iwn_read_eeprom_ht40: no entry for channel 10 iwn0: iwn_read_eeprom_ht40: no entry for channel 11 iwn0: iwn_read_eeprom_ht40: no entry for channel 12 iwn0: iwn_read_eeprom_ht40: no entry for channel 13 Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xc5bb5004 fault code = supervisor read, page not present instruction pointer = 0x20:0xc1543935 stack pointer = 0x28:0xc18207f8 frame pointer = 0x28:0xc1820858 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processsor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) trap number = 12 panic: page fault cpuid = 0 Uptime: 1s Automatic reboot in 15 seconds - press a key on the console to abort I do not remember, if the crash is the same as I had with the initial perforce version, which I cannot reproduce. Does anyone have a better version of iwn without vap? Does anyone know which current / perforce change could address the error mentioned above? There are only a few differences between the initial perforce version and the backport by gavin (besides man page). I will attach them below. Cheers, Jan Henrik diff -u perforce/sys/dev/iwn/if_iwn.c gavin/sys/dev/iwn/if_iwn.c --- perforce/sys/dev/iwn/if_iwn.c 2008-12-19 15:19:14.000000000 +0100 +++ gavin/sys/dev/iwn/if_iwn.c 2008-12-19 15:16:57.000000000 +0100 @@ -124,6 +124,7 @@ void iwn_rx_statistics(struct iwn_softc *, struct iwn_rx_desc *); void iwn_tx_intr(struct iwn_softc *, struct iwn_rx_desc *); void iwn_cmd_intr(struct iwn_softc *, struct iwn_rx_desc *); +static void iwn_bmiss(void *, int); void iwn_notif_intr(struct iwn_softc *); void iwn_intr(void *); void iwn_read_eeprom(struct iwn_softc *); @@ -292,7 +293,8 @@ taskqueue_start_threads(&sc->sc_tq, 1, PI_NET, "%s taskq", device_get_nameunit(dev)); - TASK_INIT(&sc->sc_opstask, 0, iwn_ops, sc ); + TASK_INIT(&sc->sc_ops_task, 0, iwn_ops, sc ); + TASK_INIT(&sc->sc_bmiss_task, 0, iwn_bmiss, sc ); /* * Put adapter into a known state. @@ -379,6 +381,8 @@ #endif | IEEE80211_C_WME /* WME */ ; +#if 0 + /* XXX disable until HT channel setup works */ ic->ic_htcaps = IEEE80211_HTCAP_SMPS_ENA /* SM PS mode enabled */ | IEEE80211_HTCAP_CHWIDTH40 /* 40MHz channel width */ @@ -391,7 +395,7 @@ | IEEE80211_HTC_AMPDU /* tx A-MPDU */ | IEEE80211_HTC_AMSDU /* tx A-MSDU */ ; - +#endif /* read supported channels and MAC address from EEPROM */ iwn_read_eeprom(sc); @@ -1594,6 +1598,15 @@ wakeup(&ring->cmd[desc->idx]); } +static void +iwn_bmiss(void *arg, int npending) +{ + struct iwn_softc *sc = arg; + struct ieee80211com *ic = sc->sc_ifp->if_l2com; + + ieee80211_beacon_miss(ic); +} + void iwn_notif_intr(struct iwn_softc *sc) { @@ -1652,7 +1665,8 @@ if (ic->ic_state == IEEE80211_S_RUN && misses > 5) (void) iwn_init_sensitivity(sc); if (misses >= ic->ic_bmissthreshold) - ieee80211_beacon_miss(ic); + taskqueue_enqueue(taskqueue_swi, + &sc->sc_bmiss_task); break; } case IWN_UC_READY: { @@ -2398,7 +2412,7 @@ static const struct iwn_chan_band iwn_bands[] = { { IWN_EEPROM_BAND1, IEEE80211_CHAN_G, 14, { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 } }, - { IWN_EEPROM_BAND2, IEEE80211_CHAN_A, 13, +/* { IWN_EEPROM_BAND2, IEEE80211_CHAN_A, 13, { 183, 184, 185, 187, 188, 189, 192, 196, 7, 8, 11, 12, 16 } }, { IWN_EEPROM_BAND3, IEEE80211_CHAN_A, 12, { 34, 36, 38, 40, 42, 44, 46, 48, 52, 56, 60, 64 } }, @@ -2406,11 +2420,11 @@ { 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140 } }, { IWN_EEPROM_BAND5, IEEE80211_CHAN_A, 6, { 145, 149, 153, 157, 161, 165 } }, - { IWN_EEPROM_BAND6, IEEE80211_CHAN_G | IEEE80211_CHAN_HT40, 7, +*/ { IWN_EEPROM_BAND6, IEEE80211_CHAN_G | IEEE80211_CHAN_HT40, 7, { 1, 2, 3, 4, 5, 6, 7 } }, - { IWN_EEPROM_BAND7, IEEE80211_CHAN_A | IEEE80211_CHAN_HT40, 11, +/* { IWN_EEPROM_BAND7, IEEE80211_CHAN_A | IEEE80211_CHAN_HT40, 11, { 36, 44, 52, 60, 100, 108, 116, 124, 132, 149, 157 } } - }; +*/ }; struct ieee80211com *ic = &sc->sc_ic; int i; @@ -2651,14 +2665,14 @@ /* XXX all wrong */ /* compute remaining time until next beacon */ val = (uint64_t)ni->ni_intval * 1024; /* msecs -> usecs */ - DPRINTF(sc, IWN_DEBUG_ANY, "%s: val = %llu %s\n", __func__, + DPRINTF(sc, IWN_DEBUG_ANY, "%s: val = %ju %s\n", __func__, val, val == 0 ? "correcting" : ""); if (val == 0) val = 1; mod = le64toh(tsf.tstamp) % val; tsf.binitval = htole32((uint32_t)(val - mod)); - DPRINTF(sc, IWN_DEBUG_RESET, "TSF bintval=%u tstamp=%llu, init=%u\n", + DPRINTF(sc, IWN_DEBUG_RESET, "TSF bintval=%u tstamp=%ju, init=%u\n", ni->ni_intval, le64toh(tsf.tstamp), (uint32_t)(val - mod)); if (iwn_cmd(sc, IWN_CMD_TSF, &tsf, sizeof tsf, 1) != 0) @@ -4243,7 +4257,7 @@ sc->sc_cmd_arg[sc->sc_cmd_next] = arg; sc->sc_cmd_next = (sc->sc_cmd_next + 1) % IWN_CMD_MAXOPS; } - taskqueue_enqueue(sc->sc_tq, &sc->sc_opstask); + taskqueue_enqueue(sc->sc_tq, &sc->sc_ops_task); IWN_CMD_UNLOCK(sc); return 0; } --- perforce/sys/dev/iwn/if_iwnvar.h 2008-12-19 15:19:14.000000000 +0100 +++ gavin/sys/dev/iwn/if_iwnvar.h 2008-12-19 15:16:57.000000000 +0100 @@ -197,7 +197,8 @@ struct taskqueue *sc_tq; /* Main command task queue */ /* Tasks used by the driver */ - struct task sc_opstask; /* operation handling task */ + struct task sc_ops_task; /* operation handling task */ + struct task sc_bmiss_task; /* beacon miss task */ /* Thermal calibration */ struct callout calib_to; --- perforce/sys/modules/iwn/Makefile 2008-12-19 15:19:14.000000000 +0100 +++ gavin/sys/modules/iwn/Makefile 2008-12-19 15:16:57.000000000 +0100 @@ -3,6 +3,6 @@ .PATH: ${.CURDIR}/../../dev/iwn KMOD = if_iwn -SRCS = if_iwn.c opt_bdg.h device_if.h bus_if.h pci_if.h -CFLAGS += -g -DWITNESS -DINVARIANT_SUPPORT -DINVARIANTS -I${.CURDIR}/../../ +SRCS = if_iwn.c device_if.h bus_if.h pci_if.h + .include From owner-freebsd-stable@FreeBSD.ORG Fri Dec 19 21:51:57 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 211881065670 for ; Fri, 19 Dec 2008 21:51:57 +0000 (UTC) (envelope-from bahamasfranks@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.242]) by mx1.freebsd.org (Postfix) with ESMTP id 8FF7E8FC14 for ; Fri, 19 Dec 2008 21:51:56 +0000 (UTC) (envelope-from bahamasfranks@gmail.com) Received: by an-out-0708.google.com with SMTP id c2so702747anc.13 for ; Fri, 19 Dec 2008 13:51:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=CbFMCaXSnqCyQsMJFg75av8iS/VfrRdzfoCF5WpVPzg=; b=Wuy3add+RzKKcEWelBZc2U7jokIZ+L3xsBttQTruD6Qg8k4xsXUUeHC0fRNyOLxdqd ZCh6GBEJfp4RLnSJ7UHjMjhquYNu1Cs6we5g3J1KRhDiP/lFYwE+oIKOPHGR95hB+sY6 3bR5rhgqsd3G+MC2DtKQOXSaap9422So1DxzU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=IF+JLheD6QD7ktu81lulkaBLIo9B3LQpXT3ETxhuV56aCeTeXZLrM6MHWwaQlzPT02 oLvacQ3xs67EaOwJN+EeVX0FtuPlfhF37s8L6Xw62UxD5mzj/B0NGP6LSi3KkhK0JGnO bnBmyVAVp/A8QGjf/uke4u0PTmjdPnDrXHi5g= Received: by 10.100.128.2 with SMTP id a2mr2526692and.158.1229723514081; Fri, 19 Dec 2008 13:51:54 -0800 (PST) Received: by 10.100.4.14 with HTTP; Fri, 19 Dec 2008 13:51:54 -0800 (PST) Message-ID: <539c60b90812191351i6090f24ejb9006471f74f01b9@mail.gmail.com> Date: Fri, 19 Dec 2008 14:51:54 -0700 From: "Steve Franks" To: "Paul B. Mahol" In-Reply-To: <3a142e750812140747r2eb5ebadp7ac2b2c8ae357bae@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0811241129o642dcf28re4ae177c8ccbaa25@mail.gmail.com> <200812041747.09040.gnemmi@gmail.com> <4938FE44.9090608@FreeBSD.org> <4939133E.2000701@FreeBSD.org> <493CEE90.7050104@FreeBSD.org> <3a142e750812090553l564bff84pe1f02cd1b03090ff@mail.gmail.com> <4943F43B.4060105@incunabulum.net> <3a142e750812131403p31841403ub9d5693278c74111@mail.gmail.com> <4944501E.40900@incunabulum.net> <3a142e750812140747r2eb5ebadp7ac2b2c8ae357bae@mail.gmail.com> Cc: freebsd-fs@freebsd.org, "Bruce M. Simpson" , Bruce M Simpson , freebsd-stable@freebsd.org Subject: Re: ext2fuse: user-space ext2 implementation X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2008 21:51:57 -0000 On Sun, Dec 14, 2008 at 8:47 AM, Paul B. Mahol wrote: > On 12/14/08, Bruce M Simpson wrote: >> Paul B. Mahol wrote: >>>> Can you please relay this feedback to the authors of ext2fuse? >>>> >>>> As mentioned earlier in the thread, the ext2fuse code could benefit from >>>> UBLIO-ization. Are you or any other volunteers happy to help out here? >>>> >>> >>> Well, first higher priority would be to fix existing bugs. It would be >>> very little >>> gain with user cache, because it is already too much IMHO slow and >>> adding user cache >>> will not make it faster, but that is not port problem. >>> >> >> I'm not aware of bugs with ext2fuse itself; my work on the port was >> merely to try to raise awareness that a user-space project for ext2 >> filesystem access existed. >> >> Can you elaborate further on your experience with ext2fuse which seems >> to you to be buggy, i.e. symptoms, root cause analysis etc. ? Have you >> reported these to the author(s)? > > I have read TODO. > >> Have you measured the performance? Is the performance sufficient for the >> needs of an occasional desktop user? > > Performance was not sufficient, and adding user cache will not improve access > speed on first read. > After mounting ext2fs volume (via md(4)) created with e2fsprogs port > and copying data > from ufs to ext2, reading was quite slow. Also ext2fuse after mount > doesnt exits it > is still running displaying debug data - explaining why project > itselfs is in alpha > state. > >> I realise we are largely involved in content-free argument here, however >> the trade-off of ext2fuse vs ext2fs in the FreeBSD kernel source tree, >> is that of a hopefully more actively maintained implementation vs one >> which is not maintained at all, and any alternatives for FreeBSD users >> would be welcome. > > Project itself doesnt look very active, but I may be wrong. It is in alpha state > as reported on SF. > IMHO it is better to maintain our own because it is in better shape, but I'm not > intersted in ext* as developer. AFAIK our ext* either barfs or corrupts ext3, and since linux is pretty much all using ext3 these days, we're stuck in read-only for ext3, which is rather undesirable, methinks (seems everyone's using fuse's ntfs for this same reason [which is stable, however]). Which is not to say stealing the ext3 (journal?) implementation and putting it in our code isn't a better choice, I'm just pointing out there is no good choice right now... Steve From owner-freebsd-stable@FreeBSD.ORG Sat Dec 20 17:47:13 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B380A106564A for ; Sat, 20 Dec 2008 17:47:13 +0000 (UTC) (envelope-from gmiller@classic-games.com) Received: from fmailhost06.isp.att.net (fmailhost06.isp.att.net [204.127.217.106]) by mx1.freebsd.org (Postfix) with ESMTP id A13F08FC16 for ; Sat, 20 Dec 2008 17:47:13 +0000 (UTC) (envelope-from gmiller@classic-games.com) Received: from [65.7.32.71] (host-65-7-32-71.mem.bellsouth.net?[65.7.32.71]) by isp.att.net (frfwmhc06) with ESMTP id <20081220173348H0600hat6ve>; Sat, 20 Dec 2008 17:33:48 +0000 X-Originating-IP: [65.7.32.71] Message-ID: <494D2C64.1030105@classic-games.com> Date: Sat, 20 Dec 2008 11:33:24 -0600 From: Greg Miller User-Agent: Thunderbird 2.0.0.18 (X11/20081212) MIME-Version: 1.0 To: stable-list freebsd Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Lockups and panics during multiuser boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Dec 2008 17:47:13 -0000 After upgrading to 7.1RC1, I found I could only boot single-user without a panic or a lockup. After bisecting recent changes on RELENG_7, I found that the problem occurs after 2008.10.29.15.00.00 and before 2008.10.29.18.00.00 -- http://www.velocityvector.com/ | http://www.classic-games.com/ Any government that can give you everything you want can take everything you have. From owner-freebsd-stable@FreeBSD.ORG Sat Dec 20 22:33:24 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B731106567F for ; Sat, 20 Dec 2008 22:33:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.terabit.net.ua (mail.terabit.net.ua [195.137.202.147]) by mx1.freebsd.org (Postfix) with ESMTP id 21E708FC13 for ; Sat, 20 Dec 2008 22:33:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from skuns.zoral.com.ua ([91.193.166.194] helo=mail.zoral.com.ua) by mail.terabit.net.ua with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1LEANu-0005oL-PV; Sun, 21 Dec 2008 00:33:22 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id mBKMXJqU057419 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 21 Dec 2008 00:33:20 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id mBKMXJt9083347; Sun, 21 Dec 2008 00:33:19 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id mBKMXJPp083346; Sun, 21 Dec 2008 00:33:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 21 Dec 2008 00:33:19 +0200 From: Kostik Belousov To: Greg Miller Message-ID: <20081220223319.GE2038@deviant.kiev.zoral.com.ua> References: <494D2C64.1030105@classic-games.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9Fi1pPcXZLIREiYy" Content-Disposition: inline In-Reply-To: <494D2C64.1030105@classic-games.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.94.2, clamav-milter version 0.94.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-Virus-Scanned: mail.terabit.net.ua 1LEANu-0005oL-PV 5d3ae66f141fa24702d0e432072817bb X-Terabit: YES Cc: stable-list freebsd Subject: Re: Lockups and panics during multiuser boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Dec 2008 22:33:24 -0000 --9Fi1pPcXZLIREiYy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 20, 2008 at 11:33:24AM -0600, Greg Miller wrote: >=20 > After upgrading to 7.1RC1, I found I could only boot single-user without= =20 > a panic or a lockup. After bisecting recent changes on RELENG_7, I=20 > found that the problem occurs after 2008.10.29.15.00.00 and before=20 > 2008.10.29.18.00.00 You forgot to add the actual panic messages and backtraces. --9Fi1pPcXZLIREiYy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAklNcq4ACgkQC3+MBN1Mb4h/DwCfajkP3NeOSmWtpJQQEYRo3r9t /QoAoNpURDMaWk3KpXLJAx79aZl3mzB8 =41Bw -----END PGP SIGNATURE----- --9Fi1pPcXZLIREiYy--