Date: Wed, 19 Oct 2005 17:54:50 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Philip Kizer <pckizer@nostrum.com> Cc: freebsd-current@freebsd.org Subject: Re: Problem remains with FreeBSD 6.0-RC1 as seen in RELENG_5 Message-ID: <20051019175020.S60849@fledge.watson.org> In-Reply-To: <200510191623.j9JGNSfr007356@magus.nostrum.com> References: <200510191623.j9JGNSfr007356@magus.nostrum.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 19 Oct 2005, Philip Kizer wrote: > I have a problem I reported on freebsd-stable several weeks ago in: > > > http://www.FreeBSD.org/cgi/getmsg.cgi?fetch=47770+187449+/usr/local/www/db/text/2005/freebsd-stable/20051009.freebsd-stable > > I upgraded a test box to see if all of the reports were true that > threading and most other major problems were better in the 6.x branch, > but I have had the same kind of hangs with 6.0-RC1 that I was having > with RELENG_5. > > I get notified that some of my services are unavailable and I verify > that new connection attempts from remote just hang. Attempts at issuing > commands from my existing ssh connections will let me send a <return> > and see a new prompt generated, but any attempt at execing/etc will then > hang that process. This appears to be a problem with file descriptor passing and garbage collection. I've seen one report of a lock order reversal along these lines, but it was not believed to be symptomatic of an actual hang, just an architectural issue. This could be a sign that we need to address the source of the reversal, although it sounds like you don't get a reversal warning? Could I have you try the following DDB commands also: show alllocks traceall It's possible that 'traceall' isn't present in RC1, in which case it will be present in RC2. Regardless, the output of "show alllocks" would be helpful. I'll read some code this evening and ponder. Thanks, Robert N M Watson > Moving to the console, I get the same behaviour such that if it is already > logged in, I can hit return in the shell and get a new shell prompt, but > any command I try (i.e. 'uptime' or 'uname') then results in the same lack > of any further response. [Though, obviously from below, sending a break to > activate DDB still works.] > > If the console is not already logged in and I hit return at "login:", I get > a new "login:" prompt; but, as soon as I enter a username+^M or even Ctrl-D > it, I cease to receive any feedback besides terminal echo from my input. > > > This is: > FreeBSD [host-elided] 6.0-RC1 FreeBSD 6.0-RC1 #0: Mon Oct 17 15:52:35 CDT 2005 root@[host-elided]:/usr/obj/usr/src/sys/SMP_DDB i386 > > # grep '^[a-z]' /sys/i386/conf/SMP_DDB > include GENERIC > ident DDB > options SMP # Symmetric MultiProcessor Kernel > options INCLUDE_CONFIG_FILE # Include this file in kernel > options DDB #Enable the kernel debugger > options DDB_NUMSYM #Print numerical value of symbols too > options KDB > options KDB_TRACE > options BREAK_TO_DEBUGGER > > [After notification of a problem, this is from the serial console...a few > further inline comments included in brackets "[]".] > > # > [Typed: Ctrl-T] > load: 1.82 cmd: sh 795 [ttyin] 0.00u 0.00s 0% 168k > # uptime > [Waited a while to be sure it was hung and not returning. Retyped: Ctrl-T] > load: 1.82 cmd: uptime 83303 [filelist lock] 0.00u 0.00s 0% 180k > [Sent BREAK] > KDB: enter: Line break on console > [thread pid 82885 tid 100133 ] > Stopped at 0xc0659b37 = kdb_enter+0x2b: nop > db> set $lines=0 > db> trace > Tracing pid 82885 tid 100133 td 0xc3c34c00 > kdb_enter(c0895a5e) at 0xc0659b37 = kdb_enter+0x2b > siointr1(c372ec00) at 0xc07f09b2 = siointr1+0xce > siointr(c372ec00) at 0xc07f0792 = siointr+0x5e > intr_execute_handlers(c3528090,f1325b68,4,f1325bcc,c0805073) at 0xc0808985 = intr_execute_handlers+0x85 > lapic_handle_intr(34) at 0xc080b042 = lapic_handle_intr+0x2e > Xapic_isr1() at 0xc0805073 = Xapic_isr1+0x33 > --- interrupt, eip = 0xc0685d59, esp = 0xf1325bac, ebp = 0xf1325bcc --- > unp_gc(c43be2c8) at 0xc0685d59 = unp_gc+0xf1 > unp_detach(c43b7000,f1325c08,c067a188,c43be2c8,0) at 0xc06840fa = unp_detach+0xae > uipc_detach(c43be2c8) at 0xc0682f8e = uipc_detach+0x82 > soclose(c43be2c8,c40fb6c0,0,f1325c34,c0620c10) at 0xc067a188 = soclose+0x278 > soo_close(c40fb6c0,c3c34c00) at 0xc0669fb3 = soo_close+0x63 > fdrop_locked(c40fb6c0,c3c34c00,c3a3d800,f1325cb4,c061f0fb) at 0xc0620c10 = fdrop_locked+0xac > fdrop(c40fb6c0,c3c34c00,c066371f,c40fb6c0,c878ac00) at 0xc0620b5d = fdrop+0x41 > closef(c40fb6c0,c3c34c00,0,c3c34c00,6) at 0xc061f0fb = closef+0x427 > close(c3c34c00,f1325d04,1,0,292) at 0xc061c4bd = close+0x211 > syscall(3b,1003b,bfbf003b,80d58e7,80e8780) at 0xc0818557 = syscall+0x2bf > Xint0x80_syscall() at 0xc0804d0f = Xint0x80_syscall+0x1f > --- syscall (6, FreeBSD ELF32, close), eip = 0x282cb7d3, esp = 0xbfbfae4c, ebp = 0xbfbfae68 --- > db> show reg > cs 0x20 > ds 0xc0870028 > es 0xc0650028 parse_dynamic+0x4 > fs 0xf1320008 > ss 0x28 > eax 0x22 > ecx 0xc1033000 > edx 0 > ebx 0xc3c34c00 > esp 0xf1325afc > ebp 0xf1325afc > esi 0xc372ec00 > edi 0xf9 > eip 0xc0659b37 kdb_enter+0x2b > efl 0x86 > dr0 0 > dr1 0 > dr2 0 > dr3 0 > dr4 0xffff0ff0 > dr5 0x400 > dr6 0xffff0ff0 > dr7 0x400 > 0xc0659b37 = kdb_enter+0x2b: nop > db> show witness > No such command > db> show pcpu > cpuid = 1 > curthread = 0xc3c34c00: pid 82885 \"sendmail\" > curpcb = 0xf1325d90 > fpcurthread = none > idlethread = 0xc34d8780: pid 11 \"idle: cpu1\" > APIC ID = 0 > currentldt = 0x50 > db> show pcpuu0 > cpuid = 0 > curthread = 0xc34d8900: pid 12 \"idle: cpu0\" > curpcb = 0xe4f9bd90 > fpcurthread = none > idlethread = 0xc34d8900: pid 12 \"idle: cpu0\" > APIC ID = 1 > currentldt = 0x50 > db> show pcpu 1 > cpuid = 1 > curthread = 0xc3c34c00: pid 82885 \"sendmail\" > curpcb = 0xf1325d90 > fpcurthread = none > idlethread = 0xc34d8780: pid 11 \"idle: cpu1\" > APIC ID = 0 > currentldt = 0x50 > db> show pcpu 2 > CPU 2 not found > db> print sysctllock > c093ace0 > db> ps > pid proc uid ppid pgrp flag stat wmesg wchan cmd > 83305 c43b2830 0 504 504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron > 83304 c3d91a3c 21802 605 605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 83303 c3d12c48 0 795 83303 0004002 [SLPQ filelist lock 0xc0938bd0][SLP] uptime > 83302 c5b13418 0 686 686 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83301 c3ba1c48 0 680 680 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83300 c40c320c 0 504 504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron > 83299 c43a2830 0 504 504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron > 83298 c38dec48 0 504 504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron > 83297 c3f0020c 21802 605 605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 83296 c3c36624 21802 605 605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 83295 c406620c 21802 605 605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 83294 c38de20c 21802 605 605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 83293 c41af418 0 686 686 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83292 c4244a3c 0 680 680 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83291 c40c0624 0 504 504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron > 83290 c43b1a3c 0 504 504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron > 83289 c5fffc48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83288 c3e9d830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83287 c3cbbc48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83286 c43a2000 0 636 636 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83285 c40c4c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83284 c413b624 0 636 636 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83283 c6003418 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83282 c4140000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83281 c43ac000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83280 c3d91830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83279 c43afc48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83278 c3cbb418 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83277 c4244c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83276 c40c0830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83275 c4136624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83274 c4140a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83273 c6035830 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83272 c43b120c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83271 c41ada3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83270 c3fca624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83269 c6035c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83268 c5b13c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83267 c3f82c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83266 c424320c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83265 c43a720c 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83264 c3e9d418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83263 c3d18000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83261 c6036418 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83260 c5b0120c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83259 c4244000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83258 c4243830 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83257 c43af830 0 1 83257 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83255 c5b01830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83254 c5b01000 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83248 c3b99c48 0 680 680 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83247 c42d5c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83246 c3f00a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83245 c6036000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83244 c40c3c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83243 c5fff830 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83242 c5b01418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83241 c42a2624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83240 c41af000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83239 c41ad830 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83238 c3d1820c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83237 c3f00624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83236 c5b13a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83235 c42a7a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83234 c3f82830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83233 c43a7624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83231 c4376418 0 683 683 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83230 c406e830 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83229 c42d520c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83228 c43af418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83227 c413ba3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83226 c3e9d20c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83225 c43a220c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83224 c3b99418 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83223 c3c33c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83222 c4375000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83221 c3e9d624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83220 c38de830 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83219 c3f8020c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83218 c3f02a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83213 c4375a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83212 c5b09418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83211 c43af624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83208 c4135830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83207 c41afa3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83206 c406ea3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83205 c413b418 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83195 c3f80624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83193 c5b0920c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83192 c3c33624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83191 c3c36a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83190 c4140418 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83189 c4135418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83188 c437620c 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83187 c406e624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83186 c43a7418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83185 c3c36c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83184 c3cbb830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83183 c406e000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83182 c43a2c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83181 c40c3000 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83180 c4140c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83179 c42a2418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83178 c437e000 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83175 c5fff20c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83174 c3f82624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83168 c3ba1830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83167 c4243a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83164 c6003000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83162 c4066a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83157 c40c4000 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83154 c600320c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83153 c40c0c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83152 c3d18418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83150 c41adc48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83149 c3e9dc48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83148 c3cba624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83146 c3c36000 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83145 c3f01830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83142 c41af20c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83141 c3e9da3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83140 c5fff624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83134 c41af624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83132 c4136830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83131 c603620c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83130 c3f00418 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83129 c3fcac48 0 664 664 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83128 c40c4418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83125 c42d4c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83124 c3fca20c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83123 c437e418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83121 c4135c48 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83119 c41af830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83118 c38dd000 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83117 c42d4a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83114 c42a720c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83113 c41aac48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83112 c42d420c 0 664 664 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83109 c40c4a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83108 c3f8220c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83107 c3d10830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83105 c41aa000 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail > 83101 c4066624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83100 c40c3624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83097 c42a2000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83094 c41aa830 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83093 c43a7a3c 0 664 664 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 83092 c43b1000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83091 c437ec48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83087 c5fff000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83070 c42a2830 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83069 c6036c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83056 c6035624 0 664 664 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail > 83051 c41ad20c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83050 c4244830 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83041 c40c0a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83038 c424420c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 83037 c43a2624 0 664 664 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail > 83034 c413b000 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82994 c43b2000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82993 c38d8c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82992 c5b01a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82991 c5b01c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82990 c43b2624 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82989 c5b09000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82987 c6036830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82986 c42d5624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82984 c5b09624 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82983 c5b09830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82982 c4135a3c 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 82981 c3f82000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82976 c42d5830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82975 c5b09c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82974 c43acc48 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82973 c3f02418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82972 c5b13000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82971 c43ac20c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82970 c3cbb20c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82969 c43a2418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82967 c43a7000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82966 c40c0000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82965 c4376c48 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82964 c43b1c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82962 c41ad624 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82961 c38dea3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82960 c38a820c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82959 c43b0c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82952 c42a2a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82951 c6035000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82948 c43b220c 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 82947 c6035a3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82945 c43b2418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82944 c41aaa3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82938 c40c420c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82937 c6036a3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82932 c3f01a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82931 c42d4624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82930 c41aa418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82929 c3d94c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82928 c3f02830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82927 c3d94a3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82926 c413620c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82925 c3fcbc48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82924 c42d5a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82923 c4140830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82922 c43ac830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82921 c3f0220c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82919 c43b0418 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82918 c3cbba3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82917 c4066830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82916 c3ba1624 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82915 c4376000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82914 c3f01c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82913 c43b1418 0 636 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82909 c4243c48 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82908 c3c33000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82907 c4243000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82906 c437ea3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82905 c3ba1000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82904 c43b1624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82903 c40c3a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82902 c3f82a3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82896 c43b0830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82895 c4136000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82894 c42a7000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82893 c3e9d000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82892 c3c3320c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82891 c42a220c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82890 c4375c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82889 c3ba1a3c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82888 c43af000 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82887 c42a2c48 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82886 c3c33418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82885 c3c36830 0 636 636 0000100 [CPU 1] sendmail > 82877 c42a7418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82876 c3cbaa3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82874 c413bc48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82870 c43a7c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82865 c437520c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82864 c4376830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82851 c3f80418 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail > 82837 c3f02c48 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82782 c42d5000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82781 c3d9120c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82769 c43aca3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82766 c40c020c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82765 c41aa20c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82763 c3d1220c 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82762 c3d9420c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82761 c42a7624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82740 c6036624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82738 c4136a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82737 c4243418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 82729 c3f82418 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82676 c4376624 0 664 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82621 c3c33a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 82607 c4376a3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 81310 c6035418 0 664 664 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail > 80076 c42d4000 0 650 650 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail > 79627 c4135624 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 79198 c3cba830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79197 c414020c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79193 c4066418 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 79192 c40c3830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79189 c3f00000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79188 c437e830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79186 c3d18830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79185 c3fca418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79184 c4136418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79183 c3fcaa3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79179 c3f80000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79178 c3fcb20c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79168 c603520c 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 79167 c4243624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 79083 c4375830 0 664 664 0000100 [SLPQ sbwait 0xc4401900][SLP][SWAP] sendmail > 78838 c3f01000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 78837 c3f80830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 78491 c40c4624 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 78490 c3d94830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 78484 c3f02000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 78483 c406ec48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 73847 c43ac418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 73846 c42d4418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 68027 c43b2a3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 67937 c3fcb000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 67936 c3fcb624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 67933 c42d4830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 67932 c38de624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 67904 c413b830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 67903 c3cbb624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 66287 c43afa3c 0 650 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 60279 c3f00830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 60278 c4244624 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 60277 c3fcba3c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 60276 c3fca000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 60249 c406e418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 60248 c3f00c48 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51444 c3c3620c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51443 c4066000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51430 c3b99624 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51429 c41aa624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51427 c40c4830 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51426 c3f0120c 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51231 c4375624 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 51230 c3fcb830 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51228 c3d10a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51227 c4140624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51222 c413b20c 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 51221 c38de418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51218 c43b020c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51217 c3cbac48 21802 550 550 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 51215 c43a2a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51214 c437e20c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51213 c4244418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51212 c3f02624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51211 c3f80a3c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51206 c43b0624 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51161 c413520c 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51160 c42d5418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 51153 c40c0418 21802 551 551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 50987 c3c36418 0 1 50987 0000100 [SLPQ sbwait 0xc3da3c40][SLP] sendmail > 36954 c4135000 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 36941 c38dd418 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 32803 c43af20c 0 32793 32803 0004003 [SLPQ ttyin 0xc3871810][SLP][SWAP] zsh > 32793 c43b2c48 127 32792 32793 0004002 [SLPQ pause 0xc43b2c7c][SLP][SWAP] zsh > 32792 c42a7c48 127 32781 32781 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sshd > 32781 c3fcb418 0 491 32781 0004100 [SLPQ sbwait 0xc4547e90][SLP][SWAP] sshd > 31218 c3cbb000 21802 551 551 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 31217 c3f01418 21802 550 550 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 5491 c43b0a3c 0 1 5491 0000100 [SLPQ sbwait 0xc45473e8][SLP] sendmail > 795 c3d91c48 0 778 795 0004002 [SLPQ wait 0xc3d91c48][SLP] sh > 778 c3d94418 0 760 778 0004002 [SLPQ pause 0xc3d9444c][SLP][SWAP] zsh > 760 c3ea1000 0 1 760 0004102 [SLPQ wait 0xc3ea1000][SLP][SWAP] login > 759 c3ea120c 0 1 759 0004002 [SLPQ ttyin 0xc3737410][SLP][SWAP] getty > 758 c3ea1418 0 1 758 0004002 [SLPQ ttyin 0xc3734c10][SLP][SWAP] getty > 757 c3ea1624 0 1 757 0004002 [SLPQ ttyin 0xc3734410][SLP][SWAP] getty > 756 c3ea1830 0 1 756 0004002 [SLPQ ttyin 0xc372f410][SLP][SWAP] getty > 755 c3ea1a3c 0 1 755 0004002 [SLPQ ttyin 0xc3736410][SLP][SWAP] getty > 754 c3d18c48 0 1 754 0004002 [SLPQ ttyin 0xc3730c10][SLP][SWAP] getty > 753 c3d18624 0 1 753 0004002 [SLPQ ttyin 0xc3735810][SLP][SWAP] getty > 752 c38de000 0 1 752 0004002 [SLPQ ttyin 0xc3734810][SLP][SWAP] getty > 719 c3d91418 0 1 717 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] snmpd > 709 c3d18a3c 0 1 709 0000100 [SLPQ pause 0xc3d18a70][SLP][SWAP] sendmail > 706 c3d12418 0 1 706 0000100 [SLPQ pause 0xc3d1244c][SLP][SWAP] sendmail > 703 c3d91000 0 1 703 0000100 [SLPQ pause 0xc3d91034][SLP][SWAP] sendmail > 700 c3d91624 0 1 700 0000100 [SLPQ pause 0xc3d91658][SLP][SWAP] sendmail > 696 c3d12a3c 0 1 696 0000100 [SLPQ pause 0xc3d12a70][SLP][SWAP] sendmail > 692 c3d94000 0 1 692 0000100 [SLPQ pause 0xc3d94034][SLP][SWAP] sendmail > 689 c3d94624 0 1 689 0000100 [SLPQ pause 0xc3d94658][SLP][SWAP] sendmail > 686 c3d1020c 0 1 686 0000100 [SLPQ pause 0xc3d10240][SLP] sendmail > 683 c3d10624 0 1 683 0000100 [SLPQ pause 0xc3d10658][SLP][SWAP] sendmail > 680 c3d12000 0 1 680 0000100 [SLPQ pause 0xc3d12034][SLP] sendmail > 678 c3d12624 0 1 678 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail > 664 c3d10000 0 1 664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 650 c3cba000 0 1 650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 636 c3d10418 0 1 636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail > 620 c38d8418 0 616 616 0000001 [SLPQ lockf 0xc41d3bc0][SLP][SWAP] saslauthd > 619 c3b99830 0 616 616 0000001 [SLPQ lockf 0xc878a040][SLP][SWAP] saslauthd > 618 c3cba418 0 616 616 0000001 [SLPQ lockf 0xc44dc900][SLP][SWAP] saslauthd > 617 c38d8624 0 616 616 0000001 [SLPQ accept 0xc3cf0466][SLP][SWAP] saslauthd > 616 c3d10c48 0 1 616 0000001 [SLPQ lockf 0xc44dc380][SLP][SWAP] saslauthd > 605 c3cba20c 21802 1 605 0000000 [SLPQ select 0xc0941aa4][SLP] perl > 604 c3d12830 0 577 604 0004003 [SLPQ ttyin 0xc3725810][SLP][SWAP] zsh > 577 c3c33830 127 576 577 0004002 [SLPQ pause 0xc3c33864][SLP][SWAP] zsh > 576 c3b9920c 127 574 574 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sshd > 574 c38dd624 0 491 574 0004100 [SLPQ sbwait 0xc3c30638][SLP][SWAP] sshd > 551 c38a5a3c 21802 1 551 0000001 [SLPQ select 0xc0941aa4][SLP] perl > 550 c38ddc48 21802 1 550 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl > 546 c3b99a3c 21802 1 546 0000001 [SLPQ select 0xc0941aa4][SLP] perl > 539 c38a5830 21802 534 534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd > 538 c38a8000 21802 534 534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd > 537 c38d820c 21802 534 534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd > 536 c3ba120c 21802 534 534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd > 535 c3ba1418 21802 534 534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd > 534 c38d8000 21802 1 534 0000000 [SLPQ select 0xc0941aa4][SLP] httpd > 527 c38a5624 0 1 527 0008180 (threaded) perl > thread 0xc41c1a80 ksegrp 0xc34db660 [SLPQ kserel 0xc34db694][SLP] > thread 0xc879b300 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc43a6a80 ksegrp 0xc34db660 [SLPQ kserel 0xc34db694][SLP] > thread 0xc42eec00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e8480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ade000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ee480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a9300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a2480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc423a600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44f2600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a0b780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a36600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ee780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e7a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a20000 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc87e9c00 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc42efa80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc41c0780 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc4099900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc43c9780 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc43ca180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c4180 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc41c1600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a2a80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5a20d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422d480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a20600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea2900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41a1180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a3780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4096480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a6900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442dc00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f03000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a20a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adf180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599d480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ef600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442e600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ade480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e7180 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc879ba80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc59b5180 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc4120d80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5a0b900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc87e7600 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc599d900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc423a900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc3ea2600 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc87e8780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422d780 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5a20900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5a36480 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc43ca300 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc41a1600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3c35780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3d96900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87ead80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4067000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f28180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ade600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422d180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e7480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44ef180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599d180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ade780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f92900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422d600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adf000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f8fa80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adea80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea2480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc423aa80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adf780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f03180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4099300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442d480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4068780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4095d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a2000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e6d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43ca480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3d17600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e8c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42f1600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a6780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422c600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442d300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a97000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea3c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44efd80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87ea780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ade900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c4300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a20c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e6900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3d17480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ae1c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43cad80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87ea900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422dd80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e7d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adf900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4aeb780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a20780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a2180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42f1480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c1780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c1180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44ef480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ae1a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a3900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44ef900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3d95d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599d600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422cc00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599d000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea2300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e6480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42f1d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e8600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4120780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599d780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41bfc00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4aeb900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ee900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4067a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP] > thread 0xc38df900 ksegrp 0xc354d8a0 [SLPQ ksesigwait 0xc38a575c][SLP] > 520 c38dd830 21803 1 69 000c083 (threaded) perl > thread 0xc43a9600 ksegrp 0xc354dcc0 [SLPQ kserel 0xc354dcf4][SLP] > thread 0xc8a97a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41a0c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a0bc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f8f180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a97d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599dc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ae1480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ee300 ksegrp 0xc354dcc0 [SLPQ kserel 0xc354dcf4][SLP] > thread 0xc87e8a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5600 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc87e7300 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5a0bd80 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc442da80 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc879b480 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc879b780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44f2180 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc3f90780 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5987d80 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc879bc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e7c00 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc42ee180 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc87a2300 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc59b5480 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc442e900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987780 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP] > thread 0xc5a0b480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc599dd80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87ea180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3c34780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c0900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a9900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f92c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a3480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f81d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41a0d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a36000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a9480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42f1900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a2d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c0600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4120000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e8900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e7780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a3300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc423a480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4099d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a20480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442d180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4096d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43ca600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5a36180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc423ad80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442ec00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c4c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4096a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea3600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c4000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42eed80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a9a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adfd80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ade180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41a1480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e6a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a9780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4aeb000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ef480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4120a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44ef300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a3a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a9180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e8180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adf600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc423a300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41bfd80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea3480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422c780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422c300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422d900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc44f2000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a97600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f28780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4aeb180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e8d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f28c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c0a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41bf600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422c900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42f1a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3ea3900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43caa80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43ca900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43c9c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442d600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87ea480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc43a8180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c1c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3b9a000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41bf180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4aebc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422da80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e5000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc422c180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87eaa80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42ef300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e6000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42f1c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59e7a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc41c0d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4adfc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442e780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4099000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87a4300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a97180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4121480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc59b5300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc879b600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc87e9600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f90c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a96d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc8a97300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4122a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc42eea80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc5987300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc423a780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4068300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ae1180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc3f8f480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc442e180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc4ae1900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP] > thread 0xc38e0180 ksegrp 0xc354d900 [SLPQ ksesigwait 0xc38dd968][SLP] > 519 c38a5c48 0 1 69 0004002 [SLPQ sbwait 0xc39cabc8][SLP][SWAP] perl > 504 c38a5418 0 1 504 0000000 [SLPQ nanslp 0xc093af0c][SLP] cron > 491 c38a5000 0 1 491 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sshd > 474 c38dd20c 0 1 474 0000000 [SLPQ select 0xc0941aa4][SLP] ntpd > 381 c38a520c 0 1 381 0000000 [SLPQ select 0xc0941aa4][SLP] syslogd > 352 c38d8830 0 1 352 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] devd > 268 c38dda3c 64 262 262 0000100 [SLPQ bpf 0xc38dce00][SLP] pflogd > 262 c38d8a3c 0 1 262 0000000 [SLPQ sbwait 0xc39ca370][SLP][SWAP] pflogd > 68 c38a8418 0 0 0 0000204 [SLPQ - 0xef03ed08][SLP] schedcpu > 67 c38a8624 0 0 0 0000204 [SLPQ - 0xc094a02c][SLP] nfsiod 3 > 66 c38a8830 0 0 0 0000204 [SLPQ - 0xc094a028][SLP] nfsiod 2 > 65 c38a8a3c 0 0 0 0000204 [SLPQ - 0xc094a024][SLP] nfsiod 1 > 64 c38a8c48 0 0 0 0000204 [SLPQ - 0xc094a020][SLP] nfsiod 0 > 63 c355a20c 0 0 0 0000204 [SLPQ vlruwt 0xc355a20c][SLP] vnlru > 62 c355a418 0 0 0 0000204 [SLPQ syncer 0xc093ac7c][SLP] syncer > 61 c355a624 0 0 0 0000204 [SLPQ psleep 0xc0941fec][SLP] bufdaemon > 60 c355a830 0 0 0 000020c [SLPQ pgzero 0xc09505e4][SLP] pagezero > 59 c355aa3c 0 0 0 0000204 [SLPQ psleep 0xc0950134][SLP] vmdaemon > 58 c355ac48 0 0 0 0000204 [SLPQ psleep 0xc09500f0][SLP] pagedaemon > 57 c3603000 0 0 0 0000204 [IWAIT] swi0: sio > 56 c360320c 0 0 0 0000204 [SLPQ - 0xc372903c][SLP] fdc0 > 55 c3603418 0 0 0 0000204 [SLPQ aifthd 0xc3603418][SLP] aac0aif > 54 c3603624 0 0 0 0000204 [SLPQ idle 0xc3528c00][SLP] aic_recovery0 > 53 c3603830 0 0 0 0000204 [SLPQ idle 0xc3528c00][SLP] aic_recovery0 > 52 c3603a3c 0 0 0 0000204 [IWAIT] swi6:+ > 51 c3603c48 0 0 0 0000204 [IWAIT] swi6: task queue > 9 c354b624 0 0 0 0000204 [SLPQ - 0xc3563600][SLP] acpi_task2 > 8 c354b830 0 0 0 0000204 [SLPQ - 0xc3563600][SLP] acpi_task1 > 7 c354ba3c 0 0 0 0000204 [SLPQ - 0xc3563600][SLP] acpi_task0 > 50 c354bc48 0 0 0 0000204 [IWAIT] swi2: cambio > 6 c3559000 0 0 0 0000204 [SLPQ - 0xc3563780][SLP] kqueue taskq > 49 c355920c 0 0 0 0000204 [IWAIT] swi5:+ > 5 c3559418 0 0 0 0000204 [SLPQ - 0xc3563980][SLP] thread taskq > 48 c3559624 0 0 0 0000204 [SLPQ - 0xc0935c60][SLP] yarrow > 4 c3559830 0 0 0 0000204 [SLPQ - 0xc0938428][SLP] g_down > 3 c3559a3c 0 0 0 0000204 [SLPQ - 0xc0938424][SLP] g_up > 2 c3559c48 0 0 0 0000204 [SLPQ - 0xc093841c][SLP] g_event > 47 c355a000 0 0 0 0000204 [IWAIT] swi1: net > 46 c353bc48 0 0 0 0000204 [IWAIT] swi3: vm > 45 c354a000 0 0 0 000020c [IWAIT] swi4: clock sio > 44 c354a20c 0 0 0 0000204 [IWAIT] irq0: > 43 c354a418 0 0 0 0000204 [IWAIT] irq31: aac0 > 42 c354a624 0 0 0 0000204 [IWAIT] irq30: ahc0 > 41 c354a830 0 0 0 0000204 [IWAIT] irq29: > 40 c354aa3c 0 0 0 0000204 [IWAIT] irq28: > 39 c354ac48 0 0 0 0000204 [IWAIT] irq27: > 38 c354b000 0 0 0 0000204 [IWAIT] irq26: > 37 c354b20c 0 0 0 0000204 [IWAIT] irq25: > 36 c354b418 0 0 0 0000204 [IWAIT] irq24: > 35 c3529624 0 0 0 0000204 [IWAIT] irq23: > 34 c3529830 0 0 0 0000204 [IWAIT] irq22: fxp0 > 33 c3529a3c 0 0 0 0000204 [IWAIT] irq21: > 32 c3529c48 0 0 0 0000204 [IWAIT] irq20: > 31 c353b000 0 0 0 0000204 [IWAIT] irq19: > 30 c353b20c 0 0 0 0000204 [IWAIT] irq18: fxp1 > 29 c353b418 0 0 0 0000204 [IWAIT] irq17: bge0 > 28 c353b624 0 0 0 0000204 [IWAIT] irq16: fxp2 > 27 c353b830 0 0 0 0000204 [IWAIT] irq15: ata1 > 26 c353ba3c 0 0 0 0000204 [IWAIT] irq14: ata0 > 25 c34dc20c 0 0 0 0000204 [IWAIT] irq13: > 24 c34dc418 0 0 0 0000204 [IWAIT] irq12: > 23 c34dc624 0 0 0 0000204 [IWAIT] irq11: > 22 c34dc830 0 0 0 0000204 [IWAIT] irq10: > 21 c34dca3c 0 0 0 0000204 [IWAIT] irq9: acpi0 > 20 c34dcc48 0 0 0 0000204 [IWAIT] irq8: > 19 c3529000 0 0 0 0000204 [IWAIT] irq7: ppc0 > 18 c352920c 0 0 0 0000204 [IWAIT] irq6: fdc0 > 17 c3529418 0 0 0 0000204 [IWAIT] irq5: > 16 c34d7000 0 0 0 0000204 [IWAIT] irq4: sio0 > 15 c34d720c 0 0 0 0000204 [IWAIT] irq3: sio1 > 14 c34d7418 0 0 0 0000204 [IWAIT] irq2: > 13 c34d7624 0 0 0 0000204 [IWAIT] irq1: atkbd0 > 12 c34d7830 0 0 0 000020c [CPU 0] idle: cpu0 > 11 c34d7a3c 0 0 0 000020c [Can run] idle: cpu1 > 1 c34d7c48 0 0 1 0004200 [SLPQ wait 0xc34d7c48][SLP] init > 10 c34dc000 0 0 0 0000204 [SLPQ ktrace 0xc0938e78][SLP] ktrace > 0 c0938520 0 0 0 0000200 [IWAIT] swapper > db> panic > panic: from debugger > cpuid = 1 > KDB: stack backtrace: > kdb_backtrace(100,c3c34c00,0,c089e89c,c089e8b8) at 0xc0659ab5 = kdb_backtrace+0x29 > panic(c084a8b4,f1325a04,c0468c70,c093ace0,0) at 0xc06408b8 = panic+0x114 > db_panic(c093ace0,0,ffffffff,f1325978,0) at 0xc0468cd9 = db_panic+0xd > db_command(c09217e4,c08a7820,c089e89c,c089e8b8,c084a8c2) at 0xc0468c70 = db_command+0x264 > db_command_loop(0,0,f1325a30,f1325a1c,f1325a64) at 0xc0468d38 = db_command_loop+0x5c > db_trap(3,0,c3c34c00,3,c3c34c00) at 0xc046a949 = db_trap+0xdd > kdb_trap(3,0,f1325abc) at 0xc0659dd0 = kdb_trap+0x88 > trap(f1320008,c0650028,c0870028,f9,c372ec00) at 0xc0817c88 = trap+0x498 > calltrap() at 0xc0804cba = calltrap+0x5 > --- trap 0x3, eip = 0xc0659b37, esp = 0xf1325afc, ebp = 0xf1325afc --- > kdb_enter(c0895a5e) at 0xc0659b37 = kdb_enter+0x2b > siointr1(c372ec00) at 0xc07f09b2 = siointr1+0xce > siointr(c372ec00) at 0xc07f0792 = siointr+0x5e > intr_execute_handlers(c3528090,f1325b68,4,f1325bcc,c0805073) at 0xc0808985 = intr_execute_handlers+0x85 > lapic_handle_intr(34) at 0xc080b042 = lapic_handle_intr+0x2e > Xapic_isr1() at 0xc0805073 = Xapic_isr1+0x33 > --- interrupt, eip = 0xc0685d59, esp = 0xf1325bac, ebp = 0xf1325bcc --- > unp_gc(c43be2c8) at 0xc0685d59 = unp_gc+0xf1 > unp_detach(c43b7000,f1325c08,c067a188,c43be2c8,0) at 0xc06840fa = unp_detach+0xae > uipc_detach(c43be2c8) at 0xc0682f8e = uipc_detach+0x82 > soclose(c43be2c8,c40fb6c0,0,f1325c34,c0620c10) at 0xc067a188 = soclose+0x278 > soo_close(c40fb6c0,c3c34c00) at 0xc0669fb3 = soo_close+0x63 > fdrop_locked(c40fb6c0,c3c34c00,c3a3d800,f1325cb4,c061f0fb) at 0xc0620c10 = fdrop_locked+0xac > fdrop(c40fb6c0,c3c34c00,c066371f,c40fb6c0,c878ac00) at 0xc0620b5d = fdrop+0x41 > closef(c40fb6c0,c3c34c00,0,c3c34c00,6) at 0xc061f0fb = closef+0x427 > close(c3c34c00,f1325d04,1,0,292) at 0xc061c4bd = close+0x211 > syscall(3b,1003b,bfbf003b,80d58e7,80e8780) at 0xc0818557 = syscall+0x2bf > Xint0x80_syscall() at 0xc0804d0f = Xint0x80_syscall+0x1f > --- syscall (6, FreeBSD ELF32, close), eip = 0x282cb7d3, esp = 0xbfbfae4c, ebp = 0xbfbfae68 --- > KDB: stack backtrace: > kdb_backtrace(28d5a61e,53755e05,ffc03014,c3c34c00,c084a8b4) at 0xc0659ab5 = kdb_backtrace+0x29 > mi_switch(1,0,f1325924,c0640190,c3c34c00) at 0xc0647162 = mi_switch+0xf6 > sched_bind(c3c34c00,0) at 0xc065287c = sched_bind+0x44 > boot(104,104,c3c34c00,0,c089e89c) at 0xc0640190 = boot+0x64 > panic(c084a8b4,f1325a04,c0468c70,c093ace0,0) at 0xc064095d = panic+0x1b9 > db_panic(c093ace0,0,ffffffff,f1325978,0) at 0xc0468cd9 = db_panic+0xd > db_command(c09217e4,c08a7820,c089e89c,c089e8b8,c084a8c2) at 0xc0468c70 = db_command+0x264 > db_command_loop(0,0,f1325a30,f1325a1c,f1325a64) at 0xc0468d38 = db_command_loop+0x5c > db_trap(3,0,c3c34c00,3,c3c34c00) at 0xc046a949 = db_trap+0xdd > kdb_trap(3,0,f1325abc) at 0xc0659dd0 = kdb_trap+0x88 > trap(f1320008,c0650028,c0870028,f9,c372ec00) at 0xc0817c88 = trap+0x498 > calltrap() at 0xc0804cba = calltrap+0x5 > --- trap 0x3, eip = 0xc0659b37, esp = 0xf1325afc, ebp = 0xf1325afc --- > kdb_enter(c0895a5e) at 0xc0659b37 = kdb_enter+0x2b > siointr1(c372ec00) at 0xc07f09b2 = siointr1+0xce > siointr(c372ec00) at 0xc07f0792 = siointr+0x5e > intr_execute_handlers(c3528090,f1325b68,4,f1325bcc,c0805073) at 0xc0808985 = intr_execute_handlers+0x85 > lapic_handle_intr(34) at 0xc080b042 = lapic_handle_intr+0x2e > Xapic_isr1() at 0xc0805073 = Xapic_isr1+0x33 > --- interrupt, eip = 0xc0685d59, esp = 0xf1325bac, ebp = 0xf1325bcc --- > unp_gc(c43be2c8) at 0xc0685d59 = unp_gc+0xf1 > unp_detach(c43b7000,f1325c08,c067a188,c43be2c8,0) at 0xc06840fa = unp_detach+0xae > uipc_detach(c43be2c8) at 0xc0682f8e = uipc_detach+0x82 > soclose(c43be2c8,c40fb6c0,0,f1325c34,c0620c10) at 0xc067a188 = soclose+0x278 > soo_close(c40fb6c0,c3c34c00) at 0xc0669fb3 = soo_close+0x63 > fdrop_locked(c40fb6c0,c3c34c00,c3a3d800,f1325cb4,c061f0fb) at 0xc0620c10 = fdrop_locked+0xac > fdrop(c40fb6c0,c3c34c00,c066371f,c40fb6c0,c878ac00) at 0xc0620b5d = fdrop+0x41 > closef(c40fb6c0,c3c34c00,0,c3c34c00,6) at 0xc061f0fb = closef+0x427 > close(c3c34c00,f1325d04,1,0,292) at 0xc061c4bd = close+0x211 > syscall(3b,1003b,bfbf003b,80d58e7,80e8780) at 0xc0818557 = syscall+0x2bf > Xint0x80_syscall() at 0xc0804d0f = Xint0x80_syscall+0x1f > --- syscall (6, FreeBSD ELF32, close), eip = 0x282cb7d3, esp = 0xbfbfae4c, ebp = 0xbfbfae68 --- > [Previous attempts at using "call boot(0)"/etc. produced a solid hang that > required physical access to the Reset button / power switch.] > db> call cpu_reset() > cpu_reset: Restarting BSP > cpu_reset_proxy: Stopped CPU 1 > > > [Seeing other posts that request kernel config additions to assist with > debugging, I then tried to install a kernel with the added config of: > options WITNESS > and booting failed as such:] > > /boot/kernel/acpi.ko text=0x40c0c data=0x2160+0x1090 syms=[0x4+0x7810+0x4+0xa292] > KDB: debugger backends: ddb > KDB: current backend: ddb > Copyright (c) 1992-2005 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD 6.0-RC1 #1: Wed Oct 19 09:45:23 CDT 2005 > root@[host-elided]:/usr/obj/usr/src/sys/SMP_DDB > WARNING: WITNESS option enabled, expect reduced performance. > Timecounter \"i8254\" frequency 1193182 Hz quality 0 > CPU: Intel Pentium III (997.46-MHz 686-class CPU) > Origin = \"GenuineIntel\" Id = 0x68a Stepping = 10 > Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE> > real memory = 2147418112 (2047 MB) > avail memory = 2096447488 (1999 MB) > ACPI APIC Table: <DELL PE2550 > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > cpu0 (BSP): APIC ID: 1 > cpu1 (AP): APIC ID: 0 > ioapic0: Changing APIC ID to 2 > ioapic1: Changing APIC ID to 3 > MADT: Forcing active-low polarity and level trigger for SCI > ioapic0 <Version 1.1> irqs 0-15 on motherboard > ioapic1 <Version 1.1> irqs 16-31 on motherboard > panic: spin lock hptlock not in order list > cpuid = 0 > KDB: stack backtrace: > kdb_backtrace(100,c0903380,0,c085e142,c08a28ac) at 0xc0646395 = kdb_backtrace+0x29 > panic(c084621f,c085e142,c090c6c0,0,c0845c3a) at 0xc062fd4c = panic+0x114 > enroll(c085e142,c08a28ac,c096d720,1,c0c20d5c) at 0xc064f61b = enroll+0xef > witness_init(c096d720,c34bfb74,c08dd510,c28000,c0c20d74) at 0xc064e6dd = witness_init+0x11d > mtx_init(c096d720,c085e142,0,1,c0c20d88) at 0xc0628d63 = mtx_init+0x8f > hpt_init(0,c1ec00,c1e000,0,c04457e5) at 0xc07ac352 = hpt_init+0x16 > mi_startup() at 0xc060b1be = mi_startup+0x96 > begin() at 0xc04457e5 = begin+0x2c > KDB: enter: panic > [thread pid 0 tid 0 ] > Stopped at 0xc0646417 = kdb_enter+0x2b: nop > db> > db> where > Tracing pid 0 tid 0 td 0xc0903380 > kdb_enter(c0843318) at 0xc0646417 = kdb_enter+0x2b > panic(c084621f,c085e142,c090c6c0,0,c0845c3a) at 0xc062fd5f = panic+0x127 > enroll(c085e142,c08a28ac,c096d720,1,c0c20d5c) at 0xc064f61b = enroll+0xef > witness_init(c096d720,c34bfb74,c08dd510,c28000,c0c20d74) at 0xc064e6dd = witness_init+0x11d > mtx_init(c096d720,c085e142,0,1,c0c20d88) at 0xc0628d63 = mtx_init+0x8f > hpt_init(0,c1ec00,c1e000,0,c04457e5) at 0xc07ac352 = hpt_init+0x16 > mi_startup() at 0xc060b1be = mi_startup+0x96 > begin() at 0xc04457e5 = begin+0x2c > db> show pcpu > cpuid = 0 > curthread = 0xc0903380: pid 0 \"swapper\" > curpcb = 0xc0c20d90 > fpcurthread = none > idlethread = 0xc34d8900: pid 12 \"idle: cpu0\" > APIC ID = 1 > currentldt = 0x50 > spin locks held: > db> show pcpu 0 > cpuid = 0 > curthread = 0xc0903380: pid 0 \"swapper\" > curpcb = 0xc0c20d90 > fpcurthread = none > idlethread = 0xc34d8900: pid 12 \"idle: cpu0\" > APIC ID = 1 > currentldt = 0x50 > spin locks held: > db> show pcpu 1 > cpuid = 1 > curthread = none > curpcb = 0 > fpcurthread = none > idlethread = 0xc34d8780: pid 11 \"idle: cpu1\" > APIC ID = 0 > currentldt = 0x50 > spin locks held: > db> show sysctllock > No such command > db> print sysctllock > c0905920 > db> show reg > cs 0x20 > ds 0xc0840028 > es 0xc0640028 blist_create+0x4 > fs 0xc0c20008 > ss 0x28 > eax 0x12 > ecx 0xc1033000 > edx 0 > ebx 0xc0c20d0c > esp 0xc0c20ce0 > ebp 0xc0c20ce0 > esi 0x1 > edi 0xc084621f > eip 0xc0646417 kdb_enter+0x2b > efl 0x80292 > dr0 0 > dr1 0 > dr2 0 > dr3 0 > dr4 0xffff0ff0 > dr5 0x400 > dr6 0xffff0ff0 > dr7 0x400 > 0xc0646417 = kdb_enter+0x2b: nop > db> show witness > Sleep locks: > 0 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c:115 > 0 kernel environment -- last acquired @ /usr/src/sys/kern/subr_hints.c:117 > 0 user map -- last acquired @ /usr/src/sys/vm/vm_map.c:2489 > 1 system map -- last acquired @ /usr/src/sys/vm/vm_kern.c:295 > 2 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:2631 > 3 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:1898 > 2 kmem object -- last acquired @ /usr/src/sys/vm/vm_kern.c:397 > 3 SYSMAPS -- last acquired @ /usr/src/sys/i386/i386/pmap.c:2410 > 3 vm page queue mutex -- last acquired @ /usr/src/sys/vm/vm_kern.c:404 > 4 pmap -- last acquired @ /usr/src/sys/i386/i386/pmap.c:1878 > 4 vnode interlock -- last acquired @ order list:0 > 5 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:62 > 2 kernel object -- last acquired @ /usr/src/sys/vm/vm_object.c:454 > 3 SYSMAPS -- (already displayed) > 3 vm page queue mutex -- (already displayed) > 2 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:914 > 3 UMA zone -- (already displayed) > 0 nfsd_mtx -- last acquired @ order list:0 > 3 so_snd -- last acquired @ order list:0 > 4 so_rcv -- last acquired @ order list:0 > 5 radix node head -- last acquired @ order list:0 > 6 rtentry -- last acquired @ order list:0 > 7 ifaddr -- last acquired @ order list:0 > 5 sellck -- last acquired @ order list:0 > 0 bpf global lock -- last acquired @ order list:0 > 1 bpf interface lock -- last acquired @ order list:0 > 2 bpf cdev lock -- last acquired @ order list:0 > 0 ddp_list_mtx -- last acquired @ order list:0 > 1 ddp_mtx -- last acquired @ order list:0 > 0 slip_mtx -- last acquired @ order list:0 > 1 slip sc_mtx -- last acquired @ order list:0 > 0 tcp -- last acquired @ order list:0 > 1 tcpinp -- last acquired @ order list:0 > 3 so_snd -- (already displayed) > 0 udp -- last acquired @ order list:0 > 1 udpinp -- last acquired @ order list:0 > 3 so_snd -- (already displayed) > 0 unp -- last acquired @ order list:0 > 3 so_snd -- (already displayed) > 0 in_multi_mtx -- last acquired @ order list:0 > 1 igmp_mtx -- last acquired @ order list:0 > 2 if_addr_mtx -- last acquired @ order list:0 > 0 Giant -- last acquired @ /usr/src/sys/kern/subr_witness.c:513 > 1 eventhandler list -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:132 > 1 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:229 > 1 system map -- (already displayed) > 1 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:148 > 1 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:82 > 1 filedesc structure -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1512 > 2 accept -- last acquired @ order list:0 > 3 so_snd -- (already displayed) > 2 pipe mutex -- last acquired @ order list:0 > 3 sigio lock -- last acquired @ order list:0 > 4 process group -- last acquired @ /usr/src/sys/kern/kern_fork.c:562 > 5 process lock -- last acquired @ /usr/src/sys/kern/kern_kthread.c:96 > 6 ktrace -- last acquired @ /usr/src/sys/kern/kern_fork.c:590 > 6 sigacts -- last acquired @ /usr/src/sys/kern/kern_kthread.c:98 > 6 session -- last acquired @ /usr/src/sys/kern/kern_fork.c:572 > 7 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1032 > 8 sleep mtxpool -- last acquired @ /usr/src/sys/kern/kern_resource.c:914 > 8 uidinfo struct -- last acquired @ order list:0 > 9 allprison -- last acquired @ order list:0 > 1 ithread -- last acquired @ /usr/src/sys/kern/kern_intr.c:275 > 1 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:598 > 1 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:470 > 1 ACPI semaphore -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:301 > 1 TID lock -- last acquired @ /usr/src/sys/kern/subr_unit.c:592 > 1 standard object -- last acquired @ /usr/src/sys/vm/vm_glue.c:290 > 1 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:2865 > 1 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:300 > 0 proctree -- last acquired @ /usr/src/sys/kern/kern_fork.c:561 > 4 process group -- (already displayed) > 1 allproc -- last acquired @ /usr/src/sys/kern/kern_fork.c:286 > 5 process lock -- (already displayed) > Spin locks: > 0 ap boot -- last acquired @ order list:0 > 1 rm.mutex_mtx -- last acquired @ order list:0 > 2 sio -- last acquired @ order list:0 > 3 cy -- last acquired @ order list:0 > 4 uart_hwmtx -- last acquired @ order list:0 > 5 sabtty -- last acquired @ order list:0 > 6 zstty -- last acquired @ order list:0 > 7 ng_node -- last acquired @ order list:0 > 8 ng_worklist -- last acquired @ order list:0 > 9 taskqueue_fast -- last acquired @ order list:0 > 10 intr table -- last acquired @ /usr/src/sys/i386/i386/intr_machdep.c:89 > 11 ithread table lock -- last acquired @ order list:0 > 12 sleepq chain -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:218 > 13 sched lock -- last acquired @ /usr/src/sys/kern/kern_kthread.c:116 > 14 turnstile chain -- last acquired @ order list:0 > 15 td_contested -- last acquired @ order list:0 > 16 callout -- last acquired @ /usr/src/sys/kern/kern_timeout.c:439 > 17 entropy harvest mutex -- last acquired @ order list:0 > 18 allpmaps -- last acquired @ /usr/src/sys/i386/i386/pmap.c:1429 > 19 vm page queue free mutex -- last acquired @ /usr/src/sys/vm/vm_page.c:787 > 20 icu -- last acquired @ order list:0 > 21 smp rendezvous -- last acquired @ order list:0 > 22 tlb -- last acquired @ order list:0 > 23 clk -- last acquired @ /usr/src/sys/i386/isa/clock.c:260 > 24 mutex profiling lock -- last acquired @ order list:0 > 25 kse zombie lock -- last acquired @ order list:0 > 26 ALD Queue -- last acquired @ order list:0 > 27 pcicfg -- last acquired @ order list:0 > 28 NDIS thread lock -- last acquired @ order list:0 > 29 tw_osl_io_lock -- last acquired @ order list:0 > 30 tw_osl_q_lock -- last acquired @ order list:0 > 31 tw_cl_io_lock -- last acquired @ order list:0 > 32 tw_cl_intr_lock -- last acquired @ order list:0 > 33 tw_cl_gen_lock -- last acquired @ order list:0 > Locks which were never acquired: > LED sx > LED mtx > tty > nfslock > random reseed > devd > bio queue > GEOM topology > if_cloners lock > ifnet > acpica subsystem lock > ACPI global lock > sf_buf > taskqueue list > uma object > p_peers > struct pargs.ref > pbuf mutex > bdone lock > buffer daemon lock > needsbuffer lock > runningbufspace lock > buf queue lock > callout_wait_lock > ACPI embedded controller > ACPI CPU > ACPI PCI power methods > ACPI cmbat > ACPI generic battery > ACPI AC adapter > ACPI thermal zone > ACPI PCI link > ACPI root bus > ACPI PCI bus methods > ACPI lid > ACPI power resources > umtxq_lock > bounce pages lock > arc4_mtx > securelevel mutex lock > accounting > knlist lock for lockless objects > net80211 instances > rtsock route_cb lock > Softdep Lock > domain list > rawcb > pfil_head_list lock > mountlist > pmc shared lock > accept_filter_mtx > protect sysfilt_ops > kqueue order > UUID generator mutex lock > unit# allocation > Name Cache > fifo mutex > so_glabel > encapmtx > DEVFS ruleset lock > fdesc > filelist lock > sysctl lock > phys_pager list > dev_pager list > dev_pager create > swapdev > swap_pager list > vm map sleep mutex > PMAP2 > vm86 lock > db> cont > Uptime: 1s > Automatic reboot in 15 seconds - press a key on the console to abort > Rebooting... > > > And, attempts at adding: > options INVARIANTS > > failed to compile with the following: > > /usr/src/sys/dev/ata/atapi-cd.c: In function `acd_geom_attach': > /usr/src/sys/dev/ata/atapi-cd.c:181: warning: implicit declaration of function `_sx_assert' > /usr/src/sys/dev/ata/atapi-cd.c:181: warning: nested extern declaration of `_sx_assert' > > > > Since I am having a hang that feels the same to me in both RELENG_5 and > RELENG_6, that's the main thing I would like to look at. I am not very > conversant at what I will need for kernel debugging (knowing which > addresses to dump or structures to inspect) to debug this myself, but > if you can tell me any further commands to issue in DDB or which addresses > would assist in diagnosing or config options to add, that will be no > problem and I will be more than happy to assist in any way! > > > Thank you, > Philip > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051019175020.S60849>