Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 17 Apr 2007 09:33:57 -0600
From:      Tillman Hodgson <tillman@seekingfire.com>
To:        current@freebsd.org
Subject:   Panic on boot with April 16 src (lengthy info attached)
Message-ID:  <20070417153357.GA1335@seekingfire.com>

next in thread | raw e-mail | index | archive | help
On recent -current kernels (ZFS enticed me *g*) I'm running into kernel
panics immediately upon boot. It appears to be very consistent -- if I
boot with a recent kernel, it panics as soon as the login prompt
appears. I've tried a few compiles since ZFS was announced, the kernel
that I give data on below is from src pulled from cvsup8 around 3:45pm
CST April 16.

The kernel I've gone back to (and which has been stable) is:
[root@athena ~]# uname -a
FreeBSD athena.seekingfire.prv 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Sat Mar 10 08:57:44 CST 2007
toor@athena.seekingfire.prv:/usr/obj/usr/src/sys/ATHENA  i386

The kernel config is GENERIC with the serial break to debugger enabled:
[root@athena /usr/src/sys/i386/conf]# diff ATHENA GENERIC
24c24
< ident         ATHENA
---
> ident         GENERIC
72,74d71
<
< ### Tillman added 26Feb07 as per http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html
< options         BREAK_TO_DEBUGGER

Booting single user seems to be fine. I thought it might be a PF issue,
but booting multi-user with pf_enable disbaled also panics.

Boot log (including a LOR warning) from serial console and debugger
tests as per section 11.9 of the Developers Handbook follow:

/boot/kernel/acpi.ko text=0x48038 data=0x27e0+0xdb0 syms=[0x4+0x82e0+0x4+0xac68]
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #0: Tue Apr 17 07:20:03 CST 2007
    toor@athena.seekingfire.prv:/usr/obj/usr/src/sys/ATHENA
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: MPSAFE network stack disabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Pentium III (997.16-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x68a  Stepping = 10
  Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory  = 1073676288 (1023 MB)
avail memory = 1037119488 (989 MB)
ACPI APIC Table: <VIA694 AWRDACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <VIA694 AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3fef0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0x4000-0x407f,0x4080-0x40ff,0x5000-0x500f,0x6000-0x607f on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xf4000000-0xf4ffffff,0xf6241000-0xf6241fff irq 19 at device 6.0 on pci0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd400-0xd40f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 12 at device 7.2 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 12 at device 7.3 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
pci0: <bridge> at device 7.4 (no driver attached)
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe000-0xe03f mem 0xf6240000-0xf6240fff,0xf6000000-0xf60fffff irq 17 at device 13.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:21:ad:e0
fxp0: [GIANT-LOCKED]
fxp0: [ITHREAD]
fxp1: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem 0xf6242000-0xf6242fff,0xf6100000-0xf61fffff irq 18 at device 14.0 on pci0
miibus1: <MII bus> on fxp1
inphy1: <i82555 10/100 media interface> PHY 1 on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: Ethernet address: 00:e0:81:21:ad:e1
fxp1: [GIANT-LOCKED]
fxp1: [ITHREAD]
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xe800-0xe83f mem 0xf6200000-0xf621ffff,0xf6220000-0xf623ffff irq 18 at device 16.0 on pci0
em0: Ethernet address: 00:0e:0c:c2:ce:4f
em0: [FILTER]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ad0: 38166MB <Seagate ST340016A 3.75> at ata0-master UDMA100
acd0: CDROM <CDU5211/YYS7> at ata1-master UDMA33
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
Loading configuration files.
kernel dumps on /dev/ad0s1b
Entropy harvesting: interrupts ethernet point_to_point kickstart.
swapon: adding /dev/ad0s1b as swap device
Starting file system checks:
/dev/ad0s1a: INCORRECT BLOCK COUNT I=50748 (4 should be 0) (CORRECTED)
/dev/ad0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
/dev/ad0s1a: SUMMARY INFORMATION BAD (SALVAGED)
/dev/ad0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED)
/dev/ad0s1a: 6695 files, 241552 used, 12263 free (1639 frags, 1328 blocks, 0.6% fragmentation)
/dev/ad0s1e: DEFER FOR BACKGROUND CHECKING
/dev/ad0s1f: DEFER FOR BACKGROUND CHECKING
/dev/ad0s1d: DEFER FOR BACKGROUND CHECKING
Setting hostuuid: 338939b7-e9ee-11db-94cb-00e08121ade0.
Setting hostid: 0xceaac23b.
WMounting local fAile systems:RNING: /tmp was not properly dismounted
/tmp: mount pending error: blocks 16 files 4
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 44 files 8
.
Setting hostname: athena.seekingfire.prv.
net.inet6.ip6.auto_linklocal: 1 -> 0
fxp1: link state changed to UP
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
fxp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:e0:81:21:ad:e1
        inet 24.89.83.9 netmask 0xffffff00 broadcast 24.89.83.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8843<fUP,BROADCAST,RUNlNING,SIMPLEX,MULoTICAST> metric 0g mtu 1500
        opti0ons=19b<RXCSUM,T:XCSUM,VLAN_MTU,V LAN_HWTAGGING,VLpAN_HWCSUM,TSO4>
        ether 00:0e:0co:c2:ce:4f
        inetm 192.168.23.3 neitmask 0xffffff00s broadcast 192.1c68.23.255
        mediua: Ethernet autooselect
        status:u no carrier
s mode enabled
Starting pflog.
Enabling pf.
Apr 17 08:56:11 pflogd[417]: [priv]: msg PRIV_OPEN_LOG received
No ALTQ support in kernel
ALTQ related functions disabled
No ALTQ support in kernel
ALTQ related functions disabled
No ALTQ support in kernel
ALTQ related functions disabled
pf enabled
add net default: gateway 24.89.83.1
Additional routing options: IP gateway=YES.
Starting devd.
fxp0: link state changed to DOWN
em0: link state changed to UP
hw.acpi.cpu.cx_lowest: C1
sysctl: hw.acpi.cpu.cx_lowest: Invalid argument
Initial i386 initialization:.
Additional ABI support:.
Local package initialization:.
Additional TCP options:.
Mounting NFS file systems:lock order reversal:
 1st 0xc0a9e3ac tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:617
 2nd 0xc0a9df98 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c094d94b) at db_trace_self_wrapper+0x25
kdb_backtrace(0,ffffffff,c0a5f7b8,c0a5de90,c09f7dc4,...) at kdb_backtrace+0x29
witness_checkorder(c0a9df98,1,c0958031,49) at witness_checkorder+0x586
_rw_rlock(c0a9df98,c0958031,49) at _rw_rlock+0x50
pfil_run_hooks(c0a9df80,e2991afc,c4026400,2,0,...) at pfil_run_hooks+0x2c
ip_output(c43b7800,0,e2991ac8,0,0,0) at ip_output+0x6ab
tcp_respond(0,c43ed020,c43ed034,c43b7800,98689e15,0,14) at tcp_respond+0x3a7
tcp_dropwithreset(c43b7800,c43ed034,0,0,3,...) at tcp_dropwithreset+0xdb
tcp_input(c43b7800,14,d6,9535918,0,...) at tcp_input+0xb16
ip_input(c43b7800) at ip_input+0x5c9
netisr_processqueue(c0a9d638) at netisr_processqueue+0x6e
swi_net(0) at swi_net+0x8c
ithread_execute_handlers(c3eff900,c3f3f280) at ithread_execute_handlers+0x121
ithread_loop(c3ec4950,e2991d38) at ithread_loop+0x67
fork_exit(c06adf40,c3ec4950,e2991d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2991d70, ebp = 0 ---
.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/mysql /usr/local/lib/pth
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Clearing /tmp (X related).
Starting quagga.
Starting quagga.
Starting watchquagga.
Apr 17 08:57:04 watchquagga[727]: watchquagga 0.99.6 watching [zebra ospfd], mode [monitor]
Apr 17 08:57:04 watchquagga[727]: zebra state -> up : connect succeeded
Creating and/or trimming log files:.
Starting syslogd.
Checking for core dump on /dev/ad0s1b...
savecore: no dumps found
Starting named.
Apr 17 08:57:06 athena named[812]: zone 0.0.127.in-addr.arpa/IN/us: loading master file named.loopback: file not foulnd
oApr 17 08:57:06 cathena named[812k]: zone localhos t/IN/us: loadingo master file namred.localhost: fidle not found
er reversal:
 1st 0xc4577d08 inp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:476
 2nd 0xc0a9df98 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c094d94b) at db_trace_self_wrapper+0x25
kdb_backtrace(0,ffffffff,c0a5f790,c0a5de90,c09f7dc4,...) at kdb_backtrace+0x29
witness_checkorder(c0a9df98,1,c0958031,49) at witness_checkorder+0x586
_rw_rlock(c0a9df98,c0958031,49) at _rw_rlock+0x50
pfil_run_hooks(c0a9df80,e44aab3c,c4051000,2,c4577c78,...) at pfil_run_hooks+0x2c
ip_output(c4357900,0,e44aab08,0,0,c4577c78) at ip_output+0x6ab
tcp_output(c457a9f4) at tcp_output+0x11bd
tcp_usr_connect(c45bc414,c424a130,c404fa20) at tcp_usr_connect+0xda
soconnect(c45bc414,c424a130,c404fa20) at soconnect+0x4e
kern_connect(c404fa20,1e,c424a130,c424a130,0,...) at kern_connect+0x76
connect(c404fa20,e44aad00) at connect+0x30
syscall(e44aad38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (98, FreeBSD ELF32, connect), eip = 0x2838a1b3, esp = 0xbfbfe17c, ebp = 0xbfbfe238 ---
Starting rpcbind.
NFS access cache time=60
Starting statd.
Starting lockd.
Starting local daemons:.
Starting lpd.
Updating motd.
Mounting late file systems:.
Starting ntpd.
Starting rwhod.
Apr 17 08:57:09 athena ntpd[906]: configure: keyword "authenticate" unknown, line ignored
Starting dhcpd.
Apr 17 08:57:10 athena dhcpd:
Apr 17 08:57:10 athena dhcpd: No subnet declaration for fxp1 (24.89.83.9).
Apr 17 08:57:10 athena dhcpd: ** Ignoring requests on fxp1.  If this is not what
Apr 17 08:57:10 athena dhcpd:    you want, please write a subnet declaration
Apr 17 08:57:10 athena dhcpd:    in your dhcpd.conf file for the network segment
Apr 17 08:57:10 athena dhcpd:    to which interface fxp1 is attached. **
Apr 17 08:57:10 athena dhcpd:
Starting spamd.
Starting courier_authdaemond.
postfix/postfix-script: starting the Postfix mail system
Starting courier_imap_pop3d.
Starting courier_imap_imapd.
Configuring syscons: blanktime.
Starting sshd.
Starting cron.
Starting inetd.
Starting background file system checks in 60 seconds.

Tue Apr 17 08:57:17 CST 2007

FreeBSD/i386 (athena.seekingfire.prv) (ttyd0)

login: panic: mutex Giant not owned at /usr/src/sys/netinet/tcp_timer.c:286
cpuid = 0
KDB: enter: panic
[thread pid 13 tid 100003 ]
Stopped at      kdb_enter+0x2b: nop
db>


Starting sshd.
Starting cron.
Starting inetd.
Starting background file system checks in 60 seconds.

Tue Apr 17 08:39:59 CST 2007

FreeBSD/i386 (athena.seekingfire.prv) (ttyd0)

login: panic: mutex Giant not owned at /usr/src/sys/netinet/tcp_timer.c:286
cpuid = 0
KDB: enter: panic
[thread pid 13 tid 100003 ]
Stopped at      kdb_enter+0x2b: nop

db> where
Tracing pid 13 tid 100003 td 0xc3f00a20
kdb_enter(c094addf) at kdb_enter+0x2b
panic(c094a0e2,c09627ad,c095d9a9,11e,c468a1e0,...) at panic+0x11c
_mtx_assert(c0a4efe8,1,c095d9a9,11e) at _mtx_assert+0x66
tcp_timer(c468a150) at tcp_timer+0x51
softclock(0) at softclock+0x22f
ithread_execute_handlers(c3eff6c0,c3f3f200) at ithread_execute_handlers+0x121
ithread_loop(c3ec4940,e2994d38) at ithread_loop+0x67
fork_exit(c06adf40,c3ec4940,e2994d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2994d70, ebp = 0 ---

db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
 1220  1210  1208     0  R                           cp
 1210  1208  1208     0  S       wait     0xc45666c0 sh
 1208  1206  1208     0  Ss      wait     0xc42c4900 sh
 1206  1158  1158     0  S       piperd   0xc44e9dec cron
 1205     1  1205     0  Ss+     ttyin    0xc40c2c10 getty
 1204     1  1204     0  Ss+     ttyin    0xc40d3010 getty
 1203     1  1203     0  Ss+     ttyin    0xc40d3410 getty
 1202     1  1202     0  Ss+     ttyin    0xc40d3810 getty
 1201     1  1201     0  Ss+     ttyin    0xc40d3c10 getty
 1200     1  1200     0  Ss+     ttyin    0xc40d4010 getty
 1199  1198    43     0  S+      nanslp   0xc0a4f704 sleep
 1198  1195    43     0  S+      wait     0xc4601000 sh
 1196     1    43     0  S+      piperd   0xc42a54a4 logger
 1195     1    43     0  S+      wait     0xc45646c0 sh
 1181     1  1181     0  Ss      select   0xc0a9c2bc inetd
 1158     1  1158     0  Ss      nanslp   0xc0a4f704 cron
 1151     1  1151     0  Ss      select   0xc0a9c2bc sshd
 1137  1136  1137     0  S+      select   0xc0a9c2bc couriertcpd
 1136     1  1136     0  S+      piperd   0xc42a5c60 courierlogger
 1125  1124  1125     0  S+      select   0xc0a9c2bc couriertcpd
 1124     1  1124     0  S+      piperd   0xc42a5ad4 courierlogger
 1120  1108  1108   125  S       kqread   0xc4667700 qmgr
 1119  1108  1108   125  S       kqread   0xc459e300 pickup
 1108     1  1108     0  Ss      kqread   0xc460d180 master
 1062  1051  1050     0  S+      select   0xc0a9c2bc authdaemond
 1061  1051  1050     0  S+      select   0xc0a9c2bc authdaemond
 1060  1051  1050     0  S+      select   0xc0a9c2bc authdaemond
 1059  1051  1050     0  S+      select   0xc0a9c2bc authdaemond
 1058  1051  1050     0  S+      select   0xc0a9c2bc authdaemond
 1051  1050  1050     0  S+      select   0xc0a9c2bc authdaemond
 1050     1  1050     0  S+      piperd   0xc42c218c courierlogger
 1043     1  1043     0  Ss      kqread   0xc46f6c00 perl5.8.8
 1036     1  1036  1001  Ss      select   0xc0a9c2bc dhcpd
  942     1   942     1  Ss      sbwait   0xc4559614 rwhod
  929     1   929     0  Ss      select   0xc0a9c2bc ntpd
  913     1   913     0  Ss      select   0xc0a9c2bc lpd
  891   884   884     0  S       nfslockd 0xc0aa1f68 rpc.lockd
  884     1   884     0  Ss      select   0xc0a9c2bc rpc.lockd
  879     1   879     0  Ss      select   0xc0a9c2bc rpc.statd
  845     1   845     0  Ss      select   0xc0a9c2bc rpcbind
  835     1   835    53  Ss      select   0xc0a9c2bc named
  775     1   775     0  Ss      select   0xc0a9c2bc syslogd
  750     1   750     0  Ss      select   0xc0a9c2bc watchquagga
  745     1   745     0  Ss      select   0xc0a9c2bc ospfd
  739     1   739     0  Ss      select   0xc0a9c2bc zebra
  667     1   667     0  Ss      select   0xc0a9c2bc devd
  445   440   440    64  S       bpf      0xc4247a00 pflogd
  440     1   440     0  Ss      sbwait   0xc43530a4 pflogd
  174     1   174     0  Ss      pause    0xc42454a4 adjkerntz
   42     0     0     0  SL      -        0xc0a4f534 [schedcpu]
   41     0     0     0  SL      sdflush  0xc0aa71b8 [softdepflush]
   40     0     0     0  SL      syncer   0xc0a4f52c [syncer]
   39     0     0     0  SL      vlruwt   0xc4249000 [vnlru]
   38     0     0     0  SL      psleep   0xc0a9c748 [bufdaemon]
   37     0     0     0  SL      pgzero   0xc0aafa90 [pagezero]
   36     0     0     0  SL      psleep   0xc0aa79c0 [vmdaemon]
   35     0     0     0  SL      psleep   0xc0aa7980 [pagedaemon]
   34     0     0     0  WL                          [irq7: ppc0]
   33     0     0     0  WL                          [irq1: atkbd0]
   32     0     0     0  WL                          [swi0: sio]
   31     0     0     0  SL      -        0xc40ade00 [em0 taskq]
   30     0     0     0  WL                          [irq18: fxp1 em0]
   29     0     0     0  WL                          [irq17: fxp0]
   28     0     0     0  SL      usbevt   0xc4033210 [usb1]
   27     0     0     0  SL      usbtsk   0xc0a4cf34 [usbtask-dr]
   26     0     0     0  SL      usbtsk   0xc0a4cf20 [usbtask-hc]
   25     0     0     0  SL      usbevt   0xc3ff5210 [usb0]
   24     0     0     0  WL                          [irq12: uhci0 uhci1]
   23     0     0     0  WL                          [irq15: ata1]
   22     0     0     0  WL                          [irq14: ata0]
   21     0     0     0  WL                          [irq9: acpi0]
   20     0     0     0  WL                          [swi5: +]
   19     0     0     0  SL      -        0xc3fc4680 [thread taskq]
    9     0     0     0  SL      -        0xc3fc4700 [acpi_task_2]
    8     0     0     0  SL      -        0xc3fc4700 [acpi_task_1]
    7     0     0     0  SL      -        0xc3fc4700 [acpi_task_0]
   18     0     0     0  WL                          [swi6: Giant taskq]
   17     0     0     0  WL                          [swi6: task queue]
   16     0     0     0  WL                          [swi2: cambio]
    6     0     0     0  SL      ccb_scan 0xc0a22d54 [xpt_thrd]
    5     0     0     0  SL      -        0xc3fc4b00 [kqueue taskq]
   15     0     0     0  SL      -        0xc0a4f534 [yarrow]
    4     0     0     0  SL      -        0xc0a4d6ac [g_down]
    3     0     0     0  SL      -        0xc0a4d6a8 [g_up]
    2     0     0     0  SL      -        0xc0a4d6a0 [g_event]
   14     0     0     0  WL                          [swi3: vm]
   13     0     0     0  RL      CPU 0               [swi4: clock sio]
   12     0     0     0  WL                          [swi1: net]
   11     0     0     0  RL                          [idle: cpu0]
   10     0     0     0  RL      CPU 1               [idle: cpu1]
    1     0     1     0  SLs     wait     0xc3f02000 [init]
    0     0     0     0  WLs                         [swapper]

db> show pcpu
cpuid        = 0
curthread    = 0xc3f00a20: pid 13 "swi4: clock sio"
curpcb       = 0xe2994d90
fpcurthread  = none
idlethread   = 0xc3f006c0: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

db> show allpcpu
Current CPU: 0

cpuid        = 0
curthread    = 0xc3f00a20: pid 13 "swi4: clock sio"
curpcb       = 0xe2994d90
fpcurthread  = none
idlethread   = 0xc3f006c0: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

cpuid        = 1
curthread    = 0xc3f00510: pid 10 "idle: cpu1"
curpcb       = 0xe298bd90
fpcurthread  = none
idlethread   = 0xc3f00510: pid 10 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:

db> show locks
exclusive sleep mutex inp (tcpinp) r = 0 (0xc468a1e0) locked @ /usr/src/sys/kern/kern_timeout.c:240

db> show lockedvnods
Locked vnodes

0xc42d4000: tag ufs, type VDIR
    usecount 0, writecount 0, refcount 2 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc42c5000 (pid 1220)
        ino 32897, on dev ad0s1a

db> alltrace

Tracing command cp pid 1220 tid 100067 td 0xc42c5000
sched_switch(c42c5000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c475b600) at sleepq_switch+0xc9
sleepq_wait_sig(c475b600) at sleepq_wait_sig+0x1d
_sleep(c475b600,c475b644,153,c0961c24,0) at _sleep+0x2a6
nfs_reply(c475b600,0,3fc,0,c4358800,...) at nfs_reply+0x303
nfs_request(c467edd0,c4229200,3,c42c5000,c46f0480,e68cf9a8,e68cf9ac,e68cf9b0,c42c5000,e68cf9bc) at nfs_request+0x3f9
nfs_lookup(e68cfabc) at nfs_lookup+0x2d4
VOP_LOOKUP_APV(c0a08520,e68cfabc) at VOP_LOOKUP_APV+0x87
lookup(e68cfba4) at lookup+0x4b0
namei(e68cfba4) at namei+0x2be
kern_stat(c42c5000,28216738,0,e68cfc18) at kern_stat+0x35
stat(c42c5000,e68cfd00) at stat+0x1b
syscall(e68cfd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (188, FreeBSD ELF32, stat), eip = 0x2813d2cb, esp = 0xbfbfea7c, ebp = 0xbfbfeb18 ---

Tracing command sh pid 1210 tid 100094 td 0xc4565000
sched_switch(c4565000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c45666c0) at sleepq_switch+0xc9
sleepq_wait_sig(c45666c0) at sleepq_wait_sig+0x1d
_sleep(c45666c0,c4566720,15c,c09504d6,0) at _sleep+0x2a6
kern_wait(c4565000,ffffffff,e695dc28,2,0) at kern_wait+0x7c3
wait4(c4565000,e695dd00) at wait4+0x2a
syscall(e695dd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28141c37, esp = 0xbfbfe92c, ebp = 0xbfbfe948 ---

Tracing command sh pid 1208 tid 100060 td 0xc42c5bd0
sched_switch(c42c5bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c42c4900) at sleepq_switch+0xc9
sleepq_wait_sig(c42c4900) at sleepq_wait_sig+0x1d
_sleep(c42c4900,c42c4960,15c,c09504d6,0) at _sleep+0x2a6
kern_wait(c42c5bd0,ffffffff,e68e4c28,2,0) at kern_wait+0x7c3
wait4(c42c5bd0,e68e4d00) at wait4+0x2a
syscall(e68e4d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28141c37, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 ---

Tracing command cron pid 1206 tid 100073 td 0xc4513bd0
sched_switch(c4513bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c44e9dec) at sleepq_switch+0xc9
sleepq_wait_sig(c44e9dec) at sleepq_wait_sig+0x1d
_sleep(c44e9dec,c44e9f5c,14c,c0950196,0) at _sleep+0x2a6
pipe_read(c4675318,e693cc60,c467c700,0,c4513bd0) at pipe_read+0x377
dofileread(c4513bd0,5,c4675318,e693cc60,ffffffff,...) at dofileread+0x85
kern_readv(c4513bd0,5,e693cc60,2821f000,1000,...) at kern_readv+0x36
read(c4513bd0,e693cd00) at read+0x45
syscall(e693cd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815a78f, esp = 0xbfbfe66c, ebp = 0xbfbfe688 ---

Tracing command getty pid 1205 tid 100095 td 0xc4514d80
sched_switch(c4514d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40c2c10) at sleepq_switch+0xc9
sleepq_wait_sig(c40c2c10) at sleepq_wait_sig+0x1d
_sleep(c40c2c10,0,159,c0951b1c,0) at _sleep+0x2a6
ttysleep(c40c2c00,c40c2c10,159,c0951b1c,0,...) at ttysleep+0x21
ttread(c40c2c00,e695ac60,0) at ttread+0x48f
ttyread(c40a9900,e695ac60,0,c0a4efe8,0,...) at ttyread+0x2f
giant_read(c40a9900,e695ac60,0) at giant_read+0x2d
devfs_read_f(c455b360,e695ac60,c3efd480,0,c4514d80) at devfs_read_f+0x62
dofileread(c4514d80,0,c455b360,e695ac60,ffffffff,...) at dofileread+0x85
kern_readv(c4514d80,0,e695ac60,bfbfee3f,1,...) at kern_readv+0x36
read(c4514d80,e695ad00) at read+0x45
syscall(e695ad38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815778f, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command getty pid 1204 tid 100097 td 0xc4514a20
sched_switch(c4514a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40d3010) at sleepq_switch+0xc9
sleepq_wait_sig(c40d3010) at sleepq_wait_sig+0x1d
_sleep(c40d3010,0,159,c0951b1c,0) at _sleep+0x2a6
ttysleep(c40d3000,c40d3010,159,c0951b1c,0,...) at ttysleep+0x21
ttread(c40d3000,e6954c60,0) at ttread+0x48f
ttyread(c40c6d00,e6954c60,0,e6954bb8,c069dde1,...) at ttyread+0x2f
scread(c40c6d00,e6954c60,0,c0a4efe8,0,...) at scread+0x22
giant_read(c40c6d00,e6954c60,0) at giant_read+0x2d
devfs_read_f(c4549a68,e6954c60,c3efd480,0,c4514a20) at devfs_read_f+0x62
dofileread(c4514a20,0,c4549a68,e6954c60,ffffffff,...) at dofileread+0x85
kern_readv(c4514a20,0,e6954c60,bfbfee3f,1,...) at kern_readv+0x36
read(c4514a20,e6954d00) at read+0x45
syscall(e6954d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815778f, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command getty pid 1203 tid 100089 td 0xc4565870
sched_switch(c4565870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40d3410) at sleepq_switch+0xc9
sleepq_wait_sig(c40d3410) at sleepq_wait_sig+0x1d
_sleep(c40d3410,0,159,c0951b1c,0) at _sleep+0x2a6
ttysleep(c40d3400,c40d3410,159,c0951b1c,0,...) at ttysleep+0x21
ttread(c40d3400,e696cc60,0) at ttread+0x48f
ttyread(c40c6c00,e696cc60,0,e696cbb8,c069dde1,...) at ttyread+0x2f
scread(c40c6c00,e696cc60,0,c0a4efe8,0,...) at scread+0x22
giant_read(c40c6c00,e696cc60,0) at giant_read+0x2d
devfs_read_f(c4665bd0,e696cc60,c3efd480,0,c4565870) at devfs_read_f+0x62
dofileread(c4565870,0,c4665bd0,e696cc60,ffffffff,...) at dofileread+0x85
kern_readv(c4565870,0,e696cc60,bfbfee3f,1,...) at kern_readv+0x36
read(c4565870,e696cd00) at read+0x45
syscall(e696cd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815778f, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command getty pid 1202 tid 100091 td 0xc4565510
sched_switch(c4565510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40d3810) at sleepq_switch+0xc9
sleepq_wait_sig(c40d3810) at sleepq_wait_sig+0x1d
_sleep(c40d3810,0,159,c0951b1c,0) at _sleep+0x2a6
ttysleep(c40d3800,c40d3810,159,c0951b1c,0,...) at ttysleep+0x21
ttread(c40d3800,e6966c60,0) at ttread+0x48f
ttyread(c40c6b00,e6966c60,0,e6966bb8,c069dde1,...) at ttyread+0x2f
scread(c40c6b00,e6966c60,0,c0a4efe8,0,...) at scread+0x22
giant_read(c40c6b00,e6966c60,0) at giant_read+0x2d
devfs_read_f(c42c3480,e6966c60,c3efd480,0,c4565510) at devfs_read_f+0x62
dofileread(c4565510,0,c42c3480,e6966c60,ffffffff,...) at dofileread+0x85
kern_readv(c4565510,0,e6966c60,bfbfee3f,1,...) at kern_readv+0x36
read(c4565510,e6966d00) at read+0x45
syscall(e6966d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815778f, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command getty pid 1201 tid 100051 td 0xc4248000
sched_switch(c4248000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40d3c10) at sleepq_switch+0xc9
sleepq_wait_sig(c40d3c10) at sleepq_wait_sig+0x1d
_sleep(c40d3c10,0,159,c0951b1c,0) at _sleep+0x2a6
ttysleep(c40d3c00,c40d3c10,159,c0951b1c,0,...) at ttysleep+0x21
ttread(c40d3c00,e46c8c60,0) at ttread+0x48f
ttyread(c40c6a00,e46c8c60,0,e46c8bb8,c069dde1,...) at ttyread+0x2f
scread(c40c6a00,e46c8c60,0,c0a4efe8,0,...) at scread+0x22
giant_read(c40c6a00,e46c8c60,0) at giant_read+0x2d
devfs_read_f(c42c3a20,e46c8c60,c3efd480,0,c4248000) at devfs_read_f+0x62
dofileread(c4248000,0,c42c3a20,e46c8c60,ffffffff,...) at dofileread+0x85
kern_readv(c4248000,0,e46c8c60,bfbfee3f,1,...) at kern_readv+0x36
read(c4248000,e46c8d00) at read+0x45
syscall(e46c8d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815778f, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command getty pid 1200 tid 100093 td 0xc45651b0
sched_switch(c45651b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40d4010) at sleepq_switch+0xc9
sleepq_wait_sig(c40d4010) at sleepq_wait_sig+0x1d
_sleep(c40d4010,0,159,c0951b1c,0) at _sleep+0x2a6
ttysleep(c40d4000,c40d4010,159,c0951b1c,0,...) at ttysleep+0x21
ttread(c40d4000,e6960c60,0) at ttread+0x48f
ttyread(c40c6900,e6960c60,0,e6960bb8,c069dde1,...) at ttyread+0x2f
scread(c40c6900,e6960c60,0,c0a4efe8,0,...) at scread+0x22
giant_read(c40c6900,e6960c60,0) at giant_read+0x2d
devfs_read_f(c4665ca8,e6960c60,c3efd480,0,c45651b0) at devfs_read_f+0x62
dofileread(c45651b0,0,c4665ca8,e6960c60,ffffffff,...) at dofileread+0x85
kern_readv(c45651b0,0,e6960c60,bfbfee3f,1,...) at kern_readv+0x36
read(c45651b0,e6960d00) at read+0x45
syscall(e6960d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2815778f, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command sleep pid 1199 tid 100100 td 0xc4514510
sched_switch(c4514510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4f704) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a4f704) at sleepq_timedwait_sig+0x1e
_sleep(c0a4f704,0,15c,c094bfa5,ea61,...) at _sleep+0x27c
kern_nanosleep(c4514510,e694bc70,e694bc68) at kern_nanosleep+0xab
nanosleep(c4514510,e694bd00) at nanosleep+0x4f
syscall(e694bd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812e18b, esp = 0xbfbfee3c, ebp = 0xbfbfee98 ---

Tracing command sh pid 1198 tid 100105 td 0xc4600510
sched_switch(c4600510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c4601000) at sleepq_switch+0xc9
sleepq_wait_sig(c4601000) at sleepq_wait_sig+0x1d
_sleep(c4601000,c4601060,15c,c09504d6,0) at _sleep+0x2a6
kern_wait(c4600510,ffffffff,e69adc28,2,0) at kern_wait+0x7c3
wait4(c4600510,e69add00) at wait4+0x2a
syscall(e69add38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28141c37, esp = 0xbfbfd1dc, ebp = 0xbfbfd1f8 ---

Tracing command logger pid 1196 tid 100078 td 0xc4513360
sched_switch(c4513360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c42a54a4) at sleepq_switch+0xc9
sleepq_wait_sig(c42a54a4) at sleepq_wait_sig+0x1d
_sleep(c42a54a4,c42a5614,14c,c0950196,0) at _sleep+0x2a6
pipe_read(c4665828,e692dc60,c3efd480,0,c4513360) at pipe_read+0x377
dofileread(c4513360,0,c4665828,e692dc60,ffffffff,...) at dofileread+0x85
kern_readv(c4513360,0,e692dc60,28201000,1000,...) at kern_readv+0x36
read(c4513360,e692dd00) at read+0x45
syscall(e692dd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2814978f, esp = 0xbfbfe97c, ebp = 0xbfbfe998 ---

Tracing command sh pid 1195 tid 100087 td 0xc4565bd0
sched_switch(c4565bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c45646c0) at sleepq_switch+0xc9
sleepq_wait_sig(c45646c0) at sleepq_wait_sig+0x1d
_sleep(c45646c0,c4564720,15c,c09504d6,0) at _sleep+0x2a6
kern_wait(c4565bd0,ffffffff,e6972c28,2,0) at kern_wait+0x7c3
wait4(c4565bd0,e6972d00) at wait4+0x2a
syscall(e6972d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28141c37, esp = 0xbfbfd3ec, ebp = 0xbfbfd408 ---

Tracing command inetd pid 1181 tid 100102 td 0xc45141b0
sched_switch(c45141b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c45141b0,9,bfbfe8b0,0,0,...) at kern_select+0x495
select(c45141b0,e6945d00) at select+0x44
syscall(e6945d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2816970f, esp = 0xbfbfdb8c, ebp = 0xbfbfee78 ---

Tracing command cron pid 1158 tid 100107 td 0xc46001b0
sched_switch(c46001b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4f704) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a4f704) at sleepq_timedwait_sig+0x1e
_sleep(c0a4f704,0,15c,c094bfa5,ea61,...) at _sleep+0x27c
kern_nanosleep(c46001b0,e69a7c70,e69a7c68) at kern_nanosleep+0xab
nanosleep(c46001b0,e69a7d00) at nanosleep+0x4f
syscall(e69a7d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2814018b, esp = 0xbfbfecec, ebp = 0xbfbfed18 ---

Tracing command sshd pid 1151 tid 100096 td 0xc4514bd0
sched_switch(c4514bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c4514bd0,7,2840b0bc,0,0,...) at kern_select+0x495
select(c4514bd0,e6957d00) at select+0x44
syscall(e6957d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2839070f, esp = 0xbfbfe18c, ebp = 0xbfbfee98 ---

Tracing command couriertcpd pid 1137 tid 100092 td 0xc4565360
sched_switch(c4565360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c4565360,6,bfbfe630,0,0,...) at kern_select+0x495
select(c4565360,e6963d00) at select+0x44
syscall(e6963d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815270f, esp = 0xbfbfe4ac, ebp = 0xbfbfe6c8 ---

Tracing command courierlogger pid 1136 tid 100103 td 0xc4600870
sched_switch(c4600870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c42a5c60) at sleepq_switch+0xc9
sleepq_wait_sig(c42a5c60) at sleepq_wait_sig+0x1d
_sleep(c42a5c60,c42a5dd0,14c,c0950196,0) at _sleep+0x2a6
pipe_read(c45495a0,e69b3c60,c3efd480,0,c4600870) at pipe_read+0x377
dofileread(c4600870,0,c45495a0,e69b3c60,ffffffff,...) at dofileread+0x85
kern_readv(c4600870,0,e69b3c60,28202000,1000,...) at kern_readv+0x36
read(c4600870,e69b3d00) at read+0x45
syscall(e69b3d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2814a78f, esp = 0xbfbfe32c, ebp = 0xbfbfe348 ---

Tracing command couriertcpd pid 1125 tid 100086 td 0xc42c6510
sched_switch(c42c6510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c42c6510,6,bfbfe9e0,0,0,...) at kern_select+0x495
select(c42c6510,e68f3d00) at select+0x44
syscall(e68f3d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815270f, esp = 0xbfbfe85c, ebp = 0xbfbfea78 ---

Tracing command courierlogger pid 1124 tid 100085 td 0xc42c66c0
sched_switch(c42c66c0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c42a5ad4) at sleepq_switch+0xc9
sleepq_wait_sig(c42a5ad4) at sleepq_wait_sig+0x1d
_sleep(c42a5ad4,c42a5c44,14c,c0950196,0) at _sleep+0x2a6
pipe_read(c46755a0,e68f6c60,c3efd480,0,c42c66c0) at pipe_read+0x377
dofileread(c42c66c0,0,c46755a0,e68f6c60,ffffffff,...) at dofileread+0x85
kern_readv(c42c66c0,0,e68f6c60,28202000,1000,...) at kern_readv+0x36
read(c42c66c0,e68f6d00) at read+0x45
syscall(e68f6d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2814a78f, esp = 0xbfbfe6ec, ebp = 0xbfbfe708 ---

Tracing command qmgr pid 1120 tid 100080 td 0xc4513000
sched_switch(c4513000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c4667700) at sleepq_switch+0xc9
sleepq_timedwait_sig(c4667700) at sleepq_timedwait_sig+0x1e
_sleep(c4667700,c4667700,158,c09477a9,493e1,...) at _sleep+0x27c
kqueue_scan(c4667700,64,e6927c64,e6927c1c,e6927b3c,c4513000) at kqueue_scan+0x1f7
kern_kevent(c4513000,8,0,64,e6927c64,e6927c1c) at kern_kevent+0x159
kevent(c4513000,e6927d00) at kevent+0xfa
syscall(e6927d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x28178173, esp = 0xbfbfe14c, ebp = 0xbfbfe968 ---

Tracing command pickup pid 1119 tid 100106 td 0xc4600360
sched_switch(c4600360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c459e300) at sleepq_switch+0xc9
sleepq_timedwait_sig(c459e300) at sleepq_timedwait_sig+0x1e
_sleep(c459e300,c459e300,158,c09477a9,186a1,...) at _sleep+0x27c
kqueue_scan(c459e300,64,e69aac64,e69aac1c,e69aab3c,c4600360) at kqueue_scan+0x1f7
kern_kevent(c4600360,8,0,64,e69aac64,e69aac1c) at kern_kevent+0x159
kevent(c4600360,e69aad00) at kevent+0xfa
syscall(e69aad38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x2816d173, esp = 0xbfbfe17c, ebp = 0xbfbfe998 ---

Tracing command master pid 1108 tid 100083 td 0xc42c6a20
sched_switch(c42c6a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c460d180) at sleepq_switch+0xc9
sleepq_timedwait_sig(c460d180) at sleepq_timedwait_sig+0x1e
_sleep(c460d180,c460d180,158,c09477a9,ea61,...) at _sleep+0x27c
kqueue_scan(c460d180,64,e68fcc64,e68fcc1c,e68fcb3c,c42c6a20) at kqueue_scan+0x1f7
kern_kevent(c42c6a20,c,0,64,e68fcc64,e68fcc1c) at kern_kevent+0x159
kevent(c42c6a20,e68fcd00) at kevent+0xfa
syscall(e68fcd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x28166173, esp = 0xbfbfe3dc, ebp = 0xbfbfebf8 ---

Tracing command authdaemond pid 1062 tid 100098 td 0xc4514870
sched_switch(c4514870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e6951adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,493e1) at _cv_timedwait_sig+0x188
kern_select(c4514870,6,bfbfec10,0,0,...) at kern_select+0x47f
select(c4514870,e6951d00) at select+0x44
syscall(e6951d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815f70f, esp = 0xbfbfe37c, ebp = 0xbfbfed28 ---

Tracing command authdaemond pid 1061 tid 100062 td 0xc42c5870
sched_switch(c42c5870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e68deadc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,493e1) at _cv_timedwait_sig+0x188
kern_select(c42c5870,6,bfbfec10,0,0,...) at kern_select+0x47f
select(c42c5870,e68ded00) at select+0x44
syscall(e68ded38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815f70f, esp = 0xbfbfe37c, ebp = 0xbfbfed28 ---

Tracing command authdaemond pid 1060 tid 100061 td 0xc42c5a20
sched_switch(c42c5a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e68e1adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,493e1) at _cv_timedwait_sig+0x188
kern_select(c42c5a20,6,bfbfec10,0,0,...) at kern_select+0x47f
select(c42c5a20,e68e1d00) at select+0x44
syscall(e68e1d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815f70f, esp = 0xbfbfe37c, ebp = 0xbfbfed28 ---

Tracing command authdaemond pid 1059 tid 100082 td 0xc42c6bd0
sched_switch(c42c6bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e68ffadc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,493e1) at _cv_timedwait_sig+0x188
kern_select(c42c6bd0,6,bfbfec10,0,0,...) at kern_select+0x47f
select(c42c6bd0,e68ffd00) at select+0x44
syscall(e68ffd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815f70f, esp = 0xbfbfe37c, ebp = 0xbfbfed28 ---

Tracing command authdaemond pid 1058 tid 100081 td 0xc42c6d80
sched_switch(c42c6d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e6902adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,493e1) at _cv_timedwait_sig+0x188
kern_select(c42c6d80,6,bfbfec10,0,0,...) at kern_select+0x47f
select(c42c6d80,e6902d00) at select+0x44
syscall(e6902d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815f70f, esp = 0xbfbfe37c, ebp = 0xbfbfed28 ---

Tracing command authdaemond pid 1051 tid 100104 td 0xc46006c0
sched_switch(c46006c0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e69b0adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,493e1) at _cv_timedwait_sig+0x188
kern_select(c46006c0,6,bfbfec10,0,0,...) at kern_select+0x47f
select(c46006c0,e69b0d00) at select+0x44
syscall(e69b0d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815f70f, esp = 0xbfbfe37c, ebp = 0xbfbfed28 ---

Tracing command courierlogger pid 1050 tid 100058 td 0xc42c6000
sched_switch(c42c6000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c42c218c) at sleepq_switch+0xc9
sleepq_wait_sig(c42c218c) at sleepq_wait_sig+0x1d
_sleep(c42c218c,c42c22fc,14c,c0950196,0) at _sleep+0x2a6
pipe_read(c42c3798,e68eac60,c3efd480,0,c42c6000) at pipe_read+0x377
dofileread(c42c6000,0,c42c3798,e68eac60,ffffffff,...) at dofileread+0x85
kern_readv(c42c6000,0,e68eac60,28202000,1000,...) at kern_readv+0x36
read(c42c6000,e68ead00) at read+0x45
syscall(e68ead38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2814a78f, esp = 0xbfbfe9ac, ebp = 0xbfbfe9c8 ---

Tracing command perl5.8.8 pid 1043 tid 100057 td 0xc42c61b0
sched_switch(c42c61b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c46f6c00) at sleepq_switch+0xc9
sleepq_timedwait_sig(c46f6c00) at sleepq_timedwait_sig+0x1e
_sleep(c46f6c00,c46f6c00,158,c09477a9,1389,...) at _sleep+0x27c
kqueue_scan(c46f6c00,1,e68edc64,e68edc1c,e68edb3c,c42c61b0) at kqueue_scan+0x1f7
kern_kevent(c42c61b0,9,0,1,e68edc64,e68edc1c) at kern_kevent+0x159
kevent(c42c61b0,e68edd00) at kevent+0xfa
syscall(e68edd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x28269173, esp = 0xbfbfd0ec, ebp = 0xbfbfd928 ---

Tracing command dhcpd pid 1036 tid 100088 td 0xc4565a20
sched_switch(c4565a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e696fadc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,413651) at _cv_timedwait_sig+0x188
kern_select(c4565a20,8,bfbfec30,bfbfebb0,bfbfeb30,...) at kern_select+0x47f
select(c4565a20,e696fd00) at select+0x44
syscall(e696fd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281b170f, esp = 0xbfbfeacc, ebp = 0xbfbfecc8 ---

Tracing command rwhod pid 942 tid 100077 td 0xc4513510
sched_switch(c4513510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c4559614) at sleepq_switch+0xc9
sleepq_wait_sig(c4559614) at sleepq_wait_sig+0x1d
_sleep(c4559614,c45595e4,158,c0953046,0,...) at _sleep+0x2a6
sbwait(c45595c0,3,c4513510,e6930b74,c06edf9d,...) at sbwait+0x48
soreceive_generic(c4559570,e6930be0,e6930bec,0,0,...) at soreceive_generic+0x2da
soreceive(c4559570,e6930be0,e6930bec,0,0,e6930c74) at soreceive+0x39
kern_recvit(c4513510,4,e6930c5c,0,0) at kern_recvit+0x188
recvit(c4513510,4,e6930c5c,bfbfe980) at recvit+0x1b
recvfrom(c4513510,e6930d00) at recvfrom+0x6c
syscall(e6930d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (29, FreeBSD ELF32, recvfrom), eip = 0x281081e7, esp = 0xbfbfe92c, ebp = 0xbfbfee98 ---

Tracing command ntpd pid 929 tid 100079 td 0xc45131b0
sched_switch(c45131b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c45131b0,b,bfbfed70,0,0,...) at kern_select+0x495
select(c45131b0,e692ad00) at select+0x44
syscall(e692ad38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x282cf70f, esp = 0xbfbfed2c, ebp = 0xbfbfee08 ---

Tracing command lpd pid 913 tid 100084 td 0xc42c6870
sched_switch(c42c6870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c42c6870,14,bfbfead0,0,0,...) at kern_select+0x495
select(c42c6870,e68f9d00) at select+0x44
syscall(e68f9d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815670f, esp = 0xbfbfe0ec, ebp = 0xbfbfee98 ---

Tracing command rpc.lockd pid 891 tid 100074 td 0xc4513a20
sched_switch(c4513a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0aa1f68) at sleepq_switch+0xc9
sleepq_wait_sig(c0aa1f68) at sleepq_wait_sig+0x1d
_sleep(c0aa1f68,c0aa1f4c,158,c0962385,0) at _sleep+0x2a6
nfslock_read(c3f51500,e6939c60,4) at nfslock_read+0x5f
devfs_read_f(c4549480,e6939c60,c4276c80,0,c4513a20) at devfs_read_f+0x62
dofileread(c4513a20,8,c4549480,e6939c60,ffffffff,...) at dofileread+0x85
kern_readv(c4513a20,8,e6939c60,bfbfec30,194,...) at kern_readv+0x36
read(c4513a20,e6939d00) at read+0x45
syscall(e6939d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2816478f, esp = 0xbfbfebfc, ebp = 0xbfbfedd8 ---

Tracing command rpc.lockd pid 884 tid 100053 td 0xc404fbd0
sched_switch(c404fbd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e44adadc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,7531) at _cv_timedwait_sig+0x188
kern_select(c404fbd0,7,bfbfed40,0,0,...) at kern_select+0x47f
select(c404fbd0,e44add00) at select+0x44
syscall(e44add38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2816470f, esp = 0xbfbfec7c, ebp = 0xbfbfedd8 ---

Tracing command rpc.statd pid 879 tid 100064 td 0xc42c5510
sched_switch(c42c5510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e68d8adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,7531) at _cv_timedwait_sig+0x188
kern_select(c42c5510,a,bfbfed20,0,0,...) at kern_select+0x47f
select(c42c5510,e68d8d00) at select+0x44
syscall(e68d8d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815270f, esp = 0xbfbfec5c, ebp = 0xbfbfedb8 ---

Tracing command rpcbind pid 845 tid 100068 td 0xc4248d80
sched_switch(c4248d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e46e0b28,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,7531) at _cv_timedwait_sig+0x188
poll(c4248d80,e46e0d00) at poll+0x3a3
syscall(e46e0d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2811406b, esp = 0xbfbfcc6c, ebp = 0xbfbfee38 ---

Tracing command named pid 835 tid 100066 td 0xc42c51b0
sched_switch(c42c51b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e68d2adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,3e7) at _cv_timedwait_sig+0x188
kern_select(c42c51b0,2f,bfbfed60,bfbfece0,0,...) at kern_select+0x47f
select(c42c51b0,e68d2d00) at select+0x44
syscall(e68d2d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283a370f, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 ---

Tracing command syslogd pid 775 tid 100071 td 0xc4514000
sched_switch(c4514000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c4514000,b,282200bc,0,0,...) at kern_select+0x495
select(c4514000,e6942d00) at select+0x44
syscall(e6942d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815b70f, esp = 0xbfbfd9fc, ebp = 0xbfbfee48 ---

Tracing command watchquagga pid 750 tid 100072 td 0xc4513d80
sched_switch(c4513d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e693fadc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,114a) at _cv_timedwait_sig+0x188
kern_select(c4513d80,14,bfbfecb0,bfbfec30,bfbfebb0,...) at kern_select+0x47f
select(c4513d80,e693fd00) at select+0x44
syscall(e693fd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281c370f, esp = 0xbfbfeb4c, ebp = 0xbfbfed48 ---

Tracing command ospfd pid 745 tid 100075 td 0xc4513870
sched_switch(c4513870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_timedwait_sig(c0a9c2bc,0,0,c09f7184,e6936adc,...) at sleepq_timedwait_sig+0x1e
_cv_timedwait_sig(c0a9c2bc,c0a9c2a4,3b3) at _cv_timedwait_sig+0x188
kern_select(c4513870,14,bfbfed10,bfbfec90,bfbfec10,...) at kern_select+0x47f
select(c4513870,e6936d00) at select+0x44
syscall(e6936d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2822070f, esp = 0xbfbfebac, ebp = 0xbfbfeda8 ---

Tracing command zebra pid 739 tid 100070 td 0xc4248a20
sched_switch(c4248a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c4248a20,14,bfbfed10,bfbfec90,bfbfec10,...) at kern_select+0x495
select(c4248a20,e46dad00) at select+0x44
syscall(e46dad38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281d970f, esp = 0xbfbfebac, ebp = 0xbfbfeda8 ---

Tracing command devd pid 667 tid 100038 td 0xc404c6c0
sched_switch(c404c6c0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c2bc) at sleepq_switch+0xc9
sleepq_wait_sig(c0a9c2bc,c0a9c2bc,c0a9c2a4,c094ff1f,101,...) at sleepq_wait_sig+0x1d
_cv_wait_sig(c0a9c2bc,c0a9c2a4) at _cv_wait_sig+0x17c
kern_select(c404c6c0,5,bfbfe9f0,0,0,...) at kern_select+0x495
select(c404c6c0,e4489d00) at select+0x44
syscall(e4489d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x807a89f, esp = 0xbfbfe98c, ebp = 0xbfbfee98 ---

Tracing command pflogd pid 445 tid 100059 td 0xc42c5d80
sched_switch(c42c5d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c4247a00) at sleepq_switch+0xc9
sleepq_timedwait_sig(c4247a00) at sleepq_timedwait_sig+0x1e
_sleep(c4247a00,c4247a78,11a,c0956b46,1f4) at _sleep+0x27c
bpfread(c42a3c00,e68e7c60,0,c0a4efe8,0,...) at bpfread+0x13a
giant_read(c42a3c00,e68e7c60,0) at giant_read+0x2d
devfs_read_f(c4290d80,e68e7c60,c426eb00,0,c42c5d80) at devfs_read_f+0x62
dofileread(c42c5d80,3,c4290d80,e68e7c60,ffffffff,...) at dofileread+0x85
kern_readv(c42c5d80,3,e68e7c60,2820b000,8000,...) at kern_readv+0x36
read(c42c5d80,e68e7d00) at read+0x45
syscall(e68e7d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2817c78f, esp = 0xbfbfed9c, ebp = 0xbfbfedf8 ---

Tracing command pflogd pid 440 tid 100043 td 0xc3ffdd80
sched_switch(c3ffdd80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c43530a4) at sleepq_switch+0xc9
sleepq_wait_sig(c43530a4) at sleepq_wait_sig+0x1d
_sleep(c43530a4,c4353074,158,c0953046,0,...) at _sleep+0x2a6
sbwait(c4353050,246,c0a55b74,e440eb70,c3ffde14,...) at sbwait+0x48
soreceive_generic(c4353000,0,e440ec60,0,0,...) at soreceive_generic+0x2da
soreceive(c4353000,0,e440ec60,0,0,0) at soreceive+0x39
soo_read(c42914c8,e440ec60,c3efd480,0,c3ffdd80) at soo_read+0x3d
dofileread(c3ffdd80,4,c42914c8,e440ec60,ffffffff,...) at dofileread+0x85
kern_readv(c3ffdd80,4,e440ec60,bfbfee08,4,...) at kern_readv+0x36
read(c3ffdd80,e440ed00) at read+0x45
syscall(e440ed38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2817c78f, esp = 0xbfbfedec, ebp = 0xbfbfee28 ---

Tracing command adjkerntz pid 174 tid 100044 td 0xc3ffdbd0
sched_switch(c3ffdbd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c42454a4) at sleepq_switch+0xc9
sleepq_wait_sig(c42454a4) at sleepq_wait_sig+0x1d
_sleep(c42454a4,c42454e0,168,c090e551,0) at _sleep+0x2a6
kern_sigsuspend(c3ffdbd0,0,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c3ffdbd0,e440bd00) at sigsuspend+0x33
syscall(e440bd38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280c146f, esp = 0xbfbfedac, ebp = 0xbfbfee88 ---

Tracing command schedcpu pid 42 tid 100046 td 0xc4248870
sched_switch(c4248870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4f534,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0a4f534) at sleepq_timedwait+0x4a
_sleep(c0a4f534,0,0,c0942e75,3e8,...) at _sleep+0x291
pause(c0942e75,3e8) at pause+0x26
schedcpu_thread(0,e46d7d38) at schedcpu_thread+0x19
fork_exit(c06d83a8,0,e46d7d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe46d7d70, ebp = 0 ---

Tracing command softdepflush pid 41 tid 100047 td 0xc42486c0
sched_switch(c42486c0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0aa71b8,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0aa71b8) at sleepq_timedwait+0x4a
_sleep(c0aa71b8,c0aa7190,44,c09651df,3e8) at _sleep+0x291
softdep_flush(0,e46d4d38) at softdep_flush+0x251
fork_exit(c07fb22c,0,e46d4d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe46d4d70, ebp = 0 ---

Tracing command syncer pid 40 tid 100048 td 0xc4248510
sched_switch(c4248510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4f52c,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c0a4f52c,0,0,c09f7184,c0955f6e,...) at sleepq_wait+0x46
_sleep(c0a4f52c,c0a9ca44,68,c0956378,0) at _sleep+0x2b5
sched_sync(0,e46d1d38) at sched_sync+0x2fe
fork_exit(c0725a78,0,e46d1d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe46d1d70, ebp = 0 ---

Tracing command vnlru pid 39 tid 100049 td 0xc4248360
sched_switch(c4248360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c4249000,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c4249000) at sleepq_timedwait+0x4a
_sleep(c4249000,c0a9ca18,250,c095603c,3e8,c0a9ca78) at _sleep+0x291
vnlru_proc(0,e46ced38) at vnlru_proc+0xdf
fork_exit(c0723c18,0,e46ced38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe46ced70, ebp = 0 ---

Tracing command bufdaemon pid 38 tid 100027 td 0xc3f03d80
sched_switch(c3f03d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a9c748,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0a9c748) at sleepq_timedwait+0x4a
_sleep(c0a9c748,c0a9c74c,44,c0954760,3e8) at _sleep+0x291
buf_daemon(0,e29d0d38) at buf_daemon+0x198
fork_exit(c07155dc,0,e29d0d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29d0d70, ebp = 0 ---

Tracing command pagezero pid 37 tid 100028 td 0xc3f03bd0
sched_switch(c3f03bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0aafa90,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0aafa90) at sleepq_timedwait+0x4a
_sleep(c0aafa90,c0aa7944,0,c096a366,493e0) at _sleep+0x291
vm_pagezero(0,e29cdd38) at vm_pagezero+0x6b
fork_exit(c082f4d0,0,e29cdd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29cdd70, ebp = 0 ---

Tracing command vmdaemon pid 36 tid 100029 td 0xc3f03a20
sched_switch(c3f03a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0aa79c0,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c0aa79c0,ffffffff,0,0,c06ba03c,...) at sleepq_wait+0x46
_sleep(c0aa79c0,0,68,c0954760,0) at _sleep+0x2b5
vm_daemon(0,e29cad38) at vm_daemon+0x36
fork_exit(c082e11c,0,e29cad38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29cad70, ebp = 0 ---

Tracing command pagedaemon pid 35 tid 100030 td 0xc3f03870
sched_switch(c3f03870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0aa7980,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0aa7980) at sleepq_timedwait+0x4a
_sleep(c0aa7980,c0aa7944,44,c0954760,1388) at _sleep+0x291
vm_pageout(0,e29c7d38) at vm_pageout+0x268
fork_exit(c082dda4,0,e29c7d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29c7d70, ebp = 0 ---

Tracing command irq7: ppc0 pid 34 tid 100031 td 0xc3f036c0
fork_trampoline() at fork_trampoline

Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc3f03510
fork_trampoline() at fork_trampoline

Tracing command swi0: sio pid 32 tid 100033 td 0xc404f000
sched_switch(c404f000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c40bd640,e4498d38) at ithread_loop+0xda
fork_exit(c06adf40,c40bd640,e4498d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4498d70, ebp = 0 ---

Tracing command em0 taskq pid 31 tid 100034 td 0xc404cd80
sched_switch(c404cd80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c40ade00,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c40ade00,c402ea04,c402ea04,e4495cec,c06e962e,...) at sleepq_wait+0x46
msleep_spin(c40ade00,c40ade1c,c0942e75,0) at msleep_spin+0x19e
taskqueue_thread_loop(c402ea14,e4495d38) at taskqueue_thread_loop+0x61
fork_exit(c06e9928,c402ea14,e4495d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4495d70, ebp = 0 ---

Tracing command irq18: fxp1 em0 pid 30 tid 100035 td 0xc404cbd0
sched_switch(c404cbd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c404a660,e4492d38) at ithread_loop+0xda
fork_exit(c06adf40,c404a660,e4492d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4492d70, ebp = 0 ---

Tracing command irq17: fxp0 pid 29 tid 100036 td 0xc404ca20
fork_trampoline() at fork_trampoline

Tracing command usb1 pid 28 tid 100037 td 0xc404c870
sched_switch(c404c870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c4033210,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c4033210) at sleepq_timedwait+0x4a
_sleep(c4033210,0,5c,c093fd9b,ea60,c4021980) at _sleep+0x291
usb_event_thread(c4021980,e448cd38) at usb_event_thread+0x94
fork_exit(c065af44,c4021980,e448cd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe448cd70, ebp = 0 ---

Tracing command usbtask-dr pid 27 tid 100017 td 0xc3f016c0
sched_switch(c3f016c0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4cf34,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c0a4cf34,ffffffff,0,0,c06ba03c,...) at sleepq_wait+0x46
_sleep(c0a4cf34,0,5c,c093fda2,0,...) at _sleep+0x2b5
usb_task_thread(c0a4cf34,e29a9d38) at usb_task_thread+0x47
fork_exit(c065aff8,c0a4cf34,e29a9d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29a9d70, ebp = 0 ---

Tracing command usbtask-hc pid 26 tid 100018 td 0xc3f01510
sched_switch(c3f01510,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4cf20,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c0a4cf20,ffffffff,0,0,c06ba03c,...) at sleepq_wait+0x46
_sleep(c0a4cf20,0,5c,c093fda2,0,...) at _sleep+0x2b5
usb_task_thread(c0a4cf20,e29a6d38) at usb_task_thread+0x47
fork_exit(c065aff8,c0a4cf20,e29a6d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29a6d70, ebp = 0 ---

Tracing command usb0 pid 25 tid 100019 td 0xc3f01360
sched_switch(c3f01360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3ff5210,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c3ff5210) at sleepq_timedwait+0x4a
_sleep(c3ff5210,0,5c,c093fd9b,ea60,c4024480) at _sleep+0x291
usb_event_thread(c4024480,e29a3d38) at usb_event_thread+0x94
fork_exit(c065af44,c4024480,e29a3d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29a3d70, ebp = 0 ---

Tracing command irq12: uhci0 uhci1 pid 24 tid 100020 td 0xc3f011b0
fork_trampoline() at fork_trampoline

Tracing command irq15: ata1 pid 23 tid 100021 td 0xc3ffd870
sched_switch(c3ffd870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c4028450,e4405d38) at ithread_loop+0xda
fork_exit(c06adf40,c4028450,e4405d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4405d70, ebp = 0 ---

Tracing command irq14: ata0 pid 22 tid 100022 td 0xc3ffd6c0
sched_switch(c3ffd6c0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c4022240,e4402d38) at ithread_loop+0xda
fork_exit(c06adf40,c4022240,e4402d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4402d70, ebp = 0 ---

Tracing command irq9: acpi0 pid 21 tid 100023 td 0xc3ffd510
fork_trampoline() at fork_trampoline

Tracing command swi5: + pid 20 tid 100024 td 0xc3ffd360
sched_switch(c3ffd360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c3ffe7e0,e43fcd38) at ithread_loop+0xda
fork_exit(c06adf40,c3ffe7e0,e43fcd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe43fcd70, ebp = 0 ---

Tracing command thread taskq pid 19 tid 100025 td 0xc3ffd1b0
sched_switch(c3ffd1b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3fc4680,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c3fc4680,0,0,c09f7184,c094e9d2,...) at sleepq_wait+0x46
_sleep(c3fc4680,c3fc469c,0,c0942e75,0) at _sleep+0x2b5
taskqueue_thread_loop(c0a54cd0,e43f9d38) at taskqueue_thread_loop+0x78
fork_exit(c06e9928,c0a54cd0,e43f9d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe43f9d70, ebp = 0 ---

Tracing command acpi_task_2 pid 9 tid 100026 td 0xc3ffd000
sched_switch(c3ffd000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3fc4700,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c3fc4700,0,0,c09f7184,c094e9d2,...) at sleepq_wait+0x46
_sleep(c3fc4700,c3fc471c,0,c0942e75,0) at _sleep+0x2b5
taskqueue_thread_loop(c0c12334,e43f6d38) at taskqueue_thread_loop+0x78
fork_exit(c06e9928,c0c12334,e43f6d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe43f6d70, ebp = 0 ---

Tracing command acpi_task_1 pid 8 tid 100008 td 0xc3f001b0
sched_switch(c3f001b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3fc4700,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c3fc4700,0,0,c09f7184,c094e9d2,...) at sleepq_wait+0x46
_sleep(c3fc4700,c3fc471c,0,c0942e75,0) at _sleep+0x2b5
taskqueue_thread_loop(c0c12334,e2985d38) at taskqueue_thread_loop+0x78
fork_exit(c06e9928,c0c12334,e2985d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2985d70, ebp = 0 ---

Tracing command acpi_task_0 pid 7 tid 100009 td 0xc3f00000
sched_switch(c3f00000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3fc4700,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c3fc4700,0,0,c09f7184,c094e9d2,...) at sleepq_wait+0x46
_sleep(c3fc4700,c3fc471c,0,c0942e75,0) at _sleep+0x2b5
taskqueue_thread_loop(c0c12334,e2982d38) at taskqueue_thread_loop+0x78
fork_exit(c06e9928,c0c12334,e2982d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2982d70, ebp = 0 ---

Tracing command swi6: Giant taskq pid 18 tid 100010 td 0xc3f03360
fork_trampoline() at fork_trampoline

Tracing command swi6: task queue pid 17 tid 100011 td 0xc3f031b0
sched_switch(c3f031b0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c3f85540,e29bbd38) at ithread_loop+0xda
fork_exit(c06adf40,c3f85540,e29bbd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29bbd70, ebp = 0 ---

Tracing command swi2: cambio pid 16 tid 100012 td 0xc3f03000
fork_trampoline() at fork_trampoline

Tracing command xpt_thrd pid 6 tid 100013 td 0xc3f01d80
sched_switch(c3f01d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a22d54,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c0a22d54,0,0,c09f7184,c0902309,...) at sleepq_wait+0x46
_sleep(c0a22d54,c0a22d6c,4c,c0902389,0,...) at _sleep+0x2b5
xpt_scanner_thread(0,e29b5d38) at xpt_scanner_thread+0x29
fork_exit(c045a220,0,e29b5d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29b5d70, ebp = 0 ---

Tracing command kqueue taskq pid 5 tid 100014 td 0xc3f01bd0
sched_switch(c3f01bd0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3fc4b00,c0a4efd0,0,c094e39a,21e,...) at sleepq_switch+0xc9
sleepq_wait(c3fc4b00,0,0,c09f7184,c094e9d2,...) at sleepq_wait+0x46
_sleep(c3fc4b00,c3fc4b1c,0,c0942e75,0) at _sleep+0x2b5
taskqueue_thread_loop(c0a4ddbc,e29b2d38) at taskqueue_thread_loop+0x78
fork_exit(c06e9928,c0a4ddbc,e29b2d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29b2d70, ebp = 0 ---

Tracing command yarrow pid 15 tid 100015 td 0xc3f01a20
sched_switch(c3f01a20,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4f534,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0a4f534) at sleepq_timedwait+0x4a
_sleep(c0a4f534,0,0,c0942e75,64,...) at _sleep+0x291
pause(c0942e75,64) at pause+0x26
random_kthread(0,e29afd38) at random_kthread+0x177
fork_exit(c05f79f4,0,e29afd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29afd70, ebp = 0 ---

Tracing command g_down pid 4 tid 100016 td 0xc3f01870
sched_switch(c3f01870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4d6ac,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0a4d6ac) at sleepq_timedwait+0x4a
_sleep(c0a4d6ac,c0a4d5c8,24c,c0942e75,64) at _sleep+0x291
g_io_schedule_down(c3f01870) at g_io_schedule_down+0x52
g_down_procbody(0,e29acd38) at g_down_procbody+0x5a
fork_exit(c0688198,0,e29acd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe29acd70, ebp = 0 ---

Tracing command g_up pid 3 tid 100000 td 0xc3f01000
sched_switch(c3f01000,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4d6a8,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0a4d6a8) at sleepq_timedwait+0x4a
_sleep(c0a4d6a8,c0a4d608,24c,c0942e75,64) at _sleep+0x291
g_io_schedule_up(c3f01000) at g_io_schedule_up+0x127
g_up_procbody(0,e299dd38) at g_up_procbody+0x5a
fork_exit(c0688138,0,e299dd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe299dd70, ebp = 0 ---

Tracing command g_event pid 2 tid 100001 td 0xc3f00d80
sched_switch(c3f00d80,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c0a4d6a0,c0a4efd0,0,c094e39a,243,...) at sleepq_switch+0xc9
sleepq_timedwait(c0a4d6a0) at sleepq_timedwait+0x4a
_sleep(c0a4d6a0,0,4c,c0942e75,64) at _sleep+0x291
g_event_procbody(0,e299ad38) at g_event_procbody+0x9e
fork_exit(c06881f8,0,e299ad38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe299ad70, ebp = 0 ---

Tracing command swi3: vm pid 14 tid 100002 td 0xc3f00bd0
fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 13 tid 100003 td 0xc3f00a20
kdb_enter(c094addf) at kdb_enter+0x2b
panic(c094a0e2,c09627ad,c095d9a9,11e,c468a1e0,...) at panic+0x11c
_mtx_assert(c0a4efe8,1,c095d9a9,11e) at _mtx_assert+0x66
tcp_timer(c468a150) at tcp_timer+0x51
softclock(0) at softclock+0x22f
ithread_execute_handlers(c3eff6c0,c3f3f200) at ithread_execute_handlers+0x121
ithread_loop(c3ec4940,e2994d38) at ithread_loop+0x67
fork_exit(c06adf40,c3ec4940,e2994d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2994d70, ebp = 0 ---

Tracing command swi1: net pid 12 tid 100004 td 0xc3f00870
sched_switch(c3f00870,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
ithread_loop(c3ec4950,e2991d38) at ithread_loop+0xda
fork_exit(c06adf40,c3ec4950,e2991d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2991d70, ebp = 0 ---

Tracing command idle: cpu0 pid 11 tid 100005 td 0xc3f006c0
sched_switch(c3f006c0,0,1) at sched_switch+0xff
mi_switch(1,0,c0a4efd0,0,c094c96a,...) at mi_switch+0x280
sched_idletd(0,e298ed38) at sched_idletd+0x51
fork_exit(c06d91dc,0,e298ed38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe298ed70, ebp = 0 ---

Tracing command idle: cpu1 pid 10 tid 100006 td 0xc3f00510
cpustop_handler(e298bc88,c08bf6a9,e298bc20,c06ba2c1,c3f3f580,...) at cpustop_handler+0x31
ipi_nmi_handler(e298bc20,c06ba2c1,c3f3f580,0,0,...) at ipi_nmi_handler+0x28
trap(e298bc94) at trap+0x39
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc0c0213d, esp = 0xe298bcd4, ebp = 0xe298bcd4 ---
acpi_cpu_c1(c0a4efd0,c06d91dc,e298bcf4,c06d91dc,c06d91dc,...) at acpi_cpu_c1+0x5
acpi_cpu_idle(e298bd04,c06d9205,e298bd24,c06acf80,0,...) at acpi_cpu_idle+0x14e
cpu_idle(e298bd24,c06acf80,0,e298bd38,c3effd80,...) at cpu_idle+0x28
sched_idletd(0,e298bd38) at sched_idletd+0x29
fork_exit(c06d91dc,0,e298bd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe298bd70, ebp = 0 ---

Tracing command init pid 1 tid 100007 td 0xc3f00360
sched_switch(c3f00360,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
sleepq_switch(c3f02000) at sleepq_switch+0xc9
sleepq_wait_sig(c3f02000) at sleepq_wait_sig+0x1d
_sleep(c3f02000,c3f02060,15c,c09504d6,0) at _sleep+0x2a6
kern_wait(c3f00360,ffffffff,e2988c28,0,0) at kern_wait+0x7c3
wait4(c3f00360,e2988d00) at wait4+0x2a
syscall(e2988d38) at syscall+0x252
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x80531f7, esp = 0xbfbfe92c, ebp = 0xbfbfe948 ---

Tracing command swapper pid 0 tid 0 td 0xc0a4d9a0
sched_switch(c0a4d9a0,0,1) at sched_switch+0xff
mi_switch(1,0) at mi_switch+0x280
scheduler(0,101ec00,101e000,0,c04528a5,...) at scheduler+0x195
mi_startup() at mi_startup+0x96
begin() at begin+0x2c




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070417153357.GA1335>