Date: Fri, 16 Jan 2009 09:52:14 -0700 (MST) From: <bvowk@math.ualberta.ca> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/130628: NFS / rpc.lockd deadlock on 7.1-R Message-ID: <200901161652.n0GGqEWo008423@zavijava4.math.ualberta.ca> Resent-Message-ID: <200901161730.n0GHU1tV009397@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 130628
>Category: kern
>Synopsis: NFS / rpc.lockd deadlock on 7.1-R
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Jan 16 17:30:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator: Barkley Vowk
>Release: FreeBSD 7.1-RELEASE-p1 amd64
>Organization:
University of Alberta
>Environment:
System: FreeBSD zavijava4.math.ualberta.ca 7.1-RELEASE-p1 FreeBSD 7.1-RELEASE-p1 #10: Wed Jan 14 12:13:19 MST 2009 root@zavijava4.math.ualberta.ca:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
7.1-R deadlocks when acting as an NFS server for a debian box running
2.6.18-6-amd64. The problem can be reproduced in a few minutes with
dbench running on the linux machine into an nfs mounted directory from
the 7.1 box.
>How-To-Repeat:
mount zj4:/home /mnt ; cd /mnt ; dbench -D . 5
With a generic kernel, it lasts about 5 seconds. With all the debugging
options it can last a few minutes.
I've included two dumps as requested from the debugging deadlocks
section of the handbook.
KDB: enter: Break sequence on console
[thread pid 36 tid 100035 ]
Stopped at kdb_enter_why+0x3d: movq $0,0x608808(%rip)
db> ps
pid ppid pgrp uid state wmesg wchan cmd
898 894 898 0 S+ select 0xffffffff80bc59e0 top
894 891 894 0 Ss+ pause 0xffffff000370e5c8 csh
891 818 891 0 Ss select 0xffffffff80bc59e0 sshd
890 874 48 0 S nanslp 0xffffffff80ade5e8 fsck_ufs
885 1 885 0 Ss+ ttyin 0xffffff0002f07c10 getty
884 1 884 0 Ss+ ttyin 0xffffff0002f29810 getty
883 1 883 0 Ss+ ttyin 0xffffff0002f29c10 getty
882 1 882 0 Ss+ ttyin 0xffffff0002f2a010 getty
881 1 881 0 Ss+ ttyin 0xffffff0002f2b410 getty
880 1 880 0 Ss+ ttyin 0xffffff0002f1d810 getty
879 1 879 0 Ss+ ttyin 0xffffff0002f2c010 getty
878 1 878 0 Ss+ ttyin 0xffffff0002f2bc10 getty
877 1 877 0 Ss+ ttyin 0xffffff0002f1c810 getty
874 872 48 0 S+ wait 0xffffff0003077000 fsck
873 1 48 0 S+ piperd 0xffffff000340fc60 logger
872 1 48 0 S+ wait 0xffffff00030f29b0 sh
834 1 834 0 Ss nanslp 0xffffffff80ade5e8 cron
828 1 828 25 Ss pause 0xffffff00030abaa0 sendmail
824 1 824 0 Ss select 0xffffffff80bc59e0 sendmail
818 1 818 0 Ss select 0xffffffff80bc59e0 sshd
803 802 802 123 S select 0xffffffff80bc59e0 ntpd
802 1 802 0 Ss select 0xffffffff80bc59e0 ntpd
738 1 738 0 Rs CPU 3 rpc.lockd
725 724 724 0 S kqread 0xffffff0003c5a000 rpc.statd
724 1 724 0 Ss select 0xffffffff80bc59e0 rpc.statd
711 703 703 0 S - 0xffffff000328d600 nfsd
710 703 703 0 S - 0xffffff0002fbc400 nfsd
709 703 703 0 S - 0xffffff000328c600 nfsd
708 703 703 0 S - 0xffffff0003149000 nfsd
707 703 703 0 S - 0xffffff0003114600 nfsd
706 703 703 0 S - 0xffffff0003293600 nfsd
705 703 703 0 S - 0xffffff0003112200 nfsd
704 703 703 0 S - 0xffffff0003293200 nfsd
703 1 703 0 Ss select 0xffffffff80bc59e0 nfsd
701 1 701 0 Ss select 0xffffffff80bc59e0 mountd
680 1 680 0 Ss select 0xffffffff80bc59e0 rpcbind
605 1 605 0 Ss select 0xffffffff80bc59e0 syslogd
550 1 550 0 Ss select 0xffffffff80bc59e0 devd
47 0 0 0 SL sdflush 0xffffffff80bd7418 [softdepflush]
46 0 0 0 SL syncer 0xffffffff80ade1c0 [syncer]
45 0 0 0 SL vlruwt 0xffffff0002f734d8 [vnlru]
44 0 0 0 SL psleep 0xffffffff80bc63dc [bufdaemon]
43 0 0 0 SL pgzero 0xffffffff80bd9264 [pagezero]
42 0 0 0 SL psleep 0xffffffff80bd8548 [vmdaemon]
41 0 0 0 SL psleep 0xffffffff80bd850c [pagedaemon]
40 0 0 0 SL waiting_ 0xffffffff80bca268 [sctp_iterator]
39 0 0 0 WL [irq1: atkbd0]
38 0 0 0 WL [swi0: sio]
37 0 0 0 WL [irq257: bge1]
36 0 0 0 RL CPU 2 [irq256: bge0]
35 0 0 0 WL [irq24: arcmsr0]
34 0 0 0 WL [irq16: fxp0]
33 0 0 0 SL usbevt 0xffffff00012d1420 [usb2]
32 0 0 0 SL usbevt 0xffffffff8103a420 [usb1]
31 0 0 0 SL usbtsk 0xffffffff80ad76e8 [usbtask-dr]
30 0 0 0 SL usbtsk 0xffffffff80ad76c0 [usbtask-hc]
29 0 0 0 SL usbevt 0xffffffff81038420 [usb0]
28 0 0 0 WL [irq10: ohci0 ohci+]
27 0 0 0 WL [irq15: ata1]
26 0 0 0 WL [irq14: ata0]
25 0 0 0 WL [irq11: atapci0+]
24 0 0 0 WL [irq9: acpi0]
23 0 0 0 SL - 0xffffff000116fd00 [thread taskq]
22 0 0 0 WL [swi5: +]
9 0 0 0 SL - 0xffffff0001251200 [kqueue taskq]
8 0 0 0 SL - 0xffffff0001251300 [acpi_task_2]
7 0 0 0 SL - 0xffffff0001251300 [acpi_task_1]
6 0 0 0 SL - 0xffffff0001251300 [acpi_task_0]
21 0 0 0 WL [swi2: cambio]
5 0 0 0 SL ccb_scan 0xffffffff80aa5260 [xpt_thrd]
20 0 0 0 WL [swi6: task queue]
19 0 0 0 WL [swi6: Giant taskq]
18 0 0 0 SL - 0xffffffff80ade1c8 [yarrow]
4 0 0 0 SL - 0xffffffff80ad8268 [g_down]
3 0 0 0 SL - 0xffffffff80ad8260 [g_up]
2 0 0 0 SL - 0xffffffff80ad8250 [g_event]
17 0 0 0 WL [swi3: vm]
16 0 0 0 RL CPU 1 [swi4: clock sio]
15 0 0 0 WL [swi1: net]
14 0 0 0 RL CPU 0 [idle: cpu0]
13 0 0 0 RL [idle: cpu1]
12 0 0 0 RL [idle: cpu2]
11 0 0 0 RL [idle: cpu3]
1 0 1 0 SLs wait 0xffffff00010fd9b0 [init]
10 0 0 0 SL audit_wo 0xffffffff80bd67d0 [audit]
0 0 0 0 SLs sched 0xffffffff80ad8360 [swapper]
db>
db> show pcpu
cpuid = 2
curthread = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb = 0xffffffffae62bd40
fpcurthread = none
idlethread = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show allpcpu
Current CPU: 2
cpuid = 0
curthread = 0xffffff000110c6e0: pid 14 "idle: cpu0"
curpcb = 0xffffffffac50bd40
fpcurthread = none
idlethread = 0xffffff000110c6e0: pid 14 "idle: cpu0"
spin locks held:
cpuid = 1
curthread = 0xffffff000110c000: pid 16 "swi4: clock sio"
curpcb = 0xffffffffac518d40
fpcurthread = none
idlethread = 0xffffff0001100000: pid 13 "idle: cpu1"
spin locks held:
cpuid = 2
curthread = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb = 0xffffffffae62bd40
fpcurthread = none
idlethread = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
cpuid = 3
curthread = 0xffffff00032baa50: pid 738 "rpc.lockd"
curpcb = 0xffffffffb0aa3d40
fpcurthread = none
idlethread = 0xffffff00011006e0: pid 11 "idle: cpu3"
spin locks held:
db>
db> show locks
exclusive rw tcpinp r = 0 (0xffffff00034540f0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show alllocks
Process 738 (rpc.lockd) thread 0xffffff00032baa50 (100097)
exclusive sx so_rcv_sx r = 0 (0xffffff00034f5850) locked @ /usr/src/sys/kern/uipc_sockbuf.c:150
exclusive sleep mutex xprt->xp_lock r = 0 (0xffffff00031d1000) locked @ /usr/src/sys/rpc/svc_vc.c:614
Process 36 (irq256: bge0) thread 0xffffff0002e74000 (100035)
exclusive rw tcpinp r = 0 (0xffffff00034540f0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
Process 16 (swi4: clock sio) thread 0xffffff000110c000 (100007)
shared rw tcp r = 0 (0xffffffff80bcab08) locked @ /usr/src/sys/netinet/tcp_timer.c:155
db>
db> show lockedvnods
Locked vnodes
db>
db> alltrace
Tracing command top pid 898 tid 100120 td 0xffffff0003a76370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800a8eadc, rsp = 0x7fffffffe9c8, rbp = 0x11 ---
Tracing command csh pid 894 tid 100116 td 0xffffff0003105a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80093419c, rsp = 0x7fffffffe608, rbp = 0x800c3f300 ---
Tracing command sshd pid 891 tid 100115 td 0xffffff0003103000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe4b8, rbp = 0x7fffffffe540 ---
Tracing command fsck_ufs pid 890 tid 100067 td 0xffffff000304d6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80080d8ac, rsp = 0x7fffffffdbf8, rbp = 0x1 ---
Tracing command getty pid 885 tid 100114 td 0xffffff0003103370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 884 tid 100113 td 0xffffff00031036e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 883 tid 100112 td 0xffffff0003103a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 882 tid 100111 td 0xffffff00030f6000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 881 tid 100104 td 0xffffff00032b9000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 880 tid 100066 td 0xffffff0003055000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 879 tid 100053 td 0xffffff0003055a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 878 tid 100047 td 0xffffff000304da50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 877 tid 100078 td 0xffffff00030796e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command fsck pid 874 tid 100073 td 0xffffff0003054a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80069ce8c, rsp = 0x7fffffffe378, rbp = 0x3 ---
Tracing command logger pid 873 tid 100051 td 0xffffff0002fa4a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x2bc
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800717b5c, rsp = 0x7fffffffe908, rbp = 0x7fffffffee18 ---
Tracing command sh pid 872 tid 100083 td 0xffffff00031046e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80091be8c, rsp = 0x7fffffffcf98, rbp = 0x30 ---
Tracing command cron pid 834 tid 100072 td 0xffffff00030f6370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8009188ac, rsp = 0x7fffffffec18, rbp = 0x3c ---
Tracing command sendmail pid 828 tid 100056 td 0xffffff0002e74a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1419c, rsp = 0x7fffffffda38, rbp = 0x1 ---
Tracing command sendmail pid 824 tid 100065 td 0xffffff00030f66e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800da5adc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7e0 ---
Tracing command sshd pid 818 tid 100105 td 0xffffff0003104000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe5c8, rbp = 0 ---
Tracing command ntpd pid 803 tid 100055 td 0xffffff0002fa4000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffec38, rbp = 0 ---
Tracing command ntpd pid 802 tid 100099 td 0xffffff00032ba370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
poll() at poll+0x567
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffed38, rbp = 0x323 ---
Tracing command rpc.lockd pid 738 tid 100097 td 0xffffff00032baa50
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffffaba3fff0, rbp = 0xffffffffb0aa3370 ---
_rw_wlock_hard() at _rw_wlock_hard+0x167
_rw_wlock() at _rw_wlock+0xc0
tcp_usr_rcvd() at tcp_usr_rcvd+0x3e
soreceive_generic() at soreceive_generic+0x498
svc_vc_recv() at svc_vc_recv+0x169
svc_run() at svc_run+0x146
nlm_syscall() at nlm_syscall+0x79c
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008a8d7c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee20 ---
Tracing command rpc.statd pid 725 tid 100101 td 0xffffff00032b9a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_kevent() at kern_kevent+0x324
kevent() at kevent+0x90
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800806a2c, rsp = 0x7fffffffe478, rbp = 0x800a8a000 ---
Tracing command rpc.statd pid 724 tid 100049 td 0xffffff0003056000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800821adc, rsp = 0x7fffffffebe8, rbp = 0x800960740 ---
Tracing command nfsd pid 711 tid 100093 td 0xffffff00032bca50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x7 ---
Tracing command nfsd pid 710 tid 100092 td 0xffffff00032bd000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x6 ---
Tracing command nfsd pid 709 tid 100050 td 0xffffff0002fa5000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x5 ---
Tracing command nfsd pid 708 tid 100052 td 0xffffff0002fa46e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x4 ---
Tracing command nfsd pid 707 tid 100060 td 0xffffff0002e74370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x3 ---
Tracing command nfsd pid 706 tid 100103 td 0xffffff00032b9370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x2 ---
Tracing command nfsd pid 705 tid 100074 td 0xffffff00030546e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x1 ---
Tracing command nfsd pid 704 tid 100061 td 0xffffff00030f76e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0 ---
Tracing command nfsd pid 703 tid 100068 td 0xffffff000304d370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800718adc, rsp = 0x7fffffffea38, rbp = 0x5 ---
Tracing command mountd pid 701 tid 100090 td 0xffffff0003079000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082dadc, rsp = 0x7fffffffed28, rbp = 0x7fffffffee20 ---
Tracing command rpcbind pid 680 tid 100091 td 0xffffff00032bd370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8008e4e7c, rsp = 0x7fffffffcb98, rbp = 0x800b0c080 ---
Tracing command syslogd pid 605 tid 100095 td 0xffffff00032bc370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082cadc, rsp = 0x7fffffffddd8, rbp = 0x8 ---
Tracing command devd pid 550 tid 100064 td 0xffffff00030f6a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43fd0c, rsp = 0x7fffffffe918, rbp = 0x7fffffffed40 ---
Tracing command softdepflush pid 47 tid 100046 td 0xffffff0002fa56e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
softdep_flush() at softdep_flush+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae6a1d30, rbp = 0 ---
Tracing command syncer pid 46 tid 100045 td 0xffffff0002fa5a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sched_sync() at sched_sync+0x61a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae69cd30, rbp = 0 ---
Tracing command vnlru pid 45 tid 100044 td 0xffffff0001362a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vnlru_proc() at vnlru_proc+0x696
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae697d30, rbp = 0 ---
Tracing command bufdaemon pid 44 tid 100043 td 0xffffff0002e71000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
buf_daemon() at buf_daemon+0x208
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae692d30, rbp = 0 ---
Tracing command pagezero pid 43 tid 100042 td 0xffffff0002e71370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pagezero() at vm_pagezero+0x73
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae68dd30, rbp = 0 ---
Tracing command vmdaemon pid 42 tid 100041 td 0xffffff0002e716e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae688d30, rbp = 0 ---
Tracing command pagedaemon pid 41 tid 100040 td 0xffffff0002e71a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pageout() at vm_pageout+0xaed
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae683d30, rbp = 0 ---
Tracing command sctp_iterator pid 40 tid 100039 td 0xffffff0002e73000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sctp_iterator_thread() at sctp_iterator_thread+0x54
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae67ed30, rbp = 0 ---
Tracing command irq1: atkbd0 pid 39 tid 100038 td 0xffffff0002e73370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae656d30, rbp = 0 ---
Tracing command swi0: sio pid 38 tid 100037 td 0xffffff0002e736e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae651d30, rbp = 0 ---
Tracing command irq257: bge1 pid 37 tid 100036 td 0xffffff0002e73a50
fork_trampoline() at fork_trampoline
Tracing command irq256: bge0 pid 36 tid 100035 td 0xffffff0002e74000
kdb_enter_why() at kdb_enter_why+0x3d
siointr1() at siointr1+0x238
siointr() at siointr+0x2e
intr_execute_handlers() at intr_execute_handlers+0x92
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff804b03ed, rsp = 0xffffffffae62b800, rbp = 0xffffffffae62b840 ---
_mtx_lock_sleep() at _mtx_lock_sleep+0x12d
_mtx_lock_flags() at _mtx_lock_flags+0x111
svc_vc_soupcall() at svc_vc_soupcall+0x22
sowakeup() at sowakeup+0xf8
tcp_do_segment() at tcp_do_segment+0x1e08
tcp_input() at tcp_input+0xa00
ip_input() at ip_input+0xc0
ether_demux() at ether_demux+0x1ed
ether_input() at ether_input+0x19d
bge_intr() at bge_intr+0x49f
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae62bd30, rbp = 0 ---
Tracing command irq24: arcmsr0 pid 35 tid 100034 td 0xffffff00012596e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae60ad30, rbp = 0 ---
Tracing command irq16: fxp0 pid 34 tid 100033 td 0xffffff0001259a50
fork_trampoline() at fork_trampoline
Tracing command usb2 pid 33 tid 100032 td 0xffffff0001360000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5c7d30, rbp = 0 ---
Tracing command usb1 pid 32 tid 100031 td 0xffffff0001360370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5afd30, rbp = 0 ---
Tracing command usbtask-dr pid 31 tid 100030 td 0xffffff00013606e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a6d30, rbp = 0 ---
Tracing command usbtask-hc pid 30 tid 100029 td 0xffffff0001360a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a1d30, rbp = 0 ---
Tracing command usb0 pid 29 tid 100028 td 0xffffff0001362000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae59cd30, rbp = 0 ---
Tracing command irq10: ohci0 ohci+ pid 28 tid 100027 td 0xffffff0001362370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5f5d30, rbp = 0 ---
Tracing command irq15: ata1 pid 27 tid 100026 td 0xffffff00013626e0
fork_trampoline() at fork_trampoline
Tracing command irq14: ata0 pid 26 tid 100025 td 0xffffff000111e6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5d7d30, rbp = 0 ---
Tracing command irq11: atapci0+ pid 25 tid 100024 td 0xffffff000111ea50
fork_trampoline() at fork_trampoline
Tracing command irq9: acpi0 pid 24 tid 100023 td 0xffffff0001258000
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 23 tid 100022 td 0xffffff0001258370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac567d30, rbp = 0 ---
Tracing command swi5: + pid 22 tid 100021 td 0xffffff00012586e0
fork_trampoline() at fork_trampoline
Tracing command kqueue taskq pid 9 tid 100020 td 0xffffff0001258a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac55dd30, rbp = 0 ---
Tracing command acpi_task_2 pid 8 tid 100019 td 0xffffff0001259000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac558d30, rbp = 0 ---
Tracing command acpi_task_1 pid 7 tid 100018 td 0xffffff0001259370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac553d30, rbp = 0 ---
Tracing command acpi_task_0 pid 6 tid 100017 td 0xffffff000110ca50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac54ed30, rbp = 0 ---
Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff000111d000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac549d30, rbp = 0 ---
Tracing command xpt_thrd pid 5 tid 100015 td 0xffffff000111d370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
xpt_scanner_thread() at xpt_scanner_thread+0x3a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac544d30, rbp = 0 ---
Tracing command swi6: task queue pid 20 tid 100014 td 0xffffff000111d6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac53fd30, rbp = 0 ---
Tracing command swi6: Giant taskq pid 19 tid 100013 td 0xffffff000111da50
fork_trampoline() at fork_trampoline
Tracing command yarrow pid 18 tid 100012 td 0xffffff000111e000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
random_kthread() at random_kthread+0x1ad
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac535d30, rbp = 0 ---
Tracing command g_down pid 4 tid 100011 td 0xffffff000111e370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_down() at g_io_schedule_down+0x250
g_down_procbody() at g_down_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac52cd30, rbp = 0 ---
Tracing command g_up pid 3 tid 100010 td 0xffffff0001101370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_up() at g_io_schedule_up+0x154
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac527d30, rbp = 0 ---
Tracing command g_event pid 2 tid 100009 td 0xffffff00011016e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_event_procbody() at g_event_procbody+0xa1
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac522d30, rbp = 0 ---
Tracing command swi3: vm pid 17 tid 100008 td 0xffffff0001101a50
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 16 tid 100007 td 0xffffff000110c000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffffaba35ff0, rbp = 0xffffffffac518ae0 ---
_rw_wlock_hard() at _rw_wlock_hard+0x167
_rw_wlock() at _rw_wlock+0xc0
tcp_timer_delack() at tcp_timer_delack+0x58
softclock() at softclock+0x299
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac518d30, rbp = 0 ---
Tracing command swi1: net pid 15 tid 100006 td 0xffffff000110c370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac513d30, rbp = 0 ---
Tracing command idle: cpu0 pid 14 tid 100005 td 0xffffff000110c6e0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff807685f6, rsp = 0xffffffff80be4070, rbp = 0xffffffffac50bba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac50bd30, rbp = 0 ---
Tracing command idle: cpu1 pid 13 tid 100004 td 0xffffff0001100000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac506b90, rbp = 0xffffffffac506ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac506d30, rbp = 0 ---
Tracing command idle: cpu2 pid 12 tid 100003 td 0xffffff0001100370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
critical_exit() at critical_exit+0x91
intr_execute_handlers() at intr_execute_handlers+0xe9
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac501b90, rbp = 0xffffffffac501ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac501d30, rbp = 0 ---
Tracing command idle: cpu3 pid 11 tid 100002 td 0xffffff00011006e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac4fcb90, rbp = 0xffffffffac4fcba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4fcd30, rbp = 0 ---
Tracing command init pid 1 tid 100001 td 0xffffff0001100a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffed60 ---
Tracing command audit pid 10 tid 100000 td 0xffffff0001101000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x15e
audit_worker() at audit_worker+0x77
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4f2d30, rbp = 0 ---
Tracing command swapper pid 0 tid 0 td 0xffffffff80ad8840
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
scheduler() at scheduler+0x2cd
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
db>
db>reboot
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 2
Stopped at kdb_enter_why+0x3d: movq $0,0x608808(%rip)
db> ps
pid ppid pgrp uid state wmesg wchan cmd
895 1 895 0 Ss+ ttyin 0xffffff0002f07c10 getty
894 1 894 0 Ss+ ttyin 0xffffff0002f29810 getty
893 1 893 0 Ss+ ttyin 0xffffff0002f29c10 getty
892 1 892 0 Ss+ ttyin 0xffffff0002f2a010 getty
891 1 891 0 Ss+ ttyin 0xffffff0002f2b410 getty
890 1 890 0 Ss+ ttyin 0xffffff0002f1d810 getty
889 1 889 0 Ss+ ttyin 0xffffff0002f2c010 getty
888 1 888 0 Ss+ ttyin 0xffffff0002f2bc10 getty
887 1 887 0 Ss+ ttyin 0xffffff0002f1c810 getty
844 1 844 0 Ss nanslp 0xffffffff80ade5e8 cron
838 1 838 25 Ss pause 0xffffff0002f725c8 sendmail
832 1 832 0 Ss select 0xffffffff80bc59e0 sendmail
826 1 826 0 Ss select 0xffffffff80bc59e0 sshd
811 810 810 123 S select 0xffffffff80bc59e0 ntpd
810 1 810 0 Ss select 0xffffffff80bc59e0 ntpd
746 1 746 0 Rs CPU 0 rpc.lockd
733 732 732 0 S kqread 0xffffff0003570400 rpc.statd
732 1 732 0 Ss select 0xffffffff80bc59e0 rpc.statd
720 711 711 0 S - 0xffffff000312a000 nfsd
718 711 711 0 S - 0xffffff000339d600 nfsd
717 711 711 0 S - 0xffffff0003301000 nfsd
716 711 711 0 S - 0xffffff0002fbc800 nfsd
715 711 711 0 S - 0xffffff00032ff400 nfsd
714 711 711 0 S - 0xffffff00030e2200 nfsd
713 711 711 0 S - 0xffffff00032ffc00 nfsd
712 711 711 0 S - 0xffffff00030d2c00 nfsd
711 1 711 0 Ss select 0xffffffff80bc59e0 nfsd
709 1 709 0 Ss select 0xffffffff80bc59e0 mountd
688 1 688 0 Ss select 0xffffffff80bc59e0 rpcbind
613 1 613 0 Ss select 0xffffffff80bc59e0 syslogd
558 1 558 0 Ss select 0xffffffff80bc59e0 devd
47 0 0 0 SL sdflush 0xffffffff80bd7418 [softdepflush]
46 0 0 0 SL syncer 0xffffffff80ade1c0 [syncer]
45 0 0 0 SL vlruwt 0xffffff0002f734d8 [vnlru]
44 0 0 0 SL psleep 0xffffffff80bc63dc [bufdaemon]
43 0 0 0 SL pgzero 0xffffffff80bd9264 [pagezero]
42 0 0 0 SL psleep 0xffffffff80bd8548 [vmdaemon]
41 0 0 0 SL psleep 0xffffffff80bd850c [pagedaemon]
40 0 0 0 SL waiting_ 0xffffffff80bca268 [sctp_iterator]
39 0 0 0 WL [irq1: atkbd0]
38 0 0 0 WL [swi0: sio]
37 0 0 0 WL [irq257: bge1]
36 0 0 0 RL CPU 2 [irq256: bge0]
35 0 0 0 WL [irq24: arcmsr0]
34 0 0 0 WL [irq16: fxp0]
33 0 0 0 SL usbevt 0xffffff00012d1420 [usb2]
32 0 0 0 SL usbevt 0xffffffff8103a420 [usb1]
31 0 0 0 SL usbtsk 0xffffffff80ad76e8 [usbtask-dr]
30 0 0 0 SL usbtsk 0xffffffff80ad76c0 [usbtask-hc]
29 0 0 0 SL usbevt 0xffffffff81038420 [usb0]
28 0 0 0 WL [irq10: ohci0 ohci+]
27 0 0 0 WL [irq15: ata1]
26 0 0 0 WL [irq14: ata0]
25 0 0 0 WL [irq11: atapci0+]
24 0 0 0 WL [irq9: acpi0]
23 0 0 0 SL - 0xffffff000116fd00 [thread taskq]
22 0 0 0 WL [swi5: +]
9 0 0 0 SL - 0xffffff0001251200 [kqueue taskq]
8 0 0 0 SL - 0xffffff0001251300 [acpi_task_2]
7 0 0 0 SL - 0xffffff0001251300 [acpi_task_1]
6 0 0 0 SL - 0xffffff0001251300 [acpi_task_0]
21 0 0 0 WL [swi2: cambio]
5 0 0 0 SL ccb_scan 0xffffffff80aa5260 [xpt_thrd]
20 0 0 0 WL [swi6: task queue]
19 0 0 0 WL [swi6: Giant taskq]
18 0 0 0 SL - 0xffffffff80ade1c8 [yarrow]
4 0 0 0 SL - 0xffffffff80ad8268 [g_down]
3 0 0 0 SL - 0xffffffff80ad8260 [g_up]
2 0 0 0 SL - 0xffffffff80ad8250 [g_event]
17 0 0 0 WL [swi3: vm]
16 0 0 0 RL CPU 1 [swi4: clock sio]
15 0 0 0 WL [swi1: net]
14 0 0 0 RL [idle: cpu0]
13 0 0 0 RL [idle: cpu1]
12 0 0 0 RL [idle: cpu2]
11 0 0 0 RL CPU 3 [idle: cpu3]
1 0 1 0 SLs wait 0xffffff00010fd9b0 [init]
10 0 0 0 SL audit_wo 0xffffffff80bd67d0 [audit]
0 0 0 0 SLs sched 0xffffffff80ad8360 [swapper]
db> show pcpu
cpuid = 2
curthread = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb = 0xffffffffae62bd40
fpcurthread = none
idlethread = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show allpcpu
Current CPU: 2
cpuid = 0
curthread = 0xffffff0003300000: pid 746 "rpc.lockd"
curpcb = 0xffffffffb0ac1d40
fpcurthread = none
idlethread = 0xffffff000110c6e0: pid 14 "idle: cpu0"
spin locks held:
cpuid = 1
curthread = 0xffffff000110c000: pid 16 "swi4: clock sio"
curpcb = 0xffffffffac518d40
fpcurthread = none
idlethread = 0xffffff0001100000: pid 13 "idle: cpu1"
spin locks held:
cpuid = 2
curthread = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb = 0xffffffffae62bd40
fpcurthread = none
idlethread = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
cpuid = 3
curthread = 0xffffff00011006e0: pid 11 "idle: cpu3"
curpcb = 0xffffffffac4fcd40
fpcurthread = none
idlethread = 0xffffff00011006e0: pid 11 "idle: cpu3"
spin locks held:
db> show locks
exclusive rw tcpinp r = 0 (0xffffff0003422cc0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show alllocks
Process 746 (rpc.lockd) thread 0xffffff0003300000 (100103)
exclusive sx so_rcv_sx r = 0 (0xffffff000337c130) locked @ /usr/src/sys/kern/uipc_sockbuf.c:150
exclusive sleep mutex xprt->xp_lock r = 0 (0xffffff0003bc6400) locked @ /usr/src/sys/rpc/svc_vc.c:614
Process 36 (irq256: bge0) thread 0xffffff0002e74000 (100035)
exclusive rw tcpinp r = 0 (0xffffff0003422cc0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
Process 16 (swi4: clock sio) thread 0xffffff000110c000 (100007)
shared rw tcp r = 0 (0xffffffff80bcab08) locked @ /usr/src/sys/netinet/tcp_timer.c:155
db> show lockedvnods
Locked vnodes
db> alltrace
Tracing command getty pid 895 tid 100139 td 0xffffff00034ce6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 894 tid 100138 td 0xffffff00034cea50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 893 tid 100136 td 0xffffff0003536370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 892 tid 100135 td 0xffffff00031ba6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 891 tid 100132 td 0xffffff000342f370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 890 tid 100057 td 0xffffff0003020a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 889 tid 100130 td 0xffffff000342fa50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 888 tid 100120 td 0xffffff000343e370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 887 tid 100127 td 0xffffff000344b6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command cron pid 844 tid 100093 td 0xffffff00031e36e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8009188ac, rsp = 0x7fffffffec18, rbp = 0x3c ---
Tracing command sendmail pid 838 tid 100049 td 0xffffff0002fa5370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1419c, rsp = 0x7fffffffda38, rbp = 0x1 ---
Tracing command sendmail pid 832 tid 100131 td 0xffffff000342f6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800da5adc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7e0 ---
Tracing command sshd pid 826 tid 100053 td 0xffffff00030236e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe5c8, rbp = 0 ---
Tracing command ntpd pid 811 tid 100102 td 0xffffff0003300370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffec38, rbp = 0 ---
Tracing command ntpd pid 810 tid 100126 td 0xffffff000344ba50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
poll() at poll+0x567
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffed38, rbp = 0x32b ---
Tracing command rpc.lockd pid 746 tid 100103 td 0xffffff0003300000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffff80be4070, rbp = 0xffffffffb0ac1370 ---
_rw_wlock_hard() at _rw_wlock_hard+0x167
_rw_wlock() at _rw_wlock+0xc0
tcp_usr_rcvd() at tcp_usr_rcvd+0x3e
soreceive_generic() at soreceive_generic+0x498
svc_vc_recv() at svc_vc_recv+0x169
svc_run() at svc_run+0x146
nlm_syscall() at nlm_syscall+0x79c
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008a8d7c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee20 ---
Tracing command rpc.statd pid 733 tid 100122 td 0xffffff000343da50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_kevent() at kern_kevent+0x324
kevent() at kevent+0x90
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800806a2c, rsp = 0x7fffffffe478, rbp = 0x800a8a000 ---
Tracing command rpc.statd pid 732 tid 100100 td 0xffffff00030c7a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800821adc, rsp = 0x7fffffffebe8, rbp = 0x800960740 ---
Tracing command nfsd pid 720 tid 100115 td 0xffffff00030be6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x7 ---
Tracing command nfsd pid 718 tid 100114 td 0xffffff00030bea50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x6 ---
Tracing command nfsd pid 717 tid 100113 td 0xffffff00030c7000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x5 ---
Tracing command nfsd pid 716 tid 100112 td 0xffffff00030c7370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x4 ---
Tracing command nfsd pid 715 tid 100111 td 0xffffff00030c76e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x3 ---
Tracing command nfsd pid 714 tid 100058 td 0xffffff00030206e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x2 ---
Tracing command nfsd pid 713 tid 100099 td 0xffffff00031e3000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x1 ---
Tracing command nfsd pid 712 tid 100054 td 0xffffff0002fa5000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0 ---
Tracing command nfsd pid 711 tid 100048 td 0xffffff00030246e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800718adc, rsp = 0x7fffffffea38, rbp = 0x5 ---
Tracing command mountd pid 709 tid 100083 td 0xffffff0002e74a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082dadc, rsp = 0x7fffffffed28, rbp = 0x7fffffffee20 ---
Tracing command rpcbind pid 688 tid 100098 td 0xffffff00031e3370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8008e4e7c, rsp = 0x7fffffffcb98, rbp = 0x800b0c080 ---
Tracing command syslogd pid 613 tid 100087 td 0xffffff00031ba370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082cadc, rsp = 0x7fffffffddd8, rbp = 0x8 ---
Tracing command devd pid 558 tid 100082 td 0xffffff0002fa4000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43fd0c, rsp = 0x7fffffffe918, rbp = 0x7fffffffed40 ---
Tracing command softdepflush pid 47 tid 100046 td 0xffffff0002fa56e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
softdep_flush() at softdep_flush+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae6a1d30, rbp = 0 ---
Tracing command syncer pid 46 tid 100045 td 0xffffff0002fa5a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sched_sync() at sched_sync+0x61a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae69cd30, rbp = 0 ---
Tracing command vnlru pid 45 tid 100044 td 0xffffff0001362a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vnlru_proc() at vnlru_proc+0x696
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae697d30, rbp = 0 ---
Tracing command bufdaemon pid 44 tid 100043 td 0xffffff0002e71000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
buf_daemon() at buf_daemon+0x208
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae692d30, rbp = 0 ---
Tracing command pagezero pid 43 tid 100042 td 0xffffff0002e71370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pagezero() at vm_pagezero+0x73
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae68dd30, rbp = 0 ---
Tracing command vmdaemon pid 42 tid 100041 td 0xffffff0002e716e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae688d30, rbp = 0 ---
Tracing command pagedaemon pid 41 tid 100040 td 0xffffff0002e71a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pageout() at vm_pageout+0xaed
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae683d30, rbp = 0 ---
Tracing command sctp_iterator pid 40 tid 100039 td 0xffffff0002e73000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sctp_iterator_thread() at sctp_iterator_thread+0x54
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae67ed30, rbp = 0 ---
Tracing command irq1: atkbd0 pid 39 tid 100038 td 0xffffff0002e73370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae656d30, rbp = 0 ---
Tracing command swi0: sio pid 38 tid 100037 td 0xffffff0002e736e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae651d30, rbp = 0 ---
Tracing command irq257: bge1 pid 37 tid 100036 td 0xffffff0002e73a50
fork_trampoline() at fork_trampoline
Tracing command irq256: bge0 pid 36 tid 100035 td 0xffffff0002e74000
kdb_enter_why() at kdb_enter_why+0x3d
siointr1() at siointr1+0x238
siointr() at siointr+0x2e
intr_execute_handlers() at intr_execute_handlers+0x92
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff804b03e9, rsp = 0xffffffffae62b800, rbp = 0xffffffffae62b840 ---
_mtx_lock_sleep() at _mtx_lock_sleep+0x129
_mtx_lock_flags() at _mtx_lock_flags+0x111
svc_vc_soupcall() at svc_vc_soupcall+0x22
sowakeup() at sowakeup+0xf8
tcp_do_segment() at tcp_do_segment+0x1e08
tcp_input() at tcp_input+0xa00
ip_input() at ip_input+0xc0
ether_demux() at ether_demux+0x1ed
ether_input() at ether_input+0x19d
bge_intr() at bge_intr+0x49f
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae62bd30, rbp = 0 ---
Tracing command irq24: arcmsr0 pid 35 tid 100034 td 0xffffff00012596e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae60ad30, rbp = 0 ---
Tracing command irq16: fxp0 pid 34 tid 100033 td 0xffffff0001259a50
fork_trampoline() at fork_trampoline
Tracing command usb2 pid 33 tid 100032 td 0xffffff0001360000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5c7d30, rbp = 0 ---
Tracing command usb1 pid 32 tid 100031 td 0xffffff0001360370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5afd30, rbp = 0 ---
Tracing command usbtask-dr pid 31 tid 100030 td 0xffffff00013606e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a6d30, rbp = 0 ---
Tracing command usbtask-hc pid 30 tid 100029 td 0xffffff0001360a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a1d30, rbp = 0 ---
Tracing command usb0 pid 29 tid 100028 td 0xffffff0001362000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae59cd30, rbp = 0 ---
Tracing command irq10: ohci0 ohci+ pid 28 tid 100027 td 0xffffff0001362370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5f5d30, rbp = 0 ---
Tracing command irq15: ata1 pid 27 tid 100026 td 0xffffff00013626e0
fork_trampoline() at fork_trampoline
Tracing command irq14: ata0 pid 26 tid 100025 td 0xffffff000111e6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5d7d30, rbp = 0 ---
Tracing command irq11: atapci0+ pid 25 tid 100024 td 0xffffff000111ea50
fork_trampoline() at fork_trampoline
Tracing command irq9: acpi0 pid 24 tid 100023 td 0xffffff0001258000
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 23 tid 100022 td 0xffffff0001258370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac567d30, rbp = 0 ---
Tracing command swi5: + pid 22 tid 100021 td 0xffffff00012586e0
fork_trampoline() at fork_trampoline
Tracing command kqueue taskq pid 9 tid 100020 td 0xffffff0001258a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac55dd30, rbp = 0 ---
Tracing command acpi_task_2 pid 8 tid 100019 td 0xffffff0001259000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac558d30, rbp = 0 ---
Tracing command acpi_task_1 pid 7 tid 100018 td 0xffffff0001259370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac553d30, rbp = 0 ---
Tracing command acpi_task_0 pid 6 tid 100017 td 0xffffff000110ca50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac54ed30, rbp = 0 ---
Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff000111d000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac549d30, rbp = 0 ---
Tracing command xpt_thrd pid 5 tid 100015 td 0xffffff000111d370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
xpt_scanner_thread() at xpt_scanner_thread+0x3a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac544d30, rbp = 0 ---
Tracing command swi6: task queue pid 20 tid 100014 td 0xffffff000111d6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac53fd30, rbp = 0 ---
Tracing command swi6: Giant taskq pid 19 tid 100013 td 0xffffff000111da50
fork_trampoline() at fork_trampoline
Tracing command yarrow pid 18 tid 100012 td 0xffffff000111e000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
random_kthread() at random_kthread+0x1ad
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac535d30, rbp = 0 ---
Tracing command g_down pid 4 tid 100011 td 0xffffff000111e370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_down() at g_io_schedule_down+0x250
g_down_procbody() at g_down_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac52cd30, rbp = 0 ---
Tracing command g_up pid 3 tid 100010 td 0xffffff0001101370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_up() at g_io_schedule_up+0x154
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac527d30, rbp = 0 ---
Tracing command g_event pid 2 tid 100009 td 0xffffff00011016e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_event_procbody() at g_event_procbody+0xa1
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac522d30, rbp = 0 ---
Tracing command swi3: vm pid 17 tid 100008 td 0xffffff0001101a50
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 16 tid 100007 td 0xffffff000110c000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe9b, rsp = 0xffffffffaba35ff0, rbp = 0xffffffffac518ae0 ---
_rw_wlock_hard() at _rw_wlock_hard+0x16b
_rw_wlock() at _rw_wlock+0xc0
tcp_timer_delack() at tcp_timer_delack+0x58
softclock() at softclock+0x299
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac518d30, rbp = 0 ---
Tracing command swi1: net pid 15 tid 100006 td 0xffffff000110c370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac513d30, rbp = 0 ---
Tracing command idle: cpu0 pid 14 tid 100005 td 0xffffff000110c6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac50bb90, rbp = 0xffffffffac50bba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac50bd30, rbp = 0 ---
Tracing command idle: cpu1 pid 13 tid 100004 td 0xffffff0001100000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac506b90, rbp = 0xffffffffac506ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac506d30, rbp = 0 ---
Tracing command idle: cpu2 pid 12 tid 100003 td 0xffffff0001100370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac501b90, rbp = 0xffffffffac501ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac501d30, rbp = 0 ---
Tracing command idle: cpu3 pid 11 tid 100002 td 0xffffff00011006e0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff807685f6, rsp = 0xffffffffaba3fff0, rbp = 0xffffffffac4fcba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4fcd30, rbp = 0 ---
Tracing command init pid 1 tid 100001 td 0xffffff0001100a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffed60 ---
Tracing command audit pid 10 tid 100000 td 0xffffff0001101000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x15e
audit_worker() at audit_worker+0x77
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4f2d30, rbp = 0 ---
Tracing command swapper pid 0 tid 0 td 0xffffffff80ad8840
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
scheduler() at scheduler+0x2cd
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
db>
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200901161652.n0GGqEWo008423>