From owner-freebsd-stable@FreeBSD.ORG Sat Mar 22 10:29:28 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C25671065677 for ; Sat, 22 Mar 2008 10:29:28 +0000 (UTC) (envelope-from razor@dataxnet.ro) Received: from mail.dataxnet.ro (datax28.mediasat.ro [80.96.28.28]) by mx1.freebsd.org (Postfix) with SMTP id 317988FC14 for ; Sat, 22 Mar 2008 10:29:19 +0000 (UTC) (envelope-from razor@dataxnet.ro) Received: (qmail 77340 invoked by uid 1001); 22 Mar 2008 12:29:33 +0200 Date: Sat, 22 Mar 2008 12:29:33 +0200 From: Alex Popa To: Max Laier Message-ID: <20080322102933.GA76747@dataxnet.ro> References: <20080314192359.GA4677@dataxnet.ro> <20080315203121.I42065@fledge.watson.org> <200803152217.02568.max@love2party.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline In-Reply-To: <200803152217.02568.max@love2party.net> User-Agent: Mutt/1.4.2.2i Cc: Robert Watson , freebsd-stable@freebsd.org Subject: Re: Lock Order Reversal on 7.0-STABLE with pf and ipfw / dummynet (traces) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Mar 2008 10:29:28 -0000 --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sorry for the big delay, but here are the traces you requested. The kernel/world combination is from the same sources as in my previous posts, I've only changed the kernel config and recompiled it (wiped /usr/obj, make buildkernel && make installkernel in /usr/src). Note: I haven't had access to a serial console, and experimentation late last night showed me that kdb commands and their output don't get saved in dmesg... I was about to get a camera there in order to take photos of the backtraces and such, but I remembered vidcontrol and found the "-H -P" options to have some nice potential. Long story short, I added something like "options SC_HISTORY_SIZE=4000" to the kernel and here are the results. The dmesg file is an edited output of dmesg -a > something.txt, to only contain the actual LOR related parts. The first-lor and second-lor files contain edited screen captures (as made by vidcontrol), each of them with lots of debugging info for one of the two LORs. Sorry for some wrapped lines, but I hope that's better than nothing. Hope this helps Alex -- "Computer science is no more about computers than astronomy is about telescopes" -- E. W. Dijkstra --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="dmesg-lor.txt" lock order reversal: 1st 0xffffffffa2a56680 pf task mtx (pf task mtx) @ /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729 2nd 0xffffff00013236f0 radix node head (radix node head) @ /usr/src/sys/net/route.c:147 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x539 _mtx_lock_flags() at _mtx_lock_flags+0x1f rtalloc1() at rtalloc1+0x57 rtalloc_ign() at rtalloc_ign+0x8f pf_route() at pf_route+0x4d3 pf_test() at pf_test+0x64c pf_check_in() at pf_check_in+0x2b pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 --- KDB: enter: witness_checkorder exclusive sx so_rcv_sx r = 0 (0xffffff000140f3b8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/src/sys/net/pfil.c:73 exclusive sleep mutex pf task mtx r = 0 (0xffffffffa2a56680) locked @ /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729 shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/src/sys/net/pfil.c:73 Locked vnodes 0xffffff000144aba0: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xffffff0001408000 (pid 36498) ino 2237440, on dev ufs/usr64 0xffffff00018905d0: tag ufs, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xffffff0001408000 (pid 36498) ino 2286392, on dev ufs/usr64 ... lock order reversal: 1st 0xffffffff8096eb88 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 2nd 0xffffffff8096f8a8 udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:385 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x539 _mtx_lock_flags() at _mtx_lock_flags+0x1f udp_input() at udp_input+0x1f7 ip_input() at ip_input+0xa7 dummynet_send() at dummynet_send+0xde dummynet_io() at dummynet_io+0x587 ipfw_check_in() at ipfw_check_in+0x241 pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 --- KDB: enter: witness_checkorder exclusive sx so_rcv_sx r = 0 (0xffffff000140f3b8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/src/sys/net/pfil.c:73 Locked vnodes 0xffffff001106a9b0: tag ufs, type VREG usecount 4, writecount 0, refcount 6 mountedhere 0 flags (VV_TEXT) v_object 0xffffff0011318680 ref 2 pages 35 lock type ufs: SHARED (count 1) ino 2239451, on dev ufs/usr64 0xffffff001106a3e0: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xffffff0001820000 (pid 42488) ino 2262052, on dev ufs/usr64 0xffffff0001eabba0: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xffffff001106f680 (pid 42307) with 1 pending ino 2262734, on dev ufs/usr64 Starting background file system checks in 60 seconds. ... calcru: runtime went backwards from 3023 usec to 1678 usec for pid 36225 (dhcpd) --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="first-lor.txt" lock order reversal: 1st 0xffffffffa2a56680 pf task mtx (pf task mtx) @ /usr/src/sys/modules/pf/../. ./contrib/pf/net/pf.c:6729 2nd 0xffffff00013236f0 radix node head (radix node head) @ /usr/src/sys/net/rou te.c:147 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x539 _mtx_lock_flags() at _mtx_lock_flags+0x1f rtalloc1() at rtalloc1+0x57 rtalloc_ign() at rtalloc_ign+0x8f pf_route() at pf_route+0x4d3 pf_test() at pf_test+0x64c pf_check_in() at pf_check_in+0x2b pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 --- KDB: enter: witness_checkorder [thread pid 23 tid 100022 ] Stopped at kdb_enter+0x31: popq %rbp db> show pcpu cpuid = 0 curthread = 0xffffff00011bf340: pid 23 "em0 taskq" curpcb = 0xffffffffa0555d40 fpcurthread = none idlethread = 0xffffff000107f340: pid 12 "idle: cpu0" spin locks held: db> show allpcpu Current CPU: 0 cpuid = 0 curthread = 0xffffff00011bf340: pid 23 "em0 taskq" curpcb = 0xffffffffa0555d40 fpcurthread = none idlethread = 0xffffff000107f340: pid 12 "idle: cpu0" spin locks held: cpuid = 1 curthread = 0xffffff00011bf000: pid 24 "em1 taskq" curpcb = 0xffffffffa057ad40 fpcurthread = none idlethread = 0xffffff000107f680: pid 11 "idle: cpu1" spin locks held: db> trace Tracing pid 23 tid 100022 td 0xffffff00011bf340 kdb_enter() at kdb_enter+0x31 witness_checkorder() at witness_checkorder+0x48e _mtx_lock_flags() at _mtx_lock_flags+0x1f rtalloc1() at rtalloc1+0x57 rtalloc_ign() at rtalloc_ign+0x8f pf_route() at pf_route+0x4d3 pf_test() at pf_test+0x64c pf_check_in() at pf_check_in+0x2b pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 --- db> alltrace Tracing command dhcpd pid 36121 tid 100080 td 0xffffff000146f680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec bwait() at bwait+0x55 bufwait() at bufwait+0x56 bread() at bread+0x1e ffs_read() at ffs_read+0x2f7 vn_read() at vn_read+0x1d3 dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800772acc, rsp = 0x7fffffffeba8, r bp = 0xcc5c --- Tracing command sh pid 35283 tid 100064 td 0xffffff0001420680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_wait() at kern_wait+0x3c0 wait4() at wait4+0x33 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800923abc, rsp = 0x7fffffffd2c8, rbp = 0x33 --- Tracing command ntpd pid 34406 tid 100076 td 0xffffff00014b4680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba kern_kevent() at kern_kevent+0x36a kevent() at kevent+0x85 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800bd531c, rsp = 0x7fffffffcc0 8, rbp = 0xf --- Tracing command named pid 29869 tid 100102 td 0xffffff0001426340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800c03a4c, rsp = 0x7fffff5fbcc8 , rbp = 0x26 --- Tracing command named pid 29869 tid 100101 td 0xffffff0001426680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba do_cv_wait() at do_cv_wait+0x312 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x51 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff7fc dc8, rbp = 0x7fffff7fce30 --- Tracing command named pid 29869 tid 100100 td 0xffffff00014269c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa do_cv_wait() at do_cv_wait+0x4c2 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x51 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff9fd eb8, rbp = 0 --- Tracing command named pid 29869 tid 100099 td 0xffffff0001420000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa do_cv_wait() at do_cv_wait+0x4c2 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x51 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffffbfe eb8, rbp = 0 --- Tracing command named pid 29869 tid 100075 td 0xffffff00014b49c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_sigtimedwait() at kern_sigtimedwait+0x4ea sigwait() at sigwait+0x63 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800b7dc6c, rsp = 0x7fffffffec 48, rbp = 0x800e01120 --- Tracing command syslogd pid 26784 tid 100091 td 0xffffff00017bd000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x80081aa4c, rsp = 0x7fffffffdda8 , rbp = 0x800a1c0f0 --- Tracing command accounting pid 24352 tid 100087 td 0xffffff00017be000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 acct_thread() at acct_thread+0x1c3 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa2a66d30, rbp = 0 --- Tracing command devd pid 22944 tid 100060 td 0xffffff000140a000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x43796c, rsp = 0x7fffffffe928, r bp = 0x7fffffffed50 --- Tracing command pflogd pid 14667 tid 100063 td 0xffffff00014209c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba bpfread() at bpfread+0xde devfs_read_f() at devfs_read_f+0x67 dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffed08, r bp = 0x1 --- Tracing command pflogd pid 14530 tid 100066 td 0xffffff0001404340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa soreceive_generic() at soreceive_generic+0xd1a dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffe938, r bp = 0x4 --- Tracing command pfpurge pid 14486 tid 100062 td 0xffffff0001404680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 pf_purge_thread() at pf_purge_thread+0x60 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa29c2d30, rbp = 0 --- Tracing command sh pid 51 tid 100050 td 0xffffff0001353680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_wait() at kern_wait+0x3c0 wait4() at wait4+0x33 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800923abc, rsp = 0x7fffffffe6d8, rbp = 0x33 --- Tracing command schedcpu pid 50 tid 100049 td 0xffffff00013539c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 schedcpu_thread() at schedcpu_thread+0x209 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa067ed30, rbp = 0 --- Tracing command softdepflush pid 49 tid 100048 td 0xffffff0001355000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 softdep_flush() at softdep_flush+0x2da fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0679d30, rbp = 0 --- Tracing command syncer pid 48 tid 100047 td 0xffffff0001355340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec sched_sync() at sched_sync+0x5da fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0674d30, rbp = 0 --- Tracing command vnlru pid 47 tid 100046 td 0xffffff0001355680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 vnlru_proc() at vnlru_proc+0x692 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa066fd30, rbp = 0 --- Tracing command bufdaemon pid 46 tid 100045 td 0xffffff00013559c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 buf_daemon() at buf_daemon+0x225 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa066ad30, rbp = 0 --- Tracing command pagezero pid 45 tid 100044 td 0xffffff00012479c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 vm_pagezero() at vm_pagezero+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0665d30, rbp = 0 --- Tracing command vmdaemon pid 44 tid 100043 td 0xffffff0001298000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0660d30, rbp = 0 --- Tracing command pagedaemon pid 43 tid 100042 td 0xffffff0001298340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 vm_pageout() at vm_pageout+0xa62 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa065bd30, rbp = 0 --- Tracing command dummynet pid 42 tid 100041 td 0xffffff0001298680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 msleep_spin() at msleep_spin+0x18f taskqueue_thread_loop() at taskqueue_thread_loop+0x45 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0656d30, rbp = 0 --- Tracing command irq1: atkbd0 pid 41 tid 100040 td 0xffffff00012989c0 fork_trampoline() at fork_trampoline Tracing command irq7: ppbus0 ppc0 pid 40 tid 100039 td 0xffffff0001299000 fork_trampoline() at fork_trampoline Tracing command swi0: sio pid 39 tid 100038 td 0xffffff0001299340 fork_trampoline() at fork_trampoline Tracing command fdc0 pid 38 tid 100037 td 0xffffff0001299680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 fdc_thread() at fdc_thread+0x7b5 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0622d30, rbp = 0 --- Tracing command irq15: ata1 pid 37 tid 100036 td 0xffffff00012999c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0613d30, rbp = 0 --- Tracing command irq14: ata0 pid 36 tid 100035 td 0xffffff000129b000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05fbd30, rbp = 0 --- Tracing command usb4 pid 35 tid 100034 td 0xffffff00011c0680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05e3d30, rbp = 0 --- Tracing command usb3 pid 34 tid 100033 td 0xffffff00011c09c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05ccd30, rbp = 0 --- Tracing command irq16: uhci3 pid 33 tid 100032 td 0xffffff0001246000 fork_trampoline() at fork_trampoline Tracing command usb2 pid 32 tid 100031 td 0xffffff0001246340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05c1d30, rbp = 0 --- Tracing command usb1 pid 31 tid 100030 td 0xffffff0001246680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05bbd30, rbp = 0 --- Tracing command irq19: uhci1 pid 30 tid 100029 td 0xffffff00012469c0 fork_trampoline() at fork_trampoline Tracing command usbtask-dr pid 29 tid 100028 td 0xffffff0001247000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec usb_task_thread() at usb_task_thread+0x99 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05b0d30, rbp = 0 --- Tracing command usbtask-hc pid 28 tid 100027 td 0xffffff0001247340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec usb_task_thread() at usb_task_thread+0x99 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05abd30, rbp = 0 --- Tracing command usb0 pid 27 tid 100026 td 0xffffff0001247680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05a6d30, rbp = 0 --- Tracing command irq23: uhci0 ehci0 pid 26 tid 100025 td 0xffffff00011a6680 fork_trampoline() at fork_trampoline Tracing command irq18: bge0 uhci2 pid 25 tid 100024 td 0xffffff00011a69c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa059bd30, rbp = 0 --- Tracing command em1 taskq pid 24 tid 100023 td 0xffffff00011bf000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x32 trap() at trap+0x2fd nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff8040984d, rsp = 0xffffffff9fe19ff0, rbp = 0xfffff fffa057a870 --- _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0x7a pf_test() at pf_test+0x72 pf_check_in() at pf_check_in+0x2b pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 --- Tracing command em0 taskq pid 23 tid 100022 td 0xffffff00011bf340 kdb_enter() at kdb_enter+0x31 witness_checkorder() at witness_checkorder+0x48e _mtx_lock_flags() at _mtx_lock_flags+0x1f rtalloc1() at rtalloc1+0x57 rtalloc_ign() at rtalloc_ign+0x8f pf_route() at pf_route+0x4d3 pf_test() at pf_test+0x64c pf_check_in() at pf_check_in+0x2b pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 --- Tracing command irq9: acpi0 pid 22 tid 100021 td 0xffffff00011bf680 fork_trampoline() at fork_trampoline Tracing command kqueue taskq pid 21 tid 100020 td 0xffffff00011bf9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa052bd30, rbp = 0 --- Tracing command swi6: task queue pid 20 tid 100019 td 0xffffff00011c0000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0526d30, rbp = 0 --- Tracing command swi6: Giant taskq pid 19 tid 100018 td 0xffffff00011c0340 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 9 tid 100017 td 0xffffff000108c9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa051cd30, rbp = 0 --- Tracing command swi5: + pid 18 tid 100016 td 0xffffff00011a5000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0517d30, rbp = 0 --- Tracing command swi2: cambio pid 17 tid 100015 td 0xffffff00011a5340 fork_trampoline() at fork_trampoline Tracing command xpt_thrd pid 8 tid 100014 td 0xffffff00011a5680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec xpt_scanner_thread() at xpt_scanner_thread+0x38 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa050dd30, rbp = 0 --- Tracing command acpi_task_2 pid 7 tid 100013 td 0xffffff00011a59c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0508d30, rbp = 0 --- Tracing command acpi_task_1 pid 6 tid 100012 td 0xffffff00011a6000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0503d30, rbp = 0 --- Tracing command acpi_task_0 pid 5 tid 100011 td 0xffffff00011a6340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04fed30, rbp = 0 --- Tracing command yarrow pid 16 tid 100010 td 0xffffff0001080340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 random_kthread() at random_kthread+0x1a4 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04f5d30, rbp = 0 --- Tracing command g_down pid 4 tid 100009 td 0xffffff0001080680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 g_io_schedule_down() at g_io_schedule_down+0x201 g_down_procbody() at g_down_procbody+0x55 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04f0d30, rbp = 0 --- Tracing command g_up pid 3 tid 100008 td 0xffffff00010809c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 g_io_schedule_up() at g_io_schedule_up+0x119 g_up_procbody() at g_up_procbody+0x55 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04ebd30, rbp = 0 --- Tracing command g_event pid 2 tid 100007 td 0xffffff000108c000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 g_event_procbody() at g_event_procbody+0xa9 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04e6d30, rbp = 0 --- Tracing command swi3: vm pid 15 tid 100006 td 0xffffff000108c340 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 14 tid 100005 td 0xffffff000108c680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04dcd30, rbp = 0 --- Tracing command swi1: net pid 13 tid 100004 td 0xffffff000107f000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04d7d30, rbp = 0 --- Tracing command idle: cpu0 pid 12 tid 100003 td 0xffffff000107f340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 critical_exit() at critical_exit+0x83 intr_execute_handlers() at intr_execute_handlers+0xdd Xapic_isr1() at Xapic_isr1+0x7f --- interrupt, rip = 0xffffffff805ce6a2, rsp = 0xffffffffa04d1bc0, rbp = 0xfffff fffa04d1bd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x19a sched_idletd() at sched_idletd+0x64 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04d1d30, rbp = 0 --- Tracing command idle: cpu1 pid 11 tid 100002 td 0xffffff000107f680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 critical_exit() at critical_exit+0x83 intr_execute_handlers() at intr_execute_handlers+0xdd Xapic_isr1() at Xapic_isr1+0x7f --- interrupt, rip = 0xffffffff805ce6a2, rsp = 0xffffffffa04ccbc0, rbp = 0xfffff fffa04ccbd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x19a sched_idletd() at sched_idletd+0x64 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04ccd30, rbp = 0 --- Tracing command init pid 1 tid 100001 td 0xffffff000107f9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_wait() at kern_wait+0x3c0 wait4() at wait4+0x33 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40b61c, rsp = 0x7fffffffe848, rbp = 0x33 --- Tracing command audit pid 10 tid 100000 td 0xffffff0001080000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _cv_wait() at _cv_wait+0x115 audit_worker() at audit_worker+0x47f fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04c2d30, rbp = 0 --- Tracing command swapper pid 0 tid 0 td 0xffffffff808def70 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 scheduler() at scheduler+0x39e mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> show alllocks Process 14530 (pflogd) thread 0xffffff0001404340 (100066) exclusive sx so_rcv_sx r = 0 (0xffffff00014713b8) locked @ /usr/src/sys/kern/uip c_sockbuf.c:148 Process 24 (em1 taskq) thread 0xffffff00011bf000 (100023) shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/sr c/sys/net/pfil.c:73 Process 23 (em0 taskq) thread 0xffffff00011bf340 (100022) exclusive sleep mutex pf task mtx r = 0 (0xffffffffa2a56680) locked @ /usr/src/s ys/modules/pf/../../contrib/pf/net/pf.c:6729 shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/sr c/sys/net/pfil.c:73 db> show witness Sleep locks: 0 so_rcv_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148 7 so_rcv -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2022 8 sellck -- last acquired @ /usr/src/sys/kern/sys_generic.c:759 8 radix node head -- last acquired @ /usr/src/sys/net/route.c:147 9 rtentry -- last acquired @ /usr/src/sys/netinet/ip_output.c:594 10 ifaddr -- last acquired @ /usr/src/sys/net/route.c:287 10 rts_inq -- last acquired @ /usr/src/sys/net/netisr.c:140 11 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:2335 11 UMA zone -- (already displayed) 9 UMA boot pages -- last acquired @ /usr/src/sys/vm/uma_core.c:916 11 vm page queue free mutex -- last acquired @ /usr/src/sys/vm/vm_page.c:1030 9 ifnet -- last acquired @ /usr/src/sys/net/if.c:1047 11 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:212 12 eventhandler list -- last acquired @ /usr/src/sys/kern/kern_exec.c:904 10 if_addr_mtx -- last acquired @ /usr/src/sys/net/if.c:828 10 pf task mtx -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/ne t/pf.c:6729 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 11 arc4_mtx -- last acquired @ /usr/src/sys/libkern/arc4random.c:137 11 bpf interface lock -- last acquired @ /usr/src/sys/net/bpf.c:1422 12 bpf cdev lock -- last acquired @ /usr/src/sys/net/bpf.c:1426 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 8 process lock -- last acquired @ /usr/src/sys/amd64/amd64/trap.c:622 9 session -- last acquired @ /usr/src/sys/kern/kern_fork.c:582 10 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1213 11 uidinfo struct -- last acquired @ order list:0 11 sleep mtxpool -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:508 12 kqueue -- last acquired @ /usr/src/sys/kern/kern_event.c:1182 13 struct mount mtx -- last acquired @ /usr/src/sys/kern/vfs_mount.c:440 14 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:2621 15 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:69 15 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:3040 15 Syncer mtx -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1915 10 tty -- last acquired @ /usr/src/sys/kern/kern_event.c:1664 9 sigacts -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:392 9 ktrace -- last acquired @ /usr/src/sys/kern/kern_fork.c:600 11 sleep mtxpool -- (already displayed) 11 sleep mtxpool -- (already displayed) 11 UMA zone -- (already displayed) 12 kqueue -- (already displayed) 8 process lock -- (already displayed) 2 unp_mtx -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:803 3 accept -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2837 6 so_snd -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:234 7 so_rcv -- (already displayed) 11 sleep mtxpool -- (already displayed) 7 so_rcv -- (already displayed) 7 so_rcv -- (already displayed) 6 so_snd -- (already displayed) 3 filedesc structure -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1987 14 vnode interlock -- (already displayed) 8 process lock -- (already displayed) 11 sleep mtxpool -- (already displayed) 5 Name Cache -- last acquired @ /usr/src/sys/kern/vfs_cache.c:327 14 vnode interlock -- (already displayed) 11 UMA zone -- (already displayed) 9 UMA boot pages -- (already displayed) 4 fdesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1472 15 cdev -- (already displayed) 4 Giant -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241 5 pipe mutex -- last acquired @ /usr/src/sys/kern/sys_pipe.c:1331 6 sigio lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:782 7 process group -- last acquired @ /usr/src/sys/kern/kern_fork.c:572 8 process lock -- (already displayed) 8 sellck -- (already displayed) 11 UMA zone -- (already displayed) 6 system map -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1995 8 vm page queue mutex -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3437 14 vnode interlock -- (already displayed) 9 pmap -- last acquired @ /usr/src/sys/amd64/amd64/pmap.c:3076 11 vm page queue free mutex -- (already displayed) 11 vm page queue free mutex -- (already displayed) 7 kmem object -- last acquired @ /usr/src/sys/vm/vm_object.c:441 11 vm page queue free mutex -- (already displayed) 8 vm page queue mutex -- (already displayed) 7 kernel object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3641 11 vm page queue free mutex -- (already displayed) 8 vm page queue mutex -- (already displayed) 11 vm page queue free mutex -- (already displayed) 9 pmap -- (already displayed) 7 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:2905 11 UMA zone -- (already displayed) 7 standard object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3426 11 vm page queue free mutex -- (already displayed) 14 vnode interlock -- (already displayed) 8 vm page queue mutex -- (already displayed) 8 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:225 11 UMA zone -- (already displayed) 5 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:1314 11 UMA zone -- (already displayed) 7 KMAP ENTRY -- (already displayed) 9 UMA boot pages -- (already displayed) 11 vm page queue free mutex -- (already displayed) 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 5 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:307 6 kernel environment -- last acquired @ /usr/src/sys/kern/kern_environment.c: 301 7 kernel object -- (already displayed) 8 vm object_list -- (already displayed) 7 KMAP ENTRY -- (already displayed) 10 uidinfo hash -- (already displayed) 8 process lock -- (already displayed) 11 sleep mtxpool -- (already displayed) 5 evclass_mtx -- last acquired @ /usr/src/sys/security/audit/audit_bsm_klib.c :112 5 TID lock -- last acquired @ /usr/src/sys/kern/subr_unit.c:623 7 standard object -- (already displayed) 5 intr event -- last acquired @ /usr/src/sys/kern/kern_intr.c:454 15 cdev -- (already displayed) 5 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:201 5 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:3126 10 tty -- (already displayed) 5 intr config -- last acquired @ /usr/src/sys/kern/subr_autoconf.c:73 5 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:125 5 XPT lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2668 6 XPT topology lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2673 6 kernel environment -- (already displayed) 7 taskqueue -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:73 5 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:152 5 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:539 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 5 devd -- last acquired @ /usr/src/sys/kern/subr_bus.c:489 8 sellck -- (already displayed) 8 sellck -- (already displayed) 5 ACPI semaphore -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSynch.c:30 3 5 acpi subsystem HW lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSy nch.c:377 5 acpi subsystem GPE lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdS ynch.c:377 7 taskqueue -- (already displayed) 5 msi -- last acquired @ /usr/src/sys/amd64/amd64/msi.c:389 9 ifnet -- (already displayed) 5 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:1671 11 bpf interface lock -- (already displayed) 5 bounce pages lock -- last acquired @ /usr/src/sys/amd64/amd64/busdma_machde p.c:1073 9 pmap -- (already displayed) 5 unit# allocation -- last acquired @ /usr/src/sys/kern/subr_unit.c:623 15 vnode_free_list -- (already displayed) 5 pfs_node -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_internal.h:10 3 5 pfs_fileno -- last acquired @ /usr/src/sys/kern/subr_unit.c:623 5 random reseed -- last acquired @ /usr/src/sys/dev/random/yarrow.c:278 11 arc4_mtx -- (already displayed) 5 nfsd_mtx -- last acquired @ /usr/src/sys/nfsserver/nfs_srvsock.c:796 6 so_snd -- (already displayed) 5 if_clone lock -- last acquired @ /usr/src/sys/net/if_clone.c:164 5 if_cloners lock -- last acquired @ /usr/src/sys/net/if_clone.c:252 5 domain list -- last acquired @ /usr/src/sys/kern/uipc_domain.c:228 6 pfil_head_list lock -- last acquired @ /usr/src/sys/net/pfil.c:159 5 PFil hook read/write mutex -- last acquired @ /usr/src/sys/net/pfil.c:73 6 pfil_head_list lock -- (already displayed) 10 pf task mtx -- (already displayed) 6 IPFW static rules -- last acquired @ /usr/src/sys/netinet/ip_fw2.c:2647 8 radix node head -- (already displayed) 7 if send queue -- last acquired @ /usr/src/sys/dev/em/if_em.c:949 6 network driver -- last acquired @ /usr/src/sys/dev/bge/if_bge.c:3014 10 if_addr_mtx -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 7 taskqueue -- (already displayed) 7 if send queue -- (already displayed) 5 isn_mtx -- last acquired @ /usr/src/sys/netinet/tcp_subr.c:1426 8 radix node head -- (already displayed) 6 IPFW static rules -- (already displayed) 5 lo_mtx -- last acquired @ /usr/src/sys/net/if_loop.c:161 5 ATA queue lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:177 6 ATA state lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:194 5 devstat -- last acquired @ /usr/src/sys/kern/subr_devstat.c:83 6 XPT topology lock -- (already displayed) 5 NFS iod lock -- last acquired @ /usr/src/sys/nfsclient/nfs_nfsiod.c:196 6 mountlist -- last acquired @ /usr/src/sys/kern/vfs_syscalls.c:526 13 struct mount mtx -- (already displayed) 13 struct mount mtx -- (already displayed) 5 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:460 6 mountlist -- (already displayed) 14 vnode interlock -- (already displayed) 5 buf queue lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1738 14 vnode interlock -- (already displayed) 5 bio queue -- last acquired @ /usr/src/sys/geom/geom_io.c:68 5 bdone lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3800 5 needsbuffer lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:321 5 FFS Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_alloc.c:1475 11 arc4_mtx -- (already displayed) 5 Name Cache -- (already displayed) 5 knlist lock for lockless objects -- last acquired @ /usr/src/sys/kern/kern_ event.c:1664 5 vfs hash -- last acquired @ /usr/src/sys/kern/vfs_hash.c:71 14 vnode interlock -- (already displayed) 5 devfs interlock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:194 14 vnode interlock -- (already displayed) 5 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349 6 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:457 6 dirhash -- (already displayed) 5 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:413 8 vm page queue mutex -- (already displayed) 7 process group -- (already displayed) 10 tty -- (already displayed) 9 session -- (already displayed) 5 Softdep Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:5022 6 system map -- (already displayed) 11 UMA zone -- (already displayed) 14 vnode interlock -- (already displayed) 6 buffer daemon lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:429 10 pf task mtx -- (already displayed) 6 buffer daemon lock -- (already displayed) 6 sigio lock -- (already displayed) 6 so_snd -- (already displayed) 5 pipe mutex -- (already displayed) 0 so_snd_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148 6 so_snd -- (already displayed) 1 unp_global_rwlock -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:770 2 unp_mtx -- (already displayed) 3 filedesc structure -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 8 radix node head -- (already displayed) 9 rtentry -- (already displayed) 10 rts_inq -- (already displayed) 0 pf_statetbl_lock -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/n et/pf.c:979 10 pf task mtx -- (already displayed) 6 pfil_head_list lock -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 0 devfsmount -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:201 5 devfs interlock -- (already displayed) 15 vnode_free_list -- (already displayed) 14 vnode interlock -- (already displayed) 13 struct mount mtx -- (already displayed) 15 cdev -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 1 DEVFS ruleset lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_rule.c:177 0 g_disk_done -- last acquired @ /usr/src/sys/geom/geom_disk.c:198 5 bio queue -- (already displayed) 11 UMA zone -- (already displayed) 0 dummynet -- last acquired @ /usr/src/sys/netinet/ip_dummynet.c:739 0 ipqlock -- last acquired @ /usr/src/sys/netinet/ip_input.c:1086 0 sem -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1288 0 fdc lock -- last acquired @ /usr/src/sys/dev/fdc/fdc.c:803 0 if_afdata -- last acquired @ /usr/src/sys/net/if.c:578 0 GEOM topology -- last acquired @ /usr/src/sys/geom/geom_event.c:233 5 GEOM orphanage -- (already displayed) 5 devstat -- (already displayed) 5 unit# allocation -- (already displayed) 15 cdev -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 9 UMA boot pages -- (already displayed) 8 vm object_list -- (already displayed) 14 vnode interlock -- (already displayed) 7 standard object -- (already displayed) 6 system map -- (already displayed) 0 intr sources -- last acquired @ /usr/src/sys/amd64/amd64/intr_machdep.c:593 5 msi -- (already displayed) 0 audit_mtx -- last acquired @ /usr/src/sys/security/audit/audit_worker.c:448 0 uma object -- last acquired @ /usr/src/sys/vm/vm_meter.c:115 0 p_peers -- last acquired @ /usr/src/sys/kern/kern_exit.c:284 0 runningbufspace lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:340 0 umtxql -- last acquired @ /usr/src/sys/kern/kern_umtx.c:326 0 accept_filter_mtx -- last acquired @ /usr/src/sys/kern/uipc_accf.c:116 0 clone events drain lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c :626 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 15 cdev -- (already displayed) 0 ACPI PCI bus methods -- last acquired @ /usr/src/sys/dev/acpica/acpi_pcib.c:22 1 0 rtsock route_cb lock -- last acquired @ /usr/src/sys/net/rtsock.c:236 0 rawcb -- last acquired @ /usr/src/sys/net/raw_cb.c:104 7 so_rcv -- (already displayed) 11 UMA zone -- (already displayed) 0 ACPI PCI link -- last acquired @ /usr/src/sys/dev/acpica/acpi_pci_link.c:432 5 ACPI semaphore -- (already displayed) 0 ACPI root bus -- last acquired @ /usr/src/sys/dev/acpica/acpi.c:1022 5 rman -- (already displayed) 5 ACPI semaphore -- (already displayed) 6 system map -- (already displayed) 0 acct_sx -- last acquired @ /usr/src/sys/kern/kern_acct.c:351 11 eventhandler -- (already displayed) 1 proctree -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:325 2 allproc -- last acquired @ /usr/src/sys/kern/kern_fork.c:297 3 allprison -- last acquired @ /usr/src/sys/kern/kern_jail.c:945 11 sleep mtxpool -- (already displayed) 8 process lock -- (already displayed) 4 fdesc -- (already displayed) 3 filedesc structure -- (already displayed) 14 vnode interlock -- (already displayed) 3 user map -- last acquired @ /usr/src/sys/vm/vm_map.c:3111 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 8 vm object_list -- (already displayed) 7 standard object -- (already displayed) 8 vm page queue mutex -- (already displayed) 9 UMA boot pages -- (already displayed) 9 pmap -- (already displayed) 14 vnode interlock -- (already displayed) 5 buf queue lock -- (already displayed) 5 needsbuffer lock -- (already displayed) 6 system map -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 13 struct mount mtx -- (already displayed) 6 buffer daemon lock -- (already displayed) 5 Softdep Lock -- (already displayed) 11 arc4_mtx -- (already displayed) 7 process group -- (already displayed) 4 Giant -- (already displayed) 8 process lock -- (already displayed) 9 session -- (already displayed) 6 sigio lock -- (already displayed) 3 filedesc structure -- (already displayed) 8 process lock -- (already displayed) 5 FFS Lock -- (already displayed) 13 struct mount mtx -- (already displayed) 14 vnode interlock -- (already displayed) 7 standard object -- (already displayed) 5 buf queue lock -- (already displayed) 6 system map -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 4 Giant -- (already displayed) 5 needsbuffer lock -- (already displayed) 6 buffer daemon lock -- (already displayed) 5 Softdep Lock -- (already displayed) 0 vm daemon -- last acquired @ /usr/src/sys/vm/vm_pageout.c:1531 0 protect sysfilt_ops -- last acquired @ /usr/src/sys/kern/kern_event.c:758 0 so_glabel -- last acquired @ /usr/src/sys/kern/uipc_socket.c:280 0 sysctl lock -- last acquired @ /usr/src/sys/kern/kern_sysctl.c:1396 11 arc4_mtx -- (already displayed) 2 allproc -- (already displayed) 8 process lock -- (already displayed) 3 user map -- (already displayed) 15 cdev -- (already displayed) 1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1366 3 filedesc structure -- (already displayed) 5 GEOM orphanage -- (already displayed) 4 Giant -- (already displayed) 9 ktrace -- (already displayed) 1 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:1096 11 UMA zone -- (already displayed) 2 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c: 407 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 3 filedesc structure -- (already displayed) 14 vnode interlock -- (already displayed) 5 Name Cache -- (already displayed) 5 buf queue lock -- (already displayed) 5 needsbuffer lock -- (already displayed) 5 vfs hash -- (already displayed) 15 vnode_free_list -- (already displayed) 13 struct mount mtx -- (already displayed) 5 dirhash list -- (already displayed) 11 vm page queue free mutex -- (already displayed) 6 system map -- (already displayed) 7 kernel object -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 6 dirhash -- (already displayed) 15 cdev -- (already displayed) 8 vm object_list -- (already displayed) 7 standard object -- (already displayed) 5 pbuf mutex -- (already displayed) 6 buffer daemon lock -- (already displayed) 5 kobj -- (already displayed) 1 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:741 11 UMA zone -- (already displayed) 6 system map -- (already displayed) 5 devstat -- (already displayed) 5 ttylist -- (already displayed) 8 vm object_list -- (already displayed) 5 UMA lock -- (already displayed) 15 Syncer mtx -- (already displayed) 1 unp_global_rwlock -- (already displayed) 1 tcp -- last acquired @ /usr/src/sys/netinet/tcp_usrreq.c:254 2 tcpinp -- last acquired @ /usr/src/sys/netinet/tcp_usrreq.c:255 6 so_snd -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 7 so_rcv -- (already displayed) 9 ifnet -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 1 tcp_sc_head -- last acquired @ /usr/src/sys/netinet/tcp_syncache.c:1696 1 udp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:839 2 udpinp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:843 3 in_multi_mtx -- last acquired @ /usr/src/sys/netinet/in_mcast.c:320 4 igmp_mtx -- last acquired @ /usr/src/sys/netinet/igmp.c:446 10 if_addr_mtx -- (already displayed) 10 if_addr_mtx -- (already displayed) 4 EM Core Lock -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241 5 EM TX Lock -- last acquired @ /usr/src/sys/dev/em/if_em.c:1563 7 if send queue -- (already displayed) 10 if_addr_mtx -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 9 UMA boot pages -- (already displayed) 7 taskqueue -- (already displayed) 6 network driver -- (already displayed) 6 so_snd -- (already displayed) 9 ifnet -- (already displayed) 11 arc4_mtx -- (already displayed) 8 radix node head -- (already displayed) 9 rtentry -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 6 IPFW static rules -- (already displayed) 7 if send queue -- (already displayed) 5 EM TX Lock -- (already displayed) 7 so_rcv -- (already displayed) 3 accept -- (already displayed) 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 9 UMA boot pages -- (already displayed) 8 radix node head -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 9 rtentry -- (already displayed) 6 IPFW static rules -- (already displayed) 7 if send queue -- (already displayed) 5 EM TX Lock -- (already displayed) 1 rip -- last acquired @ /usr/src/sys/netinet/raw_ip.c:613 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 2 rawinp -- last acquired @ /usr/src/sys/netinet/in_pcb.c:221 5 ACPI semaphore -- (already displayed) 3 allprison -- (already displayed) 1 cpufreq lock -- last acquired @ /usr/src/sys/kern/kern_cpu.c:413 11 UMA zone -- (already displayed) 11 vm page queue free mutex -- (already displayed) 9 UMA boot pages -- (already displayed) 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 9 ifnet -- (already displayed) 1 ACPI CPU -- last acquired @ /usr/src/sys/dev/acpica/acpi_cpu.c:1167 0 ng_node -- last acquired @ order list:0 1 ng_worklist -- last acquired @ order list:0 0 802.11 com lock -- last acquired @ order list:0 0 ddp_list_mtx -- last acquired @ order list:0 1 ddp_mtx -- last acquired @ order list:0 0 slip_mtx -- last acquired @ order list:0 1 slip sc_mtx -- last acquired @ order list:0 0 unp -- last acquired @ order list:0 6 so_snd -- (already displayed) Spin locks: Locks which were never acquired: swap_pager swhash SCSI CD Changer List MD config lock IPFW dynamic rules arp_inq tcp_hc_entry ip_inq pfs_vncache ppp_softc_list_mtx tunmtx gif_mtx msq semid nfs4dev state nfs4dev waitq nfs4dev newq ntfs nthash NFS xid lock NFS reqq lock ACPI global lock CAM SIMQ lock hptlock LED sx LED mtx audit_pipe_mtx nfslock audit_trigger_mtx ktrace_sx bpin lock UUID generator mutex lock ACPI PCI power methods ACPI lid ACPI HPET support ACPI embedded controller ACPI thermal zone ACPI Smart Battery ACPI cmbat ACPI power resources ACPI generic battery ACPI AC adapter MSDOSFS fileno pmc shared lock kqueue order firmware table securelevel mutex lock encapmtx fifo mutex phys_pager list dev_pager list swapdev swap_pager list vm map sleep mutex db> show lockedvnods Locked vnodes 0xffffff0001906000: tag ufs, type VREG usecount 1, writecount 0, refcount 6 mountedhere 0 flags () v_object 0xffffff00018714e0 ref 0 pages 13 lock type ufs: SHARED (count 1) ino 2240130, on dev ufs/usr64 db> show uma Zone Size Used Free Requests pfosfp 40 407 97 407 pfospfen 112 696 30 696 pfiaddrpl 120 0 0 0 pfstatescrub 40 0 0 0 pffrcent 24 0 0 0 pffrcache 80 0 0 0 pffrag 80 0 0 0 pffrent 32 0 0 0 pfrkentry2 216 0 0 0 pfrkentry 216 470 52 478 pfrktable 1296 12 27 28 pfpooladdrpl 88 12 72 12 pfaltqpl 240 0 0 0 pfstatepl 392 12 18 12 pfrulepl 912 69 15 69 pfsrctrpl 152 0 0 0 FFS2 dinode 256 426 39 435 FFS1 dinode 128 0 0 0 FFS inode 192 426 34 435 Mountpoints 808 6 9 6 SWAPMETA 288 0 0 0 IPFW dynamic rule 120 0 0 0 rtentry 240 24 40 26 ripcb 288 1 25 1 sackhole 32 0 0 0 tcpreass 40 0 0 0 hostcache 136 0 0 0 syncache 120 0 0 0 tcptw 88 0 0 0 tcpcb 728 9 16 11 inpcb 288 9 30 11 udpcb 288 20 32 717 ipq 56 0 0 0 unpcb 248 10 35 16 socket 696 40 15 747 KNOTE 120 2 60 2 itimer 360 0 0 0 ksiginfo 112 58 998 58 pipe 744 1 39 495 NFSNODE 664 0 0 0 NFSMOUNT 568 0 0 0 DIRHASH 1024 37 11 37 NAMEI 1024 0 36 6609 L VFS Cache 327 0 0 0 S VFS Cache 104 427 41 903 VNODEPOLL 128 0 0 0 VNODE 496 452 44 462 ata_composite 352 0 0 0 ata_request 312 1 83 1218 g_bio 216 5 355 6008 ACL UMA zone 388 0 0 0 mbuf_ext_refcnt 4 0 0 0 mbuf_jumbo_16k 16384 0 0 0 mbuf_jumbo_9k 9216 0 0 0 mbuf_jumbo_pagesize 4096 0 0 0 mbuf_cluster 2048 1024 8 1024 mbuf 256 1 385 396 mbuf_packet 256 781 243 956 audit_record 984 0 0 0 VMSPACE 416 10 44 841 SLEEPQUEUE 64 113 111 113 UPCALL 88 0 0 0 THREAD 824 103 9 103 PROC 1128 61 38 892 umtx pi 96 0 0 0 TURNSTILE 152 113 31 113 Files 120 73 82 4722 4096 4096 345 76 4049 2048 2048 65 11 302 1024 1024 60 152 2484 512 512 537 72 2490 256 256 601 74 4734 128 128 3535 90 10183 64 64 1791 113 27916 32 32 2096 328 3186 16 16 1666 350 31457 mt_zone 1024 217 7 217 DP fakepg 120 0 0 0 MAP ENTRY 112 200 196 27758 KMAP ENTRY 112 35 97 1448 MAP 248 7 23 7 VM OBJECT 208 585 99 12763 128 Bucket 1048 24 0 24 64 Bucket 536 34 1 34 32 Bucket 280 51 5 51 16 Bucket 152 73 2 73 UMA Hash 256 7 8 7 UMA RCntSlabs 128 516 6 516 UMA Slabs 128 625 13 1160 UMA Zones 216 89 13 89 UMA Kegs 216 89 13 89 db> show malloc Type InUse MemUse Requests msdosfs_mount 0 0K 0 msdosfs_fat 0 0K 0 msdosfs_fileno 0 0K 0 msdosfs_node 0 0K 0 DEVFS 27 1K 28 DEVFS_RULE 36 17K 36 DEVFS2 108 2K 108 DEVFS3 241 61K 242 DEVFS1 108 54K 108 USB 91 20K 106 USBdev 21 9K 21 atkbddev 2 1K 2 USBHC 0 0K 0 UART 0 0K 0 nexusdev 4 1K 4 msi 2 1K 2 twe_commands 0 0K 0 twa_commands 0 0K 0 firewire 0 0K 0 legacydrv 0 0K 0 io_apic 1 2K 1 entropy 1024 64K 1024 ppbusdev 3 1K 3 fw_xfer 0 0K 0 memdesc 1 4K 1 madt_table 0 0K 0 scsi_cd 0 0K 0 CAM ccb queue 0 0K 0 vm_pgdata 1 128K 1 acpipwr 0 0K 0 aaccam 0 0K 0 acpi_perf 0 0K 0 UMAHash 0 0K 0 ufs_mount 12 30K 12 ufs_quota 0 0K 0 ufs_dirhash 30 6K 30 pagedep 9 65K 16 inodedep 37 521K 49 newblk 1 1K 27 bmsafemap 6 1K 11 allocdirect 7 2K 23 indirdep 1 1K 5 allocindir 1 1K 3 freefrag 1 1K 2 freeblks 1 1K 4 freefile 3 1K 8 diradd 18 2K 28 mkdir 8 1K 12 dirrem 14 1K 25 newdirblk 0 0K 0 savedino 0 0K 12 audit_trigger 0 0K 0 audit_pipe 0 0K 0 audit_pipeent 0 0K 0 audit_pipe_preselect 0 0K 0 audit_evclass 150 5K 187 audit_bsm 0 0K 0 audit_cred 0 0K 0 audit_data 0 0K 0 audit_path 0 0K 0 audit_text 0 0K 0 rpcclnt 0 0K 0 agp 0 0K 0 nfss_srvsock 0 0K 0 nfss_srvdesc 0 0K 0 nfss_daemon 1 16K 1 nfsclient_lock 0 0K 0 nfsclient_nlminfo 0 0K 0 nfsclient_req 0 0K 0 nfsclient_bigfh 0 0K 0 nfsclient_diroff 0 0K 0 nfsclient_hash 0 0K 0 nfsclient_directio 0 0K 0 nfsclient_srvsock 0 0K 0 idmap 0 0K 0 nfs4_dev 0 0K 0 syncache 1 100K 1 tcplog 0 0K 0 aacbuf 0 0K 0 hostcache 1 36K 1 IpFw/IpAcct 1 1K 1 ipfw_tbl 0 0K 0 encap_export_host 0 0K 0 dummynet 0 0K 0 in_multi 4 1K 4 ip_moptions 0 0K 0 in_msource 0 0K 0 igmp 0 0K 0 routetbl 206 83K 465 pci_link 16 2K 16 tun 0 0K 0 SCSI SES 0 0K 0 sl 0 0K 0 ppp 0 0K 0 lo 1 1K 1 gif 0 0K 0 fw_com 0 0K 0 faith 0 0K 0 arpcom 3 1K 3 clone 6 24K 6 ifnet 7 13K 7 ifaddr 52 17K 52 ether_multi 17 1K 20 BPF 10 66K 11 subr_export_host 0 0K 0 SCSI sa 0 0K 0 mount 84 4K 138 vnodemarker 0 0K 20 vnodes 1 1K 1 vfs_hash 1 512K 1 export_host 0 0K 0 cl_savebuf 0 0K 0 vfscache 1 1024K 1 biobuf 38 76K 42 soname 5 1K 228 pcb 13 9K 15 CAM XPT 10 2K 25 mbuf_tag 12 1K 247 accf 0 0K 0 ptys 0 0K 0 ttys 403 70K 403 shm 1 16K 1 sem 4 8K 4 msg 4 30K 4 ioctlops 0 0K 1994 select 0 0K 0 iov 0 0K 1177 acpicmbat 0 0K 0 acpidev 63 4K 63 ciss_data 0 0K 0 CAM periph 2 1K 9 Unitno 7 1K 9 taskqueue 15 2K 15 stack 0 0K 0 CAM SIM 1 1K 1 scsi_da 0 0K 0 sbuf 0 0K 448 scsi_ch 0 0K 0 rman 161 20K 611 acpisem 12 2K 12 mfibuf 0 0K 0 kobj 227 908K 311 md_disk 0 0K 0 eventhandler 68 6K 68 devstat 16 33K 16 bus 961 75K 4819 bus-sc 66 83K 2400 md_sectors 0 0K 0 SWAP 0 0K 0 p1003.1b 1 1K 1 umtx 112 14K 112 sysctl 0 0K 143 sysctloid 2817 138K 2817 sysctltmp 0 0K 152 LED 0 0K 0 plimit 3 1K 254 uidinfo 4 3K 4 cred 11 3K 1266 pgrp 7 1K 7 session 7 1K 7 proc 2 16K 2 subproc 162 298K 993 kbdmux 6 9K 6 mtx_pool 1 8K 1 ipsbuf 0 0K 0 module 353 45K 353 iirbuf 0 0K 0 cache 0 0K 0 devbuf 314 1777K 321 temp 34 8K 3694 ip6opt 0 0K 0 ip6ndp 0 0K 0 CAM queue 3 1K 3 free 0 0K 0 lockf 2 1K 8 linker 96 132K 125 CAM dev queue 1 1K 1 KTRACE 100 13K 100 ast_driver 0 0K 0 afd_driver 0 0K 0 prison 0 0K 0 ithread 76 13K 76 acd_driver 0 0K 0 zombie 0 0K 0 proc-args 9 1K 327 kqueue 2 3K 2 kenv 71 11K 73 filedesc 64 33K 895 filedesc_to_leader 0 0K 0 sigio 1 1K 1 ar_driver 0 0K 10 ata_pci 0 0K 0 cdev 22 6K 22 sbp 0 0K 0 if_fwip 0 0K 0 ata_dma 2 1K 2 if_fwe 0 0K 0 ad_driver 2 1K 2 ata_generic 2 2K 2 acpitask 0 0K 1 isofs_mount 0 0K 0 isofs_node 0 0K 0 isadev 5 1K 5 acpica 1865 180K 58648 GEOM 144 25K 1163 fwmem 0 0K 0 pfs_vncache 0 0K 0 pfs_nodes 20 5K 20 ntfs_mount 0 0K 0 ntfs_ntnode 0 0K 0 ntfs_fnode 0 0K 0 ntfs_dir 0 0K 0 ntfs_vattr 0 0K 0 ntfsd_resdata 0 0K 0 ntfs_vrun 0 0K 0 ntfs_decomp 0 0K 0 ntfs_nthash 1 512K 1 db> show malloc Type InUse MemUse Requests msdosfs_mount 0 0K 0 msdosfs_fat 0 0K 0 msdosfs_fileno 0 0K 0 msdosfs_node 0 0K 0 DEVFS 27 1K 28 DEVFS_RULE 36 17K 36 DEVFS2 108 2K 108 DEVFS3 241 61K 242 DEVFS1 108 54K 108 USB 91 20K 106 USBdev 21 9K 21 atkbddev 2 1K 2 USBHC 0 0K 0 UART 0 0K 0 nexusdev 4 1K 4 msi 2 1K 2 twe_commands 0 0K 0 twa_commands 0 0K 0 firewire 0 0K 0 legacydrv 0 0K 0 io_apic 1 2K 1 entropy 1024 64K 1024 ppbusdev 3 1K 3 fw_xfer 0 0K 0 memdesc 1 4K 1 madt_table 0 0K 0 scsi_cd 0 0K 0 CAM ccb queue 0 0K 0 vm_pgdata 1 128K 1 acpipwr 0 0K 0 aaccam 0 0K 0 acpi_perf 0 0K 0 UMAHash 0 0K 0 ufs_mount 12 30K 12 ufs_quota 0 0K 0 ufs_dirhash 30 6K 30 pagedep 9 65K 16 inodedep 37 521K 49 newblk 1 1K 27 bmsafemap 6 1K 11 allocdirect 7 2K 23 indirdep 1 1K 5 allocindir 1 1K 3 freefrag 1 1K 2 freeblks 1 1K 4 freefile 3 1K 8 diradd 18 2K 28 mkdir 8 1K 12 dirrem 14 1K 25 newdirblk 0 0K 0 savedino 0 0K 12 audit_trigger 0 0K 0 audit_pipe 0 0K 0 audit_pipeent 0 0K 0 audit_pipe_preselect 0 0K 0 audit_evclass 150 5K 187 audit_bsm 0 0K 0 audit_cred 0 0K 0 audit_data 0 0K 0 audit_path 0 0K 0 audit_text 0 0K 0 rpcclnt 0 0K 0 agp 0 0K 0 nfss_srvsock 0 0K 0 nfss_srvdesc 0 0K 0 nfss_daemon 1 16K 1 nfsclient_lock 0 0K 0 nfsclient_nlminfo 0 0K 0 nfsclient_req 0 0K 0 nfsclient_bigfh 0 0K 0 nfsclient_diroff 0 0K 0 nfsclient_hash 0 0K 0 nfsclient_directio 0 0K 0 nfsclient_srvsock 0 0K 0 idmap 0 0K 0 nfs4_dev 0 0K 0 syncache 1 100K 1 tcplog 0 0K 0 aacbuf 0 0K 0 hostcache 1 36K 1 IpFw/IpAcct 1 1K 1 ipfw_tbl 0 0K 0 encap_export_host 0 0K 0 dummynet 0 0K 0 in_multi 4 1K 4 ip_moptions 0 0K 0 in_msource 0 0K 0 igmp 0 0K 0 routetbl 206 83K 465 pci_link 16 2K 16 tun 0 0K 0 SCSI SES 0 0K 0 sl 0 0K 0 ppp 0 0K 0 lo 1 1K 1 gif 0 0K 0 fw_com 0 0K 0 faith 0 0K 0 arpcom 3 1K 3 clone 6 24K 6 ifnet 7 13K 7 ifaddr 52 17K 52 ether_multi 17 1K 20 BPF 10 66K 11 subr_export_host 0 0K 0 SCSI sa 0 0K 0 mount 84 4K 138 vnodemarker 0 0K 20 vnodes 1 1K 1 vfs_hash 1 512K 1 export_host 0 0K 0 cl_savebuf 0 0K 0 vfscache 1 1024K 1 biobuf 38 76K 42 soname 5 1K 228 pcb 13 9K 15 CAM XPT 10 2K 25 mbuf_tag 12 1K 247 accf 0 0K 0 ptys 0 0K 0 ttys 403 70K 403 shm 1 16K 1 sem 4 8K 4 msg 4 30K 4 ioctlops 0 0K 1994 select 0 0K 0 iov 0 0K 1177 acpicmbat 0 0K 0 acpidev 63 4K 63 ciss_data 0 0K 0 CAM periph 2 1K 9 Unitno 7 1K 9 taskqueue 15 2K 15 stack 0 0K 0 CAM SIM 1 1K 1 scsi_da 0 0K 0 sbuf 0 0K 448 scsi_ch 0 0K 0 rman 161 20K 611 acpisem 12 2K 12 mfibuf 0 0K 0 kobj 227 908K 311 md_disk 0 0K 0 eventhandler 68 6K 68 devstat 16 33K 16 bus 961 75K 4819 bus-sc 66 83K 2400 md_sectors 0 0K 0 SWAP 0 0K 0 p1003.1b 1 1K 1 umtx 112 14K 112 sysctl 0 0K 143 sysctloid 2817 138K 2817 sysctltmp 0 0K 152 LED 0 0K 0 plimit 3 1K 254 uidinfo 4 3K 4 cred 11 3K 1266 pgrp 7 1K 7 session 7 1K 7 proc 2 16K 2 subproc 162 298K 993 kbdmux 6 9K 6 mtx_pool 1 8K 1 ipsbuf 0 0K 0 module 353 45K 353 iirbuf 0 0K 0 cache 0 0K 0 devbuf 314 1777K 321 temp 34 8K 3694 ip6opt 0 0K 0 ip6ndp 0 0K 0 CAM queue 3 1K 3 free 0 0K 0 lockf 2 1K 8 linker 96 132K 125 CAM dev queue 1 1K 1 KTRACE 100 13K 100 ast_driver 0 0K 0 afd_driver 0 0K 0 prison 0 0K 0 ithread 76 13K 76 acd_driver 0 0K 0 zombie 0 0K 0 proc-args 9 1K 327 kqueue 2 3K 2 kenv 71 11K 73 filedesc 64 33K 895 filedesc_to_leader 0 0K 0 sigio 1 1K 1 ar_driver 0 0K 10 ata_pci 0 0K 0 cdev 22 6K 22 sbp 0 0K 0 if_fwip 0 0K 0 ata_dma 2 1K 2 if_fwe 0 0K 0 ad_driver 2 1K 2 ata_generic 2 2K 2 acpitask 0 0K 1 isofs_mount 0 0K 0 isofs_node 0 0K 0 isadev 5 1K 5 acpica 1865 180K 58648 GEOM 144 25K 1163 fwmem 0 0K 0 pfs_vncache 0 0K 0 pfs_nodes 20 5K 20 ntfs_mount 0 0K 0 ntfs_ntnode 0 0K 0 ntfs_fnode 0 0K 0 ntfs_dir 0 0K 0 ntfs_vattr 0 0K 0 ntfsd_resdata 0 0K 0 ntfs_vrun 0 0K 0 ntfs_decomp 0 0K 0 ntfs_nthash 1 512K 1 db> cont [...] lock order reversal: 1st 0xffffffff8096eb88 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 2nd 0xffffffff8096f8a8 udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:385 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x539 _mtx_lock_flags() at _mtx_lock_flags+0x1f udp_input() at udp_input+0x1f7 ip_input() at ip_input+0xa7 dummynet_send() at dummynet_send+0xde dummynet_io() at dummynet_io+0x587 ipfw_check_in() at ipfw_check_in+0x241 pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 --- KDB: enter: witness_checkorder [thread pid 24 tid 100023 ] Stopped at kdb_enter+0x31: popq %rbp db> cont --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="second-lor.txt" lock order reversal: 1st 0xffffffff8096eb88 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 2nd 0xffffffff8096f8a8 udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:385 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x539 _mtx_lock_flags() at _mtx_lock_flags+0x1f udp_input() at udp_input+0x1f7 ip_input() at ip_input+0xa7 dummynet_send() at dummynet_send+0xde dummynet_io() at dummynet_io+0x587 ipfw_check_in() at ipfw_check_in+0x241 pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 --- KDB: enter: witness_checkorder [thread pid 24 tid 100023 ] Stopped at kdb_enter+0x31: popq %rbp db> show pcpu cpuid = 1 curthread = 0xffffff00011bf000: pid 24 "em1 taskq" curpcb = 0xffffffffa057ad40 fpcurthread = none idlethread = 0xffffff000107f680: pid 11 "idle: cpu1" spin locks held: db> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0xffffff0001eb3000: pid 44042 "sh" curpcb = 0xffffffffa2ad9d40 fpcurthread = none idlethread = 0xffffff000107f340: pid 12 "idle: cpu0" spin locks held: cpuid = 1 curthread = 0xffffff00011bf000: pid 24 "em1 taskq" curpcb = 0xffffffffa057ad40 fpcurthread = none idlethread = 0xffffff000107f680: pid 11 "idle: cpu1" spin locks held: db> trace Tracing pid 24 tid 100023 td 0xffffff00011bf000 kdb_enter() at kdb_enter+0x31 witness_checkorder() at witness_checkorder+0x48e _mtx_lock_flags() at _mtx_lock_flags+0x1f udp_input() at udp_input+0x1f7 ip_input() at ip_input+0xa7 dummynet_send() at dummynet_send+0xde dummynet_io() at dummynet_io+0x587 ipfw_check_in() at ipfw_check_in+0x241 pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 --- db> alltrace Tracing command sh pid 44042 tid 100110 td 0xffffff0001eb3000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x32 trap() at trap+0x2fd nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0x80052135a, rsp = 0x7fffffffcb10, rbp = 0x51df48 --- Tracing command sh pid 44022 tid 100124 td 0xffffff001106b9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa pipe_read() at pipe_read+0x264 dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800993acc, rsp = 0x7fffffffcd98, r bp = 0x800b3ee0e --- Tracing command sbcl pid 42648 tid 100119 td 0xffffff001106f000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec bwait() at bwait+0x55 bufwait() at bufwait+0x56 bread() at bread+0x1e ffs_read() at ffs_read+0x2f7 vn_read() at vn_read+0x1d3 dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800951acc, rsp = 0x7fffffffecd8, r bp = 0x7fffffffed40 --- Tracing command python pid 42488 tid 100090 td 0xffffff0001820000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec acquire() at acquire+0x5d _lockmgr() at _lockmgr+0x24f ffs_lock() at ffs_lock+0x71 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 _vn_lock() at _vn_lock+0x83 vget() at vget+0xbd cache_lookup() at cache_lookup+0x1d0 vfs_cache_lookup() at vfs_cache_lookup+0xb0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x41 lookup() at lookup+0x43d namei() at namei+0x304 kern_stat() at kern_stat+0x56 stat() at stat+0x22 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (188, FreeBSD ELF64, stat), rip = 0x800b1014c, rsp = 0x7fffffffb138, rbp = 0x800e15cb0 --- Tracing command python pid 42307 tid 100117 td 0xffffff001106f680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec bwait() at bwait+0x55 bufwait() at bufwait+0x56 bread() at bread+0x1e ffs_blkatoff() at ffs_blkatoff+0x56 ufs_lookup() at ufs_lookup+0x553 vfs_cache_lookup() at vfs_cache_lookup+0xe5 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x41 lookup() at lookup+0x43d namei() at namei+0x304 kern_stat() at kern_stat+0x56 stat() at stat+0x22 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (188, FreeBSD ELF64, stat), rip = 0x800b1014c, rsp = 0x7fffffffbff8, rbp = 0 --- Tracing command detachtty pid 42114 tid 100114 td 0xffffff0011128340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d poll() at poll+0x583 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8007c79dc, rsp = 0x7fffffffeb38, rbp = 0x7fffffffed10 --- Tracing command cron pid 41177 tid 100086 td 0xffffff0001821000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba kern_nanosleep() at kern_nanosleep+0x10a nanosleep() at nanosleep+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80090730c, rsp = 0x7fffffff ec28, rbp = 0x5 --- Tracing command httpd pid 41081 tid 100058 td 0xffffff0001408340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe7b8, rbp = 0x7fffffffea48 --- Tracing command httpd pid 41067 tid 100055 td 0xffffff000129b680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe808 --- Tracing command httpd pid 40958 tid 100104 td 0xffffff0001452000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe808 --- Tracing command httpd pid 40871 tid 100103 td 0xffffff0001452340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9f8 , rbp = 0x7fffffffea68 --- Tracing command httpd pid 40776 tid 100065 td 0xffffff00013fd340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9f8 , rbp = 0x7fffffffea68 --- Tracing command httpd pid 40720 tid 100063 td 0xffffff00013fd9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe808 --- Tracing command httpd pid 40471 tid 100095 td 0xffffff00017c69c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe808 --- Tracing command httpd pid 40372 tid 100054 td 0xffffff000129b9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe7b8, rbp = 0x7fffffffea48 --- Tracing command httpd pid 40301 tid 100052 td 0xffffff00014089c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9f8 , rbp = 0x7fffffffea68 --- Tracing command sshd pid 40142 tid 100064 td 0xffffff00013fd680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x8016d7a4c, rsp = 0x7fffffffe618 , rbp = 0x3 --- Tracing command httpd pid 39930 tid 100061 td 0xffffff00014539c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe798, rbp = 0x7fffffffea28 --- Tracing command httpd pid 39568 tid 100066 td 0xffffff00013fd000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b kern_select() at kern_select+0x833 select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800ffea4c, rsp = 0x7fffffffeab8 , rbp = 0 --- Tracing command httpd pid 39491 tid 100068 td 0xffffff0001404680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8 , rbp = 0x7fffffffea48 --- Tracing command httpd pid 39418 tid 100073 td 0xffffff000146b000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8 , rbp = 0x7fffffffea48 --- Tracing command httpd pid 39362 tid 100059 td 0xffffff0001454000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8 , rbp = 0x7fffffffea48 --- Tracing command httpd pid 39263 tid 100093 td 0xffffff00017c7340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8 , rbp = 0x7fffffffea48 --- Tracing command httpd pid 39182 tid 100051 td 0xffffff0001353340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_accept() at kern_accept+0x1cf accept() at accept+0x61 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8 , rbp = 0x7fffffffea48 --- Tracing command httpd pid 38789 tid 100062 td 0xffffff0001453680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b kern_select() at kern_select+0x833 select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800ffea4c, rsp = 0x7fffffffeab8 , rbp = 0 --- Tracing command bgpd pid 38165 tid 100076 td 0xffffff000147b340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e39dc, rsp = 0x7fffffffec18, rbp = 0x47e4a337 --- Tracing command bgpd pid 38092 tid 100067 td 0xffffff00014049c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d poll() at poll+0x583 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e39dc, rsp = 0x7fffffffec38, rbp = 0 --- Tracing command bgpd pid 38018 tid 100069 td 0xffffff0001404340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e39dc, rsp = 0x7fffffffeca8, rbp = 0x9515 --- Tracing command postgres pid 37711 tid 100053 td 0xffffff0001353000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b poll() at poll+0x1d2 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8014469dc, rsp = 0x7fffffffcff8, rbp = 0x40 --- Tracing command postgres pid 37633 tid 100078 td 0xffffff000147a9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b kern_select() at kern_select+0x833 select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd3e8 , rbp = 0 --- Tracing command postgres pid 37595 tid 100088 td 0xffffff0001820680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b kern_select() at kern_select+0x833 select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd348 , rbp = 0 --- Tracing command postgres pid 37564 tid 100075 td 0xffffff000147b680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b kern_select() at kern_select+0x833 select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd318 , rbp = 0 --- Tracing command postgres pid 37543 tid 100070 td 0xffffff0001404000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _cv_timedwait_sig() at _cv_timedwait_sig+0x12b kern_select() at kern_select+0x833 select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd9a8 , rbp = 0x6 --- Tracing command unlinkd pid 37266 tid 100083 td 0xffffff0001454680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa pipe_read() at pipe_read+0x264 dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800704acc, rsp = 0x7fffffffe808, r bp = 0x7fffffffecd8 --- Tracing command squid pid 36740 tid 100060 td 0xffffff0001408000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ast() at ast+0x159 Xfast_syscall() at Xfast_syscall+0xe0 --- syscall (72, FreeBSD ELF64, ovadvise), rip = 0x8009d7aac, rsp = 0x7fffffffe9 28, rbp = 0xbd0 --- Tracing command squid pid 36671 tid 100094 td 0xffffff00017c7000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_wait() at kern_wait+0x3c0 wait4() at wait4+0x33 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800967abc, rsp = 0x7fffffffeb18, rbp = 0x7fffffffecc8 --- Tracing command dhcpd pid 36225 tid 100081 td 0xffffff000147a000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800772a4c, rsp = 0x7fffffffe9d8 , rbp = 0 --- Tracing command ntpd pid 34475 tid 100096 td 0xffffff00017c6680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba kern_kevent() at kern_kevent+0x36a kevent() at kevent+0x85 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800bd531c, rsp = 0x7fffffffcc0 8, rbp = 0xf --- Tracing command named pid 29710 tid 100102 td 0xffffff0001452680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800c03a4c, rsp = 0x7fffff5fbcc8 , rbp = 0x26 --- Tracing command named pid 29710 tid 100101 td 0xffffff00014529c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba do_cv_wait() at do_cv_wait+0x312 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x51 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff7fc dc8, rbp = 0x7fffff7fce30 --- Tracing command named pid 29710 tid 100100 td 0xffffff0001453000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa do_cv_wait() at do_cv_wait+0x4c2 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x51 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff9fd eb8, rbp = 0 --- Tracing command named pid 29710 tid 100099 td 0xffffff0001453340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa do_cv_wait() at do_cv_wait+0x4c2 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x51 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffffbfe eb8, rbp = 0 --- Tracing command named pid 29710 tid 100087 td 0xffffff00018209c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_sigtimedwait() at kern_sigtimedwait+0x4ea sigwait() at sigwait+0x63 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800b7dc6c, rsp = 0x7fffffffec 48, rbp = 0x800e01120 --- Tracing command syslogd pid 26625 tid 100074 td 0xffffff000147b9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x80081aa4c, rsp = 0x7fffffffdda8 , rbp = 0x800a1c0f0 --- Tracing command accounting pid 24352 tid 100085 td 0xffffff0001821340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 acct_thread() at acct_thread+0x1c3 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa2a5cd30, rbp = 0 --- Tracing command devd pid 22944 tid 100082 td 0xffffff00014549c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _cv_wait_sig() at _cv_wait_sig+0x11d kern_select() at kern_select+0x85f select() at select+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x43796c, rsp = 0x7fffffffe928, r bp = 0x7fffffffed50 --- Tracing command pflogd pid 14667 tid 100080 td 0xffffff000147a340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12 _sleep() at _sleep+0x2ba bpfread() at bpfread+0xde devfs_read_f() at devfs_read_f+0x67 dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffed08, r bp = 0x1 --- Tracing command pflogd pid 14530 tid 100056 td 0xffffff000129b340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa soreceive_generic() at soreceive_generic+0xd1a dofileread() at dofileread+0x8d kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffe938, r bp = 0x4 --- Tracing command pfpurge pid 14486 tid 100071 td 0xffffff000146b680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 pf_purge_thread() at pf_purge_thread+0x60 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa29efd30, rbp = 0 --- Tracing command sh pid 51 tid 100050 td 0xffffff0001353680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_wait() at kern_wait+0x3c0 wait4() at wait4+0x33 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800923abc, rsp = 0x7fffffffe6d8, rbp = 0x33 --- Tracing command schedcpu pid 50 tid 100049 td 0xffffff00013539c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 schedcpu_thread() at schedcpu_thread+0x209 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa2981d30, rbp = 0 --- Tracing command softdepflush pid 49 tid 100048 td 0xffffff0001355000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 softdep_flush() at softdep_flush+0x2da fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0679d30, rbp = 0 --- Tracing command syncer pid 48 tid 100047 td 0xffffff0001355340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec sched_sync() at sched_sync+0x5da fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0674d30, rbp = 0 --- Tracing command vnlru pid 47 tid 100046 td 0xffffff0001355680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 vnlru_proc() at vnlru_proc+0x692 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa066fd30, rbp = 0 --- Tracing command bufdaemon pid 46 tid 100045 td 0xffffff00013559c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 buf_daemon() at buf_daemon+0x225 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa066ad30, rbp = 0 --- Tracing command pagezero pid 45 tid 100044 td 0xffffff00012479c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 vm_pagezero() at vm_pagezero+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0665d30, rbp = 0 --- Tracing command vmdaemon pid 44 tid 100043 td 0xffffff0001298000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0660d30, rbp = 0 --- Tracing command pagedaemon pid 43 tid 100042 td 0xffffff0001298340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 vm_pageout() at vm_pageout+0xa62 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa065bd30, rbp = 0 --- Tracing command dummynet pid 42 tid 100041 td 0xffffff0001298680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 msleep_spin() at msleep_spin+0x18f taskqueue_thread_loop() at taskqueue_thread_loop+0x45 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0656d30, rbp = 0 --- Tracing command irq1: atkbd0 pid 41 tid 100040 td 0xffffff00012989c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0631d30, rbp = 0 --- Tracing command irq7: ppbus0 ppc0 pid 40 tid 100039 td 0xffffff0001299000 fork_trampoline() at fork_trampoline Tracing command swi0: sio pid 39 tid 100038 td 0xffffff0001299340 fork_trampoline() at fork_trampoline Tracing command fdc0 pid 38 tid 100037 td 0xffffff0001299680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 fdc_thread() at fdc_thread+0x7b5 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0622d30, rbp = 0 --- Tracing command irq15: ata1 pid 37 tid 100036 td 0xffffff00012999c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0613d30, rbp = 0 --- Tracing command irq14: ata0 pid 36 tid 100035 td 0xffffff000129b000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05fbd30, rbp = 0 --- Tracing command usb4 pid 35 tid 100034 td 0xffffff00011c0680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05e3d30, rbp = 0 --- Tracing command usb3 pid 34 tid 100033 td 0xffffff00011c09c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05ccd30, rbp = 0 --- Tracing command irq16: uhci3 pid 33 tid 100032 td 0xffffff0001246000 fork_trampoline() at fork_trampoline Tracing command usb2 pid 32 tid 100031 td 0xffffff0001246340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05c1d30, rbp = 0 --- Tracing command usb1 pid 31 tid 100030 td 0xffffff0001246680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05bbd30, rbp = 0 --- Tracing command irq19: uhci1 pid 30 tid 100029 td 0xffffff00012469c0 fork_trampoline() at fork_trampoline Tracing command usbtask-dr pid 29 tid 100028 td 0xffffff0001247000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec usb_task_thread() at usb_task_thread+0x99 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05b0d30, rbp = 0 --- Tracing command usbtask-hc pid 28 tid 100027 td 0xffffff0001247340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec usb_task_thread() at usb_task_thread+0x99 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05abd30, rbp = 0 --- Tracing command usb0 pid 27 tid 100026 td 0xffffff0001247680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 usb_event_thread() at usb_event_thread+0xb0 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa05a6d30, rbp = 0 --- Tracing command irq23: uhci0 ehci0 pid 26 tid 100025 td 0xffffff00011a6680 fork_trampoline() at fork_trampoline Tracing command irq18: bge0 uhci2 pid 25 tid 100024 td 0xffffff00011a69c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa059bd30, rbp = 0 --- Tracing command em1 taskq pid 24 tid 100023 td 0xffffff00011bf000 kdb_enter() at kdb_enter+0x31 witness_checkorder() at witness_checkorder+0x48e _mtx_lock_flags() at _mtx_lock_flags+0x1f udp_input() at udp_input+0x1f7 ip_input() at ip_input+0xa7 dummynet_send() at dummynet_send+0xde dummynet_io() at dummynet_io+0x587 ipfw_check_in() at ipfw_check_in+0x241 pfil_run_hooks() at pfil_run_hooks+0xac ip_input() at ip_input+0x292 ether_demux() at ether_demux+0x1ac ether_input() at ether_input+0x1bf em_handle_rxtx() at em_handle_rxtx+0x1d2 taskqueue_run() at taskqueue_run+0x95 taskqueue_thread_loop() at taskqueue_thread_loop+0x53 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 --- Tracing command em0 taskq pid 23 tid 100022 td 0xffffff00011bf340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 msleep_spin() at msleep_spin+0x18f taskqueue_thread_loop() at taskqueue_thread_loop+0x45 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 --- Tracing command irq9: acpi0 pid 22 tid 100021 td 0xffffff00011bf680 fork_trampoline() at fork_trampoline Tracing command kqueue taskq pid 21 tid 100020 td 0xffffff00011bf9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa052bd30, rbp = 0 --- Tracing command swi6: task queue pid 20 tid 100019 td 0xffffff00011c0000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0526d30, rbp = 0 --- Tracing command swi6: Giant taskq pid 19 tid 100018 td 0xffffff00011c0340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0521d30, rbp = 0 --- Tracing command thread taskq pid 9 tid 100017 td 0xffffff000108c9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa051cd30, rbp = 0 --- Tracing command swi5: + pid 18 tid 100016 td 0xffffff00011a5000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0517d30, rbp = 0 --- Tracing command swi2: cambio pid 17 tid 100015 td 0xffffff00011a5340 fork_trampoline() at fork_trampoline Tracing command xpt_thrd pid 8 tid 100014 td 0xffffff00011a5680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec xpt_scanner_thread() at xpt_scanner_thread+0x38 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa050dd30, rbp = 0 --- Tracing command acpi_task_2 pid 7 tid 100013 td 0xffffff00011a59c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0508d30, rbp = 0 --- Tracing command acpi_task_1 pid 6 tid 100012 td 0xffffff00011a6000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa0503d30, rbp = 0 --- Tracing command acpi_task_0 pid 5 tid 100011 td 0xffffff00011a6340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _sleep() at _sleep+0x1ec taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04fed30, rbp = 0 --- Tracing command yarrow pid 16 tid 100010 td 0xffffff0001080340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 random_kthread() at random_kthread+0x1a4 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04f5d30, rbp = 0 --- Tracing command g_down pid 4 tid 100009 td 0xffffff0001080680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 g_io_schedule_down() at g_io_schedule_down+0x201 g_down_procbody() at g_down_procbody+0x55 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04f0d30, rbp = 0 --- Tracing command g_up pid 3 tid 100008 td 0xffffff00010809c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 g_io_schedule_up() at g_io_schedule_up+0x119 g_up_procbody() at g_up_procbody+0x55 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04ebd30, rbp = 0 --- Tracing command g_event pid 2 tid 100007 td 0xffffff000108c000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_timedwait() at sleepq_timedwait+0x31 _sleep() at _sleep+0x1d4 g_event_procbody() at g_event_procbody+0xa9 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04e6d30, rbp = 0 --- Tracing command swi3: vm pid 15 tid 100006 td 0xffffff000108c340 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 14 tid 100005 td 0xffffff000108c680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04dcd30, rbp = 0 --- Tracing command swi1: net pid 13 tid 100004 td 0xffffff000107f000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 ithread_loop() at ithread_loop+0x2d1 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04d7d30, rbp = 0 --- Tracing command idle: cpu0 pid 12 tid 100003 td 0xffffff000107f340 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sched_idletd() at sched_idletd+0x3c fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04d1d30, rbp = 0 --- Tracing command idle: cpu1 pid 11 tid 100002 td 0xffffff000107f680 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sched_idletd() at sched_idletd+0x3c fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04ccd30, rbp = 0 --- Tracing command init pid 1 tid 100001 td 0xffffff000107f9c0 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_catch_signals() at sleepq_catch_signals+0x1c5 sleepq_wait_sig() at sleepq_wait_sig+0xc _sleep() at _sleep+0x2aa kern_wait() at kern_wait+0x3c0 wait4() at wait4+0x33 syscall() at syscall+0x1b5 Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40b61c, rsp = 0x7fffffffe848, rbp = 0x33 --- Tracing command audit pid 10 tid 100000 td 0xffffff0001080000 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 sleepq_wait() at sleepq_wait+0x31 _cv_wait() at _cv_wait+0x115 audit_worker() at audit_worker+0x47f fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffa04c2d30, rbp = 0 --- Tracing command swapper pid 0 tid 0 td 0xffffffff808def70 sched_switch() at sched_switch+0x131 mi_switch() at mi_switch+0x186 scheduler() at scheduler+0x39e mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> show alllocks Process 14530 (pflogd) thread 0xffffff000129b340 (100056) exclusive sx so_rcv_sx r = 0 (0xffffff000140f3b8) locked @ /usr/src/sys/kern/uip c_sockbuf.c:148 Process 24 (em1 taskq) thread 0xffffff00011bf000 (100023) shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/sr c/sys/net/pfil.c:73 db> show witness Sleep locks: 0 so_rcv_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148 7 so_rcv -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2475 13 sellck -- last acquired @ /usr/src/sys/kern/sys_generic.c:1127 8 radix node head -- last acquired @ /usr/src/sys/net/route.c:147 11 rtentry -- last acquired @ /usr/src/sys/netinet/ip_output.c:594 12 ifaddr -- last acquired @ /usr/src/sys/net/route.c:287 12 rts_inq -- last acquired @ /usr/src/sys/net/netisr.c:140 12 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:1874 12 vm page queue free mutex -- last acquired @ /usr/src/sys/vm/vm_page.c:1030 12 UMA zone -- (already displayed) 11 UMA boot pages -- last acquired @ /usr/src/sys/vm/uma_core.c:916 12 vm page queue free mutex -- (already displayed) 9 ifnet -- last acquired @ /usr/src/sys/net/if.c:1473 11 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:212 12 eventhandler list -- last acquired @ /usr/src/sys/kern/kern_exit.c:233 10 if_addr_mtx -- last acquired @ /usr/src/sys/net/if.c:828 10 pf task mtx -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/ne t/pf.c:6729 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 UMA boot pages -- (already displayed) 11 arc4_mtx -- last acquired @ /usr/src/sys/libkern/arc4random.c:137 11 bpf interface lock -- last acquired @ /usr/src/sys/net/bpf.c:1359 12 bpf cdev lock -- last acquired @ /usr/src/sys/net/bpf.c:1363 13 sellck -- (already displayed) 11 eventhandler -- (already displayed) 11 rtentry -- (already displayed) 12 eventhandler list -- (already displayed) 8 process lock -- last acquired @ /usr/src/sys/amd64/amd64/trap.c:622 9 session -- last acquired @ /usr/src/sys/kern/kern_fork.c:582 10 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1213 11 uidinfo struct -- last acquired @ order list:0 11 sleep mtxpool -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2139 12 kqueue -- last acquired @ /usr/src/sys/kern/kern_event.c:1021 13 struct mount mtx -- last acquired @ /usr/src/sys/kern/vfs_mount.c:440 14 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2186 15 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:69 15 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:3012 15 Syncer mtx -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1915 10 tty -- last acquired @ /usr/src/sys/kern/kern_event.c:1664 9 sigacts -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:392 9 ktrace -- last acquired @ /usr/src/sys/kern/kern_fork.c:600 11 sleep mtxpool -- (already displayed) 11 sleep mtxpool -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 12 kqueue -- (already displayed) 11 UMA boot pages -- (already displayed) 8 process lock -- (already displayed) 2 unp_mtx -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:557 3 accept -- last acquired @ /usr/src/sys/kern/uipc_socket.c:685 6 so_snd -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2474 7 so_rcv -- (already displayed) 11 sleep mtxpool -- (already displayed) 8 radix node head -- (already displayed) 11 rtentry -- (already displayed) 7 tcp_hc_entry -- last acquired @ /usr/src/sys/netinet/tcp_hostcache.c:291 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 13 sellck -- (already displayed) 12 kqueue -- (already displayed) 12 UMA zone -- (already displayed) 9 vm page queue mutex -- last acquired @ /usr/src/sys/vm/vm_fault.c:887 14 vnode interlock -- (already displayed) 10 pmap -- last acquired @ /usr/src/sys/amd64/amd64/pmap.c:3076 12 vm page queue free mutex -- (already displayed) 12 vm page queue free mutex -- (already displayed) 7 so_rcv -- (already displayed) 7 so_rcv -- (already displayed) 6 so_snd -- (already displayed) 3 filedesc structure -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1025 14 vnode interlock -- (already displayed) 8 process lock -- (already displayed) 11 sleep mtxpool -- (already displayed) 5 Name Cache -- last acquired @ /usr/src/sys/kern/vfs_cache.c:327 14 vnode interlock -- (already displayed) 12 UMA zone -- (already displayed) 11 UMA boot pages -- (already displayed) 4 fdesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1472 15 cdev -- (already displayed) 4 Giant -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241 5 pipe mutex -- last acquired @ /usr/src/sys/kern/sys_pipe.c:572 6 sigio lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:782 7 process group -- last acquired @ /usr/src/sys/kern/kern_fork.c:572 8 process lock -- (already displayed) 9 session -- (already displayed) 13 sellck -- (already displayed) 12 UMA zone -- (already displayed) 7 system map -- last acquired @ /usr/src/sys/vm/vm_map.c:3111 9 vm page queue mutex -- (already displayed) 8 kmem object -- last acquired @ /usr/src/sys/vm/vm_kern.c:410 12 vm page queue free mutex -- (already displayed) 9 vm page queue mutex -- (already displayed) 8 kernel object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3641 12 vm page queue free mutex -- (already displayed) 9 vm page queue mutex -- (already displayed) 12 vm page queue free mutex -- (already displayed) 10 pmap -- (already displayed) 8 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:414 12 UMA zone -- (already displayed) 8 standard object -- last acquired @ /usr/src/sys/vm/vm_object.c:441 12 vm page queue free mutex -- (already displayed) 14 vnode interlock -- (already displayed) 9 vm page queue mutex -- (already displayed) 9 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:225 12 UMA zone -- (already displayed) 9 swap_pager swhash -- last acquired @ /usr/src/sys/vm/swap_pager.c:1888 5 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:1492 12 UMA zone -- (already displayed) 8 KMAP ENTRY -- (already displayed) 11 UMA boot pages -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 5 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:307 6 kernel environment -- last acquired @ /usr/src/sys/kern/kern_environment.c: 301 8 kernel object -- (already displayed) 9 vm object_list -- (already displayed) 8 KMAP ENTRY -- (already displayed) 10 uidinfo hash -- (already displayed) 8 process lock -- (already displayed) 11 sleep mtxpool -- (already displayed) 5 evclass_mtx -- last acquired @ /usr/src/sys/security/audit/audit_bsm_klib.c :112 5 TID lock -- last acquired @ /usr/src/sys/kern/subr_unit.c:623 8 standard object -- (already displayed) 5 intr event -- last acquired @ /usr/src/sys/kern/kern_intr.c:454 15 cdev -- (already displayed) 5 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:201 5 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:2901 10 tty -- (already displayed) 5 intr config -- last acquired @ /usr/src/sys/kern/subr_autoconf.c:73 5 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:125 5 XPT lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2668 6 XPT topology lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2673 6 kernel environment -- (already displayed) 7 taskqueue -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:73 5 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:152 5 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:539 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 5 devd -- last acquired @ /usr/src/sys/kern/subr_bus.c:489 13 sellck -- (already displayed) 13 sellck -- (already displayed) 5 ACPI semaphore -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSynch.c:30 3 5 acpi subsystem HW lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSy nch.c:377 5 acpi subsystem GPE lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdS ynch.c:377 7 taskqueue -- (already displayed) 5 msi -- last acquired @ /usr/src/sys/amd64/amd64/msi.c:389 9 ifnet -- (already displayed) 5 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:396 11 bpf interface lock -- (already displayed) 5 bounce pages lock -- last acquired @ /usr/src/sys/amd64/amd64/busdma_machde p.c:1073 10 pmap -- (already displayed) 5 unit# allocation -- last acquired @ /usr/src/sys/kern/subr_unit.c:623 15 vnode_free_list -- (already displayed) 5 pfs_node -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_internal.h:10 3 5 pfs_fileno -- last acquired @ /usr/src/sys/kern/subr_unit.c:623 6 random reseed -- last acquired @ /usr/src/sys/dev/random/yarrow.c:191 11 arc4_mtx -- (already displayed) 5 nfsd_mtx -- last acquired @ /usr/src/sys/nfsserver/nfs_srvsock.c:796 6 so_snd -- (already displayed) 5 if_clone lock -- last acquired @ /usr/src/sys/net/if_clone.c:164 5 if_cloners lock -- last acquired @ /usr/src/sys/net/if_clone.c:252 5 domain list -- last acquired @ /usr/src/sys/kern/uipc_domain.c:228 6 pfil_head_list lock -- last acquired @ /usr/src/sys/net/pfil.c:159 5 PFil hook read/write mutex -- last acquired @ /usr/src/sys/net/pfil.c:73 6 pfil_head_list lock -- (already displayed) 10 pf task mtx -- (already displayed) 6 IPFW static rules -- last acquired @ /usr/src/sys/netinet/ip_fw2.c:2647 12 UMA zone -- (already displayed) 8 radix node head -- (already displayed) 7 if send queue -- last acquired @ /usr/src/sys/dev/em/if_em.c:949 6 network driver -- last acquired @ /usr/src/sys/dev/bge/if_bge.c:3014 10 if_addr_mtx -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 7 taskqueue -- (already displayed) 7 if send queue -- (already displayed) 6 ip_inq -- last acquired @ /usr/src/sys/net/netisr.c:140 11 rtentry -- (already displayed) 6 EM TX Lock -- last acquired @ /usr/src/sys/dev/em/if_em.c:980 7 if send queue -- (already displayed) 11 bpf interface lock -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 6 dummynet -- last acquired @ /usr/src/sys/netinet/ip_dummynet.c:739 12 UMA zone -- (already displayed) 7 system map -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 eventhandler -- (already displayed) 5 isn_mtx -- last acquired @ /usr/src/sys/netinet/tcp_subr.c:1426 6 random reseed -- (already displayed) 11 arc4_mtx -- (already displayed) 8 radix node head -- (already displayed) 6 IPFW static rules -- (already displayed) 5 lo_mtx -- last acquired @ /usr/src/sys/net/if_loop.c:161 5 ATA queue lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:177 6 ATA state lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:194 5 devstat -- last acquired @ /usr/src/sys/kern/subr_devstat.c:83 6 XPT topology lock -- (already displayed) 5 NFS iod lock -- last acquired @ /usr/src/sys/nfsclient/nfs_nfsiod.c:196 6 mountlist -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:760 13 struct mount mtx -- (already displayed) 13 struct mount mtx -- (already displayed) 5 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:460 6 mountlist -- (already displayed) 14 vnode interlock -- (already displayed) 5 buf queue lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1468 14 vnode interlock -- (already displayed) 5 bio queue -- last acquired @ /usr/src/sys/geom/geom_io.c:68 5 bdone lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3800 5 needsbuffer lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:321 5 FFS Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1149 11 arc4_mtx -- (already displayed) 5 Name Cache -- (already displayed) 5 knlist lock for lockless objects -- last acquired @ /usr/src/sys/kern/kern_ event.c:1664 5 vfs hash -- last acquired @ /usr/src/sys/kern/vfs_hash.c:71 14 vnode interlock -- (already displayed) 5 devfs interlock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:194 14 vnode interlock -- (already displayed) 5 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349 6 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:368 6 dirhash -- (already displayed) 5 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:413 9 vm page queue mutex -- (already displayed) 7 process group -- (already displayed) 10 tty -- (already displayed) 9 session -- (already displayed) 5 Softdep Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:4892 7 system map -- (already displayed) 12 UMA zone -- (already displayed) 14 vnode interlock -- (already displayed) 6 buffer daemon lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:429 10 pf task mtx -- (already displayed) 6 buffer daemon lock -- (already displayed) 6 sigio lock -- (already displayed) 6 so_snd -- (already displayed) 5 pipe mutex -- (already displayed) 12 bpf cdev lock -- (already displayed) 12 kqueue -- (already displayed) 7 system map -- (already displayed) 12 UMA zone -- (already displayed) 7 so_rcv -- (already displayed) 4 protect sysfilt_ops -- last acquired @ /usr/src/sys/kern/kern_event.c:774 12 UMA zone -- (already displayed) 3 user map -- last acquired @ /usr/src/sys/vm/vm_map.c:3111 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 9 vm object_list -- (already displayed) 8 standard object -- (already displayed) 9 vm page queue mutex -- (already displayed) 10 pmap -- (already displayed) 14 vnode interlock -- (already displayed) 5 buf queue lock -- (already displayed) 5 needsbuffer lock -- (already displayed) 7 system map -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 13 struct mount mtx -- (already displayed) 6 buffer daemon lock -- (already displayed) 5 Softdep Lock -- (already displayed) 9 swap_pager swhash -- (already displayed) 2 tcpinp -- last acquired @ /usr/src/sys/netinet/tcp_input.c:479 6 so_snd -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 7 so_rcv -- (already displayed) 9 ifnet -- (already displayed) 3 tcp_sc_head -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 arc4_mtx -- (already displayed) 8 radix node head -- (already displayed) 11 rtentry -- (already displayed) 7 tcp_hc_entry -- (already displayed) 3 accept -- (already displayed) 3 so_glabel -- last acquired @ /usr/src/sys/kern/uipc_socket.c:299 8 radix node head -- (already displayed) 11 rtentry -- (already displayed) 7 tcp_hc_entry -- (already displayed) 11 arc4_mtx -- (already displayed) 5 isn_mtx -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 6 IPFW static rules -- (already displayed) 7 if send queue -- (already displayed) 6 EM TX Lock -- (already displayed) 6 ip_inq -- (already displayed) 0 so_snd_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148 6 so_snd -- (already displayed) 1 unp_global_rwlock -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:556 2 unp_mtx -- (already displayed) 3 accept -- (already displayed) 3 so_glabel -- (already displayed) 6 so_snd -- (already displayed) 3 filedesc structure -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 8 radix node head -- (already displayed) 11 rtentry -- (already displayed) 12 rts_inq -- (already displayed) 2 tcpinp -- (already displayed) 8 standard object -- (already displayed) 14 vnode interlock -- (already displayed) 5 buf queue lock -- (already displayed) 7 system map -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 5 needsbuffer lock -- (already displayed) 0 pf_statetbl_lock -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/n et/pf.c:979 10 pf task mtx -- (already displayed) 6 pfil_head_list lock -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 0 devfsmount -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:691 5 devfs interlock -- (already displayed) 15 vnode_free_list -- (already displayed) 14 vnode interlock -- (already displayed) 13 struct mount mtx -- (already displayed) 15 cdev -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 UMA boot pages -- (already displayed) 1 DEVFS ruleset lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_rule.c:177 8 process lock -- (already displayed) 3 user map -- (already displayed) 0 g_disk_done -- last acquired @ /usr/src/sys/geom/geom_disk.c:198 5 bio queue -- (already displayed) 12 UMA zone -- (already displayed) 0 ipqlock -- last acquired @ /usr/src/sys/netinet/ip_input.c:1086 0 semid -- last acquired @ /usr/src/sys/kern/sysv_sem.c:658 1 sem -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1288 0 fdc lock -- last acquired @ /usr/src/sys/dev/fdc/fdc.c:803 0 if_afdata -- last acquired @ /usr/src/sys/net/if.c:578 0 GEOM topology -- last acquired @ /usr/src/sys/geom/geom_event.c:233 5 GEOM orphanage -- (already displayed) 5 devstat -- (already displayed) 5 unit# allocation -- (already displayed) 15 cdev -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 11 UMA boot pages -- (already displayed) 9 vm object_list -- (already displayed) 14 vnode interlock -- (already displayed) 8 standard object -- (already displayed) 7 system map -- (already displayed) 0 intr sources -- last acquired @ /usr/src/sys/amd64/amd64/intr_machdep.c:593 5 msi -- (already displayed) 0 audit_mtx -- last acquired @ /usr/src/sys/security/audit/audit_worker.c:448 0 uma object -- last acquired @ /usr/src/sys/vm/vm_meter.c:115 0 p_peers -- last acquired @ /usr/src/sys/kern/kern_exit.c:284 0 runningbufspace lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:340 0 umtxql -- last acquired @ /usr/src/sys/kern/kern_umtx.c:326 0 accept_filter_mtx -- last acquired @ /usr/src/sys/kern/uipc_accf.c:116 0 ACPI PCI bus methods -- last acquired @ /usr/src/sys/dev/acpica/acpi_pcib.c:22 1 0 rtsock route_cb lock -- last acquired @ /usr/src/sys/net/rtsock.c:191 0 rawcb -- last acquired @ /usr/src/sys/net/raw_usrreq.c:81 7 so_rcv -- (already displayed) 12 UMA zone -- (already displayed) 0 ACPI PCI link -- last acquired @ /usr/src/sys/dev/acpica/acpi_pci_link.c:432 5 ACPI semaphore -- (already displayed) 0 ACPI root bus -- last acquired @ /usr/src/sys/dev/acpica/acpi.c:1022 5 rman -- (already displayed) 5 ACPI semaphore -- (already displayed) 7 system map -- (already displayed) 0 acct_sx -- last acquired @ /usr/src/sys/kern/kern_acct.c:351 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 eventhandler -- (already displayed) 5 TID lock -- (already displayed) 9 vm object_list -- (already displayed) 7 system map -- (already displayed) 8 standard object -- (already displayed) 1 proctree -- last acquired @ /usr/src/sys/kern/kern_fork.c:571 2 allproc -- last acquired @ /usr/src/sys/kern/kern_fork.c:297 3 allprison -- last acquired @ /usr/src/sys/kern/kern_jail.c:945 11 sleep mtxpool -- (already displayed) 8 process lock -- (already displayed) 4 fdesc -- (already displayed) 3 filedesc structure -- (already displayed) 14 vnode interlock -- (already displayed) 3 user map -- (already displayed) 11 arc4_mtx -- (already displayed) 7 process group -- (already displayed) 4 Giant -- (already displayed) 8 process lock -- (already displayed) 9 session -- (already displayed) 6 sigio lock -- (already displayed) 2 clone events drain lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops .c:626 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 15 cdev -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 3 filedesc structure -- (already displayed) 8 process lock -- (already displayed) 5 FFS Lock -- (already displayed) 13 struct mount mtx -- (already displayed) 14 vnode interlock -- (already displayed) 5 buf queue lock -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 4 Giant -- (already displayed) 5 needsbuffer lock -- (already displayed) 6 buffer daemon lock -- (already displayed) 5 Softdep Lock -- (already displayed) 0 vm daemon -- last acquired @ /usr/src/sys/vm/vm_pageout.c:1531 0 sysctl lock -- last acquired @ /usr/src/sys/kern/kern_sysctl.c:1396 11 arc4_mtx -- (already displayed) 2 allproc -- (already displayed) 8 process lock -- (already displayed) 3 user map -- (already displayed) 15 cdev -- (already displayed) 1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2179 3 filedesc structure -- (already displayed) 5 GEOM orphanage -- (already displayed) 4 Giant -- (already displayed) 9 ktrace -- (already displayed) 1 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:1096 12 UMA zone -- (already displayed) 2 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c: 407 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 3 filedesc structure -- (already displayed) 14 vnode interlock -- (already displayed) 5 Name Cache -- (already displayed) 5 buf queue lock -- (already displayed) 5 needsbuffer lock -- (already displayed) 5 vfs hash -- (already displayed) 15 vnode_free_list -- (already displayed) 13 struct mount mtx -- (already displayed) 5 dirhash list -- (already displayed) 12 vm page queue free mutex -- (already displayed) 7 system map -- (already displayed) 8 kernel object -- (already displayed) 5 bio queue -- (already displayed) 5 bdone lock -- (already displayed) 6 dirhash -- (already displayed) 15 cdev -- (already displayed) 9 vm object_list -- (already displayed) 8 standard object -- (already displayed) 5 pbuf mutex -- (already displayed) 6 buffer daemon lock -- (already displayed) 5 kobj -- (already displayed) 1 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:741 12 UMA zone -- (already displayed) 7 system map -- (already displayed) 5 devstat -- (already displayed) 5 ttylist -- (already displayed) 9 vm object_list -- (already displayed) 5 UMA lock -- (already displayed) 15 Syncer mtx -- (already displayed) 1 unp_global_rwlock -- (already displayed) 1 tcp -- last acquired @ /usr/src/sys/netinet/tcp_timer.c:128 2 tcpinp -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 8 radix node head -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 11 rtentry -- (already displayed) 6 IPFW static rules -- (already displayed) 7 if send queue -- (already displayed) 6 EM TX Lock -- (already displayed) 3 accept -- (already displayed) 11 sleep mtxpool -- (already displayed) 3 so_glabel -- (already displayed) 6 ip_inq -- (already displayed) 3 tcp_sc_head -- (already displayed) 1 udp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:839 2 udpinp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:843 3 in_multi_mtx -- last acquired @ /usr/src/sys/netinet/in_mcast.c:320 4 igmp_mtx -- last acquired @ /usr/src/sys/netinet/igmp.c:446 10 if_addr_mtx -- (already displayed) 10 if_addr_mtx -- (already displayed) 4 EM Core Lock -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241 6 EM TX Lock -- (already displayed) 10 if_addr_mtx -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 UMA boot pages -- (already displayed) 7 taskqueue -- (already displayed) 6 network driver -- (already displayed) 6 so_snd -- (already displayed) 9 ifnet -- (already displayed) 11 arc4_mtx -- (already displayed) 8 radix node head -- (already displayed) 11 rtentry -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 6 IPFW static rules -- (already displayed) 7 if send queue -- (already displayed) 6 EM TX Lock -- (already displayed) 7 so_rcv -- (already displayed) 3 accept -- (already displayed) 6 ip_inq -- (already displayed) 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 UMA boot pages -- (already displayed) 8 radix node head -- (already displayed) 5 PFil hook read/write mutex -- (already displayed) 11 rtentry -- (already displayed) 6 IPFW static rules -- (already displayed) 7 if send queue -- (already displayed) 6 EM TX Lock -- (already displayed) 1 rip -- last acquired @ /usr/src/sys/netinet/raw_ip.c:638 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 2 rawinp -- last acquired @ /usr/src/sys/netinet/raw_ip.c:639 7 so_rcv -- (already displayed) 5 ACPI semaphore -- (already displayed) 3 allprison -- (already displayed) 1 cpufreq lock -- last acquired @ /usr/src/sys/kern/kern_cpu.c:413 12 UMA zone -- (already displayed) 12 vm page queue free mutex -- (already displayed) 11 UMA boot pages -- (already displayed) 11 eventhandler -- (already displayed) 12 eventhandler list -- (already displayed) 9 ifnet -- (already displayed) 1 ACPI CPU -- last acquired @ /usr/src/sys/dev/acpica/acpi_cpu.c:1167 8 radix node head -- (already displayed) 14 vnode interlock -- (already displayed) 3 filedesc structure -- (already displayed) 0 ng_node -- last acquired @ order list:0 1 ng_worklist -- last acquired @ order list:0 0 802.11 com lock -- last acquired @ order list:0 0 ddp_list_mtx -- last acquired @ order list:0 1 ddp_mtx -- last acquired @ order list:0 0 slip_mtx -- last acquired @ order list:0 1 slip sc_mtx -- last acquired @ order list:0 0 unp -- last acquired @ order list:0 6 so_snd -- (already displayed) Spin locks: Locks which were never acquired: SCSI CD Changer List MD config lock IPFW dynamic rules arp_inq pfs_vncache ppp_softc_list_mtx tunmtx gif_mtx msq nfs4dev state nfs4dev waitq nfs4dev newq ntfs nthash NFS xid lock NFS reqq lock ACPI global lock CAM SIMQ lock hptlock LED sx LED mtx audit_pipe_mtx nfslock audit_trigger_mtx ktrace_sx bpin lock UUID generator mutex lock ACPI PCI power methods ACPI lid ACPI HPET support ACPI embedded controller ACPI thermal zone ACPI Smart Battery ACPI cmbat ACPI power resources ACPI generic battery ACPI AC adapter MSDOSFS fileno pmc shared lock kqueue order firmware table securelevel mutex lock encapmtx fifo mutex phys_pager list dev_pager list swapdev swap_pager list vm map sleep mutex db> show lockedvnods Locked vnodes 0xffffff001106a9b0: tag ufs, type VREG usecount 4, writecount 0, refcount 6 mountedhere 0 flags (VV_TEXT) v_object 0xffffff0011318680 ref 2 pages 35 lock type ufs: SHARED (count 1) ino 2239451, on dev ufs/usr64 0xffffff001106a3e0: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xffffff0001820000 (pid 42488) ino 2262052, on dev ufs/usr64 0xffffff0001eabba0: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xffffff001106f680 (pid 42307) with 1 pending ino 2262734, on dev ufs/usr64 db> show uma Zone Size Used Free Requests pfosfp 40 407 97 407 pfospfen 112 696 30 696 pfiaddrpl 120 0 0 0 pfstatescrub 40 0 0 0 pffrcent 24 0 0 0 pffrcache 80 0 0 0 pffrag 80 0 0 0 pffrent 32 0 0 0 pfrkentry2 216 0 0 0 pfrkentry 216 2322 72 2330 pfrktable 1296 12 27 1880 pfpooladdrpl 88 12 72 12 pfaltqpl 240 0 0 0 pfstatepl 392 460 10 466 pfrulepl 912 69 15 69 pfsrctrpl 152 0 0 0 FFS2 dinode 256 1209 66 1232 FFS1 dinode 128 0 0 0 FFS inode 192 1209 51 1232 Mountpoints 808 6 9 6 SWAPMETA 288 0 0 0 IPFW dynamic rule 120 0 0 0 rtentry 240 73 55 139 ripcb 288 1 38 2 sackhole 32 0 0 0 tcpreass 40 0 0 0 hostcache 136 2 54 2 syncache 120 0 93 86 tcptw 88 19 65 19 tcpcb 728 38 27 115 inpcb 288 57 21 115 udpcb 288 23 29 743 ipq 56 0 0 0 unpcb 248 23 22 36 socket 696 99 26 899 KNOTE 120 26 67 21403 itimer 360 0 0 0 ksiginfo 112 84 972 84 pipe 744 6 34 551 NFSNODE 664 0 0 0 NFSMOUNT 568 0 0 0 DIRHASH 1024 56 12 56 NAMEI 1024 2 34 18355 L VFS Cache 327 0 0 0 S VFS Cache 104 1400 112 2069 VNODEPOLL 128 0 0 0 VNODE 496 1240 16 1264 ata_composite 352 0 0 0 ata_request 312 2 94 4738 g_bio 216 10 332 18896 ACL UMA zone 388 0 0 0 mbuf_ext_refcnt 4 0 336 60 mbuf_jumbo_16k 16384 0 0 0 mbuf_jumbo_9k 9216 0 0 0 mbuf_jumbo_pagesize 4096 0 14 36 mbuf_cluster 2048 1152 8 1152 mbuf 256 1 392 9470 mbuf_packet 256 793 359 12023 audit_record 984 0 0 0 VMSPACE 416 45 18 1009 SLEEPQUEUE 64 145 135 145 UPCALL 88 0 0 0 THREAD 824 129 15 129 PROC 1128 96 36 1060 umtx pi 96 0 0 0 TURNSTILE 152 145 23 145 Files 120 239 102 8378 4096 4096 382 37 4229 2048 2048 65 85 2246 1024 1024 62 150 3212 512 512 625 26 6508 256 256 790 80 6895 128 128 9011 153 16531 64 64 1988 196 51814 32 32 2139 285 3341 16 16 1685 331 33999 mt_zone 1024 217 7 217 DP fakepg 120 0 0 0 MAP ENTRY 112 5149 1748 42364 KMAP ENTRY 112 36 96 1495 MAP 248 7 23 7 VM OBJECT 208 2182 122 17689 128 Bucket 1048 43 2 43 64 Bucket 536 42 0 42 32 Bucket 280 59 11 59 16 Bucket 152 80 20 80 UMA Hash 256 6 9 7 UMA RCntSlabs 128 594 15 594 UMA Slabs 128 674 22 1224 UMA Zones 216 89 13 89 UMA Kegs 216 89 13 89 db> show malloc Type InUse MemUse Requests msdosfs_mount 0 0K 0 msdosfs_fat 0 0K 0 msdosfs_fileno 0 0K 0 msdosfs_node 0 0K 0 DEVFS 27 1K 28 DEVFS_RULE 36 17K 36 DEVFS2 108 2K 108 DEVFS3 244 61K 245 DEVFS1 111 56K 111 USB 91 20K 106 USBdev 21 9K 21 atkbddev 2 1K 2 USBHC 0 0K 0 UART 0 0K 0 nexusdev 4 1K 4 msi 2 1K 2 twe_commands 0 0K 0 twa_commands 0 0K 0 firewire 0 0K 0 legacydrv 0 0K 0 io_apic 1 2K 1 entropy 1024 64K 1024 ppbusdev 3 1K 3 fw_xfer 0 0K 0 memdesc 1 4K 1 madt_table 0 0K 0 scsi_cd 0 0K 0 CAM ccb queue 0 0K 0 vm_pgdata 1 128K 1 acpipwr 0 0K 0 aaccam 0 0K 0 acpi_perf 0 0K 0 UMAHash 1 1K 1 ufs_mount 12 30K 12 ufs_quota 0 0K 0 ufs_dirhash 51 10K 51 pagedep 9 65K 24 inodedep 63 528K 89 newblk 1 1K 5144 bmsafemap 11 2K 18 allocdirect 41 11K 222 indirdep 6 1K 10 allocindir 4634 580K 4921 freefrag 1 1K 373 freeblks 12 3K 17 freefile 12 1K 22 diradd 32 2K 59 mkdir 4 1K 12 dirrem 12 1K 42 newdirblk 0 0K 0 savedino 0 0K 11 audit_trigger 0 0K 0 audit_pipe 0 0K 0 audit_pipeent 0 0K 0 audit_pipe_preselect 0 0K 0 audit_evclass 150 5K 187 audit_bsm 0 0K 0 audit_cred 0 0K 0 audit_data 0 0K 0 audit_path 0 0K 0 audit_text 0 0K 0 rpcclnt 0 0K 0 agp 0 0K 0 nfss_srvsock 0 0K 0 nfss_srvdesc 0 0K 0 nfss_daemon 1 16K 1 nfsclient_lock 0 0K 0 nfsclient_nlminfo 0 0K 0 nfsclient_req 0 0K 0 nfsclient_bigfh 0 0K 0 nfsclient_diroff 0 0K 0 nfsclient_hash 0 0K 0 nfsclient_directio 0 0K 0 nfsclient_srvsock 0 0K 0 idmap 0 0K 0 nfs4_dev 0 0K 0 syncache 1 100K 1 tcplog 0 0K 0 aacbuf 0 0K 0 hostcache 1 36K 1 IpFw/IpAcct 697 78K 697 ipfw_tbl 31 8K 31 encap_export_host 0 0K 0 dummynet 54 108K 54 in_multi 4 1K 4 ip_moptions 0 0K 0 in_msource 0 0K 0 igmp 0 0K 0 routetbl 341 97K 4442 pci_link 16 2K 16 tun 0 0K 0 SCSI SES 0 0K 0 sl 0 0K 0 ppp 0 0K 0 lo 1 1K 1 gif 0 0K 0 fw_com 0 0K 0 faith 0 0K 0 arpcom 3 1K 3 clone 6 24K 6 ifnet 7 13K 7 ifaddr 52 17K 52 ether_multi 17 1K 20 BPF 14 74K 15 subr_export_host 0 0K 0 SCSI sa 0 0K 0 mount 84 4K 138 vnodemarker 0 0K 29 vnodes 1 1K 1 vfs_hash 1 512K 1 export_host 0 0K 0 cl_savebuf 0 0K 72 vfscache 1 1024K 1 biobuf 17 34K 87 soname 8 1K 1326 pcb 24 9K 39 CAM XPT 10 2K 25 mbuf_tag 30 3K 18347 accf 0 0K 0 ptys 1 1K 1 ttys 513 85K 731 shm 6 26K 8 sem 4 8K 4 msg 4 30K 4 ioctlops 0 0K 3983 select 0 0K 0 iov 0 0K 3038 acpicmbat 0 0K 0 acpidev 63 4K 63 ciss_data 0 0K 0 CAM periph 2 1K 9 Unitno 7 1K 9 taskqueue 15 2K 15 stack 0 0K 0 CAM SIM 1 1K 1 scsi_da 0 0K 0 sbuf 0 0K 450 scsi_ch 0 0K 0 rman 161 20K 611 acpisem 12 2K 12 mfibuf 0 0K 0 kobj 227 908K 311 md_disk 0 0K 0 eventhandler 68 6K 68 devstat 16 33K 16 bus 961 75K 4819 bus-sc 66 83K 2400 md_sectors 0 0K 0 SWAP 0 0K 0 p1003.1b 1 1K 1 umtx 144 18K 144 sysctl 0 0K 192 sysctloid 2817 138K 2817 sysctltmp 0 0K 175 LED 0 0K 0 plimit 7 2K 412 uidinfo 9 3K 18 cred 34 9K 2477 pgrp 22 3K 26 session 22 3K 24 proc 2 16K 2 subproc 230 455K 1194 kbdmux 6 9K 6 mtx_pool 1 8K 1 ipsbuf 0 0K 0 module 353 45K 353 iirbuf 0 0K 0 cache 0 0K 0 devbuf 314 1777K 321 temp 34 8K 4594 ip6opt 0 0K 0 ip6ndp 0 0K 0 CAM queue 3 1K 3 free 0 0K 0 lockf 3 1K 10 linker 96 132K 125 CAM dev queue 1 1K 1 KTRACE 100 13K 100 ast_driver 0 0K 0 afd_driver 0 0K 0 prison 0 0K 0 ithread 76 13K 76 acd_driver 0 0K 0 zombie 0 0K 0 proc-args 28 3K 401 kqueue 34 39K 54 kenv 71 11K 73 filedesc 102 64K 1074 filedesc_to_leader 0 0K 0 sigio 1 1K 1 ar_driver 0 0K 10 ata_pci 0 0K 0 cdev 24 6K 24 sbp 0 0K 0 if_fwip 0 0K 0 ata_dma 2 1K 2 if_fwe 0 0K 0 ad_driver 2 1K 2 ata_generic 2 2K 2 acpitask 0 0K 1 isofs_mount 0 0K 0 isofs_node 0 0K 0 isadev 5 1K 5 acpica 1865 180K 58648 GEOM 144 25K 1163 fwmem 0 0K 0 pfs_vncache 0 0K 0 pfs_nodes 20 5K 20 ntfs_mount 0 0K 0 ntfs_ntnode 0 0K 0 ntfs_fnode 0 0K 0 ntfs_dir 0 0K 0 ntfs_vattr 0 0K 0 ntfsd_resdata 0 0K 0 ntfs_vrun 0 0K 0 ntfs_decomp 0 0K 0 ntfs_nthash 1 512K 1 db> cont Starting background file system checks in 60 seconds. --uAKRQypu60I7Lcqm--