Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 10 Mar 2010 15:37:46 +0100
From:      Borja Marcos <borjam@sarenet.es>
To:        Pawel Jakub Dawidek <pjd@FreeBSD.org>
Cc:        freebsd-fs@freebsd.org, FreeBSD Stable <freebsd-stable@freebsd.org>
Subject:   Re: Many processes stuck in zfs
Message-ID:  <548451D6-E62C-4B7F-A846-E9CF08A180C8@sarenet.es>
In-Reply-To: <20100310110202.GA1715@garage.freebsd.pl>
References:  <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <EC9BC6B4-8D0E-4FE3-852F-0E3A24569D33@sarenet.es> <20100309125815.GF3155@garage.freebsd.pl> <CB854F58-03AF-46DD-8153-85FA96037C21@sarenet.es> <BFF1E2D6-B48A-4A5E-ACEE-8577FDB07820@sarenet.es> <20100310110202.GA1715@garage.freebsd.pl>

next in thread | previous in thread | raw e-mail | index | archive | help
=09
On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote:

> Recompile your kernel and modules and reboot.
>=20
> Once the deadlock occur, enter DDB and send me the output of:
>=20
> 	ps
> 	show alllocks
> 	show lockedvnods
> 	show allchains
> 	alltrace

(copying to freebsd-fs, as this is filesystem-related)

There we go. Thanks for the textdump hint :)

Instead of a deadlock I had a crash. Maybe some of the additional =
debugging options triggered it. It is a 8-stable cvsupped on 2nd March, =
and with this patch applied:
http://people.freebsd.org/~pjd/patches/zfs_3way_deadlock.patch



This is ddb.txt:
#=20
#=20
# cat ddb.txt=20
db> set $terminal=3D0

Unknown variable
db> set $lines=3D0

db> ps

  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
 1425  1423  1423     0  S+      zfsvfs-> 0xffffff000256d310 bsdtar
 1424  1423  1423     0  S+      pipdwt   0xffffff00025d6000 bsdtar
 1423  1384  1423     0  S+      wait     0xffffff0005106000 sh
 1422  1420  1420     0  R+                                  bsdtar
 1421  1420  1420     0  S+      pipdwt   0xffffff00025d65b0 bsdtar
 1420  1384  1420     0  S+      wait     0xffffff0005107000 sh
 1419  1417  1417     0  S+      zfs      0xffffff00057ac578 bsdtar
 1418  1417  1417     0  S+      pipdwt   0xffffff00025d4000 bsdtar
 1417  1384  1417     0  S+      wait     0xffffff0002e79000 sh
 1416  1414  1414     0  S+      zio->io_ 0xffffff00026c5b28 bsdtar
 1415  1414  1414     0  S+      pipdwt   0xffffff00025d6b60 bsdtar
 1414  1384  1414     0  S+      wait     0xffffff0002e788c0 sh
 1413  1409  1413     0  S+      select   0xffffff00029d56c0 top
 1409  1408  1409     0  S+      pause    0xffffff0002d83500 csh
 1408  1405  1408  1001  S+      wait     0xffffff0002e7a000 su
 1405  1404  1405  1001  Ss+     pause    0xffffff0002e78500 csh
 1404  1401  1401  1001  S       select   0xffffff0002cce1c0 sshd
 1401  1280  1401     0  Ss      sbwait   0xffffff0002f63694 sshd
 1384  1383  1384     0  S+      ttyin    0xffffff00024b54a8 csh
 1383  1377  1383  1001  S+      wait     0xffffff0002e77460 su
 1377  1376  1377  1001  Ss+     pause    0xffffff0002e79500 csh
 1376  1373  1373  1001  S       select   0xffffff0002916440 sshd
 1373  1280  1373     0  Ss      sbwait   0xffffff0002cff3ec sshd
 1372     1  1372     0  Ss+     ttyin    0xffffff00024daca8 getty
 1371     1  1371     0  Ss+     ttyin    0xffffff00024dc0a8 getty
 1370     1  1370     0  Ss+     ttyin    0xffffff00024cdca8 getty
 1369     1  1369     0  Ss+     ttyin    0xffffff00024c94a8 getty
 1368     1  1368     0  Ss+     ttyin    0xffffff00024dd4a8 getty
 1367     1  1367     0  Ss+     ttyin    0xffffff00024c28a8 getty
 1366     1  1366     0  Ss+     ttyin    0xffffff00024db4a8 getty
 1365     1  1365     0  Ss+     ttyin    0xffffff00024dc8a8 getty
 1301     1  1301     0  Ss      nanslp   0xffffffff80c4d168 cron
 1294     1  1294    25  Ss      pause    0xffffff0002623960 sendmail
 1288     1  1288     0  Ss      select   0xffffff0002917cc0 sendmail
 1280     1  1280     0  Ss      select   0xffffff00029d5340 sshd
 1005     1  1005     0  Rs                                  syslogd
  827     1   827     0  Ss      select   0xffffff0002ccf2c0 devd
   20     0     0     0  SL      flowclea 0xffffffff80e0e950 =
[flowcleaner]
   19     0     0     0  SL      sdflush  0xffffffff80e1d8b8 =
[softdepflush]
   18     0     0     0  RL                                  [syncer]
   17     0     0     0  SL      vlruwt   0xffffff000256a8c0 [vnlru]
   16     0     0     0  SL      psleep   0xffffffff80e0e1c8 [bufdaemon]
   15     0     0     0  SL      pgzero   0xffffffff80e1f34c [pagezero]
   14     0     0     0  SL      psleep   0xffffffff80e1e6e8 [vmdaemon]
    9     0     0     0  RL                                  =
[pagedaemon]
    8     0     0     0  SL      ccb_scan 0xffffffff80c16060 [xpt_thrd]
    7     0     0     0  RL      (threaded)                  zfskern
100085                   Run     CPU 0                       =
[txg_thread_enter]
100084                   RunQ                                =
[txg_thread_enter]
100082                   D       vgeom:io 0xffffff00026acd90 [vdev da2]
100036                   D       l2arc_fe 0xffffffff81150620 =
[l2arc_feed_thread]
100035                   D       arc_recl 0xffffffff81147fe0 =
[arc_reclaim_thread]
    6     0     0     0  SL      -        0xffffff000244e648 [fdc0]
    5     0     0     0  SL      idle     0xffffff800023c000 =
[mpt_recovery0]
   13     0     0     0  SL      -        0xffffffff80c4ce44 [yarrow]
    4     0     0     0  RL                                  [g_down]
    3     0     0     0  RL                                  [g_up]
    2     0     0     0  RL                                  [g_event]
   12     0     0     0  WL      (threaded)                  intr
100033                   I                                   [swi0: uart =
uart]
100032                   I                                   [irq7: =
ppc0]
100031                   I                                   [irq12: =
psm0]
100030                   I                                   [irq1: =
atkbd0]
100024                   I                                   [irq17: em2 =
mpt0]
100023                   I                                   [irq15: =
ata1]
100022                   I                                   [irq14: =
ata0]
100021                   I                                   [irq9: =
acpi0]
100019                   I                                   [swi5: +]
100018                   I                                   [swi2: =
cambio]
100013                   I                                   [swi6: task =
queue]
100012                   I                                   [swi6: =
Giant taskq]
100006                   I                                   [swi1: =
netisr 0]
100005                   I                                   [swi4: =
clock]
100004                   I                                   [swi3: vm]
   11     0     0     0  RL                                  [idle: =
cpu0]
    1     0     1     0  SLs     wait     0xffffff00021e18c0 [init]
   10     0     0     0  SL      audit_wo 0xffffffff80e1cc10 [audit]
    0     0     0     0  SLs     (threaded)                  kernel
100088                   D       -        0xffffff0002914880 [zil_clean]
100087                   D       -        0xffffff0002916780 [zil_clean]
100086                   D       -        0xffffff0002913900 [zil_clean]
100083                   D       -        0xffffff0002628480 =
[zfs_vn_rele_taskq]
100081                   D       -        0xffffff00026acc00 [spa_zio]
100080                   D       -        0xffffff00026acb80 [spa_zio]
100079                   D       -        0xffffff00026acb00 [spa_zio]
100078                   D       -        0xffffff00026aca80 [spa_zio]
100077                   D       -        0xffffff00026aca00 [spa_zio]
100076                   D       -        0xffffff00026ac980 [spa_zio]
100075                   D       -        0xffffff00026ac900 [spa_zio]
100074                   D       -        0xffffff00026ac880 [spa_zio_7]
100073                   D       -        0xffffff00026ac880 [spa_zio_6]
100072                   D       -        0xffffff00026ac880 [spa_zio_5]
100071                   D       -        0xffffff00026ac880 [spa_zio_4]
100070                   D       -        0xffffff00026ac880 [spa_zio_3]
100069                   D       -        0xffffff00026ac880 [spa_zio_2]
100068                   D       -        0xffffff00026ac880 [spa_zio_1]
100067                   D       -        0xffffff00026ac880 [spa_zio_0]
100066                   D       -        0xffffff00026ac800 [spa_zio_7]
100065                   D       -        0xffffff00026ac800 [spa_zio_6]
100064                   D       -        0xffffff00026ac800 [spa_zio_5]
100063                   D       -        0xffffff00026ac800 [spa_zio_4]
100062                   D       -        0xffffff00026ac800 [spa_zio_3]
100061                   D       -        0xffffff00026ac800 [spa_zio_2]
100060                   D       -        0xffffff00026ac800 [spa_zio_1]
100059                   D       -        0xffffff00026ac800 [spa_zio_0]
100058                   D       -        0xffffff00026ac780 [spa_zio]
100057                   D       -        0xffffff00026ac700 [spa_zio]
100056                   D       -        0xffffff00026ac680 [spa_zio]
100029                   D       -        0xffffff0002441980 [em3 taskq]
100028                   D       -        0xffffff0002440d80 [em2 taskq]
100027                   D       -        0xffffff0002436200 [em1 taskq]
100026                   D       -        0xffffff000242fb80 [em0 taskq]
100020                   D       -        0xffffff0002358600 [thread =
taskq]
100017                   D       -        0xffffff0002316b80 =
[acpi_task_2]
100016                   D       -        0xffffff0002316b80 =
[acpi_task_1]
100015                   D       -        0xffffff0002316b80 =
[acpi_task_0]
100014                   D       -        0xffffff0002316c00 [kqueue =
taskq]
100010                   D       -        0xffffff00021e0180 [firmware =
taskq]
100000                   D       sched    0xffffffff80c49740 [swapper]
db> show alllocks

Process 1425 (bsdtar) thread 0xffffff000528b3a0 (100115)
Process 1422 (bsdtar) thread 0xffffff000528c000 (100112)
Process 1419 (bsdtar) thread 0xffffff0002d81000 (100109)
Process 1416 (bsdtar) thread 0xffffff000262a3a0 (100091)
Process 1401 (sshd) thread 0xffffff0002510740 (100055)
Process 1373 (sshd) thread 0xffffff000262a740 (100090)
Process 7 (zfskern) thread 0xffffff000262bae0 (100085)
db> show lockedvnods

Locked vnodes
db> show allchains

db> alltrace


Tracing command bsdtar pid 1425 tid 100115 td 0xffffff000528b3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_xlock_hard() at _sx_xlock_hard+0x267
_sx_xlock() at _sx_xlock+0xc1
zfs_zinactive() at zfs_zinactive+0x95
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5
vinactive() at vinactive+0x90
vputx() at vputx+0x2fc
kern_statat_vnhook() at kern_statat_vnhook+0xfa
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800e35acc, rsp =3D =
0x7fffffffe768, rbp =3D 0x801025380 ---

Tracing command bsdtar pid 1424 tid 100114 td 0xffffff000528b740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_write() at pipe_write+0x13fc
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x60
write() at write+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D =
0x7fffffffe0d8, rbp =3D 0x801004140 ---

Tracing command sh pid 1423 tid 100113 td 0xffffff000528bae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D =
0x7fffffffe8b8, rbp =3D 0x58f ---

Tracing command bsdtar pid 1422 tid 100112 td 0xffffff000528c000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
txg_wait_open() at txg_wait_open+0x95
zfs_freebsd_setattr() at zfs_freebsd_setattr+0x862
VOP_SETATTR_APV() at VOP_SETATTR_APV+0xaf
setutimes() at setutimes+0x214
kern_lutimes() at kern_lutimes+0xf2
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (276, FreeBSD ELF64, lutimes), rip =3D 0x800da66dc, rsp =3D =
0x7fffffffe7e8, rbp =3D 0x801025380 ---

Tracing command bsdtar pid 1421 tid 100111 td 0xffffff000528c3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_write() at pipe_write+0x13fc
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x60
write() at write+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D =
0x7fffffffe0d8, rbp =3D 0x801004140 ---

Tracing command sh pid 1420 tid 100110 td 0xffffff000262dae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D =
0x7fffffffe8b8, rbp =3D 0x58c ---

Tracing command bsdtar pid 1419 tid 100109 td 0xffffff0002d81000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x7a3
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vget() at vget+0x7b
cache_lookup() at cache_lookup+0x4e0
vfs_cache_lookup() at vfs_cache_lookup+0xc0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x3d3
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800e35acc, rsp =3D =
0x7fffffffe768, rbp =3D 0x801025380 ---

Tracing command bsdtar pid 1418 tid 100108 td 0xffffff0002d813a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_write() at pipe_write+0x13fc
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x60
write() at write+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D =
0x7fffffffe0d8, rbp =3D 0x801004140 ---

Tracing command sh pid 1417 tid 100101 td 0xffffff0002cfb000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D =
0x7fffffffe8b8, rbp =3D 0x589 ---

Tracing command bsdtar pid 1416 tid 100091 td 0xffffff000262a3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
zio_wait() at zio_wait+0x7d
dbuf_read() at dbuf_read+0x2bb
dnode_hold_impl() at dnode_hold_impl+0x114
dmu_bonus_hold() at dmu_bonus_hold+0x31
zfs_zget() at zfs_zget+0x259
zfs_dirent_lock() at zfs_dirent_lock+0x4a0
zfs_dirlook() at zfs_dirlook+0x90
zfs_lookup() at zfs_lookup+0x256
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x3d3
namei() at namei+0x4a9
kern_mkdirat() at kern_mkdirat+0xc3
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800e35b0c, rsp =3D =
0x7fffffffe748, rbp =3D 0x1ed ---

Tracing command bsdtar pid 1415 tid 100105 td 0xffffff0002cfa000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_write() at pipe_write+0x13fc
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x60
write() at write+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D =
0x7fffffffe0d8, rbp =3D 0x801004140 ---

Tracing command sh pid 1414 tid 100102 td 0xffffff0002cfaae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D =
0x7fffffffe8b8, rbp =3D 0x586 ---

Tracing command top pid 1413 tid 100054 td 0xffffff0002510ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x18c
seltdwait() at seltdwait+0x56
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x800ab7cfc, rsp =3D =
0x7fffffffea08, rbp =3D 0x11 ---

Tracing command csh pid 1409 tid 100093 td 0xffffff0002645ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0x9a
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D =
0x7fffffffe638, rbp =3D 0x800c5a400 ---

Tracing command su pid 1408 tid 100098 td 0xffffff0002cfbae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e854c, rsp =3D =
0x7fffffffe598, rbp =3D 0x581 ---

Tracing command csh pid 1405 tid 100103 td 0xffffff0002cfa740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0x9a
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D =
0x7fffffffe558, rbp =3D 0x800c0f900 ---

Tracing command sshd pid 1404 tid 100104 td 0xffffff0002cfa3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D =
0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 ---

Tracing command sshd pid 1401 tid 100055 td 0xffffff0002510740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d7d7c, rsp =3D =
0x7fffffffdcd8, rbp =3D 0 ---

Tracing command csh pid 1384 tid 100107 td 0xffffff0002d81740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x8009ebd7c, rsp =3D =
0x7fffffffe738, rbp =3D 0x1 ---

Tracing command su pid 1383 tid 100106 td 0xffffff0002d81ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e854c, rsp =3D =
0x7fffffffe598, rbp =3D 0x568 ---

Tracing command csh pid 1377 tid 100100 td 0xffffff0002cfb3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0x9a
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D =
0x7fffffffe558, rbp =3D 0x800c0f600 ---

Tracing command sshd pid 1376 tid 100099 td 0xffffff0002cfb740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D =
0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 ---

Tracing command sshd pid 1373 tid 100090 td 0xffffff000262a740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d7d7c, rsp =3D =
0x7fffffffdcd8, rbp =3D 0 ---

Tracing command getty pid 1372 tid 100046 td 0xffffff00025b1ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1371 tid 100095 td 0xffffff00026453a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1370 tid 100096 td 0xffffff0002da63a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1369 tid 100097 td 0xffffff0002da6000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1368 tid 100092 td 0xffffff000262a000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1367 tid 100094 td 0xffffff0002645740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1366 tid 100089 td 0xffffff000262aae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command getty pid 1365 tid 100052 td 0xffffff00025b03a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D =
0x7fffffffecc8, rbp =3D 0 ---

Tracing command cron pid 1301 tid 100053 td 0xffffff00025b0000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip =3D 0x80093efac, rsp =3D =
0x7fffffffeb28, rbp =3D 0x3b ---

Tracing command sendmail pid 1294 tid 100048 td 0xffffff00025b13a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0x9a
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (4, FreeBSD ELF64, write), rip =3D 0x800d3a19c, rsp =3D =
0x7fffffffcd68, rbp =3D 0x1 ---

Tracing command sendmail pid 1288 tid 100051 td 0xffffff00025b0740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x18c
seltdwait() at seltdwait+0x56
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x800dd8cfc, rsp =3D =
0x7fffffffc1e8, rbp =3D 0x7fffffffc280 ---

Tracing command sshd pid 1280 tid 100047 td 0xffffff00025b1740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D =
0x7fffffffddd8, rbp =3D 0x2 ---

Tracing command syslogd pid 1005 tid 100050 td 0xffffff00025b0ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x800851cfc, rsp =3D =
0x7fffffffe118, rbp =3D 0x800a310c8 ---

Tracing command devd pid 827 tid 100049 td 0xffffff00025b1000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip =3D 0x440ecc, rsp =3D =
0x7fffffffe898, rbp =3D 0x7fffffffe8b0 ---

Tracing command flowcleaner pid 20 tid 100045 td 0xffffff0002485ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
flowtable_cleaner() at flowtable_cleaner+0xde
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800018cd30, rbp =3D 0 ---

Tracing command softdepflush pid 19 tid 100044 td 0xffffff000250d000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
softdep_flush() at softdep_flush+0x259
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000187d30, rbp =3D 0 ---

Tracing command syncer pid 18 tid 100043 td 0xffffff000250d3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
sched_sync() at sched_sync+0x4db
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000182d30, rbp =3D 0 ---

Tracing command vnlru pid 17 tid 100042 td 0xffffff000250d740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
vnlru_proc() at vnlru_proc+0x5cf
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800017dd30, rbp =3D 0 ---

Tracing command bufdaemon pid 16 tid 100041 td 0xffffff000250dae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
buf_daemon() at buf_daemon+0x14a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000178d30, rbp =3D 0 ---

Tracing command pagezero pid 15 tid 100040 td 0xffffff000250e000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
vm_pagezero() at vm_pagezero+0x73
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000173d30, rbp =3D 0 ---

Tracing command vmdaemon pid 14 tid 100039 td 0xffffff000250e3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800016ed30, rbp =3D 0 ---

Tracing command pagedaemon pid 9 tid 100038 td 0xffffff000250e740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
vm_pageout() at vm_pageout+0x855
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000169d30, rbp =3D 0 ---

Tracing command xpt_thrd pid 8 tid 100037 td 0xffffff000250eae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
xpt_scanner_thread() at xpt_scanner_thread+0xd5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000164d30, rbp =3D 0 ---

Tracing command zfskern pid 7 tid 100085 td 0xffffff000262bae0
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x17b
dblfault_handler() at dblfault_handler+0x96
Xdblfault() at Xdblfault+0xac
--- trap 0x17, rip =3D 0xffffffff805d322d, rsp =3D 0xffffffff80e5ecd0, =
rbp =3D 0xffffff8012415090 ---
witness_checkorder() at witness_checkorder+0x2d
_mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x94
callout_tick() at callout_tick+0x41
hardclock_cpu() at hardclock_cpu+0xc8
hardclock() at hardclock+0x23
lapic_handle_timer() at lapic_handle_timer+0xd2
Xtimerint() at Xtimerint+0x8c
--- interrupt, rip =3D 0xffffffff80866093, rsp =3D 0xffffff8012415270, =
rbp =3D 0xffffff8012415290 ---
spinlock_exit() at spinlock_exit+0x33
putcons() at putcons+0x40
putchar() at putchar+0x6a
kvprintf() at kvprintf+0x81
vprintf() at vprintf+0x85
printf() at printf+0x67
db_putc() at db_putc+0x81
kvprintf() at kvprintf+0x81
db_printf() at db_printf+0x86
db_backtrace() at db_backtrace+0x162
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
dnode_verify() at dnode_verify+0x70
dnode_hold_impl() at dnode_hold_impl+0x73
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_enqueue() at bplist_enqueue+0x4c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119
dbuf_write() at dbuf_write+0x24c
dbuf_sync_list() at dbuf_sync_list+0x3eb
dbuf_sync_list() at dbuf_sync_list+0x17f
dbuf_sync_list() at dbuf_sync_list+0x17f
dbuf_sync_list() at dbuf_sync_list+0x17f
dbuf_sync_list() at dbuf_sync_list+0x17f
dbuf_sync_list() at dbuf_sync_list+0x17f
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xc12
dmu_objset_sync() at dmu_objset_sync+0x134
dsl_pool_sync() at dsl_pool_sync+0x88
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 ---

Tracing command zfskern pid 7 tid 100084 td 0xffffff000262d000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
txg_thread_wait() at txg_thread_wait+0x5b
txg_quiesce_thread() at txg_quiesce_thread+0xb4
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012413d30, rbp =3D 0 ---

Tracing command zfskern pid 7 tid 100082 td 0xffffff0002640000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
vdev_geom_worker() at vdev_geom_worker+0xf3
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff80123ffd30, rbp =3D 0 ---

Tracing command zfskern pid 7 tid 100036 td 0xffffff0002510000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
l2arc_feed_thread() at l2arc_feed_thread+0x162
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800015fd30, rbp =3D 0 ---

Tracing command zfskern pid 7 tid 100035 td 0xffffff0002356740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
arc_reclaim_thread() at arc_reclaim_thread+0x2b7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800015ad30, rbp =3D 0 ---

Tracing command fdc0 pid 6 tid 100034 td 0xffffff0002356ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
fdc_thread() at fdc_thread+0x774
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000155d30, rbp =3D 0 ---

Tracing command mpt_recovery0 pid 5 tid 100025 td 0xffffff0002353ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
mpt_recovery_thread() at mpt_recovery_thread+0x251
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800011ed30, rbp =3D 0 ---

Tracing command yarrow pid 13 tid 100011 td 0xffffff00021e53a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
random_kthread() at random_kthread+0x1ad
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800004cd30, rbp =3D 0 ---

Tracing command g_down pid 4 tid 100009 td 0xffffff00021e5ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
g_io_schedule_down() at g_io_schedule_down+0x236
g_down_procbody() at g_down_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000042d30, rbp =3D 0 ---

Tracing command g_up pid 3 tid 100008 td 0xffffff00021f3000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
g_io_schedule_up() at g_io_schedule_up+0x133
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800003dd30, rbp =3D 0 ---

Tracing command g_event pid 2 tid 100007 td 0xffffff00021f33a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_sleep() at _sleep+0x341
g_event_procbody() at g_event_procbody+0xa1
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000038d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100033 td 0xffffff0002483000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100032 td 0xffffff00024833a0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100031 td 0xffffff0002483740
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100030 td 0xffffff0002483ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000137d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100024 td 0xffffff0002354000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff80000d7d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100023 td 0xffffff00023543a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff80000d2d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100022 td 0xffffff0002354740
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100021 td 0xffffff0002354ae0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100019 td 0xffffff00023563a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000078d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100018 td 0xffffff00021f3ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000073d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100013 td 0xffffff0002353000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800005ad30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100012 td 0xffffff00023533a0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100006 td 0xffffff00021f3740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000033d30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100005 td 0xffffff00021e4000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
ithread_loop() at ithread_loop+0x246
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800002ed30, rbp =3D 0 ---

Tracing command intr pid 12 tid 100004 td 0xffffff00021e43a0
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100003 td 0xffffff00021e4740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
critical_exit() at critical_exit+0xa3
intr_event_handle() at intr_event_handle+0xd7
intr_execute_handlers() at intr_execute_handlers+0x5f
lapic_handle_intr() at lapic_handle_intr+0x37
Xapic_isr1() at Xapic_isr1+0xa4
--- interrupt, rip =3D 0xffffffff808545f6, rsp =3D 0xffffff8000024b50, =
rbp =3D 0xffffff8000024b60 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1f6
sched_idletd() at sched_idletd+0x123
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8000024d30, rbp =3D 0 ---

Tracing command init pid 1 tid 100002 td 0xffffff00021e4ae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x40c86c, rsp =3D =
0x7fffffffe808, rbp =3D 0x401d40 ---

Tracing command audit pid 10 tid 100001 td 0xffffff00021e5000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
audit_worker() at audit_worker+0x77
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100088 td 0xffffff000262b000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012426d30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100087 td 0xffffff000262b3a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012421d30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100086 td 0xffffff000262b740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff801241cd30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100083 td 0xffffff000263eae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012404d30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100081 td 0xffffff00026403a0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff80123fad30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100080 td 0xffffff000263d740
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff80123f5d30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100079 td 0xffffff000263dae0
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff80123f0d30, rbp =3D 0 ---

Tracing command kernel pid 0 tid 100078 td 0xffffff000263e000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleep#=20




And this is msgbuf.txt:
# cat msgbuf.txt=20
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights =
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-STABLE #3: Wed Mar 10 14:41:54 CET 2010
    =
root@pruebazfs1.sarenet.es:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG =
amd64
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz (2116.63-MHz =
K8-class CPU)
  Origin =3D "GenuineIntel"  Id =3D 0x10676  Stepping =3D 6
  =
Features=3D0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,=
MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=3D0x80082201<SSE3,SSSE3,CX16,SSE4.1,<b31>>
  AMD Features=3D0x20100800<SYSCALL,NX,LM>
  AMD Features2=3D0x1<LAHF>
  TSC: P-state invariant
real memory  =3D 536870912 (512 MB)
avail memory =3D 495988736 (473 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port =
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem =
0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on =
pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem =
0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on =
pci0
mpt0: [ITHREAD]
mpt0: MPI Version=3D1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2000-0x203f =
mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on =
pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:0c:29:11:4f:d5
em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2040-0x207f =
mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on =
pci2
em1: Memory Access and/or Bus Master bits were not set!
em1: [FILTER]
em1: Ethernet address: 00:0c:29:11:4f:cb
pci2: <multimedia, audio> at device 2.0 (no driver attached)
em2: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x20c0-0x20ff =
mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on =
pci2
em2: Memory Access and/or Bus Master bits were not set!
em2: [FILTER]
em2: Ethernet address: 00:0c:29:11:4f:df
em3: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2400-0x243f =
mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on =
pci2
em3: Memory Access and/or Bus Master bits were not set!
em3: [FILTER]
em3: Ethernet address: 00:0c:29:11:4f:e9
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on =
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
orm0: <ISA Option ROMs> at iomem =
0xc0000-0xc7fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0x=
cdfff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=3D0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on =
isa0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is =
present;
            to enable, add "vfs.zfs.prefetch_disable=3D0" to =
/boot/loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable =
behavior.
             Consider tuning vm.kmem_size and vm.kmem_size_max
             in /boot/loader.conf.
ZFS filesystem version 3
ZFS storage pool version 14
Timecounter "TSC" frequency 2116630847 Hz quality 800
Timecounters tick every 10.000 msec
Expensive timeout(9) function: 0xffffffff805ed840(0) 0.002412596 s
acd0: DVDR <VMware Virtual IDE CDROM Drive/00000001> at ata1-master =
UDMA33=20
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device=20
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device=20
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C)
da2 at mpt0 bus 0 scbus0 target 2 lun 0
da2: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device=20
da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da2: Command Queueing enabled
da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C)
da3 at mpt0 bus 0 scbus0 target 3 lun 0
da3: <VMware, VMware Virtual S 1.0> Fixed Direct Access SCSI-2 device=20
da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da3: Command Queueing enabled
da3: 12288MB (25165824 512 byte sectors: 255H 63S/T 1566C)
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
/: mount pending error: blocks 4 files 1
<118>Setting hostuuid: 564d8c47-d7e5-fea4-92ef-f17760cfe4d2.
<118>Setting hostid: 0x4d345a2c.
lock order reversal:
 1st 0xffffff000286d2e8 db->db_mtx (db->db_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:549
 2nd 0xffffff000286c0d8 dn->dn_mtx (dn->dn_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dnode.c:1173
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_block_freed() at dnode_block_freed+0x8e
dbuf_read() at dbuf_read+0x155
dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a
dmu_read() at dmu_read+0x80
load_nvlist() at load_nvlist+0x85
spa_load() at spa_load+0x49a
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D =
0x7fffffffd808, rbp =3D 0x801224140 ---
Expensive timeout(9) function: 0xffffffff80892b50(0xffffff000244fe00) =
0.005049803 s
lock order reversal:
 1st 0xffffff0002868e70 db->db_mtx (db->db_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dnode_sync.c:381
 2nd 0xffffff00026e7940 osi->os_lock (osi->os_lock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dnode.c:323
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_destroy() at dnode_destroy+0xa6
dnode_buf_pageout() at dnode_buf_pageout+0xb2
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_clear() at dbuf_clear+0x5e
dnode_evict_dbufs() at dnode_evict_dbufs+0x98
dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c
dmu_objset_evict() at dmu_objset_evict+0xbf
dsl_pool_close() at dsl_pool_close+0x52
spa_unload() at spa_unload+0xb2
spa_load() at spa_load+0x4da
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D =
0x7fffffffd808, rbp =3D 0x801224140 ---
lock order reversal:
 1st 0xffffff0002864910 buf->b_lock (buf->b_lock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/arc.c:2509
 2nd 0xffffff00028681a0 db->db_mtx (db->db_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:421
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_read_done() at dbuf_read_done+0x3b
arc_read_done() at arc_read_done+0x1d2
zio_done() at zio_done+0x308
zio_execute() at zio_execute+0xb1
arc_read_nolock() at arc_read_nolock+0x3d0
arc_read() at arc_read+0xaf
dbuf_read() at dbuf_read+0x62b
dmu_buf_hold() at dmu_buf_hold+0xcc
zap_lockdir() at zap_lockdir+0x6e
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
dsl_prop_get_dd() at dsl_prop_get_dd+0xfc
dsl_dataset_get_ref() at dsl_dataset_get_ref+0x4a7
dsl_dataset_hold() at dsl_dataset_hold+0xf2
dmu_objset_open() at dmu_objset_open+0xcf
zil_check_log_chain() at zil_check_log_chain+0x2a
dmu_objset_find_spa() at dmu_objset_find_spa+0x23f
dmu_objset_find_spa() at dmu_objset_find_spa+0x324
dmu_objset_find_spa() at dmu_objset_find_spa+0x324
dmu_objset_find() at dmu_objset_find+0x2a
spa_check_logs() at spa_check_logs+0x2a
spa_load() at spa_load+0x6ca
spa_load() at spa_load+0x4fe
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D =
0x7fffffffd808, rbp =3D 0x801224140 ---
lock order reversal:
 1st 0xffffff000286d058 db->db_mtx (db->db_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1116
 2nd 0xffffff000259ac38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1120
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_dirty() at dbuf_dirty+0x892
dnode_setdirty() at dnode_setdirty+0x1a9
dbuf_dirty() at dbuf_dirty+0xa53
bplist_vacate() at bplist_vacate+0x4d
spa_sync() at spa_sync+0x297
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 ---
lock order reversal:
 1st 0xffffff000292d538 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock =
(spa->spa_sync_bplist.bpl_lock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/bplist.c:235
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47
zio_free() at zio_free+0x105
arc_free() at arc_free+0x11c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483
dbuf_write() at dbuf_write+0x24c
dbuf_sync_list() at dbuf_sync_list+0x3eb
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xc12
dmu_objset_sync() at dmu_objset_sync+0x134
dsl_pool_sync() at dsl_pool_sync+0x200
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 ---
lock order reversal:
 1st 0xffffff0002948c38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
 2nd 0xffffff00028a7000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1859
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
dbuf_check_blkptr() at dbuf_check_blkptr+0x11e
dbuf_sync_list() at dbuf_sync_list+0x34a
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xc12
dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x60
dmu_objset_sync() at dmu_objset_sync+0x162
dsl_pool_sync() at dsl_pool_sync+0x200
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 ---
<118>Entropy harvesting:
<118> interrupts
<118> ethernet
<118> point_to_point
<118> kickstart
<118>.
<118>Starting file system checks:
<118>/dev/da0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da0s1a: SUMMARY INFORMATION BAD (SALVAGED)
<118>/dev/da0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED)
<118>/dev/da0s1a: 28961 files, 517324 used, 1004493 free (7877 frags, =
124577 blocks, 0.5% fragmentation)
<118>Mounting local file systems:
<118>.
lock order reversal:
 1st 0xffffff000263acc8 zfs (zfs) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/gfs.c:437
 2nd 0xffffff000256e310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_znode.c:866
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x23c
zfs_root() at zfs_root+0x50
zfsctl_create() at zfsctl_create+0x82
zfs_mount() at zfs_mount+0x7ef
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D =
0x7fffffffced8, rbp =3D 0x7fffffffcef8 ---
lock order reversal:
 1st 0xffffff0002996078 zp->z_name_lock (zp->z_name_lock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_dir.c:212
 2nd 0xffffff000256e330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_znode.c:866
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x23c
zfs_dirent_lock() at zfs_dirent_lock+0x4a0
zfs_dirlook() at zfs_dirlook+0x90
zfs_lookup() at zfs_lookup+0x256
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x3d3
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D =
0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 ---
lock order reversal:
 1st 0xffffff000256e210 zfsvfs->z_teardown_inactive_lock =
(zfsvfs->z_teardown_inactive_lock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_vnops.c:3724
 2nd 0xffffff000256e330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_znode.c:1027
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zinactive() at zfs_zinactive+0x95
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5
vinactive() at vinactive+0x90
vputx() at vputx+0x2fc
kern_statat_vnhook() at kern_statat_vnhook+0xfa
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D =
0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 ---
<118>/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
<118>Starting Network: lo0 em0 em1 em2 em3.
<118>lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118>	options=3D3<RXCSUM,TXCSUM>
<118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6=20
<118>	inet6 ::1 prefixlen 128=20
<118>	inet 127.0.0.1 netmask 0xff000000=20
<118>	nd6 options=3D3<PERFORMNUD,ACCEPT_RTADV>
<118>em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 =
mtu 1500
<118>	options=3D9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
<118>	ether 00:0c:29:11:4f:d5
<118>	inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
<118>	media: Ethernet autoselect (1000baseT <full-duplex>)
<118>	status: active
<118>em1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 =
mtu 1500
<118>	options=3D9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
<118>	ether 00:0c:29:11:4f:cb
<118>	inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
<118>	media: Ethernet autoselect (1000baseT <full-duplex>)
<118>	status: active
<118>em2: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 =
mtu 1500
<118>	options=3D9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
<118>	ether 00:0c:29:11:4f:df
<118>	inet 10.0.2.2 netmask 0xffffff00 broadcast 10.0.2.255
<118>	media: Ethernet autoselect (1000baseT <full-duplex>)
<118>	status: active
<118>em3: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 =
mtu 1500
<118>	options=3D9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
<118>	ether 00:0c:29:11:4f:e9
<118>	inet 172.16.2.118 netmask 0xffff0000 broadcast 172.16.255.255
<118>	media: Ethernet autoselect (1000baseT <full-duplex>)
<118>	status: active
<118>add net default: gateway 172.16.0.2
Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000272000) =
0.006700013 s
<118>Starting devd.
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based =
forwarding disabled, default to deny, logging disabled
<118>Flushed all rules.
<118>00100 allow ip from any to any via lo0
<118>00200 deny ip from any to 127.0.0.0/8
<118>00300 deny ip from 127.0.0.0/8 to any
<118>65000 allow ip from any to any
<118>Firewall rules loaded.
<118>Creating and/or trimming log files
<118>.
<118>Starting syslogd.
<118>No core dumps found.
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
<118>32-bit compatibility ldconfig path: /usr/lib32
lock order reversal:
 1st 0xffffff800a2f62e8 bufwait (bufwait) @ =
/pool/newsrc/src/sys/kern/vfs_bio.c:2559
 2nd 0xffffff0002c71400 dirhash (dirhash) @ =
/pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
ufsdirhash_acquire() at ufsdirhash_acquire+0x44
ufsdirhash_add() at ufsdirhash_add+0x19
ufs_direnter() at ufs_direnter+0x88b
ufs_mkdir() at ufs_mkdir+0x623
VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9
kern_mkdirat() at kern_mkdirat+0x264
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x80072bb0c, rsp =3D =
0x7fffffffec88, rbp =3D 0x7fffffffef66 ---
<118>Clearing /tmp (X related).
<118>Updating motd:
<118>.
<118>Configuring syscons:
<118> blanktime
<118>.
<118>Starting sshd.
<118>Starting cron.
<118>Starting background file system checks in 60 seconds.
<118>
<118>Wed Mar 10 17:29:09 CET 2010
<118>Mar 10 17:29:21  su: BAD SU borjam to root on /dev/pts/0
<118>Mar 10 17:29:24  su: BAD SU borjam to root on /dev/pts/0
<118>Mar 10 17:29:26  su: borjam to root on /dev/pts/0
<118>Mar 10 17:30:36  su: borjam to root on /dev/pts/1
lock order reversal:
 1st 0xffffff00027e2438 ds->ds_deadlist.bpl_lock =
(ds->ds_deadlist.bpl_lock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/bplist.c:189
 2nd 0xffffff000286bb88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dnode.c:130
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
dnode_verify() at dnode_verify+0x70
dnode_hold_impl() at dnode_hold_impl+0x73
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_enqueue() at bplist_enqueue+0x4c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119
dbuf_write() at dbuf_write+0x24c
dbuf_sync_list() at dbuf_sync_list+0x3eb
dbuf_sync_list() at dbuf_sync_list+0x17f
Fatal double fault
rip =3D 0xffffffff805d322d
rsp =3D 0xffffff8012414fe0
rbp =3D 0xffffff8012415090
cpuid =3D 0; apic id =3D 00
panic: double fault
cpuid =3D 0
KDB: enter: panic
shared sx zfsvfs->z_teardown_inactive_lock =
(zfsvfs->z_teardown_inactive_lock) r =3D 0 (0xffffff000256d210) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_vnops.c:3724
exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff00055b3a58) locked @ =
/pool/newsrc/src/sys/kern/vfs_subr.c:2091
exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff0005618308) locked @ =
/pool/newsrc/src/sys/kern/vfs_syscalls.c:3174
shared lockmgr zfs (zfs) r =3D 0 (0xffffff00057bda58) locked @ =
/pool/newsrc/src/sys/kern/vfs_subr.c:2091
exclusive sx zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) r =3D 0 =
(0xffffff000256d310) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_znode.c:866
shared sx zp->z_name_lock (zp->z_name_lock) r =3D 0 (0xffffff00057c7c38) =
locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/zfs_dir.c:212
exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff00057ac578) locked @ =
/pool/newsrc/src/sys/kern/vfs_subr.c:2091
exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002f63648) locked @ =
/pool/newsrc/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002cff3a0) locked @ =
/pool/newsrc/src/sys/kern/uipc_sockbuf.c:148
exclusive sx ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) r =3D 0 =
(0xffffff00027e2438) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/bplist.c:189
exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 =
(0xffffff0002948238) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 =
(0xffffff0002963538) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 =
(0xffffff0002949438) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 =
(0xffffff0002948b38) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 =
(0xffffff0002966238) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905
exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 =
(0xffffff0002963438) locked @ =
/pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common=
/fs/zfs/dbuf.c:1905

0xffffff00055b39c0: tag zfs, type VDIR
    usecount 0, writecount 0, refcount 1 mountedhere 0
    flags (VI_DOINGINACT)
    lock type zfs: EXCL by thread 0xffffff000528b3a0 (pid 1425)
#0 0xffffffff80578cef at __lockmgr_args+0x4ef
#1 0xffffffff80610a59 at vop_stdlock+0x39
#2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b
#3 0xffffffff8062d957 at _vn_lock+0x57
#4 0xffffffff806210bb at vget+0x7b
#5 0xffffffff8060e570 at cache_lookup+0x4e0
#6 0xffffffff8060e890 at vfs_cache_lookup+0xc0
#7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7
#8 0xffffffff80615513 at lookup+0x3d3
#9 0xffffffff806164e9 at namei+0x4a9
#10 0xffffffff806255cf at kern_statat_vnhook+0x8f
#11 0xffffffff80625775 at kern_statat+0x15
#12 0xffffffff8062583a at lstat+0x2a
#13 0xffffffff80876b48 at syscall+0x118
#14 0xffffffff8085cf91 at Xfast_syscall+0xe1

0xffffff0005618270: tag zfs, type VDIR
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
    lock type zfs: EXCL by thread 0xffffff000528c000 (pid 1422)
#0 0xffffffff80578f77 at __lockmgr_args+0x777
#1 0xffffffff80610a59 at vop_stdlock+0x39
#2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b
#3 0xffffffff8062d957 at _vn_lock+0x57
#4 0xffffffff806299ba at setutimes+0x7a
#5 0xffffffff80629cb2 at kern_lutimes+0xf2
#6 0xffffffff80876b48 at syscall+0x118
#7 0xffffffff8085cf91 at Xfast_syscall+0xe1

0xffffff00057bd9c0: tag zfs, type VDIR
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
    lock type zfs: SHARED (count 1)
#0 0xffffffff80578cef at __lockmgr_args+0x4ef
#1 0xffffffff80610a59 at vop_stdlock+0x39
#2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b
#3 0xffffffff8062d957 at _vn_lock+0x57
#4 0xffffffff806210bb at vget+0x7b
#5 0xffffffff8060e570 at cache_lookup+0x4e0
#6 0xffffffff8060e890 at vfs_cache_lookup+0xc0
#7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7
#8 0xffffffff80615513 at lookup+0x3d3
#9 0xffffffff806164e9 at namei+0x4a9
#10 0xffffffff806255cf at kern_statat_vnhook+0x8f
#11 0xffffffff80625775 at kern_statat+0x15
#12 0xffffffff8062583a at lstat+0x2a
#13 0xffffffff80876b48 at syscall+0x118
#14 0xffffffff8085cf91 at Xfast_syscall+0xe1

0xffffff00057ac4e0: tag zfs, type VDIR
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
    lock type zfs: EXCL by thread 0xffffff000262a3a0 (pid 1416)
 with shared waiters pending
#0 0xffffffff80578cef at __lockmgr_args+0x4ef
#1 0xffffffff80610a59 at vop_stdlock+0x39
#2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b
#3 0xffffffff8062d957 at _vn_lock+0x57
#4 0xffffffff806210bb at vget+0x7b
#5 0xffffffff8060e570 at cache_lookup+0x4e0
#6 0xffffffff8060e890 at vfs_cache_lookup+0xc0
#7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7
#8 0xffffffff80615513 at lookup+0x3d3
#9 0xffffffff806164e9 at namei+0x4a9
#10 0xffffffff80624df3 at kern_mkdirat+0xc3
#11 0xffffffff80876b48 at syscall+0x118
#12 0xffffffff8085cf91 at Xfast_syscall+0xe1
Uptime: 2m16s
#=20


Kernel configuration follows:
# cat config.txt=20
options	CONFIG_AUTOGENERATED
ident	DEBUG
machine	amd64
cpu	HAMMER
makeoptions	DEBUG=3D-g
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	ATA_STATIC_ID
options	SMP
options	DEBUG_LOCKS
options	DEBUG_VFS_LOCKS
options	DIAGNOSTIC
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DDB
options	KDB
options	WITNESS_SKIPSPIN
options	WITNESS
options	INCLUDE_CONFIG_FILE
options	FLOWTABLE
options	MAC
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=3D128
options	_KPOSIX_PRIORITY_SCHEDULING
options	P1003_1B_SEMAPHORES
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=3D5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_IA32
options	COMPAT_43TTY
options	GEOM_LABEL
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSSERVER
options	NFSCLIENT
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ata
device	atadisk
device	ataraid
device	atapicd
device	atapifd
device	atapist
device	ahc
device	ahd
device	amd
device	hptiop
device	isp
device	mpt
device	sym
device	trm
device	adv
device	adw
device	aic
device	bt
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	amr
device	arcmsr
device	ciss
device	dpt
device	hptmv
device	hptrr
device	iir
device	ips
device	mly
device	twa
device	aac
device	aacp
device	ida
device	mfi
device	mlx
device	twe
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	agp
device	cbb
device	pccard
device	cardbus
device	uart
device	ppc
device	ppbus
device	lpt
device	plip
device	ppi
device	de
device	em
device	igb
device	ixgbe
device	le
device	ti
device	txp
device	vx
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	dc
device	et
device	fxp
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	cs
device	ed
device	ex
device	ep
device	fe
device	sn
device	xe
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	an
device	ath
device	ath_hal
device	ath_rate_sample
device	ral
device	wi
device	loop
device	random
device	ether
device	vlan
device	tun
device	pty
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	usb
device	uhid
device	ukbd
device	ulpt
device	umass
device	ums
device	urio
device	uark
device	ubsa
device	uftdi
device	uipaq
device	uplcom
device	uslcom
device	uvisor
device	uvscom
device	aue
device	axe
device	cdce
device	cue
device	kue
device	rue
device	udav
device	rum
device	uath
device	ural
device	zyd
device	firewire
device	fwe
device	fwip
device	dcons
device	dcons_crom
#=20







Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?548451D6-E62C-4B7F-A846-E9CF08A180C8>