Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 4 May 2010 14:37:59 +1000
From:      Peter Jeremy <peterjeremy@acm.org>
To:        freebsd-current@freebsd.org
Subject:   Non-SUJ deadlock in -current
Message-ID:  <20100504043759.GC52686@server.vk2pj.dyndns.org>

next in thread | raw e-mail | index | archive | help

--N1GIdlSm9i+YlY4t
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I have updated a vmware guest to -current from about 48 hours ago
to play with SUJ but before I got that far, I ran into a deadlock
trying to upgrade some ports.  It gets part-way through a configure
and then stops.

VTY switching works but none of the getty's will echo input.
Ctrl-T on the active window (serial console or ssh) reports
that the curent process is runnable but not using any time.
A systat stops updating.

A couple of LORs are reported but both are well before the hang:

lock order reversal:
 1st 0xc3cb4058 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091
 2nd 0xd7a4314c bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:11362
 3rd 0xc3f0baf8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091
KDB: stack backtrace:
db_trace_self_wrapper(c077a551,e60853c8,c05a9899,c059e0b4,c077c2e9,...) at =
db_trace_self_wrapper+0x26
kdb_backtrace(c059e0b4,c077c2e9,c0945b20,c3811ef8,e608541c,...) at kdb_back=
trace+0x29
_witness_debugger(c077c2e9,c3f0baf8,c0772bf6,c3814e80,c0781760,...) at _wit=
ness_debugger+0x21
witness_checkorder(c3f0baf8,9,c0781760,82b,0,...) at witness_checkorder+0x6=
8f
__lockmgr_args(c3f0baf8,80100,c3f0bb18,0,0,...) at __lockmgr_args+0x692
ffs_lock(e6085530,e6085518,c05a98e7,c0780fc5,c077be94,...) at ffs_lock+0x7d
VOP_LOCK1_APV(c07dfba0,e6085530,e6085530,c07f7ae0,c3f0baa0,...) at VOP_LOCK=
1_APV+0x8b
_vn_lock(c3f0baa0,80100,c0781760,82b,50,...) at _vn_lock+0x3d
vget(c3f0baa0,80100,c3e94900,50,0,...) at vget+0x63
vfs_hash_get(c3b21ca8,2e8ad,80000,c3e94900,e6085680,...) at vfs_hash_get+0x=
e2
ffs_vgetf(c3b21ca8,2e8ad,80000,e6085680,1,...) at ffs_vgetf+0x43
softdep_sync_metadata(c3cb4000,0,c078ee78,147,0,...) at softdep_sync_metada=
ta+0xafc
ffs_syncvnode(c3cb4000,1,e6085718,c05654c2,c3cb40ac,...) at ffs_syncvnode+0=
x378
ffs_truncate(c3cb4000,200,0,880,c3b27d00,...) at ffs_truncate+0x7f6
ufs_direnter(c3cb4000,c3f0baa0,e6085944,e6085bd4,0,...) at ufs_direnter+0x8=
38
ufs_makeinode(e6085bd4) at ufs_makeinode+0x2b4
ufs_create(e6085b34,c3b21ca8,c07f7ee0,c3cb4000,e6085b4c,...) at ufs_create+=
0x2c
VOP_CREATE_APV(c07dfba0,e6085b34,2,c077be94,0,...) at VOP_CREATE_APV+0x7a
vn_open_cred(e6085ba8,e6085c5c,180,0,c3b27d00,...) at vn_open_cred+0x1d3
vn_open(e6085ba8,e6085c5c,180,c3b41818,c3e94900,...) at vn_open+0x3b
kern_openat(c3e94900,ffffff9c,28404080,0,a03,...) at kern_openat+0xe7
kern_open(c3e94900,28404080,0,a02,180,...) at kern_open+0x35
open(c3e94900,e6085cf8,c079a860,c077c920,c3ec2aa0,...) at open+0x30
syscall(e6085d38) at syscall+0x228
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip =3D 0x28168dd3, esp =3D 0xbfbfdd7=
c, ebp =3D 0xbfbfe228 ---


lock order reversal:
 1st 0xd7a6c2b8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2581
 2nd 0xc3d8f800 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:283
KDB: stack backtrace:
db_trace_self_wrapper(c077a551,e6058a7c,c05a9899,c059e0b4,c077c2d0,...) at =
db_trace_self_wrapper+0x26
kdb_backtrace(c059e0b4,c077c2d0,c0945d88,c3811ef8,e6058ad0,...) at kdb_back=
trace+0x29
_witness_debugger(c077c2d0,c3d8f800,c078f127,c3814ee8,c078ef60,...) at _wit=
ness_debugger+0x21
witness_checkorder(c3d8f800,9,c078ef60,11b,0,...) at witness_checkorder+0x6=
8f
_sx_xlock(c3d8f800,0,c078ef60,11b,dc57806c,...) at _sx_xlock+0x32
ufsdirhash_acquire(d7a6c258,dc57806c,c3e9cae0,c3e9c910,e6058b68,...) at ufs=
dirhash_acquire+0x31
ufsdirhash_remove(c3e9cae0,dc57806c,6c,e6058b58,e6058b54,...) at ufsdirhash=
_remove+0x14
ufs_dirremove(c3e99bb0,c3e9c910,500800c,1,c3b09480,...) at ufs_dirremove+0x=
129
ufs_rmdir(e6058c34,c3b21a20,c07f7ee0,c3e99bb0,e6058c44,...) at ufs_rmdir+0x=
d2
VOP_RMDIR_APV(c07dfba0,e6058c34,2,c3e997e8,2840d100,...) at VOP_RMDIR_APV+0=
x7c
kern_rmdirat(c3b09480,ffffff9c,2840d100,0,e6058c80,...) at kern_rmdirat+0x1=
6b
kern_rmdir(c3b09480,2840d100,0,e6058d2c,c0726b64,...) at kern_rmdir+0x27
rmdir(c3b09480,e6058cf8,c079a860,c077c8be,c3e54aa0,...) at rmdir+0x22
syscall(e6058d38) at syscall+0x228
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (137, FreeBSD ELF32, rmdir), eip =3D 0x280d7333, esp =3D 0xbfbf=
e82c, ebp =3D 0xbfbfe858 ---


With the system hung, I have:
db> show alllocks
Process 16787 (sh) thread 0xc3e94d80 (100071)
exclusive lockmgr bufwait (bufwait) r =3D 0 (0xd7a5cbb0) locked @ /usr/src/=
sys/ker
n/vfs_bio.c:2581
exclusive lockmgr ufs (ufs) r =3D 0 (0xc41266b8) locked @ /usr/src/sys/kern=
/vfs_su
br.c:2091
exclusive lockmgr bufwait (bufwait) r =3D 0 (0xd79d0f20) locked @ /usr/src/=
sys/ufs
/ffs/ffs_softdep.c:11362
exclusive lockmgr ufs (ufs) r =3D 0 (0xc419e058) locked @ /usr/src/sys/kern=
/vfs_lo
okup.c:502
Process 16 (softdepflush) thread 0xc3b09000 (100039)
exclusive sleep mutex vnode interlock (vnode interlock) r =3D 0 (0xc419e078=
) locke
d @ /usr/src/sys/kern/vfs_hash.c:79
db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
16787 16703  1123     0  R+                          sh
16703 11676  1123     0  S+      wait     0xc3f1c2a8 sh
11676 11489  1123     0  S+      wait     0xc3ec37f8 sh
11489 11375  1123     0  S+      wait     0xc3f1c550 sh
11375 11361  1123     0  S+      wait     0xc3e54000 make
11361  5540  1123     0  S+      wait     0xc3f1c000 make
 5540  5500  1123     0  S+      wait     0xc3ec2550 sh
 5500  1123  1123     0  S+      wait     0xc3f1c7f8 sh
 1128     0     0     0  RL                          [nfsiod 1]
 1127     0     0     0  RL                          [nfsiod 0]
 1123  1118  1123     0  S+      wait     0xc3e59000 sh
 1118  1099  1118     0  S+      pause    0xc3bbcda0 csh
 1106  1102  1106     0  T+                          less
 1102  1100  1102     0  Rs+                         csh
 1100   991  1100     0  Ss      select   0xc3ba7c64 sshd
 1099     1  1099     0  Ss+     wait     0xc3e59550 login
 1098     1  1098     0  Ss+     ttyin    0xc389e870 getty
 1097     1  1097     0  Ss+     ttyin    0xc389ea70 getty
 1096     1  1096     0  Ss+     ttyin    0xc389ec70 getty
 1095     1  1095     0  Ss+     ttyin    0xc389ee70 getty
 1094     1  1094     0  Ss+     ttyin    0xc3968070 getty
 1093     1  1093     0  Ss+     ttyin    0xc3968270 getty
 1092     1  1092     0  Rs+                         getty
 1091     1  1091     0  Ss+     ttyin    0xc389d270 getty
 1034     1  1034     0  Ss      select   0xc3ae5864 moused
 1010     1  1010     0  Rs                          cron
 1003     1  1003    25  Ss      pause    0xc3922850 sendmail
  999     1   999     0  Rs                          sendmail
  991     1   991     0  Ss      select   0xc3ba7624 sshd
  647     1   647     0  Rs                          syslogd
  463     1   463     0  Ss      select   0xc3ba7264 devd
  462     1   462    65  Ss      select   0xc3ae5aa4 dhclient
  446     1   446     0  Ss      select   0xc3ba7224 dhclient
  111     1   111     0  Ss      pause    0xc3b07850 adjkerntz
   16     0     0     0  RL      CPU 0               [softdepflush]
   15     0     0     0  RL                          [syncer]
   14     0     0     0  RL                          [vnlru]
    9     0     0     0  RL                          [bufdaemon]
    8     0     0     0  RL                          [pagezero]
    7     0     0     0  DL      psleep   0xc0978f58 [vmdaemon]
    6     0     0     0  RL                          [pagedaemon]
    5     0     0     0  DL      ccb_scan 0xc07fd1d4 [xpt_thrd]
   13     0     0     0  DL      (threaded)          [usb]
100026                   D       -        0xc3913dac [usbus0]
100025                   D       -        0xc3913d7c [usbus0]
100024                   D       -        0xc3913d4c [usbus0]
100023                   D       -        0xc3913d1c [usbus0]
   12     0     0     0  DL      -        0xc0814004 [yarrow]
    4     0     0     0  DL      -        0xc0811d24 [g_down]
    3     0     0     0  DL      -        0xc0811d20 [g_up]
    2     0     0     0  DL      -        0xc0811d18 [g_event]
   11     0     0     0  WL      (threaded)          [intr]
100031                   I                           [swi0: uart uart]
100030                   I                           [irq12: psm0]
100029                   I                           [irq1: atkbd0]
100028                   I                           [irq18: le0]
100027                   I                           [irq17: bt0]
100022                   I                           [irq19: pcm0 uhci0]
100021                   I                           [irq15: ata1]
100020                   I                           [irq14: ata0]
100019                   I                           [irq9: acpi0]
100017                   I                           [swi2: cambio]
100016                   I                           [swi6: task queue]
100015                   I                           [swi6: Giant taskq]
100013                   I                           [swi5: +]
100005                   I                           [swi3: vm]
100004                   I                           [swi4: clock]
100003                   I                           [swi1: netisr 0]
   10     0     0     0  RL                          [idle]
    1     0     1     0  SLs     wait     0xc384d000 [init]
    0     0     0     0  RLs     (threaded)          [kernel]
100018                   D       -        0xc38bcbc0 [kqueue taskq]
100014                   D       -        0xc38bce00 [thread taskq]
100012                   D       -        0xc38be000 [acpi_task_2]
100011                   D       -        0xc38be000 [acpi_task_1]
100010                   D       -        0xc38be000 [acpi_task_0]
100000                   RunQ                        [swapper]
db> show lockedvnods
Locked vnodes

0xc419e000: tag ufs, type VDIR
    usecount 4, writecount 0, refcount 7 mountedhere 0
    flags ()
 VI_LOCKed    v_object 0xc4191440 ref 0 pages 0
    lock type ufs: EXCL by thread 0xc3e94d80 (pid 16787)
        ino 143634, on dev ad1a

0xc4126660: tag ufs, type VREG
    usecount 2, writecount 0, refcount 2 mountedhere 0
    flags ()
    lock type ufs: EXCL by thread 0xc3e94d80 (pid 16787)
        ino 144536, on dev ad1a
db>=20

--=20
Peter Jeremy

--N1GIdlSm9i+YlY4t
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (FreeBSD)

iEYEARECAAYFAkvfpKcACgkQ/opHv/APuIfSXACfeMfvsobDy81uu9v60d78vECT
NX4AoI5bPoaGxRXwxWg+DNp+8Eb+Pl6r
=b/qX
-----END PGP SIGNATURE-----

--N1GIdlSm9i+YlY4t--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100504043759.GC52686>