Date: Sat, 19 Feb 2005 18:51:22 +0100 From: Frode Nordahl <frode@nordahl.net> To: Robert Watson <rwatson@freebsd.org> Cc: current@freebsd.org Subject: Re: VFS deadlock using dump -L on HEAD -- VOP_INACTIVE acquires snapshot file system suspend lock after vnode lock on delete-after-last-close? Message-ID: <522f0af570fa6a92db61f6dfdb9a2a33@nordahl.net> In-Reply-To: <Pine.NEB.3.96L.1050218143414.62449F-100000@fledge.watson.org> References: <Pine.NEB.3.96L.1050218143414.62449F-100000@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Feb 18, 2005, at 15:49, Robert Watson wrote:
>
> I upgraded a box to a recent 6.x this morning, and ran into the
> following
> apparently deadlock while backing up a partitionusing dump -L. It
> appears
> to have gotten stuck during mksnap_ffs. Some interpretation below, but
> the high level summary is that it looks to me like we have a lock order
> issue between the snapshot write lock for the file system and vnode
> locks,
> which occurs because vrele() calls VOP_INACTIVE with the vnode lock
> held,
> and ufs_inactive() then proceeds to get a write reference to the file
> system. It could be a fix is to grab the write reference to the file
> system in vrele() before the vnode lock, to aovid grabbing the write
> reference in ufs_inactive().
My 5.2.1 and RELENG_5 NFS servers all hang up every time I try to use
dump -L too.
However, background fsck, with some exceptions allways works. One time
a system came up after a crash just before the nightly periodic scripts
where to be run (lots of finds running etc), and that made it deadlock
in a similar way. But except for this incident I have made good use of
background fsck countless times.
I have one of the NFS servers on serial console, so I decided to retry
a dump -L now with RELENG_5, since I am doing some changes to our
backup routines anyway.
I completely forgot do call doadump though, so I probably lost some
valuable information :-(
Here is what I extracted:
db> show lockedvnods
Locked vnodes
0xc418ca50: tag ufs, type VDIR, usecount 3, writecount 0, refcount 1,
flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc8e1c320
(pid 89338)
ino 3, on dev da1s1c (4, 18)
0xc66f1528: tag ufs, type VREG, usecount 2, writecount 0, refcount
4103, lock type ufs: EXCL (count 1) by thread 0xc3e4a7d0 (pid 88381)
with 1 pending
ino 274, on dev da1s1c (4, 18)
0xc799a630: tag ufs, type VREG, usecount 1, writecount 0, refcount 147,
flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc3914af0
(pid 418) with 2 pending
ino 41430101, on dev da1s1c (4, 18)
db> ps
pid proc uarea uid ppid pgrp flag stat wmesg wchan
cmd
89351 c3def710 ee183000 0 5842 89351 0004002 [SLPQ ttyin
0xc90a9c10][SLP] bash
89338 c8e1954c ee2e2000 0 89255 89338 0004002 [SLPQ ufs
0xc66f15d4][SLP] ls
89255 c3defe20 ee187000 0 5842 89255 0004002 [SLPQ wait
0xc3defe20][SLP] bash
88630 c3dea388 ee11c000 0 85239 88630 0004002 [SLPQ select
0xc08efcc4][SLP] top
88381 c3e47e20 ee229000 0 88380 88298 0004002 [SLPQ ufs
0xc799a6dc][SLP] mksnap_ffs
88380 c3e4c000 ee233000 0 88379 88298 0004002 [SLPQ wait
0xc3e4c000][SLP] sh
88379 c398b388 ee061000 0 88298 88298 0004002 [SLPQ wait
0xc398b388][SLP] dump
88298 c3bd5000 ee078000 0 5843 88298 0004002 [SLPQ wait
0xc3bd5000][SLP] sh
85239 c3e4c54c ee236000 0 5842 85239 0004002 [SLPQ wait
0xc3e4c54c][SLP] bash
83213 c3e47a98 ee227000 0 83201 83213 0004002 [SLPQ ttyin
0xc97a0c10][SLP] bash
83201 c3e4c388 ee235000 0 496 83201 0000100 [SLPQ select
0xc08efcc4][SLP] sshd
82285 c3e47c5c ee228000 0 81759 82285 0004002 [SLPQ pause
0xc3e47c94][SLP] screen
81759 c3df88d4 ee1df000 0 81752 81759 0004002 [SLPQ wait
0xc3df88d4][SLP] bash
81752 c3df8388 ee1dc000 0 496 81752 0000100 [SLPQ select
0xc08efcc4][SLP] sshd
5843 c3def8d4 ee184000 0 5842 5843 0004002 [SLPQ wait
0xc3def8d4][SLP] bash
5842 c3e47388 ee1e5000 0 1 5842 0000000 [SLPQ select
0xc08efcc4][SLP] screen
13643 c3df8a98 ee1e0000 0 1 13643 0004002 [SLPQ ttyin
0xc38fe210][SLP] getty
920 c3deaa98 ee17c000 0 564 920 0004002 [SLPQ ttyin
0xc3619e10][SLP] bash
569 c3bda54c ee0de000 0 1 569 0004002 [SLPQ ttyin
0xc3680c10][SLP] getty
568 c3915c5c ebe2a000 0 1 568 0004002 [SLPQ ttyin
0xc3680e10][SLP] getty
567 c3bdaa98 ee0e1000 0 1 567 0004002 [SLPQ ttyin
0xc3681010][SLP] getty
566 c3bd58d4 ee0d7000 0 1 566 0004002 [SLPQ ttyin
0xc3681210][SLP] getty
565 c398b000 ee05f000 0 1 565 0004002 [SLPQ ttyin
0xc3634a10][SLP] getty
564 c398be20 ee067000 0 1 564 0004102 [SLPQ wait
0xc398be20][SLP] login
563 c3986e20 ee05e000 0 1 563 0004002 [SLPQ ttyin
0xc3619810][SLP] login
551 c39121c4 ebdfd000 0 1 551 0000000 [SLPQ select
0xc08efcc4][SLP] inetd
526 c3bd5388 ee098000 0 1 114 0000002 [SLPQ nanslp
0xc08ea56c][SLP] perl
518 c3bd5a98 ee0d8000 0 1 518 0000000 [SLPQ nanslp
0xc08ea56c][SLP] cron
506 c3bda388 ee0dd000 25 1 506 0000100 [SLPQ pause
0xc3bda3c0][SLP] sendmail
502 c3bd51c4 ee07f000 0 1 502 0000100 [SLPQ select
0xc08efcc4][SLP] sendmail
496 c398b1c4 ee060000 0 1 496 0000100 [SLPQ select
0xc08efcc4][SLP] sshd
485 c3bd5c5c ee0d9000 1 1 485 0000100 [SLPQ sbwait
0xc3bb3974][SLP] rwhod
474 c39861c4 ee057000 0 1 474 0000000 [SLPQ select
0xc08efcc4][SLP] ntpd
450 c398654c ee059000 0 1 450 0000000 [SLPQ select
0xc08efcc4][SLP] usbd
435 c3912710 ebe00000 0 429 429 0000100 [SLPQ select
0xc08efcc4][SLP] rpc.lockd
429 c398b54c ee062000 0 1 429 0000000 [SLPQ select
0xc08efcc4][SLP] rpc.lockd
424 c3912000 ebdfc000 0 1 424 0000000 [SLPQ select
0xc08efcc4][SLP] rpc.statd
418 c3912a98 ebe20000 0 415 415 0000000 [SLPQ suspfs
0xc38ef46c][SLP] nfsd
417 c398b8d4 ee064000 0 415 415 0000000 [SLPQ suspfs
0xc38ef46c][SLP] nfsd
416 c391254c ebdff000 0 415 415 0000000 [SLPQ suspfs
0xc38ef46c][SLP] nfsd
415 c39868d4 ee05b000 0 1 415 0000000 [SLPQ ufs
0xc799a6dc][SLP] nfsd
413 c39128d4 ebe01000 0 1 413 0000000 [SLPQ select
0xc08efcc4][SLP] mountd
344 c3915e20 ebe2b000 0 1 344 0000000 [SLPQ select
0xc08efcc4][SLP] rpcbind
325 c35f1e20 ebd27000 0 1 325 0000000 [SLPQ select
0xc08efcc4][SLP] syslogd
305 c3986388 ee058000 0 1 305 0000000 [SLPQ select
0xc08efcc4][SLP] devd
113 c3915000 ebe23000 0 0 0 0000204 [SLPQ -
0xebe10d18][SLP] schedcpu
112 c39151c4 ebe24000 0 0 0 0000204 [SLPQ -
0xc08f76ec][SLP] nfsiod 3
111 c3915388 ebe25000 0 0 0 0000204 [SLPQ -
0xc08f76e8][SLP] nfsiod 2
110 c391554c ebe26000 0 0 0 0000204 [SLPQ -
0xc08f76e4][SLP] nfsiod 1
109 c3915710 ebe27000 0 0 0 0000204 [SLPQ -
0xc08f76e0][SLP] nfsiod 0
108 c39158d4 ebe28000 0 0 0 0000204 [SLPQ hptrdy
0xc078fda4][SLP] hpt_wt
107 c353c1c4 ebcdb000 0 0 0 0000204 [SLPQ vlruwt
0xc353c1c4][SLP] vnlru
106 c353c388 ebcdc000 0 0 0 0000204 [SLPQ syncer
0xc08ea2ec][SLP] syncer
105 c353c54c ebcdd000 0 0 0 0000204 [SLPQ psleep
0xc08f028c][SLP] bufdaemon
104 c353c710 ebcde000 0 0 0 000020c [SLPQ pgzero
0xc08fe054][SLP] pagezero
103 c353c8d4 ebcdf000 0 0 0 0000204 [SLPQ psleep
0xc08fe0a8][SLP] vmdaemon
102 c353ca98 ebce0000 0 0 0 0000204 [SLPQ psleep
0xc08fe064][SLP] pagedaemon
101 c353cc5c ebce1000 0 0 0 0000204 [IWAIT] swi0: sio
100 c353ce20 ebd1e000 0 0 0 0000204 [SLPQ -
0xc35f303c][SLP] fdc0
99 c35f1000 ebd1f000 0 0 0 0000204 [SLPQ usbevt
0xc362b210][SLP] usb2
98 c35f11c4 ebd20000 0 0 0 0000204 [SLPQ usbevt
0xc3627210][SLP] usb1
97 c35f1388 ebd21000 0 0 0 0000204 [SLPQ usbtsk
0xc08e2138][SLP] usbtask
96 c35f154c ebd22000 0 0 0 0000204 [SLPQ usbevt
0xc3611210][SLP] usb0
95 c35f1710 ebd23000 0 0 0 0000204 [SLPQ idle
0xc361b000][SLP] aic_recovery1
94 c35f18d4 ebd24000 0 0 0 0000204 [SLPQ idle
0xc3617000][SLP] aic_recovery0
9 c35f1a98 ebd25000 0 0 0 0000204 [SLPQ actask
0xc0a34a2c][SLP] acpi_task2
8 c35f1c5c ebd26000 0 0 0 0000204 [SLPQ actask
0xc0a34a2c][SLP] acpi_task1
7 c351f54c ebcad000 0 0 0 0000204 [SLPQ actask
0xc0a34a2c][SLP] acpi_task0
93 c351f710 ebcae000 0 0 0 0000204 [IWAIT] swi6:+
6 c351f8d4 ebcaf000 0 0 0 0000204 [SLPQ -
0xc358fbc0][SLP] thread taskq
92 c351fa98 ebcb0000 0 0 0 0000204 [IWAIT] swi6: acpitaskq
91 c351fc5c ebcb1000 0 0 0 0000204 [IWAIT] swi6:+
90 c351fe20 ebcb2000 0 0 0 0000204 [IWAIT] swi6: task
queue
5 c353a000 ebcb3000 0 0 0 0000204 [SLPQ -
0xc358fd80][SLP] kqueue taskq
89 c353a1c4 ebcd2000 0 0 0 0000204 [IWAIT] swi3: cambio
88 c353a388 ebcd3000 0 0 0 0000204 [IWAIT] swi2: camnet
87 c353a54c ebcd4000 0 0 0 0000204 [SLPQ -
0xc08dfe80][SLP] yarrow
4 c353a710 ebcd5000 0 0 0 0000204 [SLPQ -
0xc08e4b28][SLP] g_down
3 c353a8d4 ebcd6000 0 0 0 0000204 [SLPQ -
0xc08e4b24][SLP] g_up
2 c353aa98 ebcd7000 0 0 0 0000204 [SLPQ -
0xc08e4b1c][SLP] g_event
86 c353ac5c ebcd8000 0 0 0 0000204 [IWAIT] swi4: vm
85 c353ae20 ebcd9000 0 0 0 000020c [IWAIT] swi5: clock sio
84 c353c000 ebcda000 0 0 0 0000204 [IWAIT] swi1: net
83 c350da98 ebc62000 0 0 0 0000204 [IWAIT] irq0: clk
82 c350dc5c ebc63000 0 0 0 0000204 [IWAIT] irq71:
81 c350de20 ebc64000 0 0 0 0000204 [IWAIT] irq70:
80 c351b000 ebc65000 0 0 0 0000204 [IWAIT] irq69:
79 c351b1c4 ebc66000 0 0 0 0000204 [IWAIT] irq68:
78 c351b388 ebc67000 0 0 0 0000204 [IWAIT] irq67:
77 c351b54c ebca4000 0 0 0 0000204 [IWAIT] irq66:
76 c351b710 ebca5000 0 0 0 0000204 [IWAIT] irq65:
75 c351b8d4 ebca6000 0 0 0 0000204 [IWAIT] irq64:
74 c351ba98 ebca7000 0 0 0 0000204 [IWAIT] irq63:
73 c351bc5c ebca8000 0 0 0 0000204 [IWAIT] irq62:
72 c351be20 ebca9000 0 0 0 0000204 [IWAIT] irq61:
71 c351f000 ebcaa000 0 0 0 0000204 [IWAIT] irq60:
70 c351f1c4 ebcab000 0 0 0 0000204 [IWAIT] irq59: em1
69 c351f388 ebcac000 0 0 0 0000204 [IWAIT] irq58: em0
68 c34fc1c4 ebc18000 0 0 0 0000204 [IWAIT] irq57:
67 c34fc388 ebc19000 0 0 0 0000204 [IWAIT] irq56:
66 c34fc54c ebc1a000 0 0 0 0000204 [IWAIT] irq55:
65 c34fc710 ebc1b000 0 0 0 0000204 [IWAIT] irq54:
64 c34fc8d4 ebc1c000 0 0 0 0000204 [IWAIT] irq53:
63 c34fca98 ebc59000 0 0 0 0000204 [IWAIT] irq52: iir0
62 c34fcc5c ebc5a000 0 0 0 0000204 [IWAIT] irq51:
61 c34fce20 ebc5b000 0 0 0 0000204 [IWAIT] irq50:
60 c350d000 ebc5c000 0 0 0 0000204 [IWAIT] irq49:
59 c350d1c4 ebc5d000 0 0 0 0000204 [IWAIT] irq48:
58 c350d388 ebc5e000 0 0 0 0000204 [IWAIT] irq47:
57 c350d54c ebc5f000 0 0 0 0000204 [IWAIT] irq46:
56 c350d710 ebc60000 0 0 0 0000204 [IWAIT] irq45:
55 c350d8d4 ebc61000 0 0 0 0000204 [IWAIT] irq44:
54 c34e6a98 ebbed000 0 0 0 0000204 [IWAIT] irq43:
53 c34e6c5c ebbee000 0 0 0 0000204 [IWAIT] irq42:
52 c34e6e20 ebbef000 0 0 0 0000204 [IWAIT] irq41:
51 c34f8000 ebbf0000 0 0 0 0000204 [IWAIT] irq40:
50 c34f81c4 ebc0f000 0 0 0 0000204 [IWAIT] irq39:
49 c34f8388 ebc10000 0 0 0 0000204 [IWAIT] irq38:
48 c34f854c ebc11000 0 0 0 0000204 [IWAIT] irq37:
47 c34f8710 ebc12000 0 0 0 0000204 [IWAIT] irq36:
46 c34f88d4 ebc13000 0 0 0 0000204 [IWAIT] irq35:
45 c34f8a98 ebc14000 0 0 0 0000204 [IWAIT] irq34:
44 c34f8c5c ebc15000 0 0 0 0000204 [IWAIT] irq33:
43 c34f8e20 ebc16000 0 0 0 0000204 [IWAIT] irq32:
42 c34fc000 ebc17000 0 0 0 0000204 [IWAIT] irq31: ahd1
41 c34dc54c ebbc3000 0 0 0 0000204 [IWAIT] irq30: ahd0
40 c34dc710 ebbc4000 0 0 0 0000204 [IWAIT] irq29:
39 c34dc8d4 ebbc5000 0 0 0 0000204 [IWAIT] irq28:
38 c34dca98 ebbe4000 0 0 0 0000204 [IWAIT] irq27:
37 c34dcc5c ebbe5000 0 0 0 0000204 [IWAIT] irq26:
36 c34dce20 ebbe6000 0 0 0 0000204 [IWAIT] irq25:
35 c34e6000 ebbe7000 0 0 0 0000204 [IWAIT] irq24:
34 c34e61c4 ebbe8000 0 0 0 0000204 [IWAIT] irq23:
33 c34e6388 ebbe9000 0 0 0 0000204 [IWAIT] irq22:
32 c34e654c ebbea000 0 0 0 0000204 [IWAIT] irq21:
31 c34e6710 ebbeb000 0 0 0 0000204 [IWAIT] irq20:
30 c34e68d4 ebbec000 0 0 0 0000204 [IWAIT] irq19: uhci1
29 c348c1c4 ea178000 0 0 0 0000204 [IWAIT] irq18: uhci2
28 c348c388 ea179000 0 0 0 0000204 [IWAIT] irq17:
27 c348c54c ea198000 0 0 0 0000204 [IWAIT] irq16: uhci0
26 c348c710 ea199000 0 0 0 0000204 [IWAIT] irq15: ata1
25 c348c8d4 ea19a000 0 0 0 0000204 [IWAIT] irq14: ata0
24 c348ca98 ea19b000 0 0 0 0000204 [IWAIT] irq13:
23 c348cc5c ea19c000 0 0 0 0000204 [IWAIT] irq12:
22 c348ce20 ea19d000 0 0 0 0000204 [IWAIT] irq11:
21 c34dc000 ebbc0000 0 0 0 0000204 [IWAIT] irq10:
20 c34dc1c4 ebbc1000 0 0 0 0000204 [IWAIT] irq9: acpi0
19 c34dc388 ebbc2000 0 0 0 0000204 [IWAIT] irq8: rtc
18 c3482000 ea132000 0 0 0 0000204 [IWAIT] irq7: ppc0
17 c34821c4 ea16f000 0 0 0 0000204 [IWAIT] irq6: fdc0
16 c3482388 ea170000 0 0 0 0000204 [IWAIT] irq5:
15 c348254c ea171000 0 0 0 0000204 [IWAIT] irq4: sio0
14 c3482710 ea172000 0 0 0 0000204 [IWAIT] irq3: sio1
13 c34828d4 ea173000 0 0 0 0000204 [IWAIT] irq1: atkbd0
12 c3482a98 ea174000 0 0 0 000020c [CPU 0] idle: cpu0
11 c3482c5c ea175000 0 0 0 000020c [CPU 1] idle: cpu1
1 c3482e20 ea176000 0 0 1 0004200 [SLPQ wait
0xc3482e20][SLP] init
10 c348c000 ea177000 0 0 0 0000204 [SLPQ ktrace
0xc08e8518][SLP] ktrace
0 c08e4c20 c0c1f000 0 0 0 0000200 [SLPQ sched
0xc08e4c20][SLP] swapper
db> trace 88381
Tracing pid 88381 tid 100235 td 0xc3e4a7d0
sched_switch(c3e4a7d0,0,1) at sched_switch+0x143
mi_switch(1,0,c3e4a7d0,1,c3e4a7d0) at mi_switch+0x1ba
sleepq_switch(c799a6dc) at sleepq_switch+0x133
sleepq_wait(c799a6dc,0,f3,0,c3c774a0) at sleepq_wait+0xb
msleep(c799a6dc,c08e93c4,50,c08303e7,0) at msleep+0x322
acquire(ee21a7c4,1000040,600,c3e4a7d0,0) at acquire+0xa2
lockmgr(c799a6dc,1010002,c799a630,c3e4a7d0,10002) at lockmgr+0x366
ufs_lock(ee21a7f0,ee21a80c,c0674b20,ee21a7f0,c08ca8e0) at ufs_lock+0x3c
ufs_vnoperate(ee21a7f0) at ufs_vnoperate+0x13
vn_lock(c799a630,10002,c3e4a7d0) at vn_lock+0x104
ffs_snapshot(c38ef400,bfbfe9eb,c3911a00,ee21aab0,0) at
ffs_snapshot+0x11d2
ffs_omount(c38ef400,c9a96480,bfbfe248,c3e4a7d0,0) at ffs_omount+0x52e
vfs_domount(c3e4a7d0,c8cc0990,c9a96480,1211118,bfbfe248) at
vfs_domount+0x7b8
mount(c3e4a7d0,ee21ad14,4,0,206) at mount+0x6c
syscall(2f,2f,2f,bfbfe498,bfbfe9e1) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (21, FreeBSD ELF32, mount), eip = 0x280c1473, esp =
0xbfbfe1bc, ebp = 0xbfbfe8b0 ---
db> trace 415
Tracing pid 415 tid 100143 td 0xc39884b0
sched_switch(c39884b0,0,1) at sched_switch+0x143
mi_switch(1,0,c39884b0,1,c39884b0) at mi_switch+0x1ba
sleepq_switch(c799a6dc) at sleepq_switch+0x133
sleepq_wait(c799a6dc,0,0,0,0) at sleepq_wait+0xb
msleep(c799a6dc,c08e93c4,50,c08303e7,0) at msleep+0x322
acquire(ee0448ec,1000040,600,c39884b0,0) at acquire+0xa2
lockmgr(c799a6dc,1010002,c799a630,c39884b0,10002) at lockmgr+0x366
ufs_lock(ee044918,ee044934,c0674b20,ee044918,c08ca8e0) at ufs_lock+0x3c
ufs_vnoperate(ee044918) at ufs_vnoperate+0x13
vn_lock(c799a630,10002,c39884b0,2782c55,4) at vn_lock+0x104
vget(c799a630,10002,c39884b0,c799a630,c39884b0) at vget+0xfd
ufs_ihashget(c393dd00,2782c55,2,ee044a0c,1010002) at ufs_ihashget+0xfe
ffs_vget(c38ef400,2782c55,2,ee044a0c) at ffs_vget+0x31
ufs_fhtovp(c38ef400,ee044bf8,ee044ae4,c38ef400,ee044a70) at
ufs_fhtovp+0x22
ffs_fhtovp(c38ef400,ee044bf8,ee044ae4,c39884b0,100) at ffs_fhtovp+0x3a
nfsrv_fhtovp(ee044bf0,1,ee044ae4,ca071380,c3bb1780) at nfsrv_fhtovp+0xf5
nfsrv_read(ca071300,c3bb1780,c39884b0,ee044ca8,ee044ca8) at
nfsrv_read+0x25d
nfssvc_nfsd(c39884b0,0,c39884b0,ee044ce8,c39868d4) at nfssvc_nfsd+0x406
nfssvc(c39884b0,ee044d14,2,1,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfedf4,8052180) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp =
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db> trace 416
Tracing pid 416 tid 100098 td 0xc353bc80
sched_switch(c353bc80,0,1) at sched_switch+0x143
mi_switch(1,0,c353bc80,1,c353bc80) at mi_switch+0x1ba
sleepq_switch(c38ef46c) at sleepq_switch+0x133
sleepq_wait(c38ef46c,0,801031d,0,0) at sleepq_wait+0xb
msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322
vn_start_write(0,ebcceabc,1) at vn_start_write+0x8b
nfsrv_write(ca071900,c3bb1780,c353bc80,ebcceca8,ebcceca0) at
nfsrv_write+0x1b4
nfssvc_nfsd(c353bc80,0,c353bc80,ebccece8,c391254c) at nfssvc_nfsd+0x406
nfssvc(c353bc80,ebcced14,2,0,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp =
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db> trace 417
Tracing pid 417 tid 100134 td 0xc3987640
sched_switch(c3987640,0,1) at sched_switch+0x143
mi_switch(1,0,c3987640,1,c3987640) at mi_switch+0x1ba
sleepq_switch(c38ef46c) at sleepq_switch+0x133
sleepq_wait(c38ef46c,0,3a,0,c364bba0) at sleepq_wait+0xb
msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322
vn_start_write(0,ee029abc,1) at vn_start_write+0x8b
nfsrv_write(ca2e2000,c3bb1780,c3987640,ee029ca8,ee029ca0) at
nfsrv_write+0x1b4
nfssvc_nfsd(c3987640,0,c3987640,ee029ce8,c398b8d4) at nfssvc_nfsd+0x406
nfssvc(c3987640,ee029d14,2,0,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp =
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db> trace 418
Tracing pid 418 tid 100127 td 0xc3914af0
sched_switch(c3914af0,0,1) at sched_switch+0x143
mi_switch(1,0,c3914af0,1,c3914af0) at mi_switch+0x1ba
sleepq_switch(c38ef46c) at sleepq_switch+0x133
sleepq_wait(c38ef46c,0,c08ca3e0,0,c364bba0) at sleepq_wait+0xb
msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322
vn_write_suspend_wait(c799a630,c38ef400,1,0,dc59b08c) at
vn_write_suspend_wait+0x89
ufs_inactive(ebe19a74,ebe19a8c,c0669ae7,ebe19a74,c08ca860) at
ufs_inactive+0x1ba
ufs_vnoperate(ebe19a74) at ufs_vnoperate+0x13
vput(c799a630) at vput+0xfb
nfsrv_read(ca10f000,c3bb1780,c3914af0,ebe19ca8,ebe19ca0) at
nfsrv_read+0xcfa
nfssvc_nfsd(c3914af0,0,c3914af0,ebe19ce8,c3912a98) at nfssvc_nfsd+0x406
nfssvc(c3914af0,ebe19d14,2,0,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp =
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db>
Regards,
Frode Nordahl
> Robert N M Watson
>
> www:~> more backup.csh
> #!/bin/csh
>
> set DATE=`date +"%Y%m%d"`
>
> dump -L -f - -a -u -0 /home/pnn | gzip -9 > ${DATE}-backup.dump.gz
> www:~> df
> Filesystem 1K-blocks Used Avail Capacity Mounted on
> /dev/twed0s1a 507630 103502 363518 22% /
> devfs 1 1 0 100% /dev
> /dev/twed0s1f 89359640 370128 81840742 0% /home/pnn
> /dev/twed0s1e 16244334 3211132 11733656 21% /usr
> /dev/twed0s1d 8122126 1541978 5930378 21% /var
> www:~> ./backup.csh
> load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
> load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
> load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
> load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
>
> On the serial console:
>
> telnet> send break
> KDB: enter: Line break on console
> [thread pid 11 tid 100004 ]
> Stopped at kdb_enter+0x2b: nop
> db> show lockedvnods
> Locked vnodes
>
> 0xc13c49b4: tag ufs, type VDIR
> usecount 153, writecount 0, refcount 1 mountedhere 0
> flags (VV_ROOT)
> v_object 0
> lock type ufs: EXCL (count 1) by thread 0xc1807a10 (pid 3057) with
> 152 pend
> ing
> ino 2, on dev twed0s1f (229, 9)
>
> 0xc15268a0: tag ufs, type VDIR
> usecount 2, writecount 0, refcount 1 mountedhere 0
> flags ()
> v_object 0
> lock type ufs: EXCL (count 1) by thread 0xc137e000 (pid 3062)
> with 1
> pendin
> g
> ino 3085322, on dev twed0s1f (229, 9)
>
> 0xc163bbdc: tag ufs, type VDIR
> usecount 2, writecount 0, refcount 1 mountedhere 0
> flags ()
> v_object 0
> lock type ufs: EXCL (count 1) by thread 0xc162dcf0 (pid 3056)
> with 1
> pendin
> g
> ino 3133686, on dev twed0s1f (229, 9)
>
> 0xc17e933c: tag ufs, type VREG
> usecount 1, writecount 0, refcount 0 mountedhere 0
> flags ()
> v_object 0xc17e7210
> lock type ufs: EXCL (count 1) by thread 0xc1996450 (pid 3045)
> with 1
> pendin
> g
> ino 3134802, on dev twed0s1f (229, 9)
>
> 0xc18419b4: tag ufs, type VREG
> usecount 0, writecount 0, refcount 0 mountedhere 0
> flags ()
> v_object 0xc1826000
> lock type ufs: EXCL (count 1) by thread 0xc19b2450 (pid 3047)
> ino 3132762, on dev twed0s1f (229, 9)
>
> 0xc1ac933c: tag ufs, type VREG
> usecount 1, writecount 0, refcount 865 mountedhere 0
> flags ()
> v_object 0
> lock type ufs: EXCL (count 1) by thread 0xc162d8a0 (pid 2993)
> ino 9891841, on dev twed0s1f (229, 9)
>
> 0xc19fd9b4: tag ufs, type VREG
> usecount 0, writecount 0, refcount 0 mountedhere 0
> flags ()
> v_object 0xc1a93948
> lock type ufs: EXCL (count 1) by thread 0xc1807b80 (pid 2609)
> ino 3197600, on dev twed0s1f (229, 9)
> db> ps
> pid proc uid ppid pgrp flag stat wmesg wchan cmd
> 3296 c1cf93f8 1062 3292 3292 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3295 c1cf91fc 1062 3293 3293 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3293 c1cf9be8 1062 3290 3293 0004000 [SLPQ wait 0xc1cf9be8][SLP] sh
> 3292 c1d66000 1062 3289 3292 0004000 [SLPQ wait 0xc1d66000][SLP] sh
> 3290 c19b0000 0 390 390 0000000 [SLPQ piperd 0xc1655480][SLP]
> cron
> 3289 c1d661fc 0 390 390 0000000 [SLPQ piperd 0xc19a3d80][SLP]
> cron
> 3271 c1d66be8 1062 3268 3268 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3268 c1d69000 1062 3267 3268 0004000 [SLPQ wait 0xc1d69000][SLP] sh
> 3267 c1d669ec 0 390 390 0000000 [SLPQ piperd 0xc13aac00][SLP]
> cron
> 3247 c1cf95f4 1062 3245 3245 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3245 c1cf99ec 1062 3243 3245 0004000 [SLPQ wait 0xc1cf99ec][SLP] sh
> 3243 c1cf9de4 0 390 390 0000000 [SLPQ piperd 0xc13aa480][SLP]
> cron
> 3227 c1d66de4 1062 3222 3222 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3226 c1c8f1fc 1062 3223 3223 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3223 c18063f8 1062 3218 3223 0004000 [SLPQ wait 0xc18063f8][SLP] sh
> 3222 c1805000 1062 3217 3222 0004000 [SLPQ wait 0xc1805000][SLP] sh
> 3218 c19ae000 0 390 390 0000000 [SLPQ piperd 0xc13a9a80][SLP]
> cron
> 3217 c18067f0 0 390 390 0000000 [SLPQ piperd 0xc13a9300][SLP]
> cron
> 3197 c1c8f3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3196 c1c8f5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3195 c1c8f7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3194 c1c8f9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3193 c1c8fbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3192 c1c8fde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3191 c1cf6000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3190 c1cf61fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3189 c1cf63f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3188 c1cf65f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3187 c1cf67f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3186 c1cf69ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3185 c1cf6be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3184 c1cf6de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3183 c1cf9000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3182 c1c8e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3181 c1c8e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3180 c1c8e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3179 c1c1e3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3178 c1c1e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3177 c1c1e7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3176 c1c1e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3175 c1c1ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3174 c1c1ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3173 c1c88000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3172 c1c881fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3171 c1c883f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3170 c1c885f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3169 c1c887f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3168 c1c889ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3167 c1c88be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3166 c1c88de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3165 c1c8e000 1062 3163 3163 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
> 3163 c1c8e3f8 1062 3161 3163 0004000 [SLPQ wait 0xc1c8e3f8][SLP] sh
> 3161 c1c8e7f0 0 390 390 0000000 [SLPQ piperd 0xc19a3000][SLP]
> cron
> 3159 c1c8ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3158 c1c8ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3157 c1c8f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3156 c1bc37f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3155 c1bc39ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3154 c1bc3be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3153 c1bc3de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3152 c1c19000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3151 c1c191fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3150 c1c193f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3149 c1c195f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3148 c1c197f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3147 c1c199ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3146 c1c19be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3145 c1c19de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3144 c1c1a000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3143 c1c1a1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3142 c1c1a3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3141 c1c1a5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3140 c1c1a7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3139 c1c1a9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3138 c1c1abe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3137 c1c1ade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3136 c1c1e000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3135 c1c1e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3134 c1bc1de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3133 c1bc1be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3132 c1b837f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3131 c1b839ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3130 c1b83be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3129 c1b83de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3128 c1bc1000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3127 c1bc11fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3126 c1bc13f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3125 c1bc15f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3124 c1bc17f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3123 c1bc19ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3120 c1bc3000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3119 c1bc31fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3118 c1bc33f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3117 c1bc35f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3116 c1abb7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3115 c1abb9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3114 c1abbbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3113 c1abbde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3112 c1b82000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3111 c1b821fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3110 c1b823f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3109 c1b825f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3108 c1b827f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3107 c1b829ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3106 c1b82be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3105 c1b82de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3104 c1b83000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3103 c1b831fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3102 c1b833f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3101 c1b835f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3100 c19b07f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3099 c19b09ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3098 c19b0be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3097 c19b0de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3096 c1aba000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3095 c1aba1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3094 c1aba3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3093 c1aba5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3092 c1aba7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3091 c1aba9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3090 c1ababe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3089 c1abade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3088 c1abb000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3087 c1abb1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3086 c1abb3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3085 c16291fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3084 c1abb5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3083 c1510be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3082 c162c3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3081 c18061fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3080 c19957f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3079 c18055f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3078 c19aede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3077 c19959ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3076 c19aebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3075 c16297f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3074 c16293f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3073 c19ae5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3072 c19ae1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3071 c19951fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3070 c19ae7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3069 c12ab7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3068 c1510de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3067 c1805be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3066 c1994000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3065 c19ae9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3064 c137d3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3063 c150f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3062 c1510000 80 2910 403 0004000 [SLPQ ufs 0xc163bca0][SLP]
> download.cgi
> 3060 c19953f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3059 c19947f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3058 c16299ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3057 c1806be8 80 403 403 0000100 [SLPQ ufs 0xc1526964][SLP]
> httpd
> 3056 c1994be8 80 403 403 0000100 [SLPQ ufs 0xc17e9400][SLP]
> httpd
> 3055 c1806000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3053 c137dde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3051 c19b05f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3048 c19941fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 3047 c19b03f8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP]
> httpd
> 3045 c1995be8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP]
> httpd
> 3036 c19945f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 2993 c162c1fc 1000 2992 2987 0004102 [SLPQ ufs 0xc13c4a78][SLP]
> mksnap_ffs
> 2992 c162cbe8 1000 2990 2987 0004002 [SLPQ wait 0xc162cbe8][SLP] sh
> 2991 c1995de4 1000 2987 2987 0004002 [SLPQ piperd 0xc13a9780][SLP]
> gzip
> 2990 c150f5f4 1000 2987 2987 0004002 [SLPQ wait 0xc150f5f4][SLP]
> dump
> 2987 c1806de4 1000 1762 2987 0004002 [SLPQ pause 0xc1806e18][SLP]
> csh
> 2910 c1994de4 80 403 403 0000100 [SLPQ wait 0xc1994de4][SLP]
> httpd
> 2909 c18053f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 2908 c19ae3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 2843 c1995000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 2839 c19955f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 2669 c1629000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 2609 c18069ec 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP]
> httpd
> 2598 c19943f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 1981 c18051fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 1899 c16295f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 1762 c162cde4 1000 1761 1762 0004002 [SLPQ pause 0xc162ce18][SLP]
> tcsh
> 1761 c162c000 1000 1759 1759 0000100 [SLPQ select 0xc078a744][SLP]
> sshd
> 1759 c1629be8 0 368 1759 0004100 [SLPQ sbwait 0xc1429480][SLP]
> sshd
> 1754 c18065f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP]
> httpd
> 526 c162c9ec 1062 516 526 0004100 [SLPQ piperd 0xc13a9900][SLP]
> sendmail
> 525 c162c7f0 1062 1 525 0004000 [SLPQ accept 0xc1429186][SLP]
> perl
> 516 c18057f0 0 390 390 0000000 [SLPQ piperd 0xc13a9c00][SLP]
> cron
> 447 c12ab5f4 0 1 447 0004002 [SLPQ ttyin 0xc1318410][SLP]
> getty
> 446 c15107f0 0 1 446 0004002 [SLPQ ttyin 0xc12f5810][SLP]
> getty
> 445 c15105f4 0 1 445 0004002 [SLPQ ttyin 0xc12f6810][SLP]
> getty
> 444 c15101fc 0 1 444 0004002 [SLPQ ttyin 0xc12f6410][SLP]
> getty
> 443 c150f9ec 0 1 443 0004002 [SLPQ ttyin 0xc12ef410][SLP]
> getty
> 442 c15103f8 0 1 442 0004002 [SLPQ ttyin 0xc12ef810][SLP]
> getty
> 441 c150f3f8 0 1 441 0004002 [SLPQ ttyin 0xc12efc10][SLP]
> getty
> 440 c150f1fc 0 1 440 0004002 [SLPQ ttyin 0xc12f5010][SLP]
> getty
> 439 c150fde4 0 1 439 0004002 [SLPQ ttyin 0xc12f5410][SLP]
> getty
> 427 c150fbe8 0 1 427 0000000 [SLPQ select 0xc078a744][SLP]
> inetd
> 403 c150f7f0 0 1 403 0000000 [SLPQ select 0xc078a744][SLP]
> httpd
> 390 c12ab9ec 0 1 390 0000000 [SLPQ nanslp 0xc0785dac][SLP]
> cron
> 378 c137d000 25 1 378 0000100 [SLPQ pause 0xc137d034][SLP]
> sendmail
> 374 c12abbe8 0 1 374 0000100 [SLPQ select 0xc078a744][SLP]
> sendmail
> 368 c137d1fc 0 1 368 0000100 [SLPQ select 0xc078a744][SLP]
> sshd
> 252 c12ab3f8 0 1 252 0000000 [SLPQ select 0xc078a744][SLP]
> syslogd
> 227 c12abde4 0 1 227 0000000 [SLPQ select 0xc078a744][SLP]
> devd
> 46 c137d5f4 0 0 0 0000204 [SLPQ - 0xcb30cd14][SLP]
> schedcpu
> 45 c137d7f0 0 0 0 0000204 [SLPQ - 0xc07923ec][SLP]
> nfsiod 3
> 44 c137d9ec 0 0 0 0000204 [SLPQ - 0xc07923e8][SLP]
> nfsiod 2
> 43 c137dbe8 0 0 0 0000204 [SLPQ - 0xc07923e4][SLP]
> nfsiod 1
> 42 c1243be8 0 0 0 0000204 [SLPQ - 0xc07923e0][SLP]
> nfsiod 0
> 41 c1243de4 0 0 0 0000204 [SLPQ vlruwt 0xc1243de4][SLP]
> vnlru
> 40 c12a8000 0 0 0 0000204 [SLPQ syncer 0xc0785b0c][SLP]
> syncer
> 39 c12a81fc 0 0 0 0000204 [SLPQ psleep 0xc078ac90][SLP]
> bufdaemon
> 38 c12a83f8 0 0 0 000020c [SLPQ pgzero 0xc07989e4][SLP]
> pagezero
> 9 c12a85f4 0 0 0 0000204 [SLPQ psleep 0xc0798534][SLP]
> vmdaemon
> 8 c12a87f0 0 0 0 0000204 [SLPQ psleep 0xc07984f0][SLP]
> pagedaemon
> 37 c12a89ec 0 0 0 0000204 [IWAIT] swi0: sio
> 7 c12a8be8 0 0 0 0000204 [SLPQ - 0xc120983c][SLP] fdc0
> 36 c12a8de4 0 0 0 0000204 [SLPQ usbtsk 0xc0780814][SLP]
> usbtask
> 35 c12ab000 0 0 0 0000204 [SLPQ usbevt 0xc12bb210][SLP]
> usb0
> 34 c12ab1fc 0 0 0 0000204 [IWAIT] swi5:+
> 6 c12335f4 0 0 0 0000204 [SLPQ - 0xc1263dc0][SLP] thread
> taskq
> 33 c12337f0 0 0 0 0000204 [IWAIT] swi6:+
> 32 c12339ec 0 0 0 0000204 [IWAIT] swi6: task queue
> 5 c1233be8 0 0 0 0000204 [SLPQ - 0xc128e040][SLP] kqueue
> taskq
> 31 c1233de4 0 0 0 0000204 [IWAIT] swi2: cambio
> 30 c1243000 0 0 0 0000204 [SLPQ - 0xc077e7a0][SLP] yarrow
> 4 c12431fc 0 0 0 0000204 [SLPQ - 0xc0783148][SLP] g_down
> 3 c12433f8 0 0 0 0000204 [SLPQ - 0xc0783144][SLP] g_up
> 2 c12435f4 0 0 0 0000204 [SLPQ - 0xc078313c][SLP]
> g_event
> 29 c12437f0 0 0 0 0000204 [IWAIT] swi3: vm
> 28 c12439ec 0 0 0 000020c [IWAIT] swi4: clock sio
> 27 c12191fc 0 0 0 0000204 [IWAIT] swi1: net
> 26 c12193f8 0 0 0 0000204 [IWAIT] irq15: ata1
> 25 c12195f4 0 0 0 0000204 [IWAIT] irq14: ata0
> 24 c12197f0 0 0 0 0000204 [IWAIT] irq13:
> 23 c12199ec 0 0 0 0000204 [IWAIT] irq12:
> 22 c1219be8 0 0 0 0000204 [IWAIT] irq11:
> 21 c1219de4 0 0 0 0000204 [IWAIT] irq10:
> 20 c1233000 0 0 0 0000204 [IWAIT] irq9: twe0
> 19 c12331fc 0 0 0 0000204 [IWAIT] irq8: rtc
> 18 c12333f8 0 0 0 0000204 [IWAIT] irq7: ppc0
> 17 c1211000 0 0 0 0000204 [IWAIT] irq6: fdc0
> 16 c12111fc 0 0 0 0000204 [IWAIT] irq5: bge0
> 15 c12113f8 0 0 0 0000204 [IWAIT] irq4: sio0
> 14 c12115f4 0 0 0 0000204 [IWAIT] irq3: uhci0
> 13 c12117f0 0 0 0 0000204 [IWAIT] irq1: atkbd0
> 12 c12119ec 0 0 0 0000204 [IWAIT] irq0: clk
> 11 c1211be8 0 0 0 000020c [CPU 0] idle
> 1 c1211de4 0 0 1 0004200 [SLPQ wait 0xc1211de4][SLP]
> init
> 10 c1219000 0 0 0 0000204 [SLPQ ktrace 0xc0783d38][SLP]
> ktrace
> 0 c07832a0 0 0 0 0000200 [SLPQ sched 0xc07832a0][SLP]
> swapper
> 517 c162c5f4 1062 516 517 0006000 zomb[INACTIVE] sh
> db> show pcpu
> cpuid = 0
> curthread = 0xc12125c0: pid 11 "idle"
> curpcb = 0xc69b7da0
> fpcurthread = none
> idlethread = 0xc12125c0: pid 11 "idle"
> APIC ID = 0
> currentldt = 0x28
>
> Joe 'sh' process in ufs:
>
> db> trace 3296
> Tracing pid 3296 tid 100255 td 0xc1cfa730
> sched_switch(c1cfa730,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c13c4a78,cbad385c,c0526901,c13c4a78,0) at
> sleepq_switch+0xe0
> sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30
> msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1
> acquire(cbad38b4,1010040,600,c1cfa730,0) at acquire+0x9a
> lockmgr(c13c4a78,1010002,c13c49b4,c1cfa730,cbad38d4) at lockmgr+0x382
> vop_stdlock(cbad3904) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cbad3904,cbad38f4,c06c279e,cbad3904) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cbad3904) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cbad3904) at VOP_LOCK_APV+0x7e
> vn_lock(c13c49b4,10002,c1cfa730,2,c136a800) at vn_lock+0xc2
> vget(c13c49b4,10002,c1cfa730,c1cfa730,c13c4bdc) at vget+0xc9
> ufs_ihashget(c136a800,2,2,cbad3a00,c06e98f4) at ufs_ihashget+0x98
> ffs_vget(c1370800,2,2,cbad3a00) at ffs_vget+0x37
> ufs_root(c1370800,cbad3a3c,c1cfa730,1,1) at ufs_root+0x18
> lookup(cbad3c34,c05197b6,1,c1cfa730,c06ea9fd) at lookup+0x593
> namei(cbad3c34,c05197b6,c1cf9460,1,0) at namei+0x34e
> do_execve(c1cfa730,cbad3cc4,0,0,cbad3cc4) at do_execve+0x15d
> kern_execve(c1cfa730,cbad3cc4,0,c6295000,c6295000) at kern_execve+0x74
> execve(c1cfa730,cbad3d14,3,0,282) at execve+0x32
> syscall(2f,2f,2f,8067058,806708c) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (59, FreeBSD ELF32, execve), eip = 0x28121d5b, esp =
> 0xbfbfec1c, ebp = 0xbfbfec48 ---
>
> httpd holding a VV_ROOT vnode lock, blocked in ufs:
>
> db> trace 3057
> Tracing pid 3057 tid 100107 td 0xc1807a10
> sched_switch(c1807a10,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c1526964,cb707994,c0526901,c1526964,0) at
> sleepq_switch+0xe0
> sleepq_wait(c1526964,0,0,c1526964,1010040) at sleepq_wait+0x30
> msleep(c1526964,c0785244,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb7079ec,1010040,600,c1807a10,0) at acquire+0x9a
> lockmgr(c1526964,1010002,c15268a0,c1807a10,cb707a0c) at lockmgr+0x382
> vop_stdlock(cb707a3c) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb707a3c,cb707a2c,c06c279e,cb707a3c) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb707a3c) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb707a3c) at VOP_LOCK_APV+0x7e
> vn_lock(c15268a0,10002,c1807a10,cb707b24,cb707c38) at vn_lock+0xc2
> vget(c15268a0,2,c1807a10,1b8,c1807a10) at vget+0xc9
> vfs_cache_lookup(cb707b24) at vfs_cache_lookup+0x1bd
> VOP_LOOKUP_APV(c076ed60,cb707b24) at VOP_LOOKUP_APV+0x7e
> lookup(cb707c10,cb707b70,1,c1807a10,1) at lookup+0x359
> namei(cb707c10,cb707bc4,c0672ebb,c19cbad0,c0703dc4) at namei+0x34e
> kern_stat(c1807a10,81541f8,0,cb707c84) at kern_stat+0x35
> stat(c1807a10,cb707d14,2,0,296) at stat+0x1b
> syscall(280b002f,2f,bfbf002f,81541f8,8170) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp =
> 0xbfbfe9dc, ebp = 0xbfbfea78 ---
>
> download.cgi blocked in ufs holding a directory vnode lock:
>
> db> trace 3062
> Tracing pid 3062 tid 100050 td 0xc137e000
> sched_switch(c137e000,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c163bca0,cb2e5844,c0526901,c163bca0,0) at
> sleepq_switch+0xe0
> sleepq_wait(c163bca0,0,0,c163bca0,1010040) at sleepq_wait+0x30
> msleep(c163bca0,c0784578,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb2e589c,1010040,600,c137e000,0) at acquire+0x9a
> lockmgr(c163bca0,1010002,c163bbdc,c137e000,cb2e58bc) at lockmgr+0x382
> vop_stdlock(cb2e58ec) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb2e58ec,cb2e58dc,c06c279e,cb2e58ec) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb2e58ec) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb2e58ec) at VOP_LOCK_APV+0x7e
> vn_lock(c163bbdc,10002,c137e000,cb2e59d4,cb2e5c04) at vn_lock+0xc2
> vget(c163bbdc,2,c137e000,1e0,c137e000) at vget+0xc9
> vfs_cache_lookup(cb2e59d4) at vfs_cache_lookup+0x1bd
> VOP_LOOKUP_APV(c076ed60,cb2e59d4) at VOP_LOOKUP_APV+0x7e
> lookup(cb2e5bdc,cb2e5a76,1,c137e000,cb2e5a80) at lookup+0x359
> namei(cb2e5bdc,0,d7ec0005,c137e000,c050f370) at namei+0x34e
> vn_open_cred(cb2e5bdc,cb2e5cdc,1a4,c19ace00,3) at vn_open_cred+0x27e
> vn_open(cb2e5bdc,cb2e5cdc,1a4,3,c07849f8) at vn_open+0x1e
> kern_open(c137e000,804a794,0,1,1b6) at kern_open+0xb6
> open(c137e000,cb2e5d14,3,1,292) at open+0x1a
> syscall(2f,2f,2f,4,2814e280) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (5, FreeBSD ELF32, open), eip = 0x280ba23b, esp =
> 0xbfbfe10c,
> ebp = 0xbfbfe138 ---
>
> httpd holding a directory lock blocked in ufs:
>
> db> trace 3056
> Tracing pid 3056 tid 100099 td 0xc162dcf0
> sched_switch(c162dcf0,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c17e9400,cb6be994,c0526901,c17e9400,0) at
> sleepq_switch+0xe0
> sleepq_wait(c17e9400,0,0,c17e9400,1010040) at sleepq_wait+0x30
> msleep(c17e9400,c0784c38,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb6be9ec,1010040,600,c162dcf0,0) at acquire+0x9a
> lockmgr(c17e9400,1010002,c17e933c,c162dcf0,cb6bea0c) at lockmgr+0x382
> vop_stdlock(cb6bea3c) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb6bea3c,cb6bea2c,c06c279e,cb6bea3c) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb6bea3c) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb6bea3c) at VOP_LOCK_APV+0x7e
> vn_lock(c17e933c,10002,c162dcf0,cb6beb24,cb6bec38) at vn_lock+0xc2
> vget(c17e933c,2,c162dcf0,210,c162dcf0) at vget+0xc9
> vfs_cache_lookup(cb6beb24) at vfs_cache_lookup+0x1bd
> VOP_LOOKUP_APV(c076ed60,cb6beb24) at VOP_LOOKUP_APV+0x7e
> lookup(cb6bec10,c17e9450,1,c162dcf0,75c) at lookup+0x359
> namei(cb6bec10,c05197b6,c0798060,1,c06ea9fd) at namei+0x34e
> kern_stat(c162dcf0,8161ad8,0,cb6bec84) at kern_stat+0x35
> stat(c162dcf0,cb6bed14,2,0,292) at stat+0x1b
> syscall(280b002f,2f,bfbf002f,8161ad8,8170) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp =
> 0xbfbfe75c, ebp = 0xbfbfe7f8 ---
>
> httpd holding a file lock, blocked in suspfs from ufs_inactive:
>
> db> trace 3045
> Tracing pid 3045 tid 100123 td 0xc1996450
> sched_switch(c1996450,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c137086c,cb6ecb8c,c0526901,c137086c,0) at
> sleepq_switch+0xe0
> sleepq_wait(c137086c,0,0,1,c153be38) at sleepq_wait+0x30
> msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
> vn_write_suspend_wait(c17e933c,c1370800,1,0,c06f2634) at
> vn_write_suspend_wait+0xe4
> ufs_inactive(cb6ecc14) at ufs_inactive+0x186
> VOP_INACTIVE_APV(c076ed60,cb6ecc14) at VOP_INACTIVE_APV+0x7e
> vrele(c17e933c,c17e7210,0,c07046ce,19d) at vrele+0x10e
> vm_object_vndeallocate(c17e7210) at vm_object_vndeallocate+0xbd
> vm_object_deallocate(c17e7210,c17e7210,0,c0703dc4,89e) at
> vm_object_deallocate+0x85
> vm_map_entry_delete(c19cbbb8,c16258c4,284c8000,c19cbbb8,c16258c4) at
> vm_map_entry_delete+0x130
> vm_map_delete(c19cbbb8,284c8000,284c9000,0,c1996450) at
> vm_map_delete+0x18f
> munmap(c1996450,cb6ecd14,2,3,292) at munmap+0xa3
> syscall(280b002f,2f,bfbf002f,2831e058,815f2d8) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
> 0xbfbfc6fc, ebp = 0xbfbfc718 ---
>
> Another httpd holding a file lock, blocked in suspfs from ufs_inactive:
>
> db> trace 3047
> Tracing pid 3047 tid 100143 td 0xc19b2450
> sched_switch(c19b2450,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c137086c,cb781b8c,c0526901,c137086c,0) at
> sleepq_switch+0xe0
> sleepq_wait(c137086c,0,0,1,c141ec08) at sleepq_wait+0x30
> msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
> vn_write_suspend_wait(c18419b4,c1370800,1,0,c06f2634) at
> vn_write_suspend_wait+0xe4
> ufs_inactive(cb781c14) at ufs_inactive+0x186
> VOP_INACTIVE_APV(c076ed60,cb781c14) at VOP_INACTIVE_APV+0x7e
> vrele(c18419b4,c1826000,0,c07046ce,19d) at vrele+0x10e
> vm_object_vndeallocate(c1826000) at vm_object_vndeallocate+0xbd
> vm_object_deallocate(c1826000,c1826000,0,c0703dc4,89e) at
> vm_object_deallocate+0x85
> vm_map_entry_delete(c19cbe10,c18bfd8c,284cd000,c19cbe10,c18bfd8c) at
> vm_map_entry_delete+0x130
> vm_map_delete(c19cbe10,284cd000,284ce000,0,c19b2450) at
> vm_map_delete+0x18f
> munmap(c19b2450,cb781d14,2,3,292) at munmap+0xa3
> syscall(280b002f,2f,bfbf002f,2831e058,815d228) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
> 0xbfbfc6fc, ebp = 0xbfbfc718 ---
>
> mksnap_ffs, creating the snapshot, but blocked on a vnode lock:
>
> db> trace 2993
> Tracing pid 2993 tid 100096 td 0xc162d8a0
> sched_switch(c162d8a0,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c13c4a78,cb6b560c,c0526901,c13c4a78,0) at
> sleepq_switch+0xe0
> sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30
> msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb6b5664,1010040,600,c162d8a0,0) at acquire+0x9a
> lockmgr(c13c4a78,1010002,c13c49b4,c162d8a0,cb6b5684) at lockmgr+0x382
> vop_stdlock(cb6b56b4) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb6b56b4,cb6b56a4,c06c279e,cb6b56b4) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb6b56b4) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb6b56b4) at VOP_LOCK_APV+0x7e
> vn_lock(c13c49b4,10002,c162d8a0) at vn_lock+0xc2
> ffs_snapshot(c1370800,c14ccbc0,c14ccbc0,c1369b00,0) at
> ffs_snapshot+0x1242
> ffs_mount(c1370800,c162d8a0,20000000,201000,c13c49b4) at
> ffs_mount+0x7a5
> vfs_domount(c162d8a0,c13207b0,c141c030,1211000,c1320670) at
> vfs_domount+0x576
> vfs_donmount(c162d8a0,1211000,cb6b5c14,c1422880,e) at vfs_donmount+0xce
> kernel_mount(c1320690,1211000,bfbfed33,0,0) at kernel_mount+0x6d
> ffs_cmount(c1320690,bfbfe578,1211000,c162d8a0,c076ea80) at
> ffs_cmount+0x5d
> mount(c162d8a0,cb6b5d14,4,2,206) at mount+0x156
> syscall(2f,2f,2f,bfbfe7c8,bfbfed29) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (21, FreeBSD ELF32, mount), eip = 0x280b80db, esp =
> 0xbfbfe4ec, ebp = 0xbfbfebe0 ---
>
> Anothr httpd blocked in suspfs while holding a file vnode lock, trying
> to
> unmap:
>
> db> trace 2609
> Tracing pid 2609 tid 100108 td 0xc1807b80
> sched_switch(c1807b80,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c137086c,cb70ab8c,c0526901,c137086c,0) at
> sleepq_switch+0xe0
> sleepq_wait(c137086c,0,0,1,c14cae38) at sleepq_wait+0x30
> msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
> vn_write_suspend_wait(c19fd9b4,c1370800,1,0,c06f2634) at
> vn_write_suspend_wait+0xe4
> ufs_inactive(cb70ac14) at ufs_inactive+0x186
> VOP_INACTIVE_APV(c076ed60,cb70ac14) at VOP_INACTIVE_APV+0x7e
> vrele(c19fd9b4,c1a93948,0,c07046ce,19d) at vrele+0x10e
> vm_object_vndeallocate(c1a93948) at vm_object_vndeallocate+0xbd
> vm_object_deallocate(c1a93948,c1a93948,0,c0703dc4,89e) at
> vm_object_deallocate+0x85
> vm_map_entry_delete(c18bd834,c18cb3b8,284c7000,c18bd834,c18cb3b8) at
> vm_map_entry_delete+0x130
> vm_map_delete(c18bd834,284c7000,284c8000,0,c1807b80) at
> vm_map_delete+0x18f
> munmap(c1807b80,cb70ad14,2,37,296) at munmap+0xa3
> syscall(2f,2f,bfbf002f,2831e058,815d0c8) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
> 0xbfbfc5cc, ebp = 0xbfbfc5e8 ---
>
> No WITNESS in this kernel.
>
> My first glance reading is that there's a deadlock between the snapshot
> lock and vnode locks, which ocurs because a vnode lock has been
> acquired
> as part of the mmap tear-down, which leads to vn_write_suspend() being
> called in ufs_inactive as the inode has a zero reference count so has
> to
> be GC'd, which involves writing to the file system. VOP_INACTIVE is
> called by vrele() with the vnode lock held. My understanding is that,
> generally, we acquire the file system snapshot lock before vnode locks
> as
> part of the VFS lock order.
>
> This could be a misinterpretation. I'll try to get a dump, but this
> is a
> twe box so I may not be able to.
>
> Robert N M Watson
>
>
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?522f0af570fa6a92db61f6dfdb9a2a33>