Date: Sat, 21 Feb 2004 01:21:23 +1100 From: Tony Frank <tfrank@optushome.com.au> To: freebsd-stable@freebsd.org Subject: ahc+vinum raid5 deadlock after msg: devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)! Message-ID: <20040220142123.GA44477@marvin.home.local> In-Reply-To: <20040219130409.GA51615@marvin.home.local> References: <20040219052336.GA28162@marvin.home.local> <20040219130409.GA51615@marvin.home.local>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi once more,
On Fri, Feb 20, 2004 at 12:04:09AM +1100, Tony Frank wrote:
> Some more updates on my problems.
I've rebuilt my world & kernel yet again from more recent stable.
No obvious changes that would affect my setup though.
This time I included the make.conf CPUTYPE=i686 flag.
make.conf options are:
WITHOUT_X11= true
IPFW2= true
CPUTYPE=i686
CFLAGS= -O -pipe
CXXFLAGS+= -fmemoize-lookups -fsave-memoized
BDECFLAGS= -W -Wall -ansi -pedantic -Wbad-function-cast -Wcast-align \
-Wcast-qual -Wchar-subscripts -Winline \
-Wmissing-prototypes -Wnested-externs -Wpointer-arith \
-Wredundant-decls -Wshadow -Wstrict-prototypes -Wwrite-strings
COPTFLAGS= -O -pipe
NOPROFILE= true # Avoid compiling profiled libraries
BOOT_COMCONSOLE_SPEED= 115200
USA_RESIDENT= NO
Based on my readings I'm not sure that this particular option will have
any impact on the kernel itself.
I am also rebuilding world without any of the make.conf options - I'll be
trying that combo tomorrow along with some vinum debugging.
As per the last two posts I had been seeing panic in relation to
ahc driver.
My new kernel has DDB compiled in, hence the 'trace' outputs in other posts.
One problem I have is that once it's in 'panic' mode I'm not sure how
exactly to force it to dump core and reboot?
I worked out the 'reboot' part from kernel debugging in handbook: "call boot(0)"
But how to force it to dump to disk so I get a vmcore file to poke at further?
Given that I have extremely limited skills at poking a core I doubt it would
help me much, but at least then I could have something to look at.
Anyway that problem aside the system behaviour has changed.
Now I no longer get a panic however the system seems to freeze/wedge/hang
during the same 'high load' test bed.
Currently the easy way to reproduce this is to try and create a local source
tree via cvsup, but just trying to untar a archive does the same thing so I
am ruling out network related issues.
I was using a Adaptec 2940UW/B (aic7880) but have switched to a known working
Adaptec 2940W (aic7870) and I get the same deadlock scenario occuring.
As per subject I get following kind of output when it hangs:
%%%%
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
%%%%
When this occurs I am writing to a vinum raid5 volume consisting of 5
subdisks, 4 on da0-da3 and 1 on ad0.
There were no problems running 'vinum init' on the volume, nor in any
of my benchmark tests (rawio, bonnie, dd)
One symptom that may be related is that when I copy my large tar file
to the volume it seems to be corrupted - md5 checksums no longer match and
tar has errors trying to extract.
This only occurs on the one system however - I 'duplicated' the scenario
with a vmware guest system using the same source (although generic kernel)
Anyway while system is in this 'hung' state I have no response on console
(I can drop into ddb but nothing else)
If I press "ctrl-t" I get back:
load: 0.00 cmd: mbmon 204 [flswai] 0.01u 0.08s 0% 500k
Network connections remain up (ssh & tcp keepalives) but cannot do anything.
I can create new SSH connections to the box which get to the point of
starting a shell and then 'hang'.
If I send certain remote commands however it does work, eg:
%%%%
tony@marvin [570]:~> ssh tony@raider 'ps -axlhu'
Password:
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
0 349 118 9 2 0 5336 1780 sbwait S ?? 0:00.14 sshd: tony [priv root 349 3.2 1.4 5336 1780 ?? S 12:43AM 0:00.14 sshd: tony [priv
0 1 0 0 10 0 548 216 wait ILs ?? 0:00.02 /sbin/init -- root 1 0.0 0.2 548 216 ?? ILs 10:02AM 0:00.02 /sbin/init --
0 2 0 0 10 0 0 0 tqthr DL ?? 0:00.00 (taskqueue) root 2 0.0 0.0 0 0 ?? DL 10:02AM 0:00.00 (taskqueue)
0 3 0 0 28 0 0 0 idle DL ?? 0:00.00 (aic_recovery0) root 3 0.0 0.0 0 0 ?? DL 10:02AM 0:00.00 (aic_recovery0)
0 4 0 0 28 0 0 0 idle DL ?? 0:00.00 (aic_recovery0) root 4 0.0 0.0 0 0 ?? DL 10:02AM 0:00.00 (aic_recovery0)
0 5 0 0 -18 0 0 0 psleep DL ?? 0:01.12 (pagedaemon) root 5 0.0 0.0 0 0 ?? DL 10:02AM 0:01.12 (pagedaemon)
0 6 0 9 18 0 0 0 psleep DL ?? 0:00.00 (vmdaemon) root 6 0.0 0.0 0 0 ?? DL 10:02AM 0:00.00 (vmdaemon)
0 7 0 0 -6 0 0 0 vrlock DL ?? 0:17.02 (bufdaemon) root 7 0.0 0.0 0 0 ?? DL 10:02AM 0:17.02 (bufdaemon)
0 8 0 0 -6 0 0 0 vrlock DL ?? 0:12.81 (syncer) root 8 0.0 0.0 0 0 ?? DL 10:02AM 0:12.81 (syncer)
0 9 0 0 -2 0 0 0 vlruwt DL ?? 0:00.21 (vnlru) root 9 0.0 0.0 0 0 ?? DL 10:02AM 0:00.21 (vnlru)
0 29 1 157 18 0 212 68 pause Is ?? 0:00.00 adjkerntz -i root 29 0.0 0.1 212 68 ?? Is 10:02AM 0:00.00 adjkerntz -i
0 69 1 0 10 0 924 552 wait Is ?? 0:00.01 /sbin/dhclient f root 69 0.0 0.4 924 552 ?? Is 11:02PM 0:00.01 /sbin/dhclient f
0 96 1 0 -2 0 996 604 flswai Ds ?? 0:00.94 /usr/sbin/syslog root 96 0.0 0.5 996 604 ?? Ds 11:02PM 0:00.94 /usr/sbin/syslog
0 101 1 0 -2 0 1328 792 flswai Ds ?? 0:00.50 /usr/sbin/ntpd - root 101 0.0 0.6 1328 792 ?? Ds 11:02PM 0:00.50 /usr/sbin/ntpd -
0 106 1 144 10 0 212 28 nfsidl I ?? 0:00.00 nfsiod -n 4 root 106 0.0 0.0 212 28 ?? I 11:02PM 0:00.00 nfsiod -n 4
0 107 1 144 10 0 212 28 nfsidl I ?? 0:00.00 nfsiod -n 4 root 107 0.0 0.0 212 28 ?? I 11:02PM 0:00.00 nfsiod -n 4
0 108 1 144 10 0 212 28 nfsidl I ?? 0:00.00 nfsiod -n 4 root 108 0.0 0.0 212 28 ?? I 11:02PM 0:00.00 nfsiod -n 4
0 109 1 144 10 0 212 28 nfsidl I ?? 0:00.00 nfsiod -n 4 root 109 0.0 0.0 212 28 ?? I 11:02PM 0:00.00 nfsiod -n 4
0 116 1 0 10 0 1024 648 nanslp Is ?? 0:00.08 /usr/sbin/cron root 116 0.0 0.5 1024 648 ?? Is 11:02PM 0:00.08 /usr/sbin/cron
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
0 118 1 0 2 0 2636 1512 select Ss ?? 0:01.41 /usr/sbin/sshd root 118 0.0 1.2 2636 1512 ?? Ss 11:02PM 0:01.41 /usr/sbin/sshd
0 150 1 0 10 0 1276 804 wait Is v0 0:00.08 login [pam] (log root 150 0.0 0.6 1276 804 v0 Is 11:02PM 0:00.08 login [pam] (log
0 151 1 0 10 0 1276 804 wait Is v1 0:00.07 login [pam] (log root 151 0.0 0.6 1276 804 v1 Is 11:02PM 0:00.07 login [pam] (log
0 152 1 0 10 0 1276 792 wait Is d0 0:00.09 login [pam] (log root 152 0.0 0.6 1276 792 d0 Is 11:02PM 0:00.09 login [pam] (log
0 159 118 0 2 0 5340 1704 sbwait I ?? 0:00.14 sshd: tony [priv root 159 0.0 1.4 5340 1704 ?? I 11:10PM 0:00.14 sshd: tony [priv
1001 161 159 0 -2 0 5336 1744 flswai D ?? 0:00.72 sshd: tony@ttyp0 tony 161 0.0 1.4 5336 1744 ?? D 11:10PM 0:00.72 sshd: tony@ttyp0
1001 162 161 1 3 0 1332 856 ttyin Is+ p0 0:00.10 -tcsh (tcsh) tony 162 0.0 0.7 1332 856 p0 Is+ 11:10PM 0:00.10 -tcsh (tcsh)
1001 169 150 0 -2 0 1928 1020 flswai D+ v0 0:08.06 top tony 169 0.0 0.8 1928 1020 v0 D+ 11:11PM 0:08.06 top
1001 172 151 0 -2 0 1520 1044 flswai D+ v1 0:09.02 systat -vmstat tony 172 0.0 0.8 1520 1044 v1 D+ 11:11PM 0:09.02 systat -vmstat
1001 196 152 0 18 0 1328 836 pause I d0 0:00.08 -tcsh (tcsh) tony 196 0.0 0.7 1328 836 d0 I 11:39PM 0:00.08 -tcsh (tcsh)
0 204 196 0 -2 0 932 500 flswai D+ d0 0:00.10 mbmon -t 60 root 204 0.0 0.4 932 500 d0 D+ 11:40PM 0:00.10 mbmon -t 60
0 205 118 4 2 0 5340 1708 sbwait I ?? 0:00.16 sshd: tony [priv root 205 0.0 1.4 5340 1708 ?? I 11:42PM 0:00.16 sshd: tony [priv
1001 207 205 0 2 0 5336 1748 select I ?? 0:15.99 sshd: tony@ttyp1 tony 207 0.0 1.4 5336 1748 ?? I 11:42PM 0:15.99 sshd: tony@ttyp1
1001 208 207 0 18 0 1332 860 pause Is p1 0:00.09 -tcsh (tcsh) tony 208 0.0 0.7 1332 860 p1 Is 11:42PM 0:00.09 -tcsh (tcsh)
1001 214 208 0 -2 0 2704 2300 flswai D+ p1 3:03.99 cvsup stable-sup tony 214 0.0 1.8 2704 2300 p1 D+ 11:42PM 3:03.99 cvsup stable-sup
0 228 118 8 2 0 5344 1696 sbwait I ?? 0:00.15 sshd: tony [priv root 228 0.0 1.4 5344 1696 ?? I 11:56PM 0:00.15 sshd: tony [priv
1001 231 228 0 -2 0 5336 1740 flswai D ?? 0:00.68 sshd: tony@ttyp2 tony 231 0.0 1.4 5336 1740 ?? D 11:56PM 0:00.68 sshd: tony@ttyp2
1001 232 231 0 18 0 1332 868 pause Is p2 0:00.08 -tcsh (tcsh) tony 232 0.0 0.7 1332 868 p2 Is 11:56PM 0:00.08 -tcsh (tcsh)
0 236 1 0 2 0 1732 1320 select Is ?? 0:00.31 screen root 236 0.0 1.1 1732 1320 ?? Is 11:56PM 0:00.31 screen
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
1001 237 236 0 18 0 1336 864 pause Is p3 0:00.10 /bin/tcsh tony 237 0.0 0.7 1336 864 p3 Is 11:56PM 0:00.10 /bin/tcsh
1001 255 237 0 -6 0 516 368 pipdwt I+ p3 0:02.32 tar -cf - -C /us tony 255 0.0 0.3 516 368 p3 I+ 11:58PM 0:02.32 tar -cf - -C /us
1001 256 237 1 -2 0 504 360 flswai D+ p3 0:03.36 tar xpf - -C /da tony 256 0.0 0.3 504 360 p3 D+ 11:58PM 0:03.36 tar xpf - -C /da
1001 257 232 0 18 0 1608 1116 pause S+ p2 0:00.04 screen -RD tony 257 0.0 0.9 1608 1116 p2 S+ 11:59PM 0:00.04 screen -RD
1001 258 236 0 18 0 1340 896 pause Is p4 0:00.11 /bin/tcsh tony 258 0.0 0.7 1340 896 p4 Is 11:59PM 0:00.11 /bin/tcsh
1001 274 258 0 -2 0 1728 1152 flswai D+ p4 0:00.05 cvsup current-su tony 274 0.0 0.9 1728 1152 p4 D+ 12:00AM 0:00.05 cvsup current-su
1001 275 236 0 18 0 1332 860 pause Is p5 0:00.07 /bin/tcsh tony 275 0.0 0.7 1332 860 p5 Is 12:00AM 0:00.07 /bin/tcsh
1001 276 275 0 3 0 1508 1184 ttyin I+ p5 0:00.05 ftp 192.168.3.2 tony 276 0.0 0.9 1508 1184 p5 I+ 12:00AM 0:00.05 ftp 192.168.3.2
0 291 118 0 -2 0 5344 1812 flswai D ?? 0:00.16 sshd: tony [priv root 291 0.0 1.4 5344 1812 ?? D 12:18AM 0:00.16 sshd: tony [priv
1001 294 291 0 2 0 5336 1848 sbwait I ?? 0:00.02 sshd: tony@ttyp6 tony 294 0.0 1.5 5336 1848 ?? I 12:18AM 0:00.02 sshd: tony@ttyp6
1001 295 294 0 -2 0 5336 1844 flswai Ds+ p6 0:00.01 sshd: tony@ttyp6 tony 295 0.0 1.5 5336 1844 p6 Ds+ 12:18AM 0:00.01 sshd: tony@ttyp6
0 296 118 0 -2 0 5344 1812 flswai D ?? 0:00.16 sshd: tony [priv root 296 0.0 1.4 5344 1812 ?? D 12:19AM 0:00.16 sshd: tony [priv
1001 299 296 0 -2 0 5336 1844 flswai D ?? 0:00.02 sshd: tony@ttyp7 tony 299 0.0 1.5 5336 1844 ?? D 12:19AM 0:00.02 sshd: tony@ttyp7
1001 300 299 0 -2 0 5336 1844 flswai Ds+ p7 0:00.01 sshd: tony@ttyp7 tony 300 0.0 1.5 5336 1844 p7 Ds+ 12:19AM 0:00.01 sshd: tony@ttyp7
0 337 69 0 10 0 644 228 wait I ?? 0:00.01 /bin/sh /sbin/dh root 337 0.0 0.2 644 228 ?? I 12:39AM 0:00.01 /bin/sh /sbin/dh
0 338 337 0 -2 0 900 516 flswai D ?? 0:00.01 /usr/bin/logger root 338 0.0 0.4 900 516 ?? D 12:39AM 0:00.01 /usr/bin/logger
1001 352 349 12 2 0 5336 1832 select S ?? 0:00.02 sshd: tony@notty tony 352 0.0 1.5 5336 1832 ?? S 12:43AM 0:00.02 sshd: tony@notty
1001 353 352 16 18 0 1108 692 pause Ss ?? 0:00.04 tcsh -c ps -axlh tony 353 0.0 0.6 1108 692 ?? Ss 12:43AM 0:00.04 tcsh -c ps -axlh
0 0 0 9 -18 0 0 0 sched DLs ?? 0:00.00 (swapper) root 0 0.0 0.0 0 0 ?? DLs 10:02AM 0:00.00 (swapper)
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
1001 354 353 16 30 0 456 244 - R ?? 0:00.00 ps -axlhu tony 354 0.0 0.2 456 244 ?? R 12:43AM 0:00.00 ps -axlhu
%%%%
This output shows me lots of processes are in 'flswai' and some of the kernel
stuff seems to be 'vrlock'
This seems to match symptoms of a vinum bug from a few years back:
(from http://www.vinumvm.org/vinum/bugs.html )
28 September 1999: We have seen hangs when perform heavy I/O to RAID-5 plexes. The symptoms are that processes hang waiting on vrlock and flswai. Use ps lax to display this information.
Technical explanation: A deadlock arose between code locking stripes on a RAID-5 plex (vrlock) and code waiting for buffers to be freed (flswai).
Status: Fix committed to FreeBSD-CURRENT on 5 January 2000 and to -STABLE on 11 May 2000.
As a test I am able to get via ssh remote command and run just about anything, so I tried 'sync'
Result (on console):
%%%%
Temp.= 29.0, 0.0, 0.0; Rot.= 0, 5192, 0
Vcore = 2.77, 2.77; Volt. = 3.46, 5.05, 11.67, -11.63, -0.10
Sat Feb 21 00:01:01 EST 2004
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
load: 0.00 cmd: mbmon 204 [flswai] 0.01u 0.08s 0% 500k
Temp.= 30.0, 0.0, 0.0; Rot.= 0, 5192, 0
Vcore = 2.77, 2.75; Volt. = 3.46, 5.05, 11.67, -11.74, -0.10
Sat Feb 21 00:50d:01 EST 2004
evstat_end_transaction: HELP!! busy_count for da0 is < 0 (-1)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-2)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-2)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-2)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-2)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-2)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-2)!
devstat_end_transaction: HELP!! busy_count for da0 is < 0 (-3)!
%%%%
Some playing I did in ddb mode - not sure whats useful but doesnt hurt none.
%%%%
db> ps
pid proc addr uid ppid pgrp flag stat wmesg wchan cmd
455 cd02d8e0 cd084000 1001 454 454 004004 3 vrlock c1048018 sync
454 cd02d740 cd090000 1001 1 454 2004084 3 pause cd090260 tcsh
449 cd02c700 cd0c4000 1001 448 448 004004 3 vrlock c1048018 sync
448 cd02c8a0 cd0b4000 1001 1 448 2004084 3 pause cd0b4260 tcsh
440 cd02cd80 cd09b000 0 439 439 004104 3 flswai c03196f0 su
439 cd02c560 cd0a5000 1001 1 439 2004084 3 pause cd0a5260 tcsh
407 cd02cf20 cd051000 1001 406 406 004004 3 flswai c03196f0 vinum
406 cd02dc20 cd074000 1001 1 406 2004084 3 pause cd074260 tcsh
397 c7f7b6c0 ccff7000 1001 396 396 004004 3 vrlock c1048018 sync
396 c7f7c560 ccdac000 1001 1 396 2004084 3 pause ccdac260 tcsh
390 cd02d0c0 cd0ae000 1001 389 389 004004 3 vrlock c1048018 sync
389 cd02d260 cd0aa000 1001 1 389 2004084 3 pause cd0aa260 tcsh
384 cd02d400 cd097000 1001 383 383 004004 3 vrlock c1048018 sync
383 cd02d5a0 cd088000 1001 1 383 2004084 3 pause cd088260 tcsh
337 cd02da80 cd077000 0 69 69 004004 3 flswai c03196f0 sh
300 cd02ddc0 cd06d000 1001 1 300 002186 3 ttywai c1550e6c sshd
296 cd02e100 cd05c000 0 118 118 000104 3 flswai c03196f0 sshd
295 cd02e2a0 cd07c000 1001 1 295 000106 3 flswai c03196f0 sshd
291 cd02eac0 cd040000 0 118 118 000104 3 flswai c03196f0 sshd
276 cd02e780 cd059000 1001 275 276 004086 3 ttywai c110b26c ftp
275 cd02e440 cd069000 1001 236 275 2004086 3 pause cd069260 tcsh
274 cd02e5e0 cd065000 1001 258 274 004006 3 flswai c03196f0 cvsup
258 cd02ec60 cd034000 1001 236 258 2004086 3 pause cd034260 tcsh
257 c7f7b380 ccfff000 1001 232 257 2004186 3 pause ccfff260 screen
256 cd02ee00 cd02f000 1001 237 255 004006 3 flswai c03196f0 tar
255 c7f7aea0 cd01e000 1001 237 255 004086 3 pipdwt ccd30ac0 tar
237 c7f7b040 cd01b000 1001 236 237 2004086 3 pause cd01b260 tcsh
236 c7f7b1e0 cd017000 1001 1 236 000104 3 flswai c03196f0 screen
232 c7f7b520 cd00a000 1001 1 232 004006 3 flswai c03196f0 tcsh
214 c7f7b860 ccfe6000 1001 208 214 004006 3 flswai c03196f0 cvsup
208 c7f7bba0 ccfde000 1001 207 208 2004086 3 pause ccfde260 tcsh
207 c7f7bd40 ccdca000 1001 205 118 000104 3 flswai c03196f0 sshd
205 c7f7bee0 ccdd5000 0 118 118 000184 3 sbwait cc5b93c8 sshd
204 c7f7ba00 ccfe9000 1001 196 204 004106 3 flswai c03196f0 mbmon
196 c7f7c080 ccdc1000 1001 152 196 2004086 3 pause ccdc1260 tcsh
172 c7f7c220 ccdb7000 1001 151 172 004106 3 flswai c03196f0 systat
169 c7f7c3c0 ccdb3000 1001 150 169 004106 3 flswai c03196f0 top
162 c7f7ddc0 ccd2a000 1001 161 162 004086 3 ttyin c0fa9a30 tcsh
161 c7f7ca40 ccd97000 1001 159 118 000104 3 flswai c03196f0 sshd
159 c7f7c8a0 ccd9c000 0 118 118 000184 3 sbwait cc5b9788 sshd
152 c7f7d5a0 ccd6b000 0 1 152 004186 3 wait c7f7d5a0 login
151 c7f7c700 ccda0000 0 1 151 004186 3 wait c7f7c700 login
150 c7f7df60 ccd22000 0 1 150 004186 3 wait c7f7df60 login
118 c7f7cbe0 ccd8c000 0 1 118 000184 3 select c03303c8 sshd
116 c7f7cd80 ccd7f000 0 1 116 000084 3 nanslp c03194f0 cron
109 c7f7cf20 ccd7b000 0 1 104 000084 3 nfsidl c033690c nfsiod
108 c7f7d0c0 ccd77000 0 1 104 000084 3 nfsidl c0336908 nfsiod
107 c7f7d260 ccd73000 0 1 104 000084 3 nfsidl c0336904 nfsiod
106 c7f7d400 ccd6f000 0 1 104 000084 3 nfsidl c0336900 nfsiod
101 c7f7dc20 ccd32000 0 1 101 000084 3 select c03303c8 ntpd
96 c7f7da80 ccd36000 0 1 96 000004 3 flswai c03196f0 syslogd
69 c7f7d740 ccd3f000 0 1 69 000084 3 wait c7f7d740 dhclient
29 c7f7d8e0 ccd3a000 0 1 29 2000084 3 pause ccd3a260 adjkerntz
9 c7f7e100 ccaba000 0 0 0 000204 3 vlruwt c7f7e100 vnlru
8 c7f7e2a0 ccab7000 0 0 0 000204 3 vrlock c1048008 syncer
7 c7f7e440 ccab4000 0 0 0 000204 3 vrlock c1048010 bufdaemon
6 c7f7e5e0 ccab1000 0 0 0 000204 3 psleep c03274c0 vmdaemon
5 c7f7e780 ccaae000 0 0 0 000204 3 psleep c030c0b8 pagedaemon
4 c7f7e920 cc5b4000 0 0 0 000204 3 idle c0fa9200 aic_recovery0
3 c7f7eac0 cc5b1000 0 0 0 000204 3 idle c0fa9200 aic_recovery0
2 c7f7ec60 c8566000 0 0 0 000204 3 tqthr c03303c4 taskqueue
1 c7f7ee00 c7f83000 0 0 1 004284 3 wait c7f7ee00 init
0 c032f6c0 c0481000 0 0 0 000204 3 sched c032f6c0 swapper
294 cd02e920 cd04d000 1001 291 118 002104 5 sshd
299 cd02df60 cd060000 1001 296 118 002104 5 sshd
db> trace 274 (cvsup)
mi_switch(0,cd067ed4,cd067f80,c166e000,68c040) at mi_switch+0x174
tsleep(c03196f0,14,c02c397d,0) at tsleep+0x1a1
bwillwrite(0,c166c380,cd067f80,ccfb40c0,cd067e1c) at bwillwrite+0x5f
vn_open(cd067ed4,603,1a4,cd02e5e0,3) at vn_open+0x51
open(cd02e5e0,cd067f80,827da24,bfbff7b0,bfbff7bc) at open+0xb8
syscall2(bfbf002f,bfbf002f,820002f,bfbff7bc,bfbff7b0) at syscall2+0x1a9
Xint0x80_syscall() at Xint0x80_syscall+0x25
db> trace 9 (vnlru)
mi_switch(135de,0,2df,c3846760,68c040) at mi_switch+0x174
tsleep(c7f7e100,14,c02c4db6,3e8,c0319810) at tsleep+0x1a1
vnlru_proc(0) at vnlru_proc+0x94
fork_trampoline() at fork_trampoline+0x8
db> trace 8 (syncer)
mi_switch(c1048008,c1072820,2,2,68c040) at mi_switch+0x174
tsleep(c1048008,10,c03af250,0,c10718fc) at tsleep+0x1a1
lockrange(1a96520,c38b86a8,c1072820,c10718fc,c1662340) at lockrange+0xd4
launch_requests(c1662340,0,c38b86a8,ccaac780,c13a0e00) at launch_requests+0x1f5
vinumstart(c38b86a8,0,c38b86a8,ccab9dd8,c01c3631) at vinumstart+0x19a
vinumstrategy(c38b86a8,c105b600,c38b86a8,1,ccab9de4) at vinumstrategy+0x92
spec_strategy(ccab9e08,ccab9df0,c02567c1,ccab9e08,ccab9e24) at spec_strategy+0x8d
spec_vnoperate(ccab9e08,ccab9e24,c0256091,ccab9e08,c38b86a8) at spec_vnoperate+0x15
ufs_vnoperatespec(ccab9e08,c38b86a8,1,68c040,c02eeba0) at ufs_vnoperatespec+0x15
ufs_strategy(ccab9e4c,ccab9e58,c01af927,ccab9e4c,1c00) at ufs_strategy+0xc5
ufs_vnoperate(ccab9e4c) at ufs_vnoperate+0x15
bwrite(c38b86a8,ccab9e70,c01b52d9,ccab9eb0,ccab9e7c) at bwrite+0x20f
vop_stdbwrite(ccab9eb0,ccab9e7c,c0256789,ccab9eb0,ccab9ebc) at vop_stdbwrite+0xf
vop_defaultop(ccab9eb0,ccab9ebc,c01b08b3,ccab9eb0,c38b86a8) at vop_defaultop+0x15
db> trace 7 (bufdaemon)
mi_switch(c1048010,c1072820,3,2,68c040) at mi_switch+0x174
tsleep(c1048010,10,c03af250,0,c10718fc) at tsleep+0x1a1
lockrange(c1ede0,c3881760,c1072820,c10718fc,c1664a80) at lockrange+0xd4
launch_requests(c1664a80,0,c3881760,ccaac780,c14ab900) at launch_requests+0x1f5
%%%%
Ideas welcome,
Tony
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040220142123.GA44477>