Date: Tue, 21 Feb 2006 12:33:57 -0500 From: "Michael R. Wayne" <wayne@staff.msen.com> To: hackers@freebsd.org Subject: Infrequent disk system hang on 5.4-RELEASE-p8 Message-ID: <20060221173357.GE71757@manor.msen.com>
next in thread | raw e-mail | index | archive | help
We have an older server, running 5.4-RELEASE-p8 and used primarily
for email, which hangs every couple of weeks. The hang seems to
be in the disk I/O system. Based on the times of the hangs, the
triggering event seems to be running dump.
We have a serial console set up, I broke to the debugger and got the
following. Since the hang is in the disk I/O system, a dump is not
possible. The many versions of inetd are likely due to users attempting
to POP their email.
Any suggestions or tips on how to track this down and get it resolved
would be appreciated.
Relevant dmesg info:
ahc0: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0xe400-0xe4ff mem 0xffafe000-0xffafefff irq 16 at device 11.0 on pci0
aic7896/97: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0xe800-0xe8ff mem 0xffaff000-0xffafffff irq 16 at device 11.1 on pci0
aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST318436LW 0010> Fixed Direct Access SCSI-3 device
da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da0: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C)
da1 at ahc1 bus 0 target 0 lun 0
da1: <SEAGATE ST336938LW 0003> Fixed Direct Access SCSI-3 device
da1: 80.000MB/s transfers (40.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da1: 35242MB (72176567 512 byte sectors: 255H 63S/T 4492C)
da2 at ahc1 bus 0 target 1 lun 0
da2: <SEAGATE ST318436LW 0010> Fixed Direct Access SCSI-3 device
da2: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da2: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C)
GEOM_MIRROR: Device gm0s1 created (id=520792649).
GEOM_MIRROR: Device gm0s1: provider da0s1 detected.
GEOM_MIRROR: Device gm0s2 created (id=3744871543).
GEOM_MIRROR: Device gm0s2: provider da0s2 detected.
GEOM_MIRROR: Device gm0s1: provider da2s1 detected.
GEOM_MIRROR: Device gm0s1: provider da2s1 activated.
GEOM_MIRROR: Device gm0s1: provider da0s1 activated.
GEOM_MIRROR: Device gm0s1: provider mirror/gm0s1 launched.
GEOM_MIRROR: Device gm0s2: provider da2s2 detected.
GEOM_MIRROR: Device gm0s2: provider da2s2 activated.
GEOM_MIRROR: Device gm0s2: provider da0s2 activated.
GEOM_MIRROR: Device gm0s2: provider mirror/gm0s2 launched.
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
67487 c5ea98d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67486 c3b8a1c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67485 c634c710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67484 c62931c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67483 c58a9388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67482 c6293710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67481 c58ab8d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67480 c6292c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67479 c62938d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67478 c634f000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67477 c62941c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67476 c5e55c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67475 c5f1fe20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67474 c5da854c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67473 c5f9ee20 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67472 c58a9e20 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67471 c602d8d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67470 c61191c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67469 c58ab1c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67468 c5f19a98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67467 c58c3388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67466 c5f1f388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67465 c5f19000 0 442 67465 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd
67464 c5fa68d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67463 c5eab54c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67462 c6294000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67461 c5fa6710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67460 c6119000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67459 c634c388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67458 c5da8710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67457 c3b8e388 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67456 c62948d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67455 c62921c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67454 c5f1954c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67453 c5eab388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67452 c5f171c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67451 c6294388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67450 c60291c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67449 c5ea9710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67448 c3b8ac5c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67447 c6293388 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67446 c5f1fa98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67445 c5e55e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67444 c6117710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67443 c5fa7a98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67442 c6026388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67441 c5e5a1c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67440 c6119710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67439 c5e5a000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67438 c3b8e8d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67437 c6293e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67436 c5dfee20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67435 c5ea8e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67434 c5fa7388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67433 c39dde20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67432 c6118000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67431 c58c31c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67430 c634fe20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67429 c5f191c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67428 c58a9c5c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67427 c63508d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67426 c3b8ee20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67425 c3b8ec5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67424 c5eabc5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67423 c5e5a54c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67422 c39de710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67421 c6117e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67420 c5da454c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67419 c5e551c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67418 c5da88d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67417 c611ce20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67416 c5e55000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67415 c5f19c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67414 c5f198d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67413 c60261c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67412 c6293c5c 0 574 67412 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67411 c58a9a98 0 574 67411 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67410 c38898d4 0 574 67410 0004000 [SLPQ suspfs 0xc37cac6c][SLP] qpopper
67409 c611c54c 0 574 67409 0004000 [SLPQ ufs 0xc38515d4][SLP] qpopper
67408 c634c1c4 0 574 67408 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67407 c5ea9000 0 574 67407 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67406 c58c3000 0 574 67406 0004000 [SLPQ ufs 0xc388d9f4][SLP] qpopper
67405 c5da41c4 2 67404 67401 0004100 [SLPQ ufs 0xc3851c04][SLP] mksnap_ffs
67404 c58c5a98 2 67403 67401 0004000 [SLPQ wait 0xc58c5a98][SLP] sh
67403 c5fa7e20 2 67402 67401 0004000 [SLPQ wait 0xc5fa7e20][SLP] dump
67402 c5fa71c4 2 67401 67401 0004000 [SLPQ piperd 0xc60fb600][SLP] gzip
67401 c5f19710 2 67400 67401 0004000 [SLPQ pause 0xc5f19748][SLP] tcsh
67400 c5e5554c 2 67398 67398 0000100 [SLPQ select 0xc08f1f44][SLP] sshd
67398 c611854c 0 442 67398 0000100 [SLPQ sbwait 0xc6273974][SLP] sshd
67322 c39dea98 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
62457 c39dee20 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
62370 c5f1f710 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
61704 c58c5c5c 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
61031 c58a91c4 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
59856 c5da48d4 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
43502 c58c3710 5147 589 43502 0004002 [SLPQ ufs 0xc3851c04][SLP] tcsh
589 c38848d4 0 1 589 0004102 [SLPQ wait 0xc38848d4][SLP] login
588 c3884c5c 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
587 c39dd000 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
586 c3889e20 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
574 c3884e20 0 1 574 0000000 [SLPQ select 0xc08f1f44][SLP] inetd
551 c39dd388 0 1 551 0000100 [SLPQ select 0xc08f1f44][SLP] sendmail
542 c39dda98 0 1 542 0008080 (threaded) spamass-milter
thread 0xc628e000 ksegrp 0xc353aaf0 [SLPQ kserel 0xc353ab30][SLP]
thread 0xc5fa1900 ksegrp 0xc353aaf0 [SLPQ select 0xc08f1f44][SLP]
thread 0xc39e0900 ksegrp 0xc353a5b0 [SLPQ ksesigwait 0xc39ddb98][SLP]
532 c39de000 25 1 532 0000100 [SLPQ pause 0xc39de038][SLP] sendmail
527 c39ddc5c 0 521 521 0000001 [SLPQ lockf 0xc38193c0][SLP] saslauthd
526 c39dd710 0 521 521 0000001 [SLPQ lockf 0xc35e4c40][SLP] saslauthd
525 c39dd54c 0 521 521 0000001 [SLPQ lockf 0xc37e2580][SLP] saslauthd
524 c39dd1c4 0 521 521 0000001 [SLPQ lockf 0xc3819140][SLP] saslauthd
521 c3889c5c 0 1 521 0000001 [SLPQ accept 0xc3997b9e][SLP] saslauthd
514 c3889a98 0 1 514 0000000 [SLPQ pause 0xc3889ad0][SLP] perl5.8.6
499 c37eda98 65534 1 499 0000100 [SLPQ select 0xc08f1f44][SLP] spamd
493 c3889000 0 1 493 0000000 [SLPQ select 0xc08f1f44][SLP] rpc.dracd
484 c3884388 106 1 484 0008180 (threaded) clamav-milter
thread 0xc5f1e900 ksegrp 0xc3448380 [SLPQ kserel 0xc34483c0][SLP]
thread 0xc6032780 ksegrp 0xc3448380 [SLPQ select 0xc08f1f44][SLP]
thread 0xc5df8780 ksegrp 0xc3448380 [SLPQ ufs 0xc3851c04][SLP]
thread 0xc39d6900 ksegrp 0xc353ae00 [SLPQ ksesigwait 0xc3884488][SLP]
477 c3884710 106 1 477 0000100 [SLPQ pause 0xc3884748][SLP] freshclam
470 c388454c 106 1 470 0008180 (threaded) clamd
thread 0xc5e34900 ksegrp 0xc3448310 [SLPQ kserel 0xc3448350][SLP]
thread 0xc5e56000 ksegrp 0xc3448310 [SLPQ accept 0xc3924e26][SLP]
thread 0xc3aa2000 ksegrp 0xc388abd0 [SLPQ ksesigwait 0xc388464c][SLP]
455 c3889388 0 1 455 0000000 [SLPQ ufs 0xc3851c04][SLP] cron
442 c38841c4 0 1 442 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd
429 c3884000 0 1 429 0000000 [SLPQ select 0xc08f1f44][SLP] ntpd
338 c37ede20 0 1 338 0000000 [SLPQ select 0xc08f1f44][SLP] rpcbind
325 c3884a98 0 1 325 0000000 [SLPQ select 0xc08f1f44][SLP] syslogd
307 c38891c4 0 1 307 0000000 [SLPQ select 0xc08f1f44][SLP] devd
58 c353c54c 0 0 0 0000204 [SLPQ - 0xe67d4d18][SLP] schedcpu
57 c353c710 0 0 0 0000204 [SLPQ - 0xc08f996c][SLP] nfsiod 3
56 c353c8d4 0 0 0 0000204 [SLPQ - 0xc08f9968][SLP] nfsiod 2
55 c353ca98 0 0 0 0000204 [SLPQ - 0xc08f9964][SLP] nfsiod 1
54 c353cc5c 0 0 0 0000204 [SLPQ - 0xc08f9960][SLP] nfsiod 0
53 c353ce20 0 0 0 0000204 [SLPQ vlruwt 0xc353ce20][SLP] vnlru
52 c37ed000 0 0 0 0000204 [SLPQ syncer 0xc08ee6cc][SLP] syncer
51 c37ed1c4 0 0 0 0000204 [SLPQ psleep 0xc08f250c][SLP] bufdaemon
50 c37ed388 0 0 0 000020c [SLPQ pgzero 0xc09002d4][SLP] pagezero
49 c37ed54c 0 0 0 0000204 [SLPQ psleep 0xc0900328][SLP] vmdaemon
48 c37ed710 0 0 0 0000204 [SLPQ psleep 0xc09002e4][SLP] pagedaemon
47 c37ed8d4 0 0 0 0000204 [SLPQ m:w2 0xc37ea000][SLP] g_mirror gm0s2
46 c349ba98 0 0 0 0000204 [SLPQ m:w2 0xc37ea500][SLP] g_mirror gm0s1
45 c349bc5c 0 0 0 0000204 [IWAIT] swi0: sio
44 c349be20 0 0 0 0000204 [SLPQ - 0xc354223c][SLP] fdc0
43 c3538000 0 0 0 0000204 [SLPQ idle 0xc3540e00][SLP] aic_recovery1
9 c35381c4 0 0 0 0000204 [SLPQ idle 0xc3540e00][SLP] aic_recovery1
8 c3538388 0 0 0 0000204 [SLPQ idle 0xc3540400][SLP] aic_recovery0
7 c353854c 0 0 0 0000204 [SLPQ idle 0xc3540400][SLP] aic_recovery0
42 c3538710 0 0 0 0000204 [IWAIT] swi6: task queue
6 c35388d4 0 0 0 0000204 [SLPQ - 0xc352e3c0][SLP] kqueue taskq
41 c3538a98 0 0 0 0000204 [IWAIT] swi3: cambio
40 c3538c5c 0 0 0 0000204 [IWAIT] swi2: camnet
39 c3538e20 0 0 0 0000204 [IWAIT] swi6:+
5 c353c000 0 0 0 0000204 [SLPQ - 0xc352ed80][SLP] thread taskq
38 c348a54c 0 0 0 0000204 [IWAIT] swi6:+
37 c348a710 0 0 0 0000204 [SLPQ - 0xc08e4660][SLP] yarrow
4 c348a8d4 0 0 0 0000204 [SLPQ - 0xc08e8fa8][SLP] g_down
3 c348aa98 0 0 0 0000204 [SLPQ - 0xc08e8fa4][SLP] g_up
2 c348ac5c 0 0 0 0000204 [SLPQ - 0xc08e8f9c][SLP] g_event
36 c348ae20 0 0 0 0000204 [IWAIT] swi1: net
35 c349b000 0 0 0 0000204 [IWAIT] swi4: vm
34 c349b1c4 0 0 0 000020c [IWAIT] swi5: clock sio
33 c349b388 0 0 0 0000204 [IWAIT] irq0: clk
32 c349b54c 0 0 0 0000204 [IWAIT] irq22:
31 c349b710 0 0 0 0000204 [IWAIT] irq21:
30 c349b8d4 0 0 0 0000204 [IWAIT] irq20:
29 c34491c4 0 0 0 0000204 [IWAIT] irq19: fxp0
28 c3449388 0 0 0 0000204 [IWAIT] irq18:
27 c344954c 0 0 0 0000204 [IWAIT] irq17:
26 c3449710 0 0 0 0000204 [IWAIT] irq16: fxp1 ahc0+
25 c34498d4 0 0 0 0000204 [IWAIT] irq15: ata1
24 c3449a98 0 0 0 0000204 [IWAIT] irq14: ata0
23 c3449c5c 0 0 0 0000204 [IWAIT] irq13:
22 c3449e20 0 0 0 0000204 [IWAIT] irq12:
21 c348a000 0 0 0 0000204 [IWAIT] irq11:
20 c348a1c4 0 0 0 0000204 [IWAIT] irq10:
19 c348a388 0 0 0 0000204 [IWAIT] irq9:
18 c3442000 0 0 0 0000204 [IWAIT] irq8: rtc
17 c34421c4 0 0 0 0000204 [IWAIT] irq7: ppc0
16 c3442388 0 0 0 0000204 [IWAIT] irq6: fdc0
15 c344254c 0 0 0 0000204 [IWAIT] irq5:
14 c3442710 0 0 0 0000204 [IWAIT] irq4: sio0
13 c34428d4 0 0 0 0000204 [IWAIT] irq3: sio1
12 c3442a98 0 0 0 0000204 [IWAIT] irq1: atkbd0
11 c3442c5c 0 0 0 000020c [CPU 0] idle
1 c3442e20 0 0 1 0004200 [SLPQ wait 0xc3442e20][SLP] init
10 c3449000 0 0 0 0000204 [SLPQ ktrace 0xc08ec8f8][SLP] ktrace
0 c08e90a0 0 0 0 0000200 [SLPQ sched 0xc08e90a0][SLP] swapper
db> where
Tracing pid 11 tid 100003 td 0xc3443480
kdb_enter(c08479e3) at kdb_enter+0x2b
siointr1(c35d6000) at siointr1+0xd5
siointr(c35d6000) at siointr+0x38
intr_execute_handlers(c343dc90,e4d53cc8,4,e4d53d0c,c07b9483) at intr_execute_handlers+0x7d
lapic_handle_intr(34) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc07c057d, esp = 0xe4d53d0c, ebp = 0xe4d53d0c ---
cpu_idle_default(e4d53d20,c0604971,c3442c5c,e4d53d34,c0604720) at cpu_idle_default+0x5
cpu_idle(c3442c5c,e4d53d34,c0604720,0,e4d53d48) at cpu_idle+0x1f
idle_proc(0,e4d53d48) at idle_proc+0x11
fork_exit(c0604960,0,e4d53d48) at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4d53d7c, ebp = 0 ---
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060221173357.GE71757>