From owner-freebsd-current@FreeBSD.ORG Sun Jun 13 05:03:33 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D9E116A4CE for ; Sun, 13 Jun 2004 05:03:33 +0000 (GMT) Received: from mail.sandvine.com (sandvine.com [199.243.201.138]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7D25543D45 for ; Sun, 13 Jun 2004 05:03:32 +0000 (GMT) (envelope-from don@sandvine.com) Received: by mail.sandvine.com with Internet Mail Service (5.5.2657.72) id ; Sun, 13 Jun 2004 00:45:49 -0400 Message-ID: From: Don Bowman To: 'Bruce Evans' , Don Bowman Date: Sun, 13 Jun 2004 00:45:42 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2657.72) Content-Type: text/plain; charset="iso-8859-1" cc: "'current@freebsd.org'" Subject: RE: kernel trap 19 with interrupts disabled: system hang X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2004 05:03:33 -0000 From: Bruce Evans [mailto:bde@zeta.org.au] > > On Wed, 9 Jun 2004, Don Bowman wrote: > > > I have a machine which is completely locking > > up solid every day or so. Its been doing this > > for a couple of months on current. It is running > > cvs current from ~2weeks ago. > > > > This time, i tried shorting the NMI out, and I > > got this message to the serial console: > > > > kernel trap 19 with interrupts disabled > > NMI ... going to debugger > > > > ... but I still can't get into the debugger > > with the key sequence, and no additional > > output came out. > > > > Can I assume from the 'with interrupts disabled' > > that it means that all interrupts are locked off? > > or that 'sti' is set? Its a MP system, a dual > > xeon (P4). > ... OK, this did the trick, i got into db. I managed to get a vmcore, and there is a kernel.debug avail. I included below the output of some commands i ran in db, can someone suggest what else i might run next time? Is anyone interested in having access to this vmcore & kernel.debug? Or can suggest where i might look in it to find this lockup? The system was locked up, so when i pressed the key sequence to enter the debugger, it timed out stopping the other cpus. Everybody is in sched_switch and idle??? Script started on Sun Jun 13 00:26:51 2004 Trying 192.168.1.8... Connected to dc-ts.sandvine.com. Escape character is '^]'. siointr1(c554d800) at siointr1+0xd0 db> t 0 sched_switch(c074bfa0) at sched_switch+0x60 mi_switch(1,0,1,c0c21d2c,c0562ba4) at mi_switch+0x1a0 sleepq_switch(c074bde0,0,c0c21d54,c054dd12,c074bde0) at sleepq_switch+0x135 sleepq_timedwait(c074bde0,0,23,0,0) at sleepq_timedwait+0xc msleep(c074bde0,0,44,c06ecd01,2710) at msleep+0x40a scheduler(0,c1ec00,c1e000,0,c0436065) at scheduler+0x167 mi_startup() at mi_startup+0x96 begin() at begin+0x2c db> t 1 sched_switch(c53e0540) at sched_switch+0x60 mi_switch(1,0,0,ed097c18,c0562b60) at mi_switch+0x1a0 sleepq_switch(c53dfdc0,0,ed097c3c,c054dd2a,c53dfdc0) at sleepq_switch+0x135 sleepq_wait_sig(c53dfdc0,c074fed0,100,0,c53dfe2c) at sleepq_wait_sig+0xc msleep(c53dfdc0,c53dfe2c,15c,c06dde84,0) at msleep+0x422 kern_wait(c53e0540,ffffffff,ed097c94,0,ed097c98) at kern_wait+0x7b5 wait4(c53e0540,ed097d14,4,7,286) at wait4+0x1f syscall(2f,2f,2f,bfbfeef8,bfbfeef8) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x80518cb, esp = 0xbfbfeddc, ebp = 0xbfbfedf8 --- db> t 2 sched_switch(c54ab7e0) at sched_switch+0x60 mi_switch(1,0,0,eec7accc,c0562ba4) at mi_switch+0x1a0 sleepq_switch(c074bce4,0,eec7acf4,c054dd12,c074bce4) at sleepq_switch+0x135 sleepq_timedwait(c074bce4,0,0,0,0) at sleepq_timedwait+0xc msleep(c074bce4,0,4c,c06c7155,64) at msleep+0x40a g_event_procbody(0,eec7ad48) at g_event_procbody+0x52 fork_exit(c05156dc,0,eec7ad48) at fork_exit+0x71 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xeec7ad7c, ebp = 0 --- db> t 3 sched_switch(c54ab930) at sched_switch+0x60 mi_switch(1,0,0,eec7dc94,c0562ba4) at mi_switch+0x1a0 sleepq_switch(c074bcec,0,eec7dcbc,c054dd12,c074bcec) at sleepq_switch+0x135 sleepq_timedwait(c074bcec,0,23,0,0) at sleepq_timedwait+0xc msleep(c074bcec,c074bc08,24c,c06c7155,64) at msleep+0x40a g_io_schedule_up(c54ab930) at g_io_schedule_up+0x140 g_up_procbody(0,eec7dd48) at g_up_procbody+0x1a fork_exit(c051569c,0,eec7dd48) at fork_exit+0x71 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xeec7dd7c, ebp = 0 --- db> p ticks c074c4b4 db> p *ticks 33f08b0 db> p/i ticks db> x ticks ticks: 33f08b0 db> x softticks softticks: 328c76a db> help print p examine x search set write w delete d break dwatch watch dhwatch hwatch step s continue c until next match trace where call show ps gdb reset kill watchdog panic ahd_set_unitahd_pause ahd_unpause ahd_in ahd_out db> show all registers breaks thread watches pciregs intrcnt pgrpdump pcpu msgbuf cbstat buffer disk lockedvnods map procvm vmochk object vmopag page pageq cyrixreg irqs idt rtc db> show all procs pid proc uarea uid ppid pgrp flag stat wmesg wchan cmd 7664 c6661dc0 f5427000 12848 2546 2546 0004100 [SLPQ select 0xc07569e4][SLP] smtp 7663 c63d4000 f52e9000 12848 2546 2546 0004100 [SLPQ select 0xc07569e4][SLP] trivial-rewrite 6853 c5a41370 f5023000 12848 2546 2546 0004100 [SLPQ select 0xc07569e4][SLP] pickup 21566 c5d6dc08 f519a000 11114 21565 21566 0004002 [SLPQ ttyin 0xc5a60610][SLP] bash 21565 c63d3a50 f52e6000 11114 21563 21563 0000100 [SLPQ select 0xc07569e4][SLP] sshd 21563 c5a3edc0 f5020000 0 2453 21563 0000100 [SLPQ sbwait 0xc5fb9810][SLP] sshd 2620 c611e898 f52a1000 0 2619 2620 0004002 [SLPQ ttyin 0xc59f5410][SLP] bash 2619 c5abfc08 f5149000 0 1 2619 0004102 [SLPQ wait 0xc5abfc08][SLP] login 2618 c5f5b370 f51f6000 0 1 2618 0004002 [SLPQ ttyin 0xc5597a10][SLP] getty 2617 c665ba50 f53e6000 0 1 2617 0004002 [SLPQ ttyin 0xc5597c10][SLP] getty 2616 c5f5d528 f5224000 0 1 2616 0004002 [SLPQ ttyin 0xc5597e10][SLP] getty 2615 c63d31b8 f52e1000 0 1 2615 0004002 [SLPQ ttyin 0xc55d2010][SLP] getty 2614 c5a9a898 f50c5000 0 1 2614 0004002 [SLPQ ttyin 0xc55d2210][SLP] getty 2613 c665b6e0 f53e4000 0 1 2613 0004002 [SLPQ ttyin 0xc55d2410][SLP] getty 2612 c63d6dc0 f531e000 0 1 2612 0004002 [SLPQ ttyin 0xc55d2610][SLP] getty 2611 c5abfa50 f5148000 0 1 2611 0004002 [SLPQ ttyin 0xc55d2a10][SLP] getty 2575 c5ac86e0 f514f000 0 1 2574 0000000 [SLPQ select 0xc07569e4][SLP] snmpd 2563 c5a37c08 f5016000 0 1 2563 0000000 [SLPQ msgwait 0xc55ac000][SLP] pamsmbd 2556 c5f5b898 f521d000 70 2555 2551 0000000 [SLPQ select 0xc07569e4][SLP] postgres --More-- 2555 c5aa3a50 f50cf000 70 2551 2551 0000000 [SLPQ select 0xc07569e4][SLP] postgres 2551 c611e528 f529f000 70 1 2551 0000000 [SLPQ select 0xc07569e4][SLP] postgres 2550 c5aa3528 f50cc000 12848 2546 2546 0004100 [SLPQ select 0xc07569e4][SLP] qmgr 2546 c5aa3370 f50cb000 0 1 2546 0004100 [CPU 1] master 2477 c648fc08 f5391000 0 1 2477 0000000 [SLPQ nanslp 0xc07515bc][SLP] cron 2453 c5f5b000 f51f4000 0 1 2453 0000100 [SLPQ select 0xc07569e4][SLP] sshd 2432 c665f000 f53f2000 0 1 2432 0000000 [SLPQ select 0xc07569e4][SLP] ntpd 2400 c5f5b6e0 f51f8000 0 1 2400 0000000 [SLPQ select 0xc07569e4][SLP] usbd 2377 c5ac81b8 f514c000 0 2373 2373 0000000 [SLPQ - 0xc5a60e00][SLP] nfsd 2376 c665f1b8 f53f3000 0 2373 2373 0000000 [SLPQ - 0xc59d0c00][SLP] nfsd 2375 c5a37a50 f5015000 0 2373 2373 0000000 [SLPQ - 0xc6198600][SLP] nfsd 2374 c611f370 f52cb000 0 2373 2373 0000000 [SLPQ - 0xc59d0200][SLP] nfsd 2373 c5f5bdc0 f5220000 0 1 2373 0000000 [SLPQ select 0xc07569e4][SLP] nfsd 2358 c611fc08 f52d0000 0 1 2358 0000000 [SLPQ select 0xc07569e4][SLP] mountd 2340 c5f5b1b8 f51f5000 0 1 2340 0000000 [SLPQ select 0xc07569e4][SLP] amd 2298 c602a1b8 f5242000 0 1 2298 0000000 [SLPQ select 0xc07569e4][SLP] rpcbind 2238 c648f6e0 f538e000 0 1 2238 0000000 [SLPQ select 0xc07569e4][SLP] syslogd 2141 c5a376e0 f5013000 0 1 2141 0000000 [SLPQ pause 0xc5a37718][SLP] adjkerntz 159 c59d9898 f2db2000 0 0 0 0000204 [SLPQ - 0xeed04d18][SLP] schedcpu 158 c59d9a50 f2db3000 0 0 0 0000204 [SLPQ - 0xc075e36c][SLP] nfsiod 3 --More-- 157 c59d9c08 f2db4000 0 0 0 0000204 [SLPQ - 0xc075e368][SLP] nfsiod 2 156 c59d9dc0 f2db5000 0 0 0 0000204 [SLPQ - 0xc075e364][SLP] nfsiod 1 155 c54d86e0 eecf2000 0 0 0 0000204 [SLPQ - 0xc075e360][SLP] nfsiod 0 154 c54d8898 eecf3000 0 0 0 0000204 [SLPQ vlruwt 0xc54d8898][SLP] vnlru 153 c54d8a50 eecf4000 0 0 0 0000204 [SLPQ syncer 0xc0751344][SLP] syncer 152 c54d8c08 eecf5000 0 0 0 0000204 [SLPQ psleep 0xc0756f30][SLP] bufdaemon 151 c54d8dc0 eecf6000 0 0 0 000020c [SLPQ pgzero 0xc0764b28][SLP] pagezero 150 c5553000 eecf7000 0 0 0 0000204 [SLPQ psleep 0xc0764b80][SLP] vmdaemon 149 c55531b8 eecf8000 0 0 0 0000204 [SLPQ psleep 0xc0764b6c][SLP] pagedaemon 148 c5553370 eecf9000 0 0 0 0000204 [IWAIT] swi0: tty:sio 147 c5553528 eecfa000 0 0 0 0000204 [SLPQ usbevt 0xc55a8210][SLP] usb2 146 c55536e0 eecfb000 0 0 0 0000204 [SLPQ usbevt 0xc556d210][SLP] usb1 145 c5553898 eecfc000 0 0 0 0000204 [SLPQ usbtsk 0xc07496ac][SLP] usbtask 144 c5553a50 eecfd000 0 0 0 0000204 [SLPQ usbevt 0xc557a210][SLP] usb0 9 c5553c08 eecfe000 0 0 0 0000204 [SLPQ aifthd 0xc5553c08][SLP] aac0aif 8 c5553dc0 eecff000 0 0 0 0000204 [SLPQ actask 0xc085a4cc][SLP] acpi_task2 7 c5554000 eed00000 0 0 0 0000204 [SLPQ actask 0xc085a4cc][SLP] acpi_task1 6 c55541b8 eed01000 0 0 0 0000204 [SLPQ actask 0xc085a4cc][SLP] acpi_task0 143 c54bea50 eecbe000 0 0 0 0000204 [IWAIT] swi3: cambio 142 c54bec08 eecbf000 0 0 0 0000204 [IWAIT] swi2: camnet --More-- 141 c54bedc0 eecc0000 0 0 0 0000204 [IWAIT] swi5:+ 140 c54d6000 eecc1000 0 0 0 0000204 [IWAIT] swi7: acpitaskq 5 c54d61b8 eecc2000 0 0 0 0000204 [SLPQ - 0xc552f240][SLP] taskqueue 139 c54d6370 eecc3000 0 0 0 0000204 [IWAIT] swi6:+ 138 c54d6528 eecc4000 0 0 0 0000204 [IWAIT] swi7: task queue 137 c54d66e0 eecc5000 0 0 0 0000204 [SLPQ - 0xc0747560][SLP] yarrow 4 c54d6898 eecc6000 0 0 0 0000204 [SLPQ - 0xc074bcf0][SLP] g_down 3 c54d6a50 eecc7000 0 0 0 0000204 [SLPQ - 0xc074bcec][SLP] g_up 2 c54d6c08 eecc8000 0 0 0 0000204 [SLPQ - 0xc074bce4][SLP] g_event 136 c54d6dc0 eecc9000 0 0 0 0000204 [IWAIT] swi4: vm 135 c54d8000 eecca000 0 0 0 000020c [LOCK Giant c5ac26c0] swi8: tty:sio clock 134 c54d81b8 eeccb000 0 0 0 0000204 [IWAIT] swi1: net 133 c54d8370 eeccc000 0 0 0 0000204 [IWAIT] irq0: clk 132 c54d8528 eeccd000 0 0 0 0000204 [IWAIT] irq119: 131 c54a8dc0 eec8a000 0 0 0 0000204 [IWAIT] irq118: 130 c54bd000 eec8b000 0 0 0 0000204 [IWAIT] irq117: 129 c54bd1b8 eec8c000 0 0 0 0000204 [IWAIT] irq116: 128 c54bd370 eec8d000 0 0 0 0000204 [IWAIT] irq115: 127 c54bd528 eec8e000 0 0 0 0000204 [IWAIT] irq114: 126 c54bd6e0 eec8f000 0 0 0 0000204 [IWAIT] irq113: --More-- 125 c54bd898 eec90000 0 0 0 0000204 [IWAIT] irq112: 124 c54bda50 eec91000 0 0 0 0000204 [IWAIT] irq111: 123 c54bdc08 eec92000 0 0 0 0000204 [IWAIT] irq110: 122 c54bddc0 eec93000 0 0 0 0000204 [IWAIT] irq109: 121 c54be000 eec94000 0 0 0 0000204 [IWAIT] irq108: 120 c54be1b8 eec95000 0 0 0 0000204 [IWAIT] irq107: 119 c54be370 eec96000 0 0 0 0000204 [IWAIT] irq106: 118 c54be528 eec97000 0 0 0 0000204 [IWAIT] irq105: 117 c54be6e0 eec98000 0 0 0 0000204 [IWAIT] irq104: 116 c54be898 eec99000 0 0 0 0000204 [IWAIT] irq103: 115 c54931b8 eec32000 0 0 0 0000204 [IWAIT] irq102: 114 c5493370 eec33000 0 0 0 0000204 [IWAIT] irq101: 113 c5493528 eec34000 0 0 0 0000204 [IWAIT] irq100: 112 c54936e0 eec35000 0 0 0 0000204 [IWAIT] irq99: 111 c5493898 eec36000 0 0 0 0000204 [IWAIT] irq98: 110 c5493a50 eec37000 0 0 0 0000204 [IWAIT] irq97: 109 c5493c08 eec38000 0 0 0 0000204 [IWAIT] irq96: aac0 108 c5493dc0 eec39000 0 0 0 0000204 [IWAIT] irq95: 107 c54a8000 eec3a000 0 0 0 0000204 [IWAIT] irq94: 106 c54a81b8 eec3b000 0 0 0 0000204 [IWAIT] irq93: --More-- 105 c54a8370 eec3c000 0 0 0 0000204 [IWAIT] irq92: 104 c54a8528 eec3d000 0 0 0 0000204 [IWAIT] irq91: 103 c54a86e0 eec3e000 0 0 0 0000204 [IWAIT] irq90: 102 c54a8898 eec3f000 0 0 0 0000204 [IWAIT] irq89: 101 c54a8a50 eec40000 0 0 0 0000204 [IWAIT] irq88: 100 c54a8c08 eec41000 0 0 0 0000204 [IWAIT] irq87: 99 c5477528 eebfe000 0 0 0 0000204 [IWAIT] irq86: 98 c54776e0 eebff000 0 0 0 0000204 [IWAIT] irq85: 97 c5477898 eec00000 0 0 0 0000204 [IWAIT] irq84: 96 c5477a50 eec01000 0 0 0 0000204 [IWAIT] irq83: 95 c5477c08 eec02000 0 0 0 0000204 [IWAIT] irq82: 94 c5477dc0 eec03000 0 0 0 0000204 [IWAIT] irq81: 93 c5492000 eec04000 0 0 0 0000204 [IWAIT] irq80: 92 c54921b8 eec05000 0 0 0 0000204 [IWAIT] irq79: 91 c5492370 eec06000 0 0 0 0000204 [IWAIT] irq78: 90 c5492528 eec07000 0 0 0 0000204 [IWAIT] irq77: 89 c54926e0 eec08000 0 0 0 0000204 [IWAIT] irq76: 88 c5492898 eec09000 0 0 0 0000204 [IWAIT] irq75: 87 c5492a50 eec0a000 0 0 0 0000204 [IWAIT] irq74: 86 c5492c08 eec0b000 0 0 0 0000204 [IWAIT] irq73: --More-- 85 c5492dc0 eec0c000 0 0 0 0000204 [IWAIT] irq72: 84 c5493000 eec0d000 0 0 0 0000204 [IWAIT] irq71: 83 c5465a50 eebcb000 0 0 0 0000204 [IWAIT] irq70: 82 c5465c08 eebcc000 0 0 0 0000204 [IWAIT] irq69: 81 c5465dc0 eebcd000 0 0 0 0000204 [IWAIT] irq68: 80 c5475000 eebce000 0 0 0 0000204 [IWAIT] irq67: 79 c54751b8 eebcf000 0 0 0 0000204 [IWAIT] irq66: 78 c5475370 eebd0000 0 0 0 0000204 [IWAIT] irq65: 77 c5475528 eebd1000 0 0 0 0000204 [IWAIT] irq64: 76 c54756e0 eebd2000 0 0 0 0000204 [IWAIT] irq63: 75 c5475898 eebd3000 0 0 0 0000204 [IWAIT] irq62: 74 c5475a50 eebd4000 0 0 0 0000204 [IWAIT] irq61: 73 c5475c08 eebd5000 0 0 0 0000204 [IWAIT] irq60: 72 c5475dc0 eebd6000 0 0 0 0000204 [IWAIT] irq59: 71 c5477000 eebd7000 0 0 0 0000204 [IWAIT] irq58: 70 c54771b8 eebd8000 0 0 0 0000204 [IWAIT] irq57: 69 c5477370 eebd9000 0 0 0 0000204 [IWAIT] irq56: 68 c54571b8 eeb75000 0 0 0 0000204 [IWAIT] irq55: 67 c5457370 eeb76000 0 0 0 0000204 [IWAIT] irq54: 66 c5457528 eeb77000 0 0 0 0000204 [IWAIT] irq53: --More-- 65 c54576e0 eeb78000 0 0 0 0000204 [IWAIT] irq52: 64 c5457898 eeb79000 0 0 0 0000204 [IWAIT] irq51: 63 c5457a50 eeb7a000 0 0 0 0000204 [IWAIT] irq50: 62 c5457c08 eeb7b000 0 0 0 0000204 [IWAIT] irq49: 61 c5457dc0 eeb7c000 0 0 0 0000204 [IWAIT] irq48: 60 c5465000 eeb7d000 0 0 0 0000204 [IWAIT] irq47: 59 c54651b8 eeb7e000 0 0 0 0000204 [IWAIT] irq46: 58 c5465370 eeb7f000 0 0 0 0000204 [IWAIT] irq45: 57 c5465528 eeb80000 0 0 0 0000204 [IWAIT] irq44: 56 c54656e0 eeb81000 0 0 0 0000204 [IWAIT] irq43: 55 c5465898 eebca000 0 0 0 0000204 [IWAIT] irq42: 54 c5441a50 eeb44000 0 0 0 0000204 [IWAIT] irq41: 53 c5441c08 eeb45000 0 0 0 0000204 [IWAIT] irq40: 52 c5441dc0 eeb46000 0 0 0 0000204 [IWAIT] irq39: 51 c5455000 eeb47000 0 0 0 0000204 [IWAIT] irq38: 50 c54551b8 eeb48000 0 0 0 0000204 [IWAIT] irq37: 49 c5455370 eeb49000 0 0 0 0000204 [IWAIT] irq36: 48 c5455528 eeb4a000 0 0 0 0000204 [IWAIT] irq35: 47 c54556e0 eeb4b000 0 0 0 0000204 [IWAIT] irq34: 46 c5455898 eeb4c000 0 0 0 0000204 [IWAIT] irq33: --More-- 45 c5455a50 eeb4d000 0 0 0 0000204 [IWAIT] irq32: 44 c5455c08 eeb72000 0 0 0 0000204 [IWAIT] irq31: 43 c5455dc0 eeb73000 0 0 0 0000204 [IWAIT] irq30: 42 c5457000 eeb74000 0 0 0 0000204 [LOCK Giant c5ac26c0] irq29: em1 41 c5438528 eeb14000 0 0 0 0000204 [LOCK Giant c5ac26c0] irq28: em0 40 c54386e0 eeb15000 0 0 0 0000204 [IWAIT] irq27: 39 c5438898 eeb16000 0 0 0 0000204 [IWAIT] irq26: 38 c5438a50 eeb17000 0 0 0 0000204 [IWAIT] irq25: 37 c5438c08 eeb18000 0 0 0 0000204 [IWAIT] irq24: 36 c5438dc0 eeb19000 0 0 0 0000204 [IWAIT] irq23: 35 c5441000 eeb1a000 0 0 0 0000204 [IWAIT] irq22: 34 c54411b8 eeb3f000 0 0 0 0000204 [IWAIT] irq21: 33 c5441370 eeb40000 0 0 0 0000204 [IWAIT] irq20: 32 c5441528 eeb41000 0 0 0 0000204 [IWAIT] irq19: uhci1 31 c54416e0 eeb42000 0 0 0 0000204 [IWAIT] irq18: uhci2 30 c5441898 eeb43000 0 0 0 0000204 [IWAIT] irq17: ichsmb0 29 c53e61b8 ed0da000 0 0 0 0000204 [IWAIT] irq16: uhci0 28 c53e6370 ed0db000 0 0 0 0000204 [IWAIT] irq15: ata1 27 c53e6528 ed0dc000 0 0 0 0000204 [IWAIT] irq14: ata0 26 c53e66e0 ed0dd000 0 0 0 0000204 [IWAIT] irq13: --More-- 25 c53e6898 ed102000 0 0 0 0000204 [IWAIT] irq12: 24 c53e6a50 ed103000 0 0 0 0000204 [IWAIT] irq11: 23 c53e6c08 ed104000 0 0 0 0000204 [IWAIT] irq10: 22 c53e6dc0 ed105000 0 0 0 0000204 [IWAIT] irq9: acpi0 21 c5438000 eeb11000 0 0 0 0000204 [IWAIT] irq8: rtc 20 c54381b8 eeb12000 0 0 0 0000204 [IWAIT] irq7: 19 c5438370 eeb13000 0 0 0 0000204 [IWAIT] irq6: fdc0 18 c53df000 ed088000 0 0 0 0000204 [IWAIT] irq5: 17 c53df1b8 ed0d1000 0 0 0 0000204 [IWAIT] irq4: sio0 16 c53df370 ed0d2000 0 0 0 0000204 [IWAIT] irq3: sio1 15 c53df528 ed0d3000 0 0 0 0000204 [IWAIT] irq1: atkbd0 14 c53df6e0 ed0d4000 0 0 0 000020c [CPU 0] idle: cpu0 13 c53df898 ed0d5000 0 0 0 000020c [Can run] idle: cpu1 12 c53dfa50 ed0d6000 0 0 0 000020c [CPU 2] idle: cpu2 11 c53dfc08 ed0d7000 0 0 0 000020c [CPU 3] idle: cpu3 1 c53dfdc0 ed0d8000 0 0 1 0004200 [SLPQ wait 0xc53dfdc0][SLP] init 10 c53e6000 ed0d9000 0 0 0 0000204 [SLPQ ktrace 0xc074f558][SLP] ktrace 0 c074bde0 c0c1f000 0 0 0 0000200 [SLPQ sched 0xc074bde0][SLP] swapper oops, ran out of processes early! db> show all registers breaks thread watches pciregs intrcnt pgrpdump pcpu msgbuf cbstat buffer disk lockedvnods map procvm vmochk object vmopag page pageq cyrixreg irqs idt rtc db> show pcpu 0 cpuid = 0 curthread = 0xc53e0a80: pid 14 "idle: cpu0" curpcb = 0xed0a3da0 fpcurthread = none idlethread = 0xc53e0a80: pid 14 "idle: cpu0" APIC ID = 0 currentldt = 0x28 db> show pcpu 1 cpuid = 1 curthread = 0xc5aa0000: pid 2546 "master" curpcb = 0xf509fda0 fpcurthread = none idlethread = 0xc53e0930: pid 13 "idle: cpu1" APIC ID = 1 currentldt = 0x28 db> show pcpu 2 cpuid = 2 curthread = 0xc53e07e0: pid 12 "idle: cpu2" curpcb = 0xed09dda0 fpcurthread = none idlethread = 0xc53e07e0: pid 12 "idle: cpu2" APIC ID = 6 currentldt = 0x28 db> show pcpu 3 cpuid = 3 curthread = 0xc53e0690: pid 11 "idle: cpu3" curpcb = 0xed09ada0 fpcurthread = none idlethread = 0xc53e0690: pid 11 "idle: cpu3" APIC ID = 7 currentldt = 0x28 db> show all registers breaks thread watches pciregs intrcnt pgrpdump pcpu msgbuf cbstat buffer disk lockedvnods map procvm vmochk object vmopag page pageq cyrixreg irqs idt rtc db> show lockedvnoe des No such command db> chow lockedvnods No such command db> show lockedvnods Locked vnodes db> show rtc 04/06/13 00:29:11, A = 29, B = 42, C = d0 db> show rtc 04/06/13 00:29:14, A = 29, B = 42, C = d0 db> show registers cs 0x8 ds 0xed0a0010 es 0x10 fs 0x18 ss 0xc5540010 eax 0x1 ecx 0xc554d9c8 edx 0x2 ebx 0xc555a303 esp 0xed0a3c48 ebp 0xed0a3c58 esi 0xc554d800 edi 0x61 eip 0xc06716d8 siointr1+0xd0 efl 0x2 dr0 0 dr1 0 dr2 0 dr3 0 dr4 0xffff0ff0 dr5 0x400 dr6 0xffff0ff0 dr7 0x400 siointr1+0xd0: movl %edi,%edx db> show all registers breaks thread watches pciregs intrcnt pgrpdump pcpu msgbuf cbstat buffer disk lockedvnods map procvm vmochk object vmopag page pageq cyrixreg irqs idt rtc db> show thread Proc 0xc53df6e0 [CPU 0] idle: cpu0 siointr1(c554d800) at siointr1+0xd0 siointr(c554d800) at siointr+0x62 intr_execute_handlers(c53d9490,ed0a3ca4,4,ed0a3ce8,c0683ed3) at intr_execute_handlers+0x85 lapic_handle_intr(34) at lapic_handle_intr+0x2e Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc084cc49, esp = 0xed0a3ce8, ebp = 0xed0a3ce8 --- acpi_cpu_c1(0,0,1,4,c53e0a80) at acpi_cpu_c1+0x5 acpi_cpu_idle(ed0a3d20,c0531d05,c53df6e0,c0531cec,ed0a3d34) at acpi_cpu_idle+0xc1 cpu_idle(c53df6e0,c0531cec,ed0a3d34,c0531a7d,0) at cpu_idle+0x28 idle_proc(0,ed0a3d48) at idle_proc+0x19 fork_exit(c0531cec,0,ed0a3d48) at fork_exit+0x71 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xed0a3d7c, ebp = 0 --- db> show thread 1 bad thread addressdb> Proc 0xc53df6e0 [CPU 0] idle: cpu0 siointr1(c554d800) at siointr1+0xd0 db> ? Bad character ? db> show all registers breaks thread watches pciregs intrcnt pgrpdump pcpu msgbuf cbstat buffer disk lockedvnods map procvm vmochk object vmopag page pageq cyrixreg irqs idt rtc db> show disk usage: show disk/devicename db> show d aacd0s1a No such command db> show disk aacd0s1a Symbol not found db> show disk/aacd0s1a dev_t = 0xc5980c00 db> show map Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x8d fault code = supervisor read, page not present instruction pointer = 0x8:0xc065671c stack pointer = 0x10:0xed0a3aa8 frame pointer = 0x10:0xed0a3ab4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 14 (idle: cpu0) timeout stopping cpus kernel: type 12 trap, code=0 Stopped at siointr1+0xd0: movl %edi,%edx db> Task map 0xc06716d8: pmap=0x1d886ff, nentries=905904234, version=3960514536 Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x741ca8d6 fault code = supervisor read, page not present instruction pointer = 0x8:0xc0656768 stack pointer = 0x10:0xed0a3aa8 frame pointer = 0x10:0xed0a3ab4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 14 (idle: cpu0) timeout stopping cpus kernel: type 12 trap, code=0 Stopped at siointr1+0xd0: movl %edi,%edx db> Task map 0xc06716d8: pmap=0x1d886ff, nentries=905904234, version=3960514536 db> ~ Bad character ? db> Task map 0xc06716d8: pmap=0x1d886ff, nentries=905904234, version=3960514536 db> ~?~ Bad character ? db> s help print p examine x search set write w delete d break dwatch watch dhwatch hwatch step s continue c until next match trace where call show ps gdb reset kill watchdog panic ahd_set_unitahd_pause ahd_unpause ahd_in ahd_out db> call doadump Dumping 3839 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976 992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152 1168 1184 1200 1216 1232 1248 1264 1280 1296 1312 1328 1344 1360 1376 1392 1408 1424 1440 1456 1472 1488 1504 1520 1536 1552 1568 1584 1600 1616 1632 1648 1664 1680 1696 1712 1728 1744 1760 1776 1792 1808 1824 1840 1856 1872 1888 1904 1920 1936 1952 1968 1984 2000 2016 2032 2048 2064 2080 2096 2112 2128 2144 2160 2176 2192 2208 2224 2240 2256 2272 2288 2304 2320 2336 2352 2368 2384 2400 2416 2432 2448 2464 2480 2496 2512 2528 2544 2560 2576 2592 2608 2624 2640 2656 2672 2688 2704 2720 2736 2752 2768 2784 2800 2816 2832 2848 2864 2880 2896 2912 2928 2944 2960 2976 2992 3008 3024 3040 3056 3072 3088 3104 3120 3136 3152 3168 3184 3200 3216 3232 3248 3264 3280 3296 3312 3328 3344 3360 3376 3392 3408 3424 3440 3456 3472 3488 3504 3520 3536 3552 3568 3584 3600 3616 3632 3648 3664 3680 3696 3712 3728 3744 3760 3776 3792 3808 3824 Dump complete 0xf db> x ticks ticks: 33f08b0 db> c whoa, other_cpus: 0x0000000e, stopped_cpus: 0x00000004 timeout stopping cpus Stopped at siointr1+0xd0: movl %edi,%edx db> x ticks ticks: 33f162a db> x softticks softticks: 328c76a db> call cpu_reset cpu_reset called on cpu#0 cpu_reset: Stopping other CPUs telnet> quit Connection closed. Script done on Sun Jun 13 00:34:44 2004