Date: Sun, 8 Sep 2002 19:30:04 -0700 (PDT) From: El Vampiro <vampiro@rootshell.ru> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed. Message-ID: <200209090230.g892U4Cl067557@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/42277; it has been noted by GNATS. From: El Vampiro <vampiro@rootshell.ru> To: freebsd-gnats-submit@FreeBSD.org Cc: Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed. Date: Mon, 9 Sep 2002 09:25:46 +0700 Two more panisc during holidays. First: IdlePTD at phsyical address 0x003f0000 initial pcb at physical address 0x0035e080 panicstr: softdep_lock: locking against myself panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0xffff000a fault code = supervisor read, page not present instruction pointer = 0x8:0xc022f84c stack pointer = 0x10:0xcd0c6d10 frame pointer = 0x10:0xcd0c6d10 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 6 (syncer) interrupt mask = bio trap number = 12 panic: page fault syncing disks... panic: softdep_lock: locking against myself Uptime: 1d2h45m48s dumping to dev #da/0x20009, offset 128 dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) where #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc0191837 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0191c5c in poweroff_wait (junk=0xc03224c0, howto=-1052082688) at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc022f57e in acquire_lock (lk=0xc034c6bc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261 #4 0xc0233698 in softdep_update_inodeblock (ip=0xc14a7e00, bp=0xc64dce38, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813 #5 0xc022e6cd in ffs_update (vp=0xcd869980, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106 #6 0xc02380c5 in ffs_fsync (ap=0xcd0c6bb0) at /usr/src/sys/ufs/ffs/ffs_vnops.c:273 #7 0xc02369a3 in ffs_sync (mp=0xc12a7200, waitfor=2, cred=0xc0a42680, p=0xc0372b60) at vnode_if.h:558 #8 0xc01c19db in sync (p=0xc0372b60, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576 #9 0xc01915aa in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235 #10 0xc0191c5c in poweroff_wait (junk=0xc032c60c, howto=-1070415569) at /usr/src/sys/kern/kern_shutdown.c:595 #11 0xc0278df6 in trap_fatal (frame=0xcd0c6cd0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974 #12 0xc0278ac9 in trap_pfault (frame=0xcd0c6cd0, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867 #13 0xc02786b3 in trap (frame={tf_fs = -854851568, tf_es = -1052377072, tf_ds = 16, tf_edi = 0, tf_esi = -1052351744, tf_ebp = -854823664, tf_isp = -854823684, tf_ebx = -65536, tf_edx = -1051899520, tf_ecx = -65536, tf_eax = -1051899520, tf_trapno = 12, tf_err = 0, tf_eip = -1071450036, tf_cs = 8, tf_eflags = 66067, tf_esp = -854823632, tf_ss = -1071433906}) at /usr/src/sys/i386/i386/trap.c:466 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467 #15 0xc023374e in softdep_update_inodeblock (ip=0xc1466300, bp=0xc64cde4c, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847 #16 0xc022e6cd in ffs_update (vp=0xcdde9200, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106 #17 0xc022e9b1 in ffs_truncate (vp=0xcdde9200, length=0, flags=0, cred=0x0, p=0xcbfd75e0) at /usr/src/sys/ufs/ffs/ffs_inode.c:201 #18 0xc0238dc8 in ufs_inactive (ap=0xcd0c6ed8) at /usr/src/sys/ufs/ufs/ufs_inode.c:89 #19 0xc023e2c1 in ufs_vnoperate (ap=0xcd0c6ed8) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422 #20 0xc01bfb2c in vput (vp=0xcdde9200) at vnode_if.h:815 #21 0xc0232554 in handle_workitem_remove (dirrem=0xc151a960) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852 #22 0xc022fbcd in process_worklist_item (matchmnt=0x0, flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:716 #23 0xc022fa72 in softdep_process_worklist (matchmnt=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:622 #24 0xc01bf453 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177 (kgdb) up 14 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467 467 panic("worklist_remove: lock not held"); (kgdb) l 462 worklist_remove(item) 463 struct worklist *item; 464 { 465 466 if (lk.lkt_held == -1) 467 panic("worklist_remove: lock not held"); 468 if ((item->wk_state & ONWORKLIST) == 0) { 469 FREE_LOCK(&lk); 470 panic("worklist_remove: not on list"); 471 } (kgdb) p item $1 = (struct worklist *) 0x0 (kgdb) p lk $2 = {lkt_spl = 0, lkt_held = -1} (kgdb) up #15 0xc023374e in softdep_update_inodeblock (ip=0xc1466300, bp=0xc64cde4c, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847 3847 WORKLIST_REMOVE(wk); (kgdb) l 3842 * operations dependent on the inode being written to disk 3843 * can be moved to the id_bufwait so that they will be 3844 * processed when the buffer I/O completes. 3845 */ 3846 while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) { 3847 WORKLIST_REMOVE(wk); 3848 WORKLIST_INSERT(&inodedep->id_bufwait, wk); 3849 } 3850 /* 3851 * Newly allocated inodes cannot be written until the bitmap (kgdb) p wk $3 = (struct worklist *) 0x68c040 (kgdb) p *wk Cannot access memory at address 0x68c040. (kgdb) p *inodedep $4 = {id_list = {wk_list = {le_next = 0xc1943000, le_prev = 0xc64cdf8c}, wk_type = 1, wk_state = 32777}, id_hash = { le_next = 0x0, le_prev = 0xc1222e94}, id_fs = 0xc12c7000, id_ino = 22082, id_nlinkdelta = 0, id_savedino = 0x0, id_deps = {le_next = 0x0, le_prev = 0x0}, id_buf = 0x0, id_savedsize = -1, id_pendinghd = {lh_first = 0x0}, id_bufwait = { lh_first = 0xc14d4c00}, id_inowait = {lh_first = 0xffff0000}, id_inoupdt = {tqh_first = 0x0, tqh_last = 0xc14d49c4}, id_newinoupdt = {tqh_first = 0x0, tqh_last = 0xc14d49cc}} # vmstat -m -M vmcore.2 Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 952 328 874701 1280 0 32 536 360 4176708 640 0 64 16355 5917 31821882 320 9082 128 1247 513 6295455 160 59721 256 17477 779 4662394 80 635 512 443 213 18611 40 0 1K 102 18 3986474 20 19 2K 42 138 2378 10 1463 4K 30 2 658022 5 0 8K 3 1 40 5 0 16K 5 0 245 5 0 32K 8 0 16 5 0 64K 10 0 10 5 0 128K 3 0 4 5 0 Memory usage type by bucket size Size Type(s) 16 uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl, ether_multi, BPF, vnodes, mount, pcb, soname, accf, shm, rman, bus, sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld 32 atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag, indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl, ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname, accf, taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl, uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld 64 isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon, NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes, vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc, session, temp, devbuf, lockf, proc-args, file 128 ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF, vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred, temp, devbuf, zombie, proc-args, dev_t, timecounter, kld 256 UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon, NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes, ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue, file desc 512 UFS dirhash, UFS mount, NFS daemon, BPF, mount, BIO buffer, ptys, msg, bus, uidinfo, ATA generic, temp, devbuf, prison, file desc 1K uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, ioctlops, bus, temp, devbuf, kqueue, file desc 2K UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver, shm, ioctlops, bus, proc, devbuf, file desc 4K memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus, temp, devbuf 8K pagedep, syncache, shm, bus, temp 16K VM pgdata, indirdep, sem, msg, temp, devbuf 32K BPF, sem, msg, temp, devbuf 64K ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf 128K vfscache, msg, SWAP Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) atkbddev 1 1K 1K 42378K 1 0 0 32 uc_devlist 14 2K 2K 42378K 14 0 0 16,1K memdesc 1 4K 4K 42378K 1 0 0 4K mbuf 1 4K 4K 42378K 1 0 0 4K isadev 10 1K 1K 42378K 10 0 0 64 ZONE 16 2K 2K 42378K 16 0 0 128 VM pgdata 1 16K 16K 42378K 1 0 0 16K ISOFS mount 1 64K 64K 42378K 1 0 0 64K UFS dirhash 776 153K 264K 42378K 2226 0 0 16,32,64,128,256,512,1K,2K,4K UFS mount 21 46K 46K 42378K 39 0 0 512,2K,4K UFS ihash 1 64K 64K 42378K 1 0 0 64K FFS node 17135 4284K 4485K 42378K 520818 0 0 256 dirrem 2 1K 3K 42378K 25912 0 0 32 mkdir 0 0K 1K 42378K 124 0 0 32 diradd 0 0K 3K 42378K 26553 0 0 32 freefile 0 0K 2K 42378K 13804 0 0 32 freeblks 1 1K 5K 42378K 12336 0 0 128 freefrag 0 0K 3K 42378K 5170 0 0 32 allocindir 1 1K 152K 42378K 20648 0 0 64 indirdep 1 1K 33K 42378K 1158 0 0 32,16K allocdirect 1 1K 6K 42378K 22316 0 0 64 bmsafemap 3 1K 1K 42378K 11096 0 0 32 newblk 1 1K 1K 42378K 42965 0 0 32,256 inodedep 4 65K 77K 42378K 23254 0 0 128,64K pagedep 3 9K 11K 42378K 12282 0 0 64,8K p1003.1b 1 1K 1K 42378K 1 0 0 16 NFS hash 1 64K 64K 42378K 1 0 0 64K NQNFS Lease 1 1K 1K 42378K 1 0 0 1K NFSV3 srvdesc 0 0K 1K 42378K 62136 0 0 16,256 NFS daemon 71 8K 8K 42378K 71 0 0 64,256,512 NFS req 0 0K 1K 42378K 2015016 0 0 64 NFS srvsock 1 1K 1K 42378K 1 0 0 256 syncache 1 8K 8K 42378K 1 0 0 8K tseg_qent 0 0K 2K 42378K 30284 0 0 32 IpFw/IpAcct 79 11K 53K 42378K 14699 0 0 64,128,256,1K in_multi 3 1K 1K 42378K 3 0 0 32 routetbl 163 23K 126K 42378K 3651 0 0 16,32,64,128,256 ether_multi 12 1K 1K 42378K 12 0 0 16,32,64 ifaddr 27 6K 6K 42378K 27 0 0 32,64,256,2K BPF 15 137K 266K 42378K 31 0 0 16,32,128,512,4K,32K Export Host 6 2K 2K 42378K 6 0 0 256 vnodes 26 6K 6K 42378K 337 0 0 16,32,64,128,256 mount 16 8K 8K 42378K 24 0 0 16,128,512 cluster_save buffer 0 0K 1K 42378K 1521 0 0 32 vfscache 15065 1073K 1332K 42378K 549206 0 0 64,128,128K BIO buffer 4 8K 286K 42378K 1184 0 0 512,2K pcb 69 6K 8K 42378K 14966 0 0 16,32,64,2K soname 38 4K 6K 42378K 839397 0 0 16,32,64,128 ACD driver 2 3K 3K 42378K 2 0 0 256,2K accf 3 1K 1K 42378K 3 0 0 16,32 ptys 5 3K 3K 42378K 5 0 0 512 ttys 210 27K 47K 42378K 2355 0 0 128,256 shm 16 37K 51K 42378K 1085 0 0 16,2K,8K sem 3 88K 88K 42378K 3 0 0 16K,32K,64K msg 4 137K 137K 42378K 4 0 0 512,16K,32K,128K rman 58 4K 4K 42378K 401 0 0 16,64 iov 0 0K 1K 42378K 406 0 0 128 ioctlops 0 0K 4K 42378K 7 0 0 1K,2K,4K taskqueue 1 1K 1K 42378K 1 0 0 32 SWAP 2 73K 145K 42378K 4 0 0 32,128K ATAPI generic 1 1K 1K 42378K 2 0 0 32,128 kobj 1 4K 4K 42378K 1 0 0 4K eventhandler 15 1K 1K 42378K 15 0 0 32,64 bus 375 34K 37K 42378K 696 0 0 16,32,64,128,256,512,1K,2K,4K,8K sysctloid 10 1K 1K 42378K 10 0 0 16,64 sysctl 0 0K 1K 42378K 4771 0 0 16,32 uidinfo 11 1K 1K 42378K 1271 0 0 32,512 cred 67 9K 13K 42378K 715419 0 0 128 subproc 241 17K 26K 42378K 8001154 0 0 32,64,256 proc 2 4K 4K 42378K 2 0 0 2K session 47 3K 4K 42378K 4512 0 0 64 pgrp 47 2K 2K 42378K 4620 0 0 32 ATA generic 2 1K 1K 42378K 2 0 0 16,512 temp 305 138K 153K 42378K 8754546 0 0 16,32,64,128,256,512,1K,4K,8K,16K,32K,64K devbuf 506 412K 413K 42378K 1141 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K,64K lockf 42 3K 6K 42378K 1564032 0 0 64 prison 3 2K 2K 42378K 3 0 0 512 atexit 1 1K 1K 42378K 1 0 0 16 zombie 0 0K 1K 42378K 3997472 0 0 128 proc-args 81 5K 7K 42378K 4062493 0 0 16,32,64,128,256 kqueue 5 5K 22K 42378K 7104 0 0 256,1K sigio 1 1K 1K 42378K 133 0 0 32 file 728 46K 67K 42378K 17098401 0 0 64 file desc 147 40K 62K 42378K 4000832 0 0 256,512,1K,2K dev_t 663 83K 83K 42378K 663 0 0 128 timecounter 10 2K 2K 42378K 10 0 0 128 kld 4 1K 1K 42378K 35 0 0 16,32,128 Memory Totals: In Use Free Requests 7243K 1062K 52496940 And second panic: IdlePTD at phsyical address 0x003f0000 initial pcb at physical address 0x0035e080 panicstr: softdep_lock: locking against myself panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0xffff000a fault code = supervisor read, page not present instruction pointer = 0x8:0xc022f84c stack pointer = 0x10:0xcda0ace8 frame pointer = 0x10:0xcda0ace8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 28039 (nmbd) interrupt mask = bio trap number = 12 panic: page fault syncing disks... panic: softdep_lock: locking against myself Uptime: 1d5h27m47s dumping to dev #da/0x20009, offset 128 dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) where #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc0191837 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0191c5c in poweroff_wait (junk=0xc03224c0, howto=-1052660736) at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc022f57e in acquire_lock (lk=0xc034c6bc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261 #4 0xc0233698 in softdep_update_inodeblock (ip=0xc141ac00, bp=0xc65508dc, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813 #5 0xc022e6cd in ffs_update (vp=0xcd67e780, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106 #6 0xc02380c5 in ffs_fsync (ap=0xcda0ab88) at /usr/src/sys/ufs/ffs/ffs_vnops.c:273 #7 0xc02369a3 in ffs_sync (mp=0xc12a7400, waitfor=2, cred=0xc0a42680, p=0xc0372b60) at vnode_if.h:558 #8 0xc01c19db in sync (p=0xc0372b60, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576 #9 0xc01915aa in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235 #10 0xc0191c5c in poweroff_wait (junk=0xc032c60c, howto=-1070415569) at /usr/src/sys/kern/kern_shutdown.c:595 #11 0xc0278df6 in trap_fatal (frame=0xcda0aca8, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974 #12 0xc0278ac9 in trap_pfault (frame=0xcda0aca8, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867 #13 0xc02786b3 in trap (frame={tf_fs = -845152240, tf_es = -1049231344, tf_ds = 16, tf_edi = 0, tf_esi = -1049228288, tf_ebp = -845107992, tf_isp = -845108012, tf_ebx = -65536, tf_edx = -1045960576, tf_ecx = -65536, tf_eax = -1045960576, tf_trapno = 12, tf_err = 0, tf_eip = -1071450036, tf_cs = 8, tf_eflags = 66071, tf_esp = -845107960, tf_ss = -1071433906}) at /usr/src/sys/i386/i386/trap.c:466 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467 #15 0xc023374e in softdep_update_inodeblock (ip=0xc1760c00, bp=0xc6564930, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847 #16 0xc022e6cd in ffs_update (vp=0xcdd3a440, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106 #17 0xc022e9b1 in ffs_truncate (vp=0xcdd3a440, length=0, flags=0, cred=0x0, p=0xcd93e8a0) at /usr/src/sys/ufs/ffs/ffs_inode.c:201 #18 0xc0238dc8 in ufs_inactive (ap=0xcda0aeb0) at /usr/src/sys/ufs/ufs/ufs_inode.c:89 #19 0xc023e2c1 in ufs_vnoperate (ap=0xcda0aeb0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422 #20 0xc01bfb2c in vput (vp=0xcdd3a440) at vnode_if.h:815 #21 0xc01c2e15 in unlink (p=0xcd93e8a0, uap=0xcda0af80) at /usr/src/sys/kern/vfs_syscalls.c:1504 #22 0xc027901a in syscall2 (frame={tf_fs = -1078001617, tf_es = 47, tf_ds = -1078001617, tf_edi = 672627448, tf_esi = 0, tf_ebp = -1077937936, tf_isp = -845107244, tf_ebx = 1, tf_edx = 672628888, tf_ecx = 135524352, tf_eax = 10, tf_trapno = 12, tf_err = 2, tf_eip = 672239144, tf_cs = 31, tf_eflags = 647, tf_esp = -1077940028, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175 #23 0xc026cd75 in Xint0x80_syscall () #24 0x805d23c in ?? () #25 0x804b714 in ?? () #26 0x804c40d in ?? () #27 0x804a871 in ?? () (kgdb) up 14 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467 467 panic("worklist_remove: lock not held"); (kgdb) l 462 worklist_remove(item) 463 struct worklist *item; 464 { 465 466 if (lk.lkt_held == -1) 467 panic("worklist_remove: lock not held"); 468 if ((item->wk_state & ONWORKLIST) == 0) { 469 FREE_LOCK(&lk); 470 panic("worklist_remove: not on list"); 471 } (kgdb) p *item Cannot access memory at address 0x0. (kgdb) p lk $1 = {lkt_spl = 0, lkt_held = -1} (kgdb) up #15 0xc023374e in softdep_update_inodeblock (ip=0xc1760c00, bp=0xc6564930, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847 3847 WORKLIST_REMOVE(wk); (kgdb) l 3842 * operations dependent on the inode being written to disk 3843 * can be moved to the id_bufwait so that they will be 3844 * processed when the buffer I/O completes. 3845 */ 3846 while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) { 3847 WORKLIST_REMOVE(wk); 3848 WORKLIST_INSERT(&inodedep->id_bufwait, wk); 3849 } 3850 /* 3851 * Newly allocated inodes cannot be written until the bitmap (kgdb) p wk $2 = (struct worklist *) 0x68c040 (kgdb) p *wk Cannot access memory at address 0x68c040. # vmstat -m -M vmcore.3 Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 1116 164 424115 1280 0 32 488 280 4533610 640 0 64 19590 6586 35473023 320 18958 128 1244 4100 6899026 160 348464 256 17591 905 5822574 80 2631 512 548 124 15562 40 277 1K 101 7 4403849 20 13 2K 61 99 2956 10 1304 4K 30 3 624293 5 0 8K 3 1 10 5 0 16K 5 0 85 5 0 32K 8 0 8 5 0 64K 10 0 10 5 0 128K 3 0 4 5 0 Memory usage type by bucket size Size Type(s) 16 uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl, ether_multi, BPF, vnodes, mount, pcb, soname, accf, shm, rman, bus, sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld 32 atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag, indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl, ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname, accf, taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl, uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld 64 isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon, NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes, vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc, session, temp, devbuf, lockf, proc-args, file 128 ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF, vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred, temp, devbuf, zombie, proc-args, dev_t, timecounter, kld 256 UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon, NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes, ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue, file desc 512 UFS dirhash, UFS mount, NFS daemon, mount, BIO buffer, ptys, msg, ioctlops, bus, uidinfo, ATA generic, temp, devbuf, prison, file desc 1K uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, ioctlops, bus, temp, devbuf, kqueue 2K UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver, shm, ioctlops, bus, proc, devbuf 4K memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus, temp, devbuf 8K pagedep, syncache, shm, bus, temp 16K VM pgdata, indirdep, sem, msg, devbuf 32K BPF, sem, msg, temp, devbuf 64K ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf 128K vfscache, msg, SWAP Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) atkbddev 1 1K 1K 42378K 1 0 0 32 uc_devlist 14 2K 2K 42378K 14 0 0 16,1K memdesc 1 4K 4K 42378K 1 0 0 4K mbuf 1 4K 4K 42378K 1 0 0 4K isadev 10 1K 1K 42378K 10 0 0 64 ZONE 16 2K 2K 42378K 16 0 0 128 VM pgdata 1 16K 16K 42378K 1 0 0 16K ISOFS mount 1 64K 64K 42378K 1 0 0 64K UFS dirhash 1088 216K 274K 42378K 4095 0 0 16,32,64,128,256,512,1K,2K,4K UFS mount 21 46K 46K 42378K 21 0 0 512,2K,4K UFS ihash 1 64K 64K 42378K 1 0 0 64K FFS node 17283 4321K 4487K 42378K 1282170 0 0 256 dirrem 0 0K 3K 42378K 14638 0 0 32 mkdir 0 0K 1K 42378K 130 0 0 32 diradd 2 1K 3K 42378K 14962 0 0 32 freefile 0 0K 2K 42378K 8476 0 0 32 freeblks 1 1K 3K 42378K 6793 0 0 128 freefrag 0 0K 10K 42378K 3611 0 0 32 allocindir 0 0K 425K 42378K 32022 0 0 64 indirdep 0 0K 33K 42378K 410 0 0 32,16K allocdirect 1 1K 4K 42378K 12040 0 0 64 bmsafemap 1 1K 1K 42378K 7652 0 0 32 newblk 1 1K 1K 42378K 44063 0 0 32,256 inodedep 4 65K 78K 42378K 13879 0 0 128,64K pagedep 2 9K 10K 42378K 8084 0 0 64,8K p1003.1b 1 1K 1K 42378K 1 0 0 16 NFS hash 1 64K 64K 42378K 1 0 0 64K NQNFS Lease 1 1K 1K 42378K 1 0 0 1K NFSV3 srvdesc 0 0K 1K 42378K 8 0 0 16,256 NFS daemon 11 4K 4K 42378K 11 0 0 64,256,512 NFS req 0 0K 1K 42378K 2212294 0 0 64 NFS srvsock 1 1K 1K 42378K 1 0 0 256 syncache 1 8K 8K 42378K 1 0 0 8K tseg_qent 0 0K 1K 42378K 185 0 0 32 IpFw/IpAcct 53 8K 15K 42378K 2652 0 0 64,128,256,1K in_multi 3 1K 1K 42378K 3 0 0 32 routetbl 140 20K 120K 42378K 2327 0 0 16,32,64,128,256 ether_multi 12 1K 1K 42378K 12 0 0 16,32,64 ifaddr 27 6K 6K 42378K 27 0 0 32,64,256,2K BPF 15 137K 137K 42378K 15 0 0 16,32,128,4K,32K Export Host 6 2K 2K 42378K 6 0 0 256 vnodes 26 6K 6K 42378K 337 0 0 16,32,64,128,256 mount 16 8K 8K 42378K 18 0 0 16,128,512 cluster_save buffer 0 0K 1K 42378K 1031 0 0 32 vfscache 18465 1287K 1696K 42378K 1339707 0 0 64,128,128K BIO buffer 25 50K 248K 42378K 1868 0 0 512,2K pcb 63 6K 6K 42378K 5919 0 0 16,32,64,2K soname 34 3K 5K 42378K 426863 0 0 16,32,64,128 ACD driver 2 3K 3K 42378K 2 0 0 256,2K accf 3 1K 1K 42378K 3 0 0 16,32 ptys 1 1K 1K 42378K 1 0 0 512 ttys 210 27K 27K 42378K 639 0 0 128,256 shm 14 33K 43K 42378K 1141 0 0 16,2K,8K sem 3 88K 88K 42378K 3 0 0 16K,32K,64K msg 4 137K 137K 42378K 4 0 0 512,16K,32K,128K rman 58 4K 4K 42378K 401 0 0 16,64 iov 0 0K 1K 42378K 316 0 0 128 ioctlops 0 0K 4K 42378K 14 0 0 512,1K,2K,4K taskqueue 1 1K 1K 42378K 1 0 0 32 SWAP 2 73K 145K 42378K 4 0 0 32,128K ATAPI generic 1 1K 1K 42378K 2 0 0 32,128 kobj 1 4K 4K 42378K 1 0 0 4K eventhandler 15 1K 1K 42378K 15 0 0 32,64 bus 375 34K 37K 42378K 696 0 0 16,32,64,128,256,512,1K,2K,4K,8K sysctloid 10 1K 1K 42378K 10 0 0 16,64 sysctl 0 0K 1K 42378K 4790 0 0 16,32 uidinfo 10 1K 1K 42378K 708 0 0 32,512 cred 68 9K 12K 42378K 722640 0 0 128 subproc 210 14K 21K 42378K 8845542 0 0 32,64,256 proc 2 4K 4K 42378K 2 0 0 2K session 46 3K 4K 42378K 4248 0 0 64 pgrp 46 2K 2K 42378K 4248 0 0 32 ATA generic 2 1K 1K 42378K 2 0 0 16,512 temp 299 138K 152K 42378K 9568290 0 0 16,32,64,128,256,512,1K,4K,8K,32K,64K devbuf 506 412K 412K 42378K 1168 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K,64K lockf 36 3K 4K 42378K 1546084 0 0 64 prison 3 2K 2K 42378K 3 0 0 512 atexit 1 1K 1K 42378K 1 0 0 16 zombie 0 0K 1K 42378K 4419948 0 0 128 proc-args 70 4K 6K 42378K 4491591 0 0 16,32,64,128,256 kqueue 4 4K 10K 42378K 5529 0 0 256,1K sigio 1 1K 1K 42378K 121 0 0 32 file 646 41K 52K 42378K 18711429 0 0 64 file desc 120 31K 45K 42378K 4422440 0 0 256,512 dev_t 663 83K 83K 42378K 663 0 0 128 timecounter 10 2K 2K 42378K 10 0 0 128 kld 4 1K 1K 42378K 35 0 0 16,32,128 Memory Totals: In Use Free Requests 7564K 1449K 58199125 Here is the kernel: machine i386 cpu I586_CPU cpu I686_CPU ident NEWMONSTER maxusers 128 makeoptions DEBUG=-g options CPU_ENABLE_SSE options INET options FFS options FFS_ROOT options NFS options SOFTUPDATES options PROCFS options CD9660 options COMPAT_43 options UCONSOLE options USERCONFIG options VISUAL_USERCONFIG options P1003_1B options _KPOSIX_PRIORITY_SCHEDULING options _KPOSIX_VERSION=199309L options KTRACE options PERFMON options SYSVSHM options SYSVMSG options SYSVSEM options SHMMAXPGS=8192 # max amount of shared memory pages (4k on i386) options SHMALL=33554432 # max amount of shared memory (bytes) options SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)" options SHMMNI=256 # max shared mem id's per system options SHMSEG=256 # max shared mem id's per process options MSGMNB=8192 # max # of bytes in a queue options MSGMNI=256 # number of message queue identifiers options MSGSEG=256 # number of message segments per queue options MSGSSZ=64 # size of a message segment options MSGTQL=8192 # max messages in system options SEMMAP=256 options SEMMNI=256 options SEMMNS=512 options SEMMNU=256 options INCLUDE_CONFIG_FILE options IPFILTER options IPFILTER_LOG options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFW2 options RANDOM_IP_ID options ICMP_BANDLIM options ACCEPT_FILTER_HTTP options VESA options PANIC_REBOOT_WAIT_TIME=20 options SMBFS options LIBMCHAIN options LIBICONV options NETSMB options NETSMBCRYPTO options UFS_DIRHASH options SHOW_BUSYBUFS options HZ=1000 device isa device pci device fdc0 at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 device fd1 at fdc0 drive 1 device ata0 at isa? port IO_WD1 irq 14 device ata1 at isa? port IO_WD2 irq 15 device ata device atadisk device atapicd options ATA_STATIC_ID device ahc device aic0 at isa? device scbus device da device sa device cd device pass device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 device vga0 at isa? device sc0 at isa? flags 0x100 device npx0 at nexus? port IO_NPX irq 13 device sio0 at isa? port IO_COM1 flags 0x10 irq 4 device sio1 at isa? port IO_COM2 irq 3 # device sio2 at isa? disable port IO_COM3 irq 5 # device sio3 at isa? disable port IO_COM4 irq 9 device miibus device dc device fxp pseudo-device loop pseudo-device ether pseudo-device tun pseudo-device pty pseudo-device bpf 8 pseudo-device vn 2 pseudo-device gzip pseudo-device splash device smbus device intpm device alpm device ichsmb device viapm device smb device iicbus device iicbb device ic device iic device iicsmb device apm0 at nexus? This machine runs squid && pgsql - I cleared softupdates from squid and pgsql filesystems but the panic chase me. -- VAMPIRO-RIPN To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200209090230.g892U4Cl067557>