From owner-freebsd-current@FreeBSD.ORG Wed Aug 6 22:30:32 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9271937B404 for ; Wed, 6 Aug 2003 22:30:32 -0700 (PDT) Received: from phk.freebsd.dk (phk.freebsd.dk [212.242.86.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id 158E043F85 for ; Wed, 6 Aug 2003 22:30:31 -0700 (PDT) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by phk.freebsd.dk (8.12.8/8.12.8) with ESMTP id h775UTV3046052 for ; Thu, 7 Aug 2003 05:30:29 GMT (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.9/8.12.9) with ESMTP id h775USPI051751 for ; Thu, 7 Aug 2003 07:30:29 +0200 (CEST) (envelope-from phk@phk.freebsd.dk) To: current@freebsd.org From: Poul-Henning Kamp Date: Thu, 07 Aug 2003 07:30:28 +0200 Message-ID: <51750.1060234228@critter.freebsd.dk> Subject: reproducible deadlock... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2003 05:30:32 -0000 I am still able to reproduce a deadlock by running a "make -j 21 buildworld" on a 48MB physmem configuration. The first LOR is a false positive according to Alan Cox. Interestingly, gstat(8) over an ssh session continues to run, so Giant is presumably not involved. I can reproduce it with both SCHED_ULE and SCHED_4BSD. I've left the machine hanging in case anybody has any requests for further DDB probing. Poul-Henning syv# sh t82.sh lock order reversal 1st 0xc135db90 vm object (vm object) @ vm/vm_object.c:434 2nd 0xc082f110 system map (system map) @ vm/vm_kern.c:328 Stack backtrace: backtrace(c038747a,c082f110,c0392feb,c0392feb,c0392e93) at 0xc01fe7a7 = backtrace+0x17 witness_lock(c082f110,8,c0392e93,148,c1455130) at 0xc02239f2 = witness_lock+0x672 _mtx_lock_flags(c082f110,0,c0392e8a,148,3) at 0xc01f496a = _mtx_lock_flags+0xba _vm_map_lock(c082f0b0,c0392e8a,148,2b3,c03e61e0) at 0xc02ef006 = _vm_map_lock+0x36 kmem_malloc(c082f0b0,1000,101,c5616ac4,c02fef72) at 0xc02ee1da = kmem_malloc+0x3a page_alloc(c083a1c0,1000,c5616ab7,101,c03dfc98) at 0xc02ff1e7 = page_alloc+0x27 slab_zalloc(c083a1c0,101,c039477e,682,c1244c54) at 0xc02fef72 = slab_zalloc+0xc2 uma_zone_slab(c083a1c0,101,c0394775,682,0) at 0xc0300138 = uma_zone_slab+0xe8 uma_zalloc_internal(c083a1c0,0,101,706,0) at 0xc030045f = uma_zalloc_internal+0x4f uma_zfree_arg(c1244c40,c54498a0,0,773,0) at 0xc03007e2 = uma_zfree_arg+0x2c2 swp_pager_meta_free_all(c135db90,c039278c,c03926f5,20f) at 0xc02e9482 = swp_pager_meta_free_all+0xf2 swap_pager_dealloc(c135db90,1,c039472e,104,0) at 0xc02e769a = swap_pager_dealloc+0x11a vm_pager_deallocate(c135db90,0,c039391c,261,c0394775) at 0xc02fdbed = vm_pager_deallocate+0x3d vm_object_terminate(c135db90,0,c039391c,1b2,c01f4a50) at 0xc02f5953 = vm_object_terminate+0x1f3 vm_object_deallocate(c135db90,c1205654,c135db90,c1205654,c5616c64) at 0xc02f5701 = vm_object_deallocate+0x371 vm_map_entry_delete(c16bc100,c1205654,c0393059,897,0) at 0xc02f186b = vm_map_entry_delete+0x3b vm_map_delete(c16bc100,0,bfc00000,c16bc100,c125c000) at 0xc02f1c70 = vm_map_delete+0x3e0 vm_map_remove(c16bc100,0,bfc00000,11d,c0382132) at 0xc02f1ce8 = vm_map_remove+0x58 exit1(c164e260,0,c0382132,65,c5616d40) at 0xc01e6f36 = exit1+0x686 sys_exit(c164e260,c5616d10,c039884a,3ee,1) at 0xc01e68a1 = sys_exit+0x41 syscall(2f,2f,2f,bfbff8dc,0) at 0xc033e213 = syscall+0x273 Xint0x80_syscall() at 0xc032d6bd = Xint0x80_syscall+0x1d --- syscall (1), eip = 0x80b2d63, esp = 0xbfbff85c, ebp = 0xbfbff878 --- Debugger("witness_lock") Stopped at 0xc032bf34 = Debugger+0x54: xchgl %ebx,0xc0431044 = in_Debugger.0 db> cont load: 0.00 cmd: sh 26559 [ufs] 0.00u 0.00s 0% 32k load: 0.00 cmd: sh 26559 [ufs] 0.00u 0.00s 0% 32k load: 0.00 cmd: sh 26559 [ufs] 0.00u 0.00s 0% 32k Stopped at 0xc031845c = siointr1+0xdc: jmp 0xc0318590 = siointr1+0x210 db> ps pid proc addr uid ppid pgrp flag stat wmesg wchan cmd 26559 c12f6000 c5809000 0 26251 498 0004002 [SLP]ufs 0xc149fd48] sh 26558 c13843c8 c57e5000 0 26546 498 4000002 [SLP]ufs 0xc140b1e0] sh 26557 c195fd3c c59cc000 0 26545 498 4000002 [SLP]ufs 0xc1773670] sh 26556 c12e1000 c5773000 0 26251 498 0004002 [SLP]ufs 0xc149fd48] sh 26555 c12e13c8 c5775000 0 26251 498 0004002 [SLP]ufs 0xc149fd48] sh 26554 c16b9000 c56b4000 0 26251 498 0004002 [SLP]ufs 0xc149fd48] sh 26553 c13583c8 c59e4000 0 26251 498 0004002 [SLP]ufs 0xc14b1d48] sh 26552 c12a23c8 c576d000 0 26543 498 4000002 [SLP]thrd_sleep 0xc09820ec] sh 26551 c12e05ac c56fe000 0 26251 498 0004002 [SLP]ufs 0xc149fd48] sh 26550 c195d3c8 c5969000 0 26251 498 0004002 [SLP]ufs 0xc14b8304] sh 26549 c14b51e4 c5880000 0 26251 498 0004002 [SLP]ufs 0xc14b1b00] sh 26548 c12f6b58 c580f000 0 26539 498 4000002 [SLP]thrd_sleep 0xc09820ec] sh 26547 c164dd3c c563b000 0 26544 498 4000002 [SLP]ufs 0xc1773670] sh 26546 c195d790 c596b000 0 26251 498 0004002 [SLP]wait 0xc195d790] sh 26545 c14b5000 c587f000 0 26251 498 0004002 [SLP]wait 0xc14b5000] sh 26544 c13725ac c590d000 0 26251 498 0004002 [SLP]wait 0xc13725ac] sh 26543 c1372b58 c5910000 0 26251 498 0004002 [SLP]wait 0xc1372b58] sh 26542 c19583c8 c5961000 0 1 498 0004002 [SLP]ufs 0xc1773670] cc 26540 c1491000 c5829000 0 26535 498 0004002 [SLP]ufs 0xc1773670] cc 26539 c12a2974 c5770000 0 26251 498 0004002 [SLP]wait 0xc12a2974][SWAP] sh 26537 c14915ac c587a000 0 26536 498 0004002 [SLP]ufs 0xc1773670] cc 26536 c139b3c8 c5a3f000 0 26531 498 0004002 [SLP]wait 0xc139b3c8][SWAP] sh 26535 c12e1b58 c5779000 0 26251 498 0004002 [SLP]wait 0xc12e1b58][SWAP] sh 26534 c1958974 c5964000 0 26532 498 0004002 [SLP]ufs 0xc1773670] cc 26533 c195dd3c c596e000 0 26530 498 0004002 [SLP]ufs 0xc1773670] cc 26532 c12a2b58 c5771000 0 26251 498 0004002 [SLP]wait 0xc12a2b58][SWAP] sh 26531 c12eed3c c5808000 0 26397 498 0004002 [CV]select 0xc04024b4] make 26530 c13705ac c5905000 0 26251 498 0004002 [SLP]wait 0xc13705ac][SWAP] sh 26529 c12e1d3c c577a000 0 26527 498 0004002 [SLP]piperd 0xc1779f20] as 26528 c12a21e4 c576c000 0 26527 498 0004002 [SLP]ufs 0xc149fd48] cc1 26527 c12ce5ac c57fc000 0 1 498 0004002 [SLP]wait 0xc12ce5ac][SWAP] cc 26524 c12deb58 c56f9000 0 26518 498 0004002 [SLP]piperd 0xc194af20] as 26523 c12a2000 c576b000 0 26518 498 0004002 [SLP]ufs 0xc149fd48] cc1 26522 c160e5ac c588a000 0 26516 498 0004002 [SLP]piperd 0xc194adc0] as 26521 c195e974 c5974000 0 26516 498 0004002 [SLP]ufs 0xc149fd48] cc1 26520 c195f790 c597b000 0 26513 498 0004002 [SLP]piperd 0xc194a000] as 26519 c12f61e4 c580a000 0 26513 498 0004002 [SLP]ufs 0xc149fd48] cc1 26518 c1372974 c590f000 0 1 498 0004002 [SLP]wait 0xc1372974][SWAP] cc 26516 c13675ac c58fd000 0 1 498 0004002 [SLP]wait 0xc13675ac][SWAP] cc 26515 c1958790 c5963000 0 26512 498 0004002 [SLP]piperd 0xc1343b00] as 26514 c12de5ac c56f6000 0 26512 498 0004002 [SLP]ufs 0xc149fd48] cc1 26513 c139b1e4 c5a3e000 0 1 498 0004002 [SLP]wait 0xc139b1e4][SWAP] cc 26512 c168e974 c569d000 0 1 498 0004002 [SLP]wait 0xc168e974][SWAP] cc 26511 c12de790 c56f7000 0 26506 498 0004002 [SLP]piperd 0xc135b210] as 26509 c1358974 c5a32000 0 26506 498 0004002 [SLP]ufs 0xc149fd48] cc1 26506 c12911e4 c5764000 0 1 498 0004002 [SLP]wait 0xc12911e4][SWAP] cc 26505 c16b9d3c c56bb000 0 26501 498 0004002 [SLP]piperd 0xc135b8f0] as 26504 c1384b58 c57e9000 0 26501 498 0004002 [SLP]ufs 0xc149fd48] cc1 26503 c195fb58 c59cb000 0 26497 498 0004002 [SLP]piperd 0xc135b0b0] as 26502 c160e974 c588c000 0 26497 498 0004002 [SLP]ufs 0xc149fd48] cc1 26501 c12ce790 c57fd000 0 1 498 0004002 [SLP]wait 0xc12ce790][SWAP] cc 26499 c13581e4 c59a3000 0 26493 498 0004002 [SLP]piperd 0xc135bf20] as 26498 c168d1e4 c5643000 0 26493 498 0004002 [SLP]ufs 0xc149fd48] cc1 26497 c195e3c8 c5971000 0 1 498 0004002 [SLP]wait 0xc195e3c8][SWAP] cc 26495 c160e790 c588b000 0 26484 498 0004002 [SLP]piperd 0xc1779580] as 26494 c168d5ac c5645000 0 26484 498 0004002 [SLP]ufs 0xc149fd48] cc1 26493 c164c974 c560a000 0 1 498 0004002 [SLP]wait 0xc164c974][SWAP] cc 26492 c1358790 c5a31000 0 26485 498 0004002 [SLP]piperd 0xc1338210] as 26491 c195db58 c596d000 0 26485 498 0004002 [SLP]ufs 0xc14b1c24] cc1 26490 c1372d3c c5911000 0 26478 498 0004002 [SLP]piperd 0xc1338d10] as 26489 c1291790 c5767000 0 26478 498 0004002 [SLP]ufs 0xc149fd48] cc1 26487 c1291000 c5715000 0 26483 498 0004002 [SLP]piperd 0xc1338a50] as 26486 c1370790 c5906000 0 26483 498 0004002 [SLP]ufs 0xc149fd48] cc1 26485 c164d790 c5638000 0 1 498 0004002 [SLP]wait 0xc164d790][SWAP] cc 26484 c139b790 c5a41000 0 1 498 0004002 [SLP]wait 0xc139b790][SWAP] cc 26483 c12913c8 c5765000 0 1 498 0004002 [SLP]wait 0xc12913c8][SWAP] cc 26478 c14b5b58 c5885000 0 1 498 0004002 [SLP]wait 0xc14b5b58][SWAP] cc 26476 c168d000 c5642000 0 1 0 0002204 [CV]allproc 0xc03dc838] nfsiod 9 26475 c139b000 c5a3d000 0 1 0 0002204 [CV]allproc 0xc03dc838] nfsiod 8 26474 c139ab58 c5a3b000 0 1 0 0002204 [CV]allproc 0xc03dc838] nfsiod 7 26473 c16b83c8 c56ae000 0 1 0 0002204 [CV]allproc 0xc03dc838] nfsiod 6 26467 c1358d3c c5a34000 0 23685 498 0004002 [SLP]ufs 0xc149fd48] cc1 26466 c168db58 c5648000 0 26456 498 0004002 [SLP]piperd 0xc1333a50] as 26465 c12de1e4 c56f4000 0 26456 498 0004002 [SLP]ufs 0xc149fd48] cc1 26464 c12e0974 c5700000 0 26454 498 0004002 [SLP]piperd 0xc13389a0] as 26463 c12e1974 c5778000 0 26451 498 0004002 [SLP]piperd 0xc1338630] as 26462 c195e000 c596f000 0 26458 498 0004002 [SLP]piperd 0xc18fbdc0] as 26461 c16b9790 c56b8000 0 26454 498 0004002 [SLP]ufs 0xc149fd48] cc1 26460 c168e3c8 c564c000 0 26451 498 0004002 [SLP]ufs 0xc149fd48] cc1 26459 c14913c8 c5879000 0 26458 498 0004002 [SLP]ufs 0xc149fd48] cc1 26458 c12e0d3c c5702000 0 26455 498 0004002 [SLP]wait 0xc12e0d3c][SWAP] cc 26457 c1370974 c5907000 0 26452 498 0004002 [SLP]ufs 0xc1773670] cc 26456 c12ce1e4 c57fa000 0 26450 498 0004002 [SLP]wait 0xc12ce1e4][SWAP] cc 26455 c1491d3c c587e000 0 26251 498 0004002 [SLP]wait 0xc1491d3c][SWAP] sh 26454 c16b91e4 c56b5000 0 26449 498 0004002 [SLP]wait 0xc16b91e4][SWAP] cc 26452 c168d3c8 c5644000 0 26251 498 0004002 [SLP]wait 0xc168d3c8][SWAP] sh 26451 c195eb58 c5975000 0 26447 498 0004002 [SLP]wait 0xc195eb58][SWAP] cc 26450 c1358b58 c5a33000 0 26251 498 0004002 [SLP]wait 0xc1358b58][SWAP] sh 26449 c14b5d3c c5886000 0 26251 498 0004002 [SLP]wait 0xc14b5d3c][SWAP] sh 26448 c164c000 c5605000 0 23611 498 0004002 [SLP]ufs 0xc149fd48] cc1 26447 c1372790 c590e000 0 26251 498 0004002 [SLP]wait 0xc1372790][SWAP] sh 26445 c12e0000 c56fb000 0 23742 498 0004002 [SLP]ufs 0xc149fd48] cc1 26444 c12ee974 c5806000 0 23700 498 0004002 [SLP]ufs 0xc149fd48] cc1 26442 c12f6790 c580d000 0 26438 498 0006002 [CV]allproc 0xc03dc838] as 26441 c1491790 c587b000 0 26438 498 0006002 [CV]allproc 0xc03dc838] cc1 26440 c12ee790 c5805000 0 26432 498 0006002 [CV]allproc 0xc03dc838] as 26439 c1358000 c5991000 0 26432 498 0006002 [CV]allproc 0xc03dc838] cc1 26438 c12ee5ac c5804000 0 1 498 0004002 [SLP]wait 0xc12ee5ac][SWAP] cc 26436 c13723c8 c590c000 0 26434 498 0006002 [CV]allproc 0xc03dc838] as 26435 c195f3c8 c5979000 0 26434 498 0006002 [CV]allproc 0xc03dc838] cc1 26434 c19581e4 c5960000 0 1 498 0004002 [SLP]wait 0xc19581e4][SWAP] cc 26432 c195e790 c5973000 0 1 498 0004002 [SLP]wait 0xc195e790][SWAP] cc 26431 c12ced3c c5800000 0 26428 498 0006002 [CV]allproc 0xc03dc838] as 26430 c14b5790 c5883000 0 26428 498 0006002 [CV]allproc 0xc03dc838] cc1 26428 c1958b58 c5965000 0 1 498 0004002 [SLP]wait 0xc1958b58][SWAP] cc 26426 c139a5ac c5a38000 0 26424 498 0006002 [CV]allproc 0xc03dc838] as 26425 c139a790 c5a39000 0 26424 498 0006002 [CV]allproc 0xc03dc838] cc1 26424 c12de974 c56f8000 0 1 498 0004002 [SLP]wait 0xc12de974][SWAP] cc 26423 c168e000 c564a000 0 26420 498 0006002 [CV]allproc 0xc03dc838] as 26422 c1384790 c57e7000 0 26420 498 0006002 [CV]allproc 0xc03dc838] cc1 26420 c1370b58 c5908000 0 1 498 0004002 [SLP]wait 0xc1370b58][SWAP] cc 26415 c195e1e4 c5970000 0 1 498 0004002 [SLP]ufs 0xc1773670] cc 26397 c164cd3c c560c000 0 26394 498 0004002 [SLP]wait 0xc164cd3c][SWAP] sh 26394 c12e0b58 c5701000 0 26393 498 0004002 [CV]select 0xc04024b4] make 26393 c160e000 c5887000 0 26391 498 0004002 [SLP]wait 0xc160e000][SWAP] sh 26391 c139a3c8 c5a37000 0 23509 498 0004002 [CV]select 0xc04024b4] make 26378 c16b8790 c56b0000 0 26374 498 0006002 [CV]allproc 0xc03dc838] as 26377 c13585ac c59e5000 0 26374 498 0006002 [CV]allproc 0xc03dc838] cc1 26376 c160ed3c c588e000 0 26372 498 0006002 [CV]allproc 0xc03dc838] as 26375 c12ce974 c57fe000 0 26372 498 0006002 [CV]allproc 0xc03dc838] cc1 26374 c12e01e4 c56fc000 0 1 498 0004002 [SLP]wait 0xc12e01e4][SWAP] cc 26372 c1384d3c c57ea000 0 1 498 0004002 [SLP]wait 0xc1384d3c][SWAP] cc 26369 c12a2d3c c5772000 0 26363 498 0006002 [CV]allproc 0xc03dc838] as 26368 c168eb58 c569e000 0 26363 498 0006002 [CV]allproc 0xc03dc838] cc1 26367 c195d974 c596c000 0 26360 498 0006002 [CV]allproc 0xc03dc838] as 26365 c139b974 c5a42000 0 26360 498 0006002 [CV]allproc 0xc03dc838] cc1 26364 c195f5ac c597a000 0 26351 498 0006002 [CV]allproc 0xc03dc838] as 26363 c12f65ac c580c000 0 1 498 0004002 [SLP]wait 0xc12f65ac][SWAP] cc 26360 c168ed3c c569f000 0 1 498 0004002 [SLP]wait 0xc168ed3c][SWAP] cc 26358 c195e5ac c5972000 0 26355 498 0006002 [CV]allproc 0xc03dc838] as 26357 c139a1e4 c5a36000 0 26355 498 0006002 [CV]allproc 0xc03dc838] cc1 26355 c12f6d3c c5810000 0 1 498 0004002 [SLP]wait 0xc12f6d3c][SWAP] cc 26353 c16b9b58 c56ba000 0 26351 498 0006002 [CV]allproc 0xc03dc838] cc1 26351 c168d790 c5646000 0 1 498 0004002 [SLP]wait 0xc168d790][SWAP] cc 26251 c1367d3c c5901000 0 23519 498 0004002 [CV]select 0xc04024b4] make 26230 c1370000 c5902000 0 1 0 0002204 [CV]allproc 0xc03dc838] nfsiod 5 26229 c195d000 c5967000 0 1 0 0002204 [CV]allproc 0xc03dc838] nfsiod 4 26119 c1372000 c590a000 0 26108 498 0004002 [SLP]ufs 0xc1773670] cc 26108 c1291974 c5768000 0 26105 498 0004002 [SLP]wait 0xc1291974][SWAP] sh 26105 c164d974 c5639000 0 23499 498 0004002 [SLP]wait 0xc164d974][SWAP] sh 23742 c12ce000 c57f9000 0 23736 498 0004002 [SLP]wait 0xc12ce000][SWAP] cc 23736 c1491974 c587c000 0 23614 498 0004002 [SLP]wait 0xc1491974][SWAP] sh 23700 c16b8d3c c56b3000 0 23549 498 0004002 [SLP]wait 0xc16b8d3c][SWAP] cc 23685 c12e0790 c56ff000 0 23682 498 0004002 [SLP]wait 0xc12e0790][SWAP] cc 23682 c13841e4 c57a1000 0 23676 498 0004002 [SLP]wait 0xc13841e4][SWAP] sh 23676 c164db58 c563a000 0 23529 498 0004002 [SLP]wait 0xc164db58][SWAP] sh 23614 c14b55ac c5882000 0 23493 498 0004002 [SLP]wait 0xc14b55ac][SWAP] sh 23611 c143d000 c55ec000 0 23604 498 0004002 [SLP]wait 0xc143d000][SWAP] cc 23604 c12ceb58 c57ff000 0 23581 498 0004002 [SLP]wait 0xc12ceb58][SWAP] sh 23581 c14911e4 c5878000 0 23496 498 0004002 [SLP]wait 0xc14911e4][SWAP] sh 23549 c12a2790 c576f000 0 23526 498 0004002 [SLP]wait 0xc12a2790][SWAP] sh 23529 c168dd3c c5649000 0 23523 498 0004002 [CV]select 0xc04024b4] make 23526 c13845ac c57e6000 0 23492 498 0004002 [SLP]wait 0xc13845ac][SWAP] sh 23523 c16b81e4 c56ad000 0 23481 498 0004002 [SLP]wait 0xc16b81e4][SWAP] sh 23519 c12ee1e4 c5802000 0 23481 498 0004002 [SLP]wait 0xc12ee1e4][SWAP] sh 23509 c160e1e4 c5888000 0 23481 498 0004002 [SLP]wait 0xc160e1e4][SWAP] sh 23499 c14b5974 c5884000 0 23495 498 0004002 [CV]select 0xc04024b4] make 23496 c164d1e4 c5635000 0 23494 498 0004002 [CV]select 0xc04024b4] make 23495 c12e15ac c5776000 0 23481 498 0004002 [SLP]wait 0xc12e15ac][SWAP] sh 23494 c1291b58 c5769000 0 23481 498 0004002 [SLP]wait 0xc1291b58][SWAP] sh 23493 c12f6974 c580e000 0 23491 498 0004002 [CV]select 0xc04024b4] make 23492 c1384974 c57e8000 0 23490 498 0004002 [CV]select 0xc04024b4] make 23491 c164c3c8 c5607000 0 23481 498 0004002 [SLP]wait 0xc164c3c8][SWAP] sh 23490 c16b95ac c56b7000 0 23481 498 0004002 [SLP]wait 0xc16b95ac][SWAP] sh 23481 c12de000 c56f3000 0 22718 498 0004002 [CV]select 0xc04024b4] make 22718 c168e5ac c564d000 0 22711 498 0004002 [SLP]wait 0xc168e5ac][SWAP] sh 22711 c16b8000 c56ac000 0 22710 498 0004002 [CV]select 0xc04024b4] make 22710 c168d974 c5647000 0 593 498 0004002 [SLP]wait 0xc168d974][SWAP] sh 712 c168e1e4 c564b000 0 709 712 0004002 [CV]sysctl lock 0xc03dd238] top 709 c164c790 c5609000 0 703 709 0004002 [SLP]pause 0xc164b000][SWAP] csh 703 c143db58 c55f2000 0 427 703 0000100 [CV]select 0xc04024b4] sshd 635 c164c1e4 c5606000 0 582 635 0004002 [SLP]thrd_sleep 0xc16bc63c] systat 593 c143d974 c55f1000 0 591 498 0004002 [CV]select 0xc04024b4] make 591 c143dd3c c55f3000 0 562 498 0004002 [SLP]wait 0xc143dd3c][SWAP] sh 582 c164d5ac c5637000 0 567 582 0004002 [SLP]pause 0xc1646000][SWAP] csh 567 c164c5ac c5608000 0 427 567 0000100 [CV]select 0xc04024b4] sshd 562 c164d000 c560d000 0 498 498 0004002 [CV]select 0xc04024b4] make 556 c11f93c8 c53ae000 0 0 0 0000204 [SLP]mdwait 0xc15bf400] md21 498 c11f9b58 c53d9000 0 491 498 0004002 [SLP]wait 0xc11f9b58][SWAP] sh 497 c14385ac c5599000 0 495 497 0004002 [SLP]nanslp 0xc03dd3ec] gstat 495 c143d1e4 c55ed000 0 493 495 0004002 [SLP]pause 0xc1443000][SWAP] csh 493 c143d790 c55f0000 0 427 493 0000100 [CV]select 0xc04024b4] sshd 491 c14381e4 c5597000 0 490 491 0004002 [SLP]pause 0xc1453000][SWAP] csh 490 c1438b58 c559c000 0 1 490 0004102 [SLP]wait 0xc1438b58][SWAP] login 489 c11f95ac c53af000 0 1 489 0004002 [SLP]ttyin 0xc13ab210][SWAP] getty 488 c11f9974 c53b1000 0 1 488 0004002 [SLP]ttyin 0xc13ab010][SWAP] getty 487 c13e6d3c c5401000 0 1 487 0004002 [SLP]ttyin 0xc13ac410][SWAP] getty 486 c11f9790 c53b0000 0 1 486 0004002 [SLP]ttyin 0xc13bf810][SWAP] getty 485 c14383c8 c5598000 0 1 485 0004002 [SLP]ttyin 0xc13bf210][SWAP] getty 484 c1438974 c559b000 0 1 484 0004002 [SLP]ttyin 0xc1375410][SWAP] getty 483 c1438000 c5596000 0 1 483 0004002 [SLP]ttyin 0xc13be810][SWAP] getty 482 c1438790 c559a000 0 1 482 0004002 [SLP]ttyin 0xc11fc010][SWAP] getty 427 c143d5ac c55ef000 0 1 427 0000100 [CV]select 0xc04024b4][SWAP] sshd 283 c143d3c8 c55ee000 0 1 283 0000000 [CV]select 0xc04024b4] rpcbind 269 c1438d3c c55eb000 0 1 269 0000000 [CV]select 0xc04024b4] syslogd 50 c11f9d3c c53da000 0 0 0 0000204 [SLP]- 0xc04068cc] nfsiod 3 49 c13e6000 c53fa000 0 0 0 0000204 [SLP]- 0xc04068c8] nfsiod 2 48 c13e61e4 c53fb000 0 0 0 0000204 [SLP]- 0xc04068c4] nfsiod 1 47 c13e63c8 c53fc000 0 0 0 0000204 [SLP]- 0xc04068c0] nfsiod 0 46 c13e65ac c53fd000 0 0 0 0000204 [SLP]vlruwt 0xc13e65ac] vnlru 45 c13e6790 c53fe000 0 0 0 0000204 [SLP]syncer 0xc03dcda0] syncer 44 c13e6974 c53ff000 0 0 0 0000204 [SLP]psleep 0xc0402950] bufdaemon 43 c13e6b58 c5400000 0 0 0 000020c [SLP]pgzero 0xc040cbc8] pagezero 42 c11b8b58 c538d000 0 0 0 0000204 [SLP]psleep 0xc040cc20] vmdaemon 41 c11b8d3c c538e000 0 0 0 0000204 [SLP]psleep 0xc040cc0c] pagedaemon 40 c11f7000 c53a4000 0 0 0 0000204 new [IWAIT] irq8: rtc 39 c11f71e4 c53a5000 0 0 0 0000204 new [IWAIT] irq0: clk 38 c11f73c8 c53a6000 0 0 0 0000204 [IWAIT] irq7: ppc0 37 c11f75ac c53a7000 0 0 0 0000204 new [IWAIT] irq3: sio1 36 c11f7790 c53a8000 0 0 0 0000204 new [IWAIT] irq4: sio0 35 c11f7974 c53a9000 0 0 0 0000204 [IWAIT] swi0: tty:sio 34 c11f7b58 c53aa000 0 0 0 0000204 [IWAIT] irq6: fdc0 33 c11f7d3c c53ab000 0 0 0 0000204 [IWAIT] irq1: atkbd0 32 c11f9000 c53ac000 0 0 0 0000204 [CV]isp_kthread_cv 0xc0985874] isp1: fc_thrd 31 c11f91e4 c53ad000 0 0 0 0000204 [IWAIT] irq10: isp1 30 c112b5ac c5356000 0 0 0 0000204 [CV]isp_kthread_cv 0xc0985474] isp0: fc_thrd 29 c112b790 c5357000 0 0 0 0000204 [IWAIT] irq11: isp0 28 c112b974 c5358000 0 0 0 0000204 [SLP]usbevt 0xc11cf210] usb1 27 c112bb58 c5359000 0 0 0 0000204 [SLP]usbtsk 0xc03d3c88] usbtask 26 c112bd3c c535a000 0 0 0 0000204 [SLP]usbevt 0xc11cd210] usb0 25 c11b8000 c5360000 0 0 0 0000204 [IWAIT] irq12: fxp0 uhci0+ 24 c11b81e4 c5361000 0 0 0 0000204 [IWAIT] irq15: ata1 23 c11b83c8 c5362000 0 0 0 0000204 [IWAIT] irq14: ata0 9 c11b85ac c5363000 0 0 0 0000204 [SLP]actask 0xc04e99ac] acpi_task2 8 c11b8790 c5364000 0 0 0 0000204 [SLP]actask 0xc04e99ac] acpi_task1 7 c11b8974 c5365000 0 0 0 0000204 [SLP]actask 0xc04e99ac] acpi_task0 22 c09901e4 c4f24000 0 0 0 0000204 new [IWAIT] irq9: acpi0 21 c09903c8 c4f25000 0 0 0 0000204 new [IWAIT] irq13: 20 c09905ac c4f26000 0 0 0 0000204 new [IWAIT] swi5:+ 19 c0990790 c4f27000 0 0 0 0000204 new [IWAIT] swi6: task queue 18 c0990974 c4f28000 0 0 0 0000204 [IWAIT] swi3: cambio 17 c0990b58 c4f29000 0 0 0 0000204 new [IWAIT] swi2: camnet 16 c0990d3c c4f51000 0 0 0 0000204 new [IWAIT] swi6: acpitaskq 15 c112b000 c5353000 0 0 0 0000204 [SLP]- 0xc03c9660] random 6 c112b1e4 c5354000 0 0 0 0000204 [SLP]crypto_ret_wait 0xc040bc54] crypto returns 5 c112b3c8 c5355000 0 0 0 0000204 [SLP]crypto_wait 0xc040bbf4] crypto 4 c0989000 c4ecd000 0 0 0 0000204 [SLP]- 0xc03d6220] g_down 3 c09891e4 c4f1c000 0 0 0 0000204 [SLP]- 0xc03d621c] g_up 2 c09893c8 c4f1d000 0 0 0 0000204 [SLP]- 0xc03d6214] g_event 14 c09895ac c4f1e000 0 0 0 0000204 new [IWAIT] swi4: vm 13 c0989790 c4f1f000 0 0 0 000020c [IWAIT] swi7: tty:sio clock 12 c0989974 c4f20000 0 0 0 0000204 [IWAIT] swi1: net 11 c0989b58 c4f21000 0 0 0 000020c [CPU 0] idle 1 c0989d3c c4f22000 0 0 1 0004200 [SLP]wait 0xc0989d3c] init 10 c0990000 c4f23000 0 0 0 0000204 [CV]ktrace 0xc03d9854] ktrace 0 c03d6340 c0519000 0 0 0 0000200 [SLP]sched 0xc03d6340] swapper db> show lock Ambiguous db> show locks db> show witness Sleep locks: 0 g_xdown -- last acquired @ geom/geom_io.c:359 2 Giant -- last acquired @ kern/kern_timeout.c:216 3 eventhandler -- last acquired @ kern/subr_eventhandler.c:213 4 eventhandler list -- last acquired @ kern/kern_exit.c:210 3 g_disk_done -- last acquired @ geom/geom_disk.c:179 4 bio queue -- last acquired @ geom/geom_io.c:70 10 UMA pcpu -- last acquired @ vm/uma_core.c:255 11 KMAP ENTRY -- last acquired @ vm/uma_core.c:265 11 UMA zone -- last acquired @ vm/uma_core.c:265 3 UMA lock -- last acquired @ vm/uma_core.c:1167 7 system map -- last acquired @ vm/vm_map.c:2228 8 kmem object -- last acquired @ vm/vm_kern.c:436 9 vm page queue mutex -- last acquired @ vm/vm_pageout.c:1464 10 UMA pcpu -- (already displayed) 10 vnode interlock -- last acquired @ kern/vfs_subr.c:3228 11 Syncer mtx -- last acquired @ kern/vfs_subr.c:1798 11 vnode_free_list -- last acquired @ kern/vfs_subr.c:760 11 spechash -- last acquired @ kern/vfs_subr.c:2009 10 CMAPCADDR12 -- last acquired @ i386/i386/pmap.c:2582 8 vm object -- last acquired @ vm/vm_object.c:434 9 swap_pager list -- last acquired @ vm/swap_pager.c:519 9 vm page queue mutex -- (already displayed) 7 Malloc Stats -- last acquired @ kern/kern_malloc.c:324 4 vm object_list -- last acquired @ vm/vm_object.c:621 8 kmem object -- (already displayed) 8 vm object -- (already displayed) 3 malloc -- last acquired @ kern/kern_malloc.c:561 3 md bio queue -- last acquired @ dev/md/md.c:605 3 vm86 lock -- last acquired @ i386/i386/vm86.c:583 3 GEOM orphanage -- last acquired @ geom/geom_event.c:169 3 ithread -- last acquired @ kern/kern_intr.c:265 3 pbuf mutex -- last acquired @ vm/vm_pager.c:443 3 taskqueue list -- last acquired @ kern/subr_taskqueue.c:87 3 sf_bufs list lock -- last acquired @ kern/uipc_syscalls.c:1642 7 Malloc Stats -- (already displayed) 7 system map -- (already displayed) 3 kernel linker -- last acquired @ kern/kern_linker.c:1032 3 rman head -- last acquired @ kern/subr_rman.c:110 3 ACPI semaphore -- last acquired @ /bang/src/sys/dev/acpica/Osd/OsdSynch.c:319 3 acpica subsystem lock -- last acquired @ /bang/src/sys/dev/acpica/Osd/OsdSynch.c:380 3 rman -- last acquired @ kern/subr_rman.c:445 7 Malloc Stats -- (already displayed) 7 system map -- (already displayed) 3 pseudofs_vncache -- last acquired @ fs/pseudofs/pseudofs_vncache.c:227 3 devd -- last acquired @ kern/subr_bus.c:479 7 Malloc Stats -- (already displayed) 10 UMA pcpu -- (already displayed) 3 devstat -- last acquired @ kern/subr_devstat.c:398 3 pseudofs -- last acquired @ fs/pseudofs/pseudofs_fileno.c:86 3 mntid -- last acquired @ kern/vfs_subr.c:583 4 mountlist -- last acquired @ kern/vfs_subr.c:3524 3 isp -- last acquired @ dev/isp/isp_freebsd.c:2696 7 Malloc Stats -- (already displayed) 10 UMA pcpu -- (already displayed) 3 bpf global lock -- last acquired @ net/bpf.c:1382 3 bdone lock -- last acquired @ kern/vfs_bio.c:2953 3 netisr lock -- last acquired @ net/netisr.c:215 4 tcp -- last acquired @ netinet/tcp_timer.c:141 5 inp -- last acquired @ netinet/tcp_input.c:650 6 network driver -- last acquired @ dev/fxp/if_fxp.c:1798 7 mbuf PCPU list lock -- last acquired @ kern/subr_mbuf.c:926 8 mbuf subsystem general lists lock -- last acquired @ kern/subr_mbuf.c:676 7 Malloc Stats -- (already displayed) 7 if send queue -- last acquired @ dev/fxp/if_fxp.c:1317 7 system map -- (already displayed) 6 random reseed -- last acquired @ dev/random/yarrow.c:170 6 sellck -- last acquired @ kern/sys_generic.c:816 6 radix node head -- last acquired @ net/route.c:139 7 Malloc Stats -- (already displayed) 7 ifnet -- last acquired @ net/if.c:1170 7 ifaddr -- last acquired @ net/route.c:671 7 system map -- (already displayed) 11 sleep mtxpool -- last acquired @ kern/kern_descrip.c:1808 4 udp -- last acquired @ netinet/udp_usrreq.c:263 5 inp -- (already displayed) 4 arp_inq -- last acquired @ net/netisr.c:233 4 ip_inq -- last acquired @ net/netisr.c:233 3 sem -- last acquired @ kern/sysv_sem.c:1157 3 mntvnode -- last acquired @ kern/vfs_subr.c:3218 10 vnode interlock -- (already displayed) 3 crypto driver table -- last acquired @ opencrypto/crypto.c:498 3 p_peers -- last acquired @ kern/kern_exit.c:252 3 buffer daemon lock -- last acquired @ kern/vfs_bio.c:2093 3 fdesc -- last acquired @ kern/kern_descrip.c:1516 4 filedesc structure -- last acquired @ kern/sys_generic.c:895 5 pipe mutex -- last acquired @ kern/sys_pipe.c:1303 6 sellck -- (already displayed) 6 sigio lock -- last acquired @ kern/kern_descrip.c:588 7 process group -- last acquired @ kern/tty.c:2407 8 process lock -- last acquired @ kern/kern_condvar.c:149 9 ktrace -- last acquired @ kern/kern_exit.c:353 9 sigacts -- last acquired @ kern/kern_condvar.c:150 9 struct pargs.ref -- last acquired @ kern/kern_proc.c:1065 9 session -- last acquired @ kern/kern_proc.c:790 10 vnode interlock -- (already displayed) 10 uidinfo hash -- last acquired @ kern/kern_resource.c:878 11 sleep mtxpool -- (already displayed) 11 uidinfo struct -- last acquired @ order list:0 12 allprison -- last acquired @ kern/kern_jail.c:414 3 ufs ihash -- last acquired @ ufs/ufs/ufs_ihash.c:124 10 vnode interlock -- (already displayed) 3 needsbuffer lock -- last acquired @ kern/vfs_bio.c:292 3 runningbufspace lock -- last acquired @ kern/vfs_bio.c:310 3 buf queue lock -- last acquired @ kern/vfs_bio.c:1493 10 vnode interlock -- (already displayed) 3 accounting -- last acquired @ kern/kern_acct.c:232 3 dirhash list -- last acquired @ ufs/ufs/ufs_dirhash.c:342 4 dirhash -- last acquired @ ufs/ufs/ufs_dirhash.c:450 0 g_xup -- last acquired @ geom/geom_io.c:378 2 Giant -- (already displayed) 0 ACPI task -- last acquired @ /bang/src/sys/dev/acpica/Osd/OsdSchedule.c:110 0 crypto return queues -- last acquired @ opencrypto/crypto.c:1197 0 crypto op queues -- last acquired @ opencrypto/crypto.c:1073 0 GEOM event stalling -- last acquired @ geom/geom_event.c:157 1 GEOM topology -- last acquired @ geom/geom_event.c:158 2 Giant -- (already displayed) 0 module subsystem sx lock -- last acquired @ kern/kern_module.c:331 0 kernel environment -- last acquired @ kern/kern_environment.c:288 0 sysctl lock -- last acquired @ kern/kern_sysctl.c:1269 1 filelist lock -- last acquired @ kern/kern_descrip.c:1234 4 filedesc structure -- (already displayed) 1 allproc -- last acquired @ kern/sched_4bsd.c:254 2 Giant -- (already displayed) 0 dev_pager create -- last acquired @ vm/device_pager.c:150 1 dev_pager list -- last acquired @ vm/device_pager.c:164 4 vm object_list -- (already displayed) 0 proctree -- last acquired @ kern/kern_exit.c:573 1 allproc -- (already displayed) Spin locks: Locks which were never acquired: rip pseudofs_fileno msq semid cd9660_ihash msdosfs dehash bpf interface lock ACPI global lock taskqueue strategy bounce pages lock jumbo mutex securelevel mutex lock UUID generator mutex lock umtx phys_pager list vm map sleep mutex lockmgr db> -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.