From owner-freebsd-stable@FreeBSD.ORG Thu May 10 21:43:49 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9116216A403 for ; Thu, 10 May 2007 21:43:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 7BA1B13C4BC for ; Thu, 10 May 2007 21:43:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id BB1041A3C1A; Thu, 10 May 2007 14:44:33 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id A76D5513BC; Thu, 10 May 2007 17:43:48 -0400 (EDT) Date: Thu, 10 May 2007 17:43:48 -0400 From: Kris Kennaway To: Scott Swanson Message-ID: <20070510214348.GA68925@xor.obsecurity.org> References: <46438569.1000506@inetz.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46438569.1000506@inetz.com> User-Agent: Mutt/1.4.2.2i Cc: freebsd-stable@freebsd.org Subject: Re: panic: spin lock held too long (w/ backtrace) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 21:43:49 -0000 On Thu, May 10, 2007 at 02:49:45PM -0600, Scott Swanson wrote: > Hello all, > > I have a couple dozen SuperMicro servers in production and have had > reoccurring crashing issues on a couple of them that are under higher > load. > > After managing to pry one out of production, I have been able to > re-create the problem fairly consistently just by looping a script that > tars up a large directory and writes the the file to a different > partition on the same raid array. > > Can anyone point me towards the next step to debug this issue? I can > pursue any other debugging info that may be required. > > Regards; > Scott Swanson > Inetz Media > > > > uname -a > FreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue > May 8 13:08:22 MDT 2007 > root@biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE i386 > > > root@pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug > /var/crash/vmcore.0 > [GDB will not be able to debug user-mode threads: > /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] > GNU gdb 6.1.1 [FreeBSD] > > Unread portion of the kernel message buffer: > spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds What is thread 0xc95a5900 doing? Kris > panic: spin lock held too long > cpuid = 0 > Uptime: 1d3h51m44s > Dumping 3967 MB (3 chunks) > chunk 0: 1MB (158 pages) ... ok > chunk 1: 3966MB (1015280 pages) 3950 3934 3918 3902 3886 3870 3854 > 3838 3822 3806 3790 3774 3758 3742 3726 3710 3694 3678 3662 3646 3630 > 3614 3598 3582 3566 3550 3534 3518 3502 3486 3470 3454 3438 3422 3406 > 3390 3374 3358 3342 3326 3310 3294 3278 3262 3246 3230 3214 3198 3182 > 3166 3150 3134 3118 3102 3086 3070 3054 3038 3022 3006 2990 2974 2958 > 2942 2926 2910 2894 2878 2862 2846 2830 2814 2798 2782 2766 2750 2734 > 2718 2702 2686 2670 2654 2638 2622 2606 2590 2574 2558 2542 2526 2510 > 2494 2478 2462 2446 2430 2414 2398 2382 2366 2350 2334 2318 2302 2286 > 2270 2254 2238 2222 2206 2190 2174 2158 2142 2126 2110 2094 2078 2062 > 2046 2030 2014 1998 1982 1966 1950 1934 1918 1902 1886 1870 1854 1838 > 1822 1806 1790 1774 1758 1742 1726 1710 1694 1678 1662 1646 1630 1614 > 1598 1582 1566 1550 1534 1518 1502 1486 1470 1454 1438 1422 1406 1390 > 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 > 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 > 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 > 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 > 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 > 30 14 ... ok > chunk 2: 1MB (128 pages) > > #0 doadump () at pcpu.h:165 > 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); > (kgdb) > (kgdb) backtrace > #0 doadump () at pcpu.h:165 > #1 0xc0632e8a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 > #2 0xc06331e1 in panic (fmt=0xc087937c "spin lock held too long") at > /usr/src/sys/kern/kern_shutdown.c:565 > #3 0xc062a1bd in _mtx_lock_spin (m=0xc094f420, tid=3376316416, opts=0, > file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:633 > #4 0xc08147ce in pmap_invalidate_range (pmap=0xc0984420, > sva=3820650496, eva=3820666880) at /usr/src/sys/i386/i386/pmap.c:602 > #5 0xc0814fad in pmap_qremove (sva=3820650496, count=0) at > /usr/src/sys/i386/i386/pmap.c:974 > #6 0xc067ba30 in vfs_vmio_release (bp=0xdd53ae40) at > /usr/src/sys/kern/vfs_bio.c:1498 > #7 0xc067c0d8 in getnewbuf (slpflag=0, slptimeo=0, size=16384, > maxsize=16384) at /usr/src/sys/kern/vfs_bio.c:1779 > #8 0xc067d7e8 in geteblk (size=16384) at /usr/src/sys/kern/vfs_bio.c:2580 > #9 0xc0775df1 in ffs_bufwrite (bp=0xdd4a7778) at > /usr/src/sys/ufs/ffs/ffs_vfsops.c:1700 > #10 0xc067a9ab in bawrite (bp=0x0) at buf.h:410 > #11 0xc0682f00 in vop_stdfsync (ap=0xeaa0acc0) at > /usr/src/sys/kern/vfs_default.c:431 > #12 0xc05e5c8f in devfs_fsync (ap=0xeaa0acc0) at > /usr/src/sys/fs/devfs/devfs_vnops.c:379 > #13 0xc082c3cc in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020 > #14 0xc068bf87 in sync_vnode (bo=0xc97491d0, td=0xc93e8000) at > vnode_if.h:537 > #15 0xc068c2a5 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698 > #16 0xc061c5cd in fork_exit (callout=0xc068c04c , arg=0x0, > frame=0xeaa0ad38) at /usr/src/sys/kern/kern_fork.c:821 > #17 0xc080690c in fork_trampoline () at > /usr/src/sys/i386/i386/exception.s:208 > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >