Date: Thu, 10 May 2007 14:49:45 -0600 From: Scott Swanson <scotts@inetz.com> To: freebsd-stable@freebsd.org Subject: panic: spin lock held too long (w/ backtrace) Message-ID: <46438569.1000506@inetz.com>
next in thread | raw e-mail | index | archive | help
Hello all,
I have a couple dozen SuperMicro servers in production and have had
reoccurring crashing issues on a couple of them that are under higher
load.
After managing to pry one out of production, I have been able to
re-create the problem fairly consistently just by looping a script that
tars up a large directory and writes the the file to a different
partition on the same raid array.
Can anyone point me towards the next step to debug this issue? I can
pursue any other debugging info that may be required.
Regards;
Scott Swanson
Inetz Media
> uname -a
FreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue
May 8 13:08:22 MDT 2007
root@biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE i386
root@pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug
/var/crash/vmcore.0
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Unread portion of the kernel message buffer:
spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds
panic: spin lock held too long
cpuid = 0
Uptime: 1d3h51m44s
Dumping 3967 MB (3 chunks)
chunk 0: 1MB (158 pages) ... ok
chunk 1: 3966MB (1015280 pages) 3950 3934 3918 3902 3886 3870 3854
3838 3822 3806 3790 3774 3758 3742 3726 3710 3694 3678 3662 3646 3630
3614 3598 3582 3566 3550 3534 3518 3502 3486 3470 3454 3438 3422 3406
3390 3374 3358 3342 3326 3310 3294 3278 3262 3246 3230 3214 3198 3182
3166 3150 3134 3118 3102 3086 3070 3054 3038 3022 3006 2990 2974 2958
2942 2926 2910 2894 2878 2862 2846 2830 2814 2798 2782 2766 2750 2734
2718 2702 2686 2670 2654 2638 2622 2606 2590 2574 2558 2542 2526 2510
2494 2478 2462 2446 2430 2414 2398 2382 2366 2350 2334 2318 2302 2286
2270 2254 2238 2222 2206 2190 2174 2158 2142 2126 2110 2094 2078 2062
2046 2030 2014 1998 1982 1966 1950 1934 1918 1902 1886 1870 1854 1838
1822 1806 1790 1774 1758 1742 1726 1710 1694 1678 1662 1646 1630 1614
1598 1582 1566 1550 1534 1518 1502 1486 1470 1454 1438 1422 1406 1390
1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166
1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926
910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638
622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350
334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46
30 14 ... ok
chunk 2: 1MB (128 pages)
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb)
(kgdb) backtrace
#0 doadump () at pcpu.h:165
#1 0xc0632e8a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc06331e1 in panic (fmt=0xc087937c "spin lock held too long") at
/usr/src/sys/kern/kern_shutdown.c:565
#3 0xc062a1bd in _mtx_lock_spin (m=0xc094f420, tid=3376316416, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:633
#4 0xc08147ce in pmap_invalidate_range (pmap=0xc0984420,
sva=3820650496, eva=3820666880) at /usr/src/sys/i386/i386/pmap.c:602
#5 0xc0814fad in pmap_qremove (sva=3820650496, count=0) at
/usr/src/sys/i386/i386/pmap.c:974
#6 0xc067ba30 in vfs_vmio_release (bp=0xdd53ae40) at
/usr/src/sys/kern/vfs_bio.c:1498
#7 0xc067c0d8 in getnewbuf (slpflag=0, slptimeo=0, size=16384,
maxsize=16384) at /usr/src/sys/kern/vfs_bio.c:1779
#8 0xc067d7e8 in geteblk (size=16384) at /usr/src/sys/kern/vfs_bio.c:2580
#9 0xc0775df1 in ffs_bufwrite (bp=0xdd4a7778) at
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1700
#10 0xc067a9ab in bawrite (bp=0x0) at buf.h:410
#11 0xc0682f00 in vop_stdfsync (ap=0xeaa0acc0) at
/usr/src/sys/kern/vfs_default.c:431
#12 0xc05e5c8f in devfs_fsync (ap=0xeaa0acc0) at
/usr/src/sys/fs/devfs/devfs_vnops.c:379
#13 0xc082c3cc in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020
#14 0xc068bf87 in sync_vnode (bo=0xc97491d0, td=0xc93e8000) at
vnode_if.h:537
#15 0xc068c2a5 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698
#16 0xc061c5cd in fork_exit (callout=0xc068c04c <sched_sync>, arg=0x0,
frame=0xeaa0ad38) at /usr/src/sys/kern/kern_fork.c:821
#17 0xc080690c in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:208
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46438569.1000506>