Date: Mon, 13 Nov 2006 04:49:25 GMT From: Jeff Lawson<bovine@distributed.net> To: freebsd-gnats-submit@FreeBSD.org Subject: misc/105463: [panis] "softdep_setup_inomapdep: found inode" on 3ware RAID volume [6.0-STABLE] Message-ID: <200611130449.kAD4nPIU040685@www.freebsd.org> Resent-Message-ID: <200611130450.kAD4o9c0059731@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 105463
>Category: misc
>Synopsis: [panis] "softdep_setup_inomapdep: found inode" on 3ware RAID volume [6.0-STABLE]
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Nov 13 04:50:09 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator: Jeff Lawson
>Release: 6.0-STABLE
>Organization:
Distributed Computing Technologies Inc.
>Environment:
FreeBSD fritz.distributed.net 6.0-STABLE FreeBSD 6.0-STABLE #6: Fri Dec 9 19:14:19 UTC 2005 root@fritz.distributed.net:/usr/obj/usr/src/sys/FRITZ amd64
>Description:
Can someone tell me if this more likely suggests a hardware problem, or a software problem?
root:/usr/obj/usr/src/sys/FRITZ>kgdb kernel.debug /var/crash/vmcore.3
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".
Unread portion of the kernel message buffer:
panic: softdep_setup_inomapdep: found inode
cpuid = 1
Uptime: 7h24m20s
Dumping 3967 MB (2 chunks)
chunk 0: 1MB (156 pages) ... ok
chunk 1: 3967MB (1015536 pages) 3951 3935 3919 3903 3887 3871 3855 3839 3823 3807 3791 3775 3759 3743 3727 3711 3695 3679 3663 3647 3631 3615 3599 3583 3567 3551 3535 3519 3503 3487 3471 3455 3439 3423 3407 3391 3375 3359 3343 3327 3311 3295 3279 3263 3247 3231 3215 3199 3183 3167 3151 3135 3119 3103 3087 3071 3055 3039 3023 3007 2991 2975 2959 2943 2927 2911 2895 2879 2863 2847 2831 2815 2799 2783 2767 2751 2735 2719 2703 2687 2671 2655 2639 2623 2607 2591 2575 2559 2543 2527 2511 2495 2479 2463 2447 2431 2415 2399 2383 2367 2351 2335 2319 2303 2287 2271 2255 2239 2223 2207 2191 2175 2159 2143 2127 2111 2095 2079 2063 2047 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879
863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:172
172 __asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) where
#0 doadump () at pcpu.h:172
#1 0x0000000000000004 in ?? ()
#2 0xffffffff80240347 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
#3 0xffffffff80240ab1 in panic (fmt=0xffffff009c4d0000 "\200V«ç")
at /usr/src/sys/kern/kern_shutdown.c:555
#4 0xffffffff80307ec9 in softdep_setup_inomapdep (bp=0xffffffffa13cec48,
ip=0xffffff00bc3d3a80, newinum=305018) at /usr/src/sys/ufs/ffs/ffs_softdep.c:1388
#5 0xffffffff802f1c0c in ffs_nodealloccg (ip=0xffffff00bc3d3a80, cg=9531, ipref=26,
mode=32768) at /usr/src/sys/ufs/ffs/ffs_alloc.c:1747
#6 0xffffffff802f1068 in ffs_hashalloc (ip=0xffffff00bc3d3a80, cg=9531, pref=0,
size=33152, allocator=0xffffffff802f17e0 <ffs_nodealloccg>)
at /usr/src/sys/ufs/ffs/ffs_alloc.c:1243
#7 0xffffffff802f4087 in ffs_valloc (pvp=0xffffff00bc2bc9b0, mode=33152, cred=0x0,
vpp=0xffffffffacd575a8) at /usr/src/sys/ufs/ffs/ffs_alloc.c:920
#8 0xffffffff8031869a in ufs_makeinode (mode=33152, dvp=0xffffff00bc2bc9b0,
vpp=0xffffffffacd579e8, cnp=0xffffffffacd57a10)
at /usr/src/sys/ufs/ufs/ufs_vnops.c:2165
#9 0xffffffff80318c18 in ufs_create (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:171
#10 0xffffffff80384146 in VOP_CREATE_APV (vop=0x0, a=0xffffffffacd577a0)
at vnode_if.c:204
#11 0xffffffff802b8e2e in vn_open_cred (ndp=0xffffffffacd579c0,
flagp=0xffffffffacd5790c, cmode=-1137981008, cred=0xffffff00be3fe100, fdidx=43)
at vnode_if.h:111
#12 0xffffffff802ade3f in kern_open (td=0xffffff009c4d0000,
path=0xa75ba0 <Address 0xa75ba0 out of bounds>, pathseg=UIO_USERSPACE,
flags=2563, mode=-1395295808) at /usr/src/sys/kern/vfs_syscalls.c:979
#13 0xffffffff802ae5e2 in open (td=0x0, uap=0xffffffffacd57bc0)
at /usr/src/sys/kern/vfs_syscalls.c:945
#14 0xffffffff80354bb1 in syscall (frame=
{tf_rdi = 10967968, tf_rsi = 2562, tf_rdx = 384, tf_rcx = 36, tf_r8 = -1361024, tf_r9 = 0, tf_rax = 5, tf_rbx = 38, tf_rbp = 10962672, tf_r10 = 9616448, tf_r11 = 10879304, tf_r12 = 384, tf_r13 = 2562, tf_r14 = 10967968, tf_r15 = 9606944, tf_trapno = 12, t---Type <return> to continue, or q <return> to quit---
f_addr = 10879032, tf_flags = 0, tf_err = 2, tf_rip = 34381032172, tf_cs = 43, tf_rflags = 582, tf_rsp = 140737488343064, tf_ss = 35}) at /usr/src/sys/amd64/amd64/trap.c:792
#15 0xffffffff80341608 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:270
#16 0x000000080144eaec in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) frame 4
#4 0xffffffff80307ec9 in softdep_setup_inomapdep (bp=0xffffffffa13cec48,
ip=0xffffff00bc3d3a80, newinum=305018) at /usr/src/sys/ufs/ffs/ffs_softdep.c:1388
1388 panic("softdep_setup_inomapdep: found inode");
(kgdb) list
1383 * Otherwise add it to the dependency list for the buffer holding
1384 * the cylinder group map from which it was allocated.
1385 */
1386 ACQUIRE_LOCK(&lk);
1387 if ((inodedep_lookup(ip->i_fs, newinum, DEPALLOC|NODELAY, &inodedep)))
1388 panic("softdep_setup_inomapdep: found inode");
1389 inodedep->id_buf = bp;
1390 inodedep->id_state &= ~DEPCOMPLETE;
1391 bmsafemap = bmsafemap_lookup(bp);
1392 LIST_INSERT_HEAD(&bmsafemap->sm_inodedephd, inodedep, id_deps);
(kgdb) p newinum
$1 = 305018
(kgdb) p ip->i_fs
$2 = (struct fs *) 0xffffff004c768800
(kgdb) p *ip->i_fs
$3 = {fs_firstfield = 0, fs_unused_1 = 0, fs_sblkno = 72, fs_cblkno = 80,
fs_iblkno = 88, fs_dblkno = 96, fs_old_cgoffset = 0, fs_old_cgmask = 0,
fs_old_time = 0, fs_old_size = 0, fs_old_dsize = 0, fs_ncg = 10374,
fs_bsize = 8192, fs_fsize = 1024, fs_frag = 8, fs_minfree = 8, fs_old_rotdelay = 0,
fs_old_rps = 0, fs_bmask = -8192, fs_fmask = -1024, fs_bshift = 13, fs_fshift = 10,
fs_maxcontig = 16, fs_maxbpg = 2048, fs_fragshift = 3, fs_fsbtodb = 1,
fs_sbsize = 2048, fs_spare1 = {0, 0}, fs_nindir = 1024, fs_inopb = 32,
fs_old_nspf = 0, fs_optim = 0, fs_old_npsect = 0, fs_old_interleave = 0,
fs_old_trackskew = 0, fs_id = {1133907622, 1076285835}, fs_old_csaddr = 0,
fs_cssize = 166912, fs_cgsize = 8192, fs_spare2 = 0, fs_old_nsect = 0,
fs_old_spc = 0, fs_old_ncyl = 0, fs_old_cpg = 0, fs_ipg = 32, fs_fpg = 56480,
fs_old_cstotal = {cs_ndir = 0, cs_nbfree = 0, cs_nifree = 0, cs_nffree = 0},
fs_fmod = 1 '\001', fs_clean = 0 '\0', fs_ronly = 0 '\0',
fs_old_flags = -128 '\200',
fs_fsmnt = "/usr/local/raid10", '\0' <repeats 450 times>,
fs_volname = '\0' <repeats 31 times>, fs_swuid = 0, fs_pad = 0, fs_cgrotor = 9534,
fs_ocsp = {0x0 <repeats 12 times>}, fs_contigdirs = 0xffffffff8845ee18 "",
fs_csp = 0xffffffff8842c000, fs_maxcluster = 0xffffffff88454c00, fs_active = 0x0,
fs_old_cpc = 0, fs_maxbsize = 8192, fs_sparecon64 = {0 <repeats 17 times>},
fs_sblockloc = 65536, fs_cstotal = {cs_ndir = 41, cs_nbfree = 26965666,
cs_nifree = 329692, cs_nffree = 294, cs_numclusters = 0, cs_spare = {0, 0, 0}},
fs_time = 1162613781, fs_size = 585898551, fs_dsize = 585649340, fs_csaddr = 96,
fs_pendingblocks = -26944, fs_pendinginodes = 4, fs_snapinum = {
0 <repeats 20 times>}, fs_avgfilesize = 16384, fs_avgfpdir = 64,
fs_save_cgsize = 0, fs_sparecon32 = {0 <repeats 26 times>}, fs_flags = 7,
fs_contigsumsize = 16, fs_maxsymlinklen = 120, fs_old_inodefmt = 0,
fs_maxfilesize = 8804691443711, fs_qbmask = 8191, fs_qfmask = 1023, fs_state = 0,
fs_old_postblformat = 0, fs_old_nrpos = 0, fs_spare5 = {0, 0}, fs_magic = 424935705}
(kgdb)
Please note that the "/usr/local/raid10" volume that is being accessed is a RAID-10 volume hosted by a 3Ware 9550SX-8LP RAID Controller.
>How-To-Repeat:
Machine crashes randomly, anywhere between a few hours of uptime to several months. Usually we are not able to retrieve a coredump, but this time we were lucky and were able to.
>Fix:
Not yet known.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200611130449.kAD4nPIU040685>