Date: Fri, 20 Jun 2008 11:31:14 +0000 From: Chameeya Software Services Ltd. <chameeyass@hotmail.com> To: <remko@elvandar.org>, <freebsd-bugs@freebsd.org> Subject: RE: kern/124670: large file operation on RAID cause many GEOM errors - crash Message-ID: <BLU121-W49DCBF9AD1B58C7EF3718AD0A50@phx.gbl> In-Reply-To: <fcafb04e5d6e2e6845adfd3dfda51840.squirrel@galain.elvandar.org> References: <200806171930.m5HJU45s014686@freefall.freebsd.org> <BLU121-W29537AD2A92063D65E7830D0AA0@phx.gbl> <fcafb04e5d6e2e6845adfd3dfda51840.squirrel@galain.elvandar.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Okay managed to build a debug kernel..took two hours!
and started kgdb.. and this is what I got
[<code>]
GEOM_MIRROR: Device dat: rebuilding provider ad4 stopped.
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...GEOM_MIRR=
OR: Device dat: provider mirror/dat destroyed.
Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address =3D 0xf000fff0
fault code =3D supervisor read, page not present
instruction pointer =3D 0x20:0xc0747f04
stack pointer =3D 0x28:0xd92dfc3c
frame pointer =3D 0x28:0xd92dfc54
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, def32 1, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 3 (g_up)
trap number =3D 12
panic: page fault
cpuid =3D 0
Uptime: 8m38s
Physical memory: 627 MB
Dumping 35 MB: 20 4
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td));
(kgdb) backtrace
#0 doadump () at pcpu.h:195
#1 0xc0754457 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4=
09
#2 0xc0754719 in panic (fmt=3DVariable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc0a4905c in trap_fatal (frame=3D0xd92dfbfc, eva=3D4026597360)
at /usr/src/sys/i386/i386/trap.c:899
#4 0xc0a492e0 in trap_pfault (frame=3D0xd92dfbfc, usermode=3D0, eva=3D4026=
597360)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc0a49c8c in trap (frame=3D0xd92dfbfc) at /usr/src/sys/i386/i386/trap.=
c:490
#6 0xc0a2fc0b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc0747f04 in _mtx_lock_sleep (m=3D0x58, tid=3D3275608064, opts=3D0,=20
file=3D0xc0d18eda "/usr/src/sys/modules/geom/geom_mirror/../../../geom/=
mirror/g_mirror.c", line=3D1010) at /usr/src/sys/kern/kern_mutex.c:335
#8 0xc0748392 in _mtx_lock_flags (m=3D0x58, opts=3D0,=20
file=3D0xc0d18eda "/usr/src/sys/modules/geom/geom_mirror/../../../geom/=
mirror/g_mirror.c", line=3D1010) at /usr/src/sys/kern/kern_mutex.c:186
#9 0xc0d0efca in ?? ()
#10 0x00000058 in ?? ()
#11 0x00000000 in ?? ()
#12 0xc0d18eda in ?? ()
#13 0x000003f2 in ?? ()
#14 0xc0d0ef80 in ?? ()
#15 0xc3665210 in ?? ()
#16 0xc33dd000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#17 0xd92dfcb8 in ?? ()
#18 0xc07b3c9d in biodone (bp=3D0x58) at /usr/src/sys/kern/vfs_bio.c:3009
Previous frame identical to this frame (corrupt stack?)
(kgdb)=20
(kgdb) list *0xc0747f04
0xc0747f04 is in _mtx_lock_sleep (/usr/src/sys/kern/kern_mutex.c:337).
332 */
333 v =3D m->mtx_lock;
334 if (v !=3D MTX_UNOWNED) {
335 owner =3D (struct thread *)(v & ~MTX_FLAGMA=
SK);
336 #ifdef ADAPTIVE_GIANT
337 if (TD_IS_RUNNING(owner)) {
338 #else
339 if (m !=3D &Giant && TD_IS_RUNNING(owner)) =
{
340 #endif
341 if (LOCK_LOG_TEST(&m->lock_object, =
0))
(kgdb)=20
[</CODE>]
That was one core. and this is the other
[<CODE>]
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address =3D 0xc0
fault code =3D supervisor read, page not present
instruction pointer =3D 0x20:0xc06e9331
stack pointer =3D 0x28:0xdaab0a5c
frame pointer =3D 0x28:0xdaab0a5c
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, def32 1, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 1065 (Thunar)
trap number =3D 12
panic: page fault
cpuid =3D 0
Uptime: 1h24m0s
Physical memory: 627 MB
Dumping 62 MB: 47 31 15
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td));
(kgdb) backtrace
#0 doadump () at pcpu.h:195
#1 0xc0754457 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4=
09
#2 0xc0754719 in panic (fmt=3DVariable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc0a4905c in trap_fatal (frame=3D0xdaab0a1c, eva=3D192)
at /usr/src/sys/i386/i386/trap.c:899
#4 0xc0a492e0 in trap_pfault (frame=3D0xdaab0a1c, usermode=3D0, eva=3D192)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc0a49c8c in trap (frame=3D0xdaab0a1c) at /usr/src/sys/i386/i386/trap.=
c:490
#6 0xc0a2fc0b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc06e9331 in dev2udev (x=3D0xc3653200)
at /usr/src/sys/fs/devfs/devfs_vnops.c:1325
#8 0xc095e1fb in ufs_getattr (ap=3D0xdaab0aa0)
at /usr/src/sys/ufs/ufs/ufs_vnops.c:401
#9 0xc0a5d8e2 in VOP_GETATTR_APV (vop=3D0xc0b93c60, a=3D0xdaab0aa0)
at vnode_if.c:530
#10 0xc07d9b69 in vn_stat (vp=3D0xc3b4cbb0, sb=3D0xdaab0b94,=20
active_cred=3D0xc3a40e00, file_cred=3D0x0, td=3D0xc3adfc60) at vnode_if=
.h:286
#11 0xc07d08d5 in kern_lstat (td=3D0xc3adfc60,=20
path=3D0x29c21440 <Address 0x29c21440 out of bounds>, pathseg=3DUIO_USE=
RSPACE,=20
sbp=3D0xdaab0c18) at /usr/src/sys/kern/vfs_syscalls.c:2165
#12 0xc07d099f in lstat (td=3D0xc3adfc60, uap=3D0xdaab0cfc)
at /usr/src/sys/kern/vfs_syscalls.c:2144
#13 0xc0a49635 in syscall (frame=3D0xdaab0d38)
---Type <return> to continue, or q <return> to quit---
at /usr/src/sys/i386/i386/trap.c:1035
#14 0xc0a2fc70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s=
:196
#15 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)=20
(kgdb) list *0xc06e9331
0xc06e9331 is in dev2udev (/usr/src/sys/fs/devfs/devfs_vnops.c:1325).
1320 dev_t
1321 dev2udev(struct cdev *x)
1322 {
1323 if (x =3D=3D NULL)
1324 return (NODEV);
1325 return (x->si_priv->cdp_inode);
1326 }
1327
1328 static struct fileops devfs_ops_f =3D {
1329 .fo_read =3D devfs_read_f,
[</CODE>]
A third core
[<CODE>]
Unread portion of the kernel message buffer:
ad4: FAILURE - device detached
subdisk4: detached
ad4: detached
GEOM_MIRROR: Device dat: provider ad4 disconnected.
GEOM_MIRROR: Device dat: provider mirror/dat destroyed.
GEOM_MIRROR: Device dat destroyed.
g_vfs_done():mirror/dats1d[WRITE(offset=3D3862298624, length=3D16384)]error=
=3D 6
g_vfs_done():mirror/dats1d[WRITE(offset=3D4432019456, length=3D16384)]error=
=3D 6
panic: softdep_move_dependencies: need merge code
cpuid =3D 0
Uptime: 5m34s
Physical memory: 627 MB
Dumping 111 MB: 96 80 64 48 32 16
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td));
(kgdb) backtrace
#0 doadump () at pcpu.h:195
#1 0xc0754457 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4=
09
#2 0xc0754719 in panic (fmt=3DVariable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc0944e44 in softdep_move_dependencies (oldbp=3D0xd01b6da8,=20
newbp=3D0xd0178730) at /usr/src/sys/ufs/ffs/ffs_softdep.c:997
#4 0xc094c816 in ffs_backgroundwritedone (bp=3D0xd01b6da8)
at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1669
#5 0xc07b9b77 in bufdone (bp=3D0xd01b6da8) at /usr/src/sys/kern/vfs_bio.c:=
3167
#6 0xc0708928 in g_vfs_done (bip=3D0xc39dfd68)
at /usr/src/sys/geom/geom_vfs.c:87
#7 0xc07b3c9d in biodone (bp=3D0xc39dfd68) at /usr/src/sys/kern/vfs_bio.c:=
3009
#8 0xc0704ddf in g_io_schedule_up (tp=3D0xc33dd000)
at /usr/src/sys/geom/geom_io.c:587
#9 0xc070512e in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:95
#10 0xc0734479 in fork_exit (callout=3D0xc07050c0 <g_up_procbody>, arg=3D0x=
0,=20
frame=3D0xd92dfd38) at /usr/src/sys/kern/kern_fork.c:781
#11 0xc0a2fc80 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:=
205
(kgdb)=20
[</CODE>]
Hope that helps someone out
I think I may have another core as well. similar stuff I think.
Salik.
_________________________________________________________________
Great deals on almost anything at eBay.co.uk. Search, bid, find and win on =
eBay today!
http://clk.atdmt.com/UKM/go/msnnkmgl0010000004ukm/direct/01/=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BLU121-W49DCBF9AD1B58C7EF3718AD0A50>