From owner-freebsd-current@FreeBSD.ORG Sun May 17 20:54:57 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1B38106566C for ; Sun, 17 May 2009 20:54:57 +0000 (UTC) (envelope-from lwindschuh@googlemail.com) Received: from mail-gx0-f214.google.com (mail-gx0-f214.google.com [209.85.217.214]) by mx1.freebsd.org (Postfix) with ESMTP id 838728FC30 for ; Sun, 17 May 2009 20:54:57 +0000 (UTC) (envelope-from lwindschuh@googlemail.com) Received: by gxk10 with SMTP id 10so1964290gxk.19 for ; Sun, 17 May 2009 13:54:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=nJa6Pergz10rJtSBjwi6acRpipISRmsBaxlzD2W44tI=; b=S38wRydiGAGlV+Vk7v1JBpsyW6izIWAZrVYKRlCZXNfJuJ/sawd4JJdvuFtZhnt1Fa AOugyxAUi8bNZ346KNqqK+/pEByc/XCvBfpFcRvMMSBlrGPmSeHQ0T3/PzttF8/PAmZB L3n/5bBESsOHR6bcLnz+zwIgkX+Qt+31KR0Hw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=m62SLFEQexDX1XL5Ox67cUi5ENvO5HigC+qACImgS68fmLsvnFw7+rls++KmoT6z1t 7Pnx6NN1ze4j7qz77fOJ0oJr21aguzrg/ZVx7r6Gd4pCdcLg7YWpEzvtL5nNeWu4oPVt zbOLv1FjL7HMc7tUDtqyUTVMHIbBmlSmTXNsQ= MIME-Version: 1.0 Received: by 10.151.130.8 with SMTP id h8mr10856922ybn.247.1242593696755; Sun, 17 May 2009 13:54:56 -0700 (PDT) Date: Sun, 17 May 2009 22:54:56 +0200 Message-ID: <90a5caac0905171354k6e7c008eye18bd69aa543eaa6@mail.gmail.com> From: Lucius Windschuh To: current@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: Panics and potential memory corruption when pulling out a uath device X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2009 20:54:58 -0000 With the newly imported uath driver, I was able to produce five different panics. Since four of them occur in unrelated kernel parts, this looks to me like some kernel part is corrupting memory. But since I am not an expert, here are backtraces for them: First, the one which seems to be without memory corruption (minidump availa= ble): panic: mtx_lock() of destroyed mutex @ /usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1697 (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc04949c9 in db_fncall (dummy1=3D-979506816, dummy2=3D0, dummy3=3D-1068655593, dummy4=3D0xf3c47988 "@\231\235=EF=BF=BD001") at /usr/src/sys/ddb/db_command.c:548 #2 0xc0494dc1 in db_command (last_cmdp=3D0xc0989c9c, cmd_table=3D0x0, dopager=3D1) at /usr/src/sys/ddb/db_command.c:445 #3 0xc0494f1a in db_command_loop () at /usr/src/sys/ddb/db_command.c:498 #4 0xc0496d7d in db_trap (type=3D3, code=3D0) at /usr/src/sys/ddb/db_main.= c:229 #5 0xc06579d6 in kdb_trap (type=3D3, code=3D0, tf=3D0xf3c47b2c) at /usr/src/sys/kern/subr_kdb.c:534 #6 0xc088bdce in trap (frame=3D0xf3c47b2c) at /usr/src/sys/i386/i386/trap.= c:685 #7 0xc086f6fb in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #8 0xc0657b5a in kdb_enter (why=3D0xc08f8592 "panic", msg=3D0xc08f8592 "panic") at cpufunc.h:71 #9 0xc062a1a6 in panic (fmt=3D0xc08f6f47 "mtx_lock() of destroyed mutex @ %s:%d") at /usr/src/sys/kern/kern_shutdown.c:559 #10 0xc061a925 in _mtx_lock_flags (m=3D0xc6af26b8, opts=3D0, file=3D0xc858faff "/usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c", line=3D1697) at /usr/src/sys/kern/kern_mutex.c:174 #11 0xc857445e in ieee80211_node_delucastkey (ni=3D0xc6af8000) at /usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1697 #12 0xc85760d6 in node_free (ni=3D0xc6af8000) at /usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:999 #13 0xc8573992 in _ieee80211_free_node (ni=3D0xc6af8000) at /usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1622 #14 0xc84f5af0 in uath_bulk_tx_callback () from /boot/kernel/if_uath.ko #15 0xc0594d27 in usb2_callback_wrapper (pq=3D0xc9448030) at /usr/src/sys/dev/usb/usb_transfer.c:1962 #16 0xc0592716 in usb2_command_wrapper (pq=3D0xc9448030, xfer=3D0x0) at /usr/src/sys/dev/usb/usb_transfer.c:2538 #17 0xc05927f8 in usb2_callback_proc (_pm=3D0xc9448044) at /usr/src/sys/dev/usb/usb_transfer.c:1834 #18 0xc058febe in usb2_process (arg=3D0xc58d8ca4) at /usr/src/sys/dev/usb/usb_process.c:139 #19 0xc06036e8 in fork_exit (callout=3D0xc058fde0 , arg=3D0xc58d8ca4, frame=3D0xf3c47d38) at /usr/src/sys/kern/kern_fork.c:830 #20 0xc086f7a0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:= 270 Now the strange faults: 2nd: (minidump available) Fatal trap 12: page fault while in kernel mode (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc04949c9 in db_fncall (dummy1=3D-979506816, dummy2=3D0, dummy3=3D-1068655593, dummy4=3D0xc4eb3a20 "@\231\235=EF=BF=BD001") at /usr/src/sys/ddb/db_command.c:548 #2 0xc0494dc1 in db_command (last_cmdp=3D0xc0989c9c, cmd_table=3D0x0, dopager=3D1) at /usr/src/sys/ddb/db_command.c:445 #3 0xc0494f1a in db_command_loop () at /usr/src/sys/ddb/db_command.c:498 #4 0xc0496d7d in db_trap (type=3D12, code=3D0) at /usr/src/sys/ddb/db_main= .c:229 #5 0xc06579d6 in kdb_trap (type=3D12, code=3D0, tf=3D0xc4eb3c08) at /usr/src/sys/kern/subr_kdb.c:534 #6 0xc088afcf in trap_fatal (frame=3D0xc4eb3c08, eva=3D3735929062) at /usr/src/sys/i386/i386/trap.c:924 #7 0xc088b963 in trap (frame=3D0xc4eb3c08) at /usr/src/sys/i386/i386/trap.= c:325 #8 0xc086f6fb in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #9 0xc063cad1 in softclock (arg=3D0xc09a4ea0) at /usr/src/sys/kern/kern_timeout.c:335 #10 0xc0605975 in intr_event_execute_handlers (p=3D0xc516aa90, ie=3D0xc51aa000) at /usr/src/sys/kern/kern_intr.c:1134 #11 0xc06065df in ithread_loop (arg=3D0xc50e7ca0) at /usr/src/sys/kern/kern_intr.c:1147 #12 0xc06036e8 in fork_exit (callout=3D0xc0606540 , arg=3D0xc50e7ca0, frame=3D0xc4eb3d38) at /usr/src/sys/kern/kern_fork.c:830 #13 0xc086f7a0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:= 270 3rd: (minidump available) panic: Bad tailq NEXT(0xe59b4e40->tqh_last) !=3D NULL (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc04949c9 in db_fncall (dummy1=3D1, dummy2=3D0, dummy3=3D-1061793024, dummy4=3D0xc4eb39d8 "") at /usr/src/sys/ddb/db_command.c:548 #2 0xc0494dc1 in db_command (last_cmdp=3D0xc0989c9c, cmd_table=3D0x0, dopager=3D1) at /usr/src/sys/ddb/db_command.c:445 #3 0xc0494f1a in db_command_loop () at /usr/src/sys/ddb/db_command.c:498 #4 0xc0496d7d in db_trap (type=3D3, code=3D0) at /usr/src/sys/ddb/db_main.= c:229 #5 0xc06579d6 in kdb_trap (type=3D3, code=3D0, tf=3D0xc4eb3b7c) at /usr/src/sys/kern/subr_kdb.c:534 #6 0xc088bdce in trap (frame=3D0xc4eb3b7c) at /usr/src/sys/i386/i386/trap.= c:685 #7 0xc086f6fb in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #8 0xc0657b5a in kdb_enter (why=3D0xc08f8592 "panic", msg=3D0xc08f8592 "panic") at cpufunc.h:71 #9 0xc062a1a6 in panic (fmt=3D0xc08c0c8d "Bad tailq NEXT(%p->tqh_last) !=3D NULL") at /usr/src/sys/kern/kern_shutdown.c:559 #10 0xc063c780 in callout_reset_on (c=3D0xc09903a0, to_ticks=3D10, ftn=3D0xc04d9c20 , arg=3D0xc580ae00, cpu=3D0) at /usr/src/sys/kern/kern_timeout.c:626 #11 0xc04d9cf4 in dcons_timeout (v=3D0xc580ae00) at /usr/src/sys/dev/dcons/dcons_os.c:241 #12 0xc063ccd4 in softclock (arg=3D0xc09a4ea0) at /usr/src/sys/kern/kern_timeout.c:411 #13 0xc0605975 in intr_event_execute_handlers (p=3D0xc516aa90, ie=3D0xc51aa000) at /usr/src/sys/kern/kern_intr.c:1134 #14 0xc06065df in ithread_loop (arg=3D0xc50e7ca0) at /usr/src/sys/kern/kern_intr.c:1147 #15 0xc06036e8 in fork_exit (callout=3D0xc0606540 , arg=3D0xc50e7ca0, frame=3D0xc4eb3d38) at /usr/src/sys/kern/kern_fork.c:830 #16 0xc086f7a0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:= 270 4th: (only textdump; PID 1368 is fsck_ufs) panic: Bad link elm 0xc67e5f28 prev->next !=3D elm db:0:kdb.enter.panic> bt Tracing pid 1368 tid 100086 td 0xc67e5d80 kdb_enter(c09c58b4,c09c58b4,c09875f4,eae86b50,0,...) at kdb_enter+0x3a panic(c09875f4,c67e5f28,100,c67e5d80,c67e5d80,...) at panic+0x136 _callout_stop_safe(c67e5f28,0,c09c9bf3,208,0,...) at _callout_stop_safe+0x3= 91 sleepq_check_timeout(b,c06d2380,c67e5d80,0,100,...) at sleepq_check_timeout= +0x73 sleepq_timedwait_sig(c0a7be84,5c,c09c6aa3,100,0,...) at sleepq_timedwait_sig+0x21 _sleep(c0a7be84,0,15c,c09c6aa3,b,...) at _sleep+0x30e kern_nanosleep(c67e5d80,eae86c64,eae86c6c,0,5dfc8c0,...) at kern_nanosleep+= 0xc1 nanosleep(c67e5d80,eae86cf8,8,c09cc50a,c0a2d800,...) at nanosleep+0x6f syscall(eae86d38) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip =3D 0x281724ef, esp =3D 0xbfbfda1c, ebp =3D 0xbfbfda48 --- 5th: (only textdump; PID 11 is "intr") panic: Bad link elm 0xc6f54568 next->prev !=3D elm db:0:kdb.enter.panic> bt Tracing pid 11 tid 100006 td 0xc6176480 kdb_enter(c09c58b4,c09c58b4,c09875d2,c5f3ec54,0,...) at kdb_enter+0x3a panic(c09875d2,c6f54568,c09c6bbc,145,c0a7bef4,...) at panic+0x136 softclock(c0a7bec0,c5f3ecc8,c068cda4,c0a7fe00,c61b5c38,...) at softclock+0x= 10a intr_event_execute_handlers(c6174a90,c61b5c00,c09c1671,4dd,c61b5c70,...) at intr_event_execute_handlers+0x125 ithread_loop(c610fba0,c5f3ed38,c09c13ec,336,c6174a90,...) at ithread_loop+0= x9f fork_exit(c0679190,c610fba0,c5f3ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc5f3ed70, ebp =3D 0 --- The last two panics are from a differenct machine ("t400"), so I exclude faulty memory. The first three are from my machine "current". Kernel config, etc: http://sites.google.com/site/lwfreebsd/Home/files/ Kernel version: CURRENT r192252 Any ideas? Lucius