Date: Fri, 21 Jan 2011 18:30:13 GMT From: Juergen Lock <nox@jelal.kn-bremen.de> To: freebsd-net@FreeBSD.org Subject: Re: kern/153938: [run] [panic] [patch] Workaround for use-after-free panic Message-ID: <201101211830.p0LIUDZJ006253@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/153938; it has been noted by GNATS. From: Juergen Lock <nox@jelal.kn-bremen.de> To: PseudoCylon <moonlightakkiy@yahoo.ca> Cc: bug-followup@freebsd.org, Juergen Lock <nox@jelal.kn-bremen.de> Subject: Re: kern/153938: [run] [panic] [patch] Workaround for use-after-free panic Date: Fri, 21 Jan 2011 19:21:20 +0100 On Thu, Jan 20, 2011 at 04:35:48PM -0800, PseudoCylon wrote: > Hello, > > I have applied changes. Please check it out. > http://gitorious.org/run/run/trees/ratectl_fix/dev/usb/wlan I added debug output again and then after a while got a deadlock [1] that I suspect is caused by a lor, see below. (lock order reversal between "run0" and "run0_node_lock" i.e. RUN_LOCK and IEEE80211_NODE_LOCK.) It's possible this was triggered by the first DPRINTFN() in run_node_cleanup() (that I turned into a device_printf() and meanwhile have disabled, maybe it caused a taskswitch) - but in any case I'd say this is not safe i.e. needs to be fixed. :) [1] box stayed up but several things got stuck so in the end I had to drop to ddb and do a `call doadump', and fortunately this time the dump worked too... (kgdb) info threads [...] at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 121 Thread 100418 (PID=31634: hostapd) sched_switch ( td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 [...] 72 Thread 100064 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c21000, newtd=0xffffff0005c20ba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 71 Thread 100063 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c213e0, newtd=0xffffff00018837c0, flags=Variable "flags" is not available. ) ---Type <return> to continue, or q <return> to quit--- at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 70 Thread 100062 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c217c0, newtd=0xffffff0005c213e0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 69 Thread 100061 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c21ba0, newtd=0xffffff0005c217c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 68 Thread 100057 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005c25ba0, newtd=0xffffff00018907c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 67 Thread 100056 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005a853e0, newtd=0xffffff00018833e0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 66 Thread 100055 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005a857c0, newtd=0xffffff00018907c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 65 Thread 100054 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005a85ba0, ---Type <return> to continue, or q <return> to quit--- newtd=0xffffff0005a857c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 64 Thread 100052 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b403e0, newtd=0xffffff0005a85ba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 63 Thread 100051 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b407c0, newtd=0xffffff00018833e0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 62 Thread 100050 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b40ba0, newtd=0xffffff0005b407c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 61 Thread 100049 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b41000, newtd=0xffffff0005b40ba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 60 Thread 100048 (PID=14: usb/usbus3) sched_switch (td=0xffffff0005b413e0, newtd=0xffffff0005b41000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 ---Type <return> to continue, or q <return> to quit--- 59 Thread 100047 (PID=14: usb/usbus3) sched_switch (td=0xffffff0005b417c0, newtd=0xffffff0001883000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 58 Thread 100046 (PID=14: usb/usbus3) sched_switch (td=0xffffff0005b41ba0, newtd=0xffffff0005b417c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 57 Thread 100045 (PID=14: usb/usbus3) sched_switch (td=0xffffff0001a2cba0, newtd=0xffffff0005b41ba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 56 Thread 100043 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a813e0, newtd=0xffffff00018837c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 55 Thread 100042 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a817c0, newtd=0xffffff00018907c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 54 Thread 100041 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a81ba0, newtd=0xffffff0001883000, flags=Variable "flags" is not available. ) ---Type <return> to continue, or q <return> to quit--- at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 53 Thread 100040 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a83000, newtd=0xffffff0005a81ba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 52 Thread 100039 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a833e0, newtd=0xffffff00018907c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 51 Thread 100038 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a837c0, newtd=0xffffff00018837c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 50 Thread 100037 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a83ba0, newtd=0xffffff0005a837c0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 49 Thread 100036 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a85000, newtd=0xffffff0005a83ba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 48 Thread 100035 (PID=14: usb/usbus0) sched_switch (td=0xffffff00019fe7c0, ---Type <return> to continue, or q <return> to quit--- newtd=0xffffff0005a85000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 47 Thread 100034 (PID=14: usb/usbus0) sched_switch (td=0xffffff00019feba0, newtd=0xffffff0001883000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 46 Thread 100033 (PID=14: usb/usbus0) sched_switch (td=0xffffff0001a2a000, newtd=0xffffff00019feba0, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 45 Thread 100032 (PID=14: usb/usbus0) sched_switch (td=0xffffff0001a2a3e0, newtd=0xffffff0001a2a000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 (kgdb) thread 121 [Switching to thread 121 (Thread 100418)]#0 sched_switch ( td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 1850 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 #1 0xffffffff805f90ef in mi_switch (flags=259, newtd=0x0) at /data2v/home/nox/src-r81/src/sys/kern/kern_synch.c:449 #2 0xffffffff80630fb6 in turnstile_wait (ts=Variable "ts" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/subr_turnstile.c:746 #3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a6c330, tid=18446742976169653216, opts=Variable "opts" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447 #4 0xffffffff805e14b3 in _mtx_lock_flags (m=Variable "m" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:203 #5 0xffffffff8117839b in run_node_cleanup (ni=0xffffff8000f83000) at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:1719 #6 0xffffffff806db816 in ieee80211_sta_leave (ni=0xffffff8000f83000) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_node.c:834 #7 0xffffffff806db94e in ieee80211_node_leave (ni=0xffffff8000f83000) ---Type <return> to continue, or q <return> to quit--- at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_node.c:2508 #8 0xffffffff806d2c13 in setmlme_common (vap=0xffffff013e1e2000, op=Variable "op" is not available. ) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_ioctl.c:1327 #9 0xffffffff806d2db5 in ieee80211_ioctl_setmlme (vap=0xffffff013e1e2000, ireq=Variable "ireq" is not available. ) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_ioctl.c:1512 #10 0xffffffff806d405a in ieee80211_ioctl_set80211 (vap=0xffffff013e1e2000, cmd=Variable "cmd" is not available. ) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_ioctl.c:2721 #11 0xffffffff806f7b7b in in_control (so=0xffffff01e7ef3d48, cmd=2149607914, data=0xffffff0007832460 "wlan0", ifp=0xffffff013e2c7800, td=0xffffff00758633e0) at /data2v/home/nox/src-r81/src/sys/netinet/in.c:290 #12 0xffffffff806a27b7 in ifioctl (so=0xffffff01e7ef3d48, cmd=2149607914, data=0xffffff0007832460 "wlan0", td=0xffffff00758633e0) at /data2v/home/nox/src-r81/src/sys/net/if.c:2523 #13 0xffffffff80632bc6 in kern_ioctl (td=0xffffff00758633e0, fd=3, com=2149607914, data=0xffffff0007832460 "wlan0") at file.h:262 #14 0xffffffff80632e0d in ioctl (td=0xffffff00758633e0, ---Type <return> to continue, or q <return> to quit--- uap=0xffffff80ee69ebf0) at /data2v/home/nox/src-r81/src/sys/kern/sys_generic.c:678 #15 0xffffffff808e5407 in syscall (frame=0xffffff80ee69ec80) at /data2v/home/nox/src-r81/src/sys/amd64/amd64/trap.c:945 #16 0xffffffff808cac31 in Xfast_syscall () at /data2v/home/nox/src-r81/src/sys/amd64/amd64/exception.S:374 #17 0x0000000800ca438c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) fr 5 #5 0xffffffff8117839b in run_node_cleanup (ni=0xffffff8000f83000) at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:1719 1719 RUN_LOCK(sc); (kgdb) l 1714 wcid = rn->wcid; 1715 /* sc_ni[0] is not used */ 1716 if (wcid != 0 && wcid <= RT2870_WCID_MAX) 1717 sc->sc_ni[wcid] = NULL; 1718 } else { 1719 RUN_LOCK(sc); 1720 wcid = rn->wcid; 1721 if (wcid != 0 && wcid <= RT2870_WCID_MAX) 1722 sc->sc_ni[wcid] = NULL; 1723 RUN_UNLOCK(sc); (kgdb) down #4 0xffffffff805e14b3 in _mtx_lock_flags (m=Variable "m" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:203 203 _get_sleep_lock(m, curthread, opts, file, line); (kgdb) #3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a6c330, tid=18446742976169653216, opts=Variable "opts" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447 447 turnstile_wait(ts, mtx_owner(m), TS_EXCLUSIVE_QUEUE); (kgdb) p m $1 = (struct mtx *) 0xffffff8000a6c330 (kgdb) p *m $2 = {lock_object = {lo_name = 0xffffff0005e799e0 "run0", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 18446742974292827042} (kgdb) p m.mtx_lock & 0xfffffffffffffff $3 = 1152920405190122402 (kgdb) p m.mtx_lock & 0xffffffffffffffff $4 = 18446742974292827042 (kgdb) p m.mtx_lock & 0xfffffffffffffff8 $5 = 18446742974292827040 (kgdb) p (struct thread *)m.mtx_lock & 0xfffffffffffffff8 Argument to arithmetic operation not a number or boolean. (kgdb) p (struct thread *)(m.mtx_lock & 0xfffffffffffffff8) $6 = (struct thread *) 0xffffff0005a81ba0 (kgdb) thr 54 [Switching to thread 54 (Thread 100041)]#0 sched_switch ( td=0xffffff0005a81ba0, newtd=0xffffff0001883000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 1850 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xffffff0005a81ba0, newtd=0xffffff0001883000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 #1 0xffffffff805f90ef in mi_switch (flags=259, newtd=0x0) at /data2v/home/nox/src-r81/src/sys/kern/kern_synch.c:449 #2 0xffffffff80630fb6 in turnstile_wait (ts=Variable "ts" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/subr_turnstile.c:746 #3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a717c8, tid=18446742974292827040, opts=Variable "opts" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447 #4 0xffffffff806dad00 in ieee80211_free_node (ni=0xffffff8000f83000) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_node.c:1682 #5 0xffffffff81172e1a in run_tx_free (pq=0xffffff8000a6c350, data=0xffffff8000a6c660, txerr=Variable "txerr" is not available. ) at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:2759 #6 0xffffffff8117783d in run_bulk_tx_callbackN (xfer=0xffffff8000d1e148, error=USB_ERR_NORMAL_COMPLETION, index=0) ---Type <return> to continue, or q <return> to quit--- at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:2793 #7 0xffffffff8052a92d in usbd_callback_wrapper (pq=Variable "pq" is not available. ) at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_transfer.c:2136 #8 0xffffffff80526fa6 in usb_command_wrapper (pq=0xffffff8000d1e060, xfer=Variable "xfer" is not available. ) at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_transfer.c:2745 #9 0xffffffff80529a70 in usb_callback_proc (_pm=Variable "_pm" is not available. ) at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_transfer.c:2005 #10 0xffffffff80524633 in usb_process (arg=Variable "arg" is not available. ) at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_process.c:166 #11 0xffffffff805c64a8 in fork_exit ( callout=0xffffffff80524560 <usb_process>, arg=0xffffff80003e8d10, frame=0xffffff80e97efc80) at /data2v/home/nox/src-r81/src/sys/kern/kern_fork.c:844 #12 0xffffffff808cae2e in fork_trampoline () at /data2v/home/nox/src-r81/src/sys/amd64/amd64/exception.S:562 #13 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- #14 0x0000000000000000 in ?? () #15 0x0000000000000001 in ?? () #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit---q Quit (kgdb) fr 3 #3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a717c8, tid=18446742974292827040, opts=Variable "opts" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447 447 turnstile_wait(ts, mtx_owner(m), TS_EXCLUSIVE_QUEUE); (kgdb) p m $7 = (struct mtx *) 0xffffff8000a717c8 (kgdb) p *m $8 = {lock_object = {lo_name = 0xffffff8000a717b8 "run0_node_lock", lo_flags = 17498112, lo_data = 0, lo_witness = 0x0}, mtx_lock = 18446742976169653218} (kgdb) p (struct thread *)(m.mtx_lock & 0xfffffffffffffff8) $9 = (struct thread *) 0xffffff00758633e0 (kgdb) thread 121 [Switching to thread 121 (Thread 100418)]#0 sched_switch ( td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available. ) at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850 1850 cpuid = PCPU_GET(cpuid); (kgdb) q
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201101211830.p0LIUDZJ006253>