Date: Wed, 29 Feb 2012 02:26:51 +0900 (JST) From: Hiroki Sato <hrs@FreeBSD.org> To: kostikbel@gmail.com Cc: stable@FreeBSD.org Subject: Re: another panic in 8.3-PRERELEASE Message-ID: <20120229.022651.1585266709145027511.hrs@allbsd.org> In-Reply-To: <20120228130838.GN55074@deviant.kiev.zoral.com.ua> References: <20120224150259.GV55074@deviant.kiev.zoral.com.ua> <20120225.025828.128418237042325597.hrs@allbsd.org> <20120228130838.GN55074@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
----Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)--
Content-Type: Multipart/Mixed;
boundary="--Next_Part(Wed_Feb_29_02_26_51_2012_369)--"
Content-Transfer-Encoding: 7bit
----Next_Part(Wed_Feb_29_02_26_51_2012_369)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Konstantin Belousov <kostikbel@gmail.com> wrote
in <20120228130838.GN55074@deviant.kiev.zoral.com.ua>:
ko> I can see the race in how the wiring of the sysctl buffers is done, but the
ko> race can only realize for the multithreaded process.
ko>
ko> Can you, please, further show me two things:
ko> - the p/x *(td->td_pcb)
ko> - (this is somewhat laborous) Please find the vm map entry in the process
ko> vm_map which covers the range [0x800e96000, 0x800ea6a79) and print it out.
ko> You need to walk the td->td_proc->p_vmspace.vm_map.header list using
ko> the next link, looking for the entry start/end values.
The results and gdb commands I used are attached. In the linked-list
there seem two entries that covers the range.
-- Hiroki
----Next_Part(Wed_Feb_29_02_26_51_2012_369)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="result.txt"
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address = 0x800e96000
fault code = supervisor write data, protection violation
instruction pointer = 0x20:0xffffffff809440cb
stack pointer = 0x28:0xffffff86c63890b0
frame pointer = 0x28:0xffffff86c6389100
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 47211 (top)
lock order reversal: (Giant after non-sleepable)
1st 0xffffff0244b85568 process lock (process lock) @ /usr/src/sys/kern/kern_proc.c:1211
2nd 0xffffffff80d74c80 Giant (Giant) @ /usr/src/sys/dev/usb/input/ukbd.c:2018
KDB: stack backtrace:
Dumping 23903 out of 24550 MB:..1%..11%..21%..31% (CTRL-C to abort) (CTRL-C to abort) ..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:263
263 if (textdump_pending)
#16 0xffffffff80675e3a in __sysctl (td=0xffffff0396ec5460,
uap=0xffffff86c6389bc0) at /usr/src/sys/kern/kern_sysctl.c:1491
1491 error = userland_sysctl(td, name, uap->namelen,
--------
p/x *(td->td_pcb):
$1 = {pcb_r15 = 0xffffff03969bf470, pcb_r14 = 0x0,
pcb_r13 = 0xffffffff80d7f540, pcb_r12 = 0xffffff00057a18c0,
pcb_rbp = 0xffffff86c6389700, pcb_rsp = 0xffffff86c63896a8,
pcb_rbx = 0xffffff0396ec5460, pcb_rip = 0xffffffff80691367,
pcb_fsbase = 0x800542398, pcb_gsbase = 0x0, pcb_kgsbase = 0x0,
pcb_cr0 = 0x0, pcb_cr2 = 0x0, pcb_cr3 = 0x6793f000, pcb_cr4 = 0x0,
pcb_dr0 = 0x0, pcb_dr1 = 0x0, pcb_dr2 = 0x0, pcb_dr3 = 0x0, pcb_dr6 = 0x0,
pcb_dr7 = 0x0, pcb_gdt = {rd_limit = 0x0, rd_base = 0x0}, pcb_idt = {
rd_limit = 0x0, rd_base = 0x0}, pcb_ldt = {rd_limit = 0x0, rd_base = 0x0},
pcb_tr = 0x0, pcb_flags = 0x18, pcb_initial_fpucw = 0x37f,
pcb_onfault = 0xffffffff809440f0, pcb_gs32sd = {sd_lolimit = 0x0,
sd_lobase = 0x0, sd_type = 0x0, sd_dpl = 0x0, sd_p = 0x0,
sd_hilimit = 0x0, sd_xx = 0x0, sd_long = 0x0, sd_def32 = 0x0,
sd_gran = 0x0, sd_hibase = 0x0}, pcb_tssp = 0x0,
pcb_save = 0xffffff86c6389e00, pcb_user_save = {sv_env = {en_cw = 0x37f,
en_sw = 0x0, en_tw = 0x0, en_zero = 0x0, en_opcode = 0x0, en_rip = 0x0,
en_rdp = 0x0, en_mxcsr = 0x1fa4, en_mxcsr_mask = 0xffff}, sv_fp = {{
fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {
fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}},
fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0,
0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0,
0x0}}, {fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {
fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {
fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}},
fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0,
0x0, 0x0, 0x0, 0x0}}}, sv_xmm = {{xmm_bytes = {
0x0 <repeats 16 times>}}, {xmm_bytes = {0x0 <repeats 16 times>}}, {
xmm_bytes = {0x0 <repeats 16 times>}}, {xmm_bytes = {0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0xe0, 0x3f, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}},
{xmm_bytes = {0x0 <repeats 16 times>}} <repeats 12 times>}, sv_pad = {
0x0 <repeats 96 times>}}}
--------
#11 0xffffffff8065f6a6 in sysctl_out_proc_copyout (ki=0xffffff86c6389470,
req=0xffffff86c63899c0) at /usr/src/sys/kern/kern_proc.c:1085
1085 error = SYSCTL_OUT(req, ki, sizeof(struct kinfo_proc));
--------
range start:
$2 = 0x800e96000
range end:
$3 = 0x800ea6a79
--------
#16 0xffffffff80675e3a in __sysctl (td=0xffffff0396ec5460,
uap=0xffffff86c6389bc0) at /usr/src/sys/kern/kern_sysctl.c:1491
1491 error = userland_sysctl(td, name, uap->namelen,
--------
td->td_proc->p_vmspace.vm_map.header:
$4 = 0xffffff03d98bedc8
::start
$5 = 0x1000
::end
$6 = 0x800000000000
--------
--------
next:
$7 = 0xffffff01f943bb40
::start
$8 = 0x400000
::end
$9 = 0x40c000
--------
--------
next:
$10 = 0xffffff01f94cb780
::start
$11 = 0x50c000
::end
$12 = 0x50d000
--------
--------
next:
$13 = 0xffffff01f9452690
::start
$14 = 0x50d000
::end
$15 = 0x600000
--------
--------
next:
$16 = 0xffffff01f9452ca8
::start
$17 = 0x80050c000
::end
$18 = 0x80053c000
--------
--------
next:
$19 = 0xffffff007d349ca8
::start
$20 = 0x80053c000
::end
$21 = 0x800544000
--------
--------
next:
$22 = 0xffffff007d3295a0
::start
$23 = 0x80063c000
::end
$24 = 0x800644000
--------
--------
next:
$25 = 0xffffff000cf09ac8
::start
$26 = 0x800644000
::end
$27 = 0x800653000
--------
--------
next:
$28 = 0xffffff01f9581348
::start
$29 = 0x800653000
::end
$30 = 0x800697000
--------
--------
next:
$31 = 0xffffff04d28094b0
::start
$32 = 0x800697000
::end
$33 = 0x800796000
--------
--------
next:
$34 = 0xffffff01f9698708
::start
$35 = 0x800796000
::end
$36 = 0x8007a0000
--------
--------
next:
$37 = 0xffffff01f94cb708
::start
$38 = 0x8007a0000
::end
$39 = 0x8007be000
--------
--------
next:
$40 = 0xffffff012beda348
::start
$41 = 0x8007be000
::end
$42 = 0x8008be000
--------
--------
next:
$43 = 0xffffff01f94cc780
::start
$44 = 0x8008be000
::end
$45 = 0x8008c0000
--------
--------
next:
$46 = 0xffffff007d330528
::start
$47 = 0x8008c0000
::end
$48 = 0x8008c8000
--------
--------
next:
$49 = 0xffffff03f03347f8
::start
$50 = 0x8008c8000
::end
$51 = 0x8009c8000
--------
--------
next:
$52 = 0xffffff012beda960
::start
$53 = 0x8009c8000
::end
$54 = 0x8009c9000
--------
--------
next:
$55 = 0xffffff01f94b2348
::start
$56 = 0x8009c9000
::end
$57 = 0x800ad2000
--------
--------
next:
$58 = 0xffffff052b8144b0
::start
$59 = 0x800ad2000
::end
$60 = 0x800bd1000
--------
--------
next:
$61 = 0xffffff007d349d20
::start
$62 = 0x800bd1000
::end
$63 = 0x800bf0000
--------
--------
next:
$64 = 0xffffff01f94b2ca8
::start
$65 = 0x800bf0000
::end
$66 = 0x800c0b000
--------
--------
next:
$67 = 0xffffff01f943b1e0
::start
$68 = 0x800e00000
::end
$69 = 0x800e96000
::this entry covers the range
$70 = {prev = 0xffffff01f94b2ca8, next = 0xffffff00054f7960,
left = 0xffffff01f94b2ca8, right = 0x0, start = 0x800e00000,
end = 0x800e96000, avail_ssize = 0x0, adj_free = 0x0,
max_free = 0x7fff0c000, object = {vm_object = 0xffffff0342935000,
sub_map = 0xffffff0342935000}, offset = 0x210000, eflags = 0x0,
protection = 0x3, max_protection = 0x7, inheritance = 0x1,
wired_count = 0x0, lastr = 0x2c2, uip = 0x0}
--------
--------
next:
$71 = 0xffffff00054f7960
::start
$72 = 0x800e96000
::end
$73 = 0x800ea7000
::this entry covers the range
$74 = {prev = 0xffffff01f943b1e0, next = 0xffffff056f97b690,
left = 0xffffff01f943b1e0, right = 0xffffff056f97b690, start = 0x800e96000,
end = 0x800ea7000, avail_ssize = 0x0, adj_free = 0x0,
max_free = 0x7ff7fefe0000, object = {vm_object = 0xffffff0342935000,
sub_map = 0xffffff0342935000}, offset = 0x2a6000, eflags = 0x0,
protection = 0x3, max_protection = 0x7, inheritance = 0x1,
wired_count = 0x1, lastr = 0x2c2, uip = 0x0}
--------
--------
next:
$75 = 0xffffff056f97b690
::start
$76 = 0x800ea7000
::end
$77 = 0x801000000
--------
--------
next:
$78 = 0xffffff01f94cc8e8
::start
$79 = 0x7ffffffe0000
::end
$80 = 0x800000000000
--------
----Next_Part(Wed_Feb_29_02_26_51_2012_369)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; filename="gdb.cmd"
set height 0
f 16
echo --------\n
echo p/x *(td->td_pcb):\n
p/x *(td->td_pcb)
echo --------\n
f 11
set $start = req->oldptr
set $end = $start + req->oldlen
echo --------\n
echo range start:\n
p/x $start
echo range end:\n
p/x $end
echo --------\n
f 16
set $h = &td->td_proc->p_vmspace.vm_map.header
set $p = $h
set $x = 1
echo --------\n
echo td->td_proc->p_vmspace.vm_map.header:\n
p/x $h
echo ::start\n
p/x $h->start
echo ::end\n
p/x $h->end
set $map = 0
if ($p->start >= $start)
if ($p->start < $end)
set $map = 1
end
end
if ($p->end >= $start)
if ($p->end < $end)
set $map = 1
end
end
if ($map > 0)
echo ::this entry covers the range\n
p/x *$p
set $map = 0
end
echo --------\n
set $p = $p->next
while ($x > 0)
echo --------\n
echo next:\n
p/x $p
echo ::start\n
p/x $p->start
echo ::end\n
p/x $p->end
set $map = 0
if ($p->start >= $start)
if ($p->start < $end)
set $map = 1
end
end
if ($p->end >= $start)
if ($p->end < $end)
set $map = 1
end
end
if ($map > 0)
echo ::this entry covers the range\n
p/x *$p
set $map = 0
end
set $p = $p->next
if ($p == $h)
set $x = 0
end
echo --------\n
end
quit
----Next_Part(Wed_Feb_29_02_26_51_2012_369)----
----Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)--
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (FreeBSD)
iEYEABECAAYFAk9NDlsACgkQTyzT2CeTzy3TZgCfdpFiMmQ+aaD2XhQMs69Zcd4d
8K0An1HF6L/sW5MbZ/J5o2+929h3WvtB
=FQ1R
-----END PGP SIGNATURE-----
----Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120229.022651.1585266709145027511.hrs>