Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 04 Mar 2013 16:49:41 -0800
From:      Thomas Skibo <ThomasSkibo@sbcglobal.net>
To:        freebsd-arm@freebsd.org
Subject:   Weird kernel mode data abort panic on Zedboard.
Message-ID:  <51354125.4060500@sbcglobal.net>

next in thread | raw e-mail | index | archive | help

Hello, again.

I thought I'd describe this kernel panic and see if anyone has seen 
anything similar.  This is on the Zedboard (ARM Cortex-A9 armv6).

I can consistently panic the kernel by ftp'ing files to a local 
filesystem.  It crashes every fourth time or so in the same manner.  The 
specific fault is a 'Permission Fault (P)' on the first page of data in 
a file buf.

The fault occurs in ffs_write() when it attempts to uiomove/copyin data 
from user space to the file buf.  The first write faults but copyin() 
catches the fault and then calls vfs_bio_clrbuf() which tries to zero 
out the buf data.  It's the second fault that causes the panic.

Using Xilinx's XMD tool, I can walk the page table in physical memory 
and find the PTE.  It is always okay and has priviledged read and write 
permissions.  Curiously, sometimes I get another permission fault if I 
try to read from the fault address in the debugger.  That tells me there 
is a stale TLB entry without read permissions either.  The other buf 
pages seem okay.

Any ideas?  I can duplicate this with l2 cache turned off.  I can also 
duplicate it using a USB-ethernet interface instead of the Zynq cgem 
ethernet interface so it's not that driver.

Thanks,
--Thomas

============================================================

(ftp data from a remote system to Zedboard...)

230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd tftpboot
250 Directory successfully changed.
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||30574|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA100% |***********************************|  4078 KiB   27.55 
MiB/s    00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.45 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||32024|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA100% |***********************************|  4078 KiB   27.79 
MiB/s    00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.71 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||59472|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA100% |***********************************|  4078 KiB   27.81 
MiB/s    00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.73 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||18618|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA100% |***********************************|  4078 KiB   27.90 
MiB/s    00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.81 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||8385|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA100% |***********************************|  4078 KiB   27.83 
MiB/s    00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.75 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||45403|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA100% |***********************************|  4078 KiB   27.92 
MiB/s    00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.83 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||34019|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
   0% |                                   |     0        0.00 KiB/s 
--:-- ETA
vm_fault(0xc0610000, cdf2c000, 2, 0) -> 2
Fatal kernel mode data abort: 'Permission Fault (P)'
trapframe: 0xd67eda68
FSR=0000080f, FAR=cdf2c000, spsr=20000013
r0 =cdf2c000, r1 =00000f80, r2 =00000000, r3 =00000000
r4 =cdf2c000, r5 =00000000, r6 =00000000, r7 =cdf2d000
r8 =000000ff, r9 =00000000, r10=cd385660, r11=d67edae0
r12=cdf2c000, ssp=d67edab4, slr=c02d27a8, pc =c043039c

[ thread pid 619 tid 100040 ]
Stopped at      memset+0x48:    undge   0xa0cc20f8
db> show buffer $r10

===== faulty address is always first page of buf =====

buf at 0xcd385660
b_flags = 0x20000000<vmio>, b_xflags=0x2<clean>, b_vflags=0x0
b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0
b_bufobj = (0xc3028954), b_data = 0xcdf2c000, b_blkno = 2432, b_lblkno = 
24, b_dep = 0
b_npages = 8, pages(OBJ, IDX, PA): (0xc301dc24, 0xc0, 
0x402a000),(0xc301dc24, 0xc1, 0x402b000),(0xc301dc24, 0xc2, 
0x4250000),(0xc301dc24, 0xc3, 0x4251000),(0xc301dc24, 0xc4, 
0x4252000),(0xc301dc24, 0xc5, 0x4253000),(0xc301dc24, 0xc6, 
0x4274000),(0xc301dc24, 0xc7, 0x4275000)
lock type bufwait: EXCL by thread 0xc2ef7000 (pid 619, ftp, tid 100040)

db> bt
Tracing pid 619 tid 100040 td 0xc2ef7000
db_trace_self() at db_trace_self+0xc
scp=0xc04236fc rlv=0xc0423748 (db_trace_thread+0x38)
         rsp=0xd67ed77c rfp=0xd67ed788
db_trace_thread() at db_trace_thread+0xc
scp=0xc042371c rlv=0xc012b708 (db_command_init+0x354)
         rsp=0xd67ed78c rfp=0xd67ed7a8
db_command_init() at db_command_init+0x27c
scp=0xc012b630 rlv=0xc012b10c (db_skip_to_eol+0x4a0)
         rsp=0xd67ed7ac rfp=0xd67ed850
         r5=0x00000000 r4=0xc04d382c
db_skip_to_eol() at db_skip_to_eol+0x1d4
scp=0xc012ae40 rlv=0xc012b278 (db_command_loop+0x60)
         rsp=0xd67ed854 rfp=0xd67ed860
         r10=0x60000093 r8=0x0000080f
         r7=0x00000000 r6=0xcdf2c000 r5=0xc04d3af4 r4=0xd67ed86c
db_command_loop() at db_command_loop+0xc
scp=0xc012b224 rlv=0xc012d758 (X_db_sym_numargs+0xf4)
         rsp=0xd67ed864 rfp=0xd67ed980
X_db_sym_numargs() at X_db_sym_numargs+0x14
--More--        scp=0xc012d678 rlv=0xc02897a4 (kdb_trap+0xa4)
         rsp=0xd67ed984 rfp=0xd67ed9a8
         r4=0xd67eda68
kdb_trap() at kdb_trap+0xc
scp=0xc028970c rlv=0xc04329b4 (badaddr_read+0x284)
         rsp=0xd67ed9ac rfp=0xd67ed9c8
         r10=0x00000000 r8=0xd67eda68
         r7=0xc2ef7000 r6=0xcdf2c000 r5=0x0000080f r4=0xd67eda68
badaddr_read() at badaddr_read+0xfc
scp=0xc043282c rlv=0xc0432ef0 (data_abort_handler+0x4e4)
         rsp=0xd67ed9cc rfp=0xd67eda64
         r6=0xd67edef8 r5=0xc2eec8ac
         r4=0xcdf2c000
data_abort_handler() at data_abort_handler+0xc
scp=0xc0432a18 rlv=0xc0424f00 (address_exception_entry+0x50)
         rsp=0xd67eda68 rfp=0xd67edae0
         r10=0xcd385660 r9=0x00000000
         r8=0x000000ff r7=0xcdf2d000 r6=0x00000000 r5=0x00000000
         r4=0xcdf2c000
vfs_bio_clrbuf() at vfs_bio_clrbuf+0xc
--More--        scp=0xc02d2594 rlv=0xc03d7970 (ffs_syncvnode+0xb4c)
         rsp=0xd67edae4 rfp=0xd67edb5c
         r10=0x00008000 r9=0x00000000
         r8=0x00008000 r7=0x0000000f r6=0x00000000 r5=0x00008000
         r4=0xc2f61000
ffs_syncvnode() at ffs_syncvnode+0x6bc
scp=0xc03d74e0 rlv=0xc04448c8 (VOP_WRITE_APV+0x104)
         rsp=0xd67edb60 rfp=0xd67edc08
         r10=0x00000000 r9=0x000c0000
         r8=0x00000000 r7=0x00000000 r6=0x00000000 r5=0xd67edc24
         r4=0xc04f5f00
VOP_WRITE_APV() at VOP_WRITE_APV+0xc
scp=0xc04447d0 rlv=0xc02ff4b8 (vn_extattr_get+0x2e0)
         rsp=0xd67edc0c rfp=0xd67edc6c
         r8=0xc2f25ce8 r7=0xc30288e0
         r6=0x00000000 r5=0xd67edc24 r4=0x00000000
vn_extattr_get() at vn_extattr_get+0x158
scp=0xc02ff330 rlv=0xc02fd9a0 (foffset_lock_uio+0x1f8)
         rsp=0xd67edc70 rfp=0xd67edd40
         r10=0xc30288e0 r9=0x00010000
--More--                r8=0x00000000 r7=0x00000000 r6=0xc2f25ce8 
r5=0xd67eddb4
         r4=0x7fffffff
foffset_lock_uio() at foffset_lock_uio+0x38
scp=0xc02fd7e0 rlv=0xc029ce48 (sys_ioctl+0x210)
         rsp=0xd67edd44 rfp=0xd67edd70
         r10=0x00010000 r9=0x00000005
         r8=0x00000000 r7=0xc2f25ce8 r6=0xc2ef7000 r5=0xd67eddb4
         r4=0xffffffff
sys_ioctl() at sys_ioctl+0x174
scp=0xc029cdac rlv=0xc029d190 (kern_writev+0x60)
         rsp=0xd67edd74 rfp=0xd67edda8
         r10=0x00000000 r9=0x00000000
         r8=0xc2ef7000 r7=0xd67eddb4 r6=0x00000005 r5=0x00000000
         r4=0x00000000
kern_writev() at kern_writev+0xc
scp=0xc029d13c rlv=0xc029d230 (sys_write+0x58)
         rsp=0xd67eddac rfp=0xd67edde0
         r8=0x00000000 r7=0x00000000
         r6=0x00000000 r5=0xc2ef7000 r4=0xc2eec8ac
sys_write() at sys_write+0xc
--More--        scp=0xc029d1e4 rlv=0xc0433570 (swi_handler+0x49c)
         rsp=0xd67edde4 rfp=0xd67edea4
swi_handler() at swi_handler+0xc
scp=0xc04330e0 rlv=0xc0424d34 (swi_entry+0x30)
         rsp=0xd67edea8 rfp=0xbfffe2e0
         r7=0x00000000 r6=0xd67edeac
         r5=0x00010000 r4=0x2089d000
fiqvector() at 0x170f8
scp=0x000170f8 rlv=0x00019f94 (0x19f94)
         rsp=0xbfffe2e4 rfp=0xbfffe3d8
         r10=0x20803130 r9=0x00036614
         r8=0x00024058 r7=0x00000000 r6=0x00000001 r5=0x00000000
         r4=0x00000004
fiqvector() at 0x19824
scp=0x00019824 rlv=0x00010424 (0x10424)
         rsp=0xbfffe3dc rfp=0xbfffe884
         r7=0x20803130 r6=0x00000001
         r5=0x00000000 r4=0xbfffe8a0
fiqvector() at 0x101b4
scp=0x000101b4 rlv=0x00010634 (0x10634)
--More--                rsp=0xbfffe888 rfp=0xbfffe894
         r10=0x00000000 r9=0x00000000
         r8=0x00000000 r7=0xbfffee64 r6=0x20808400 r5=0x00030048
         r4=0xbfffe8a0
fiqvector() at 0x10614
scp=0x00010614 rlv=0x0001c82c (0x1c82c)
         rsp=0xbfffe898 rfp=0xbfffe8c8
fiqvector() at 0x1c4a0
scp=0x0001c4a0 rlv=0x0001d72c (0x1d72c)
         rsp=0xbfffe8cc rfp=0xbfffed10
         r6=0x00000001 r5=0x00037354
         r4=0x00034bfc
fiqvector() at 0x1c94c
scp=0x0001c94c rlv=0x0000aa58 (0xaa58)
         rsp=0xbfffed14 rfp=0xbfffed34
         r10=0x2004eb90 r8=0x00000000
         r7=0x00000000 r6=0xbfffed54 r5=0xbfffed48 r4=0x00000002
fiqvector() at 0xa928
scp=0x0000a928 rlv=0x20038144 (0x20038144)
         rsp=0xbfffed38 rfp=0x00000000
--More--                r8=0x00000000 r7=0x00000000
         r6=0xbfffed44 r5=0x20037db8 r4=0x20053000


====== (now debugger faults trying to read the same address) ======

db> x 0xcdf2c000
0xcdf2c000:
vm_fault(0xc0610000, cdf2c000, 2, 0) -> 2
Fatal kernel mode data abort: 'Permission Fault (P)'
trapframe: 0xd67ed6f0
FSR=0000000f, FAR=cdf2c000, spsr=60000093
r0 =00000000, r1 =c2ef7000, r2 =00000004, r3 =c051bac8
r4 =00000004, r5 =cdf2c000, r6 =00000000, r7 =d67ed760
r8 =cdf2c000, r9 =00000002, r10=00000010, r11=d67ed75c
r12=00000001, ssp=d67ed73c, slr=c0429c04, pc =c0423174

panic: Fatal abort
Uptime: 49s
Automatic reboot in 15 seconds - press a key on the console to abort


-- 
--------
Thomas Skibo
ThomasSkibo@sbcglobal.net




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51354125.4060500>