From owner-freebsd-arm@FreeBSD.ORG Sun Jul 5 10:36:48 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41E9A1065673; Sun, 5 Jul 2009 10:36:48 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe07.swip.net [212.247.154.193]) by mx1.freebsd.org (Postfix) with ESMTP id 44CE48FC12; Sun, 5 Jul 2009 10:36:46 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=Hrwt8fWgTlIA:10 a=MXw7gxVQKqGXY79tIT8aFQ==:17 a=dT19lrtdAAAA:8 a=MinmzMF1LL3qrF-msNAA:9 a=D_bpiQ21t--K_77gx9EA:7 a=h5JnGOmdAB9tnkmLPbytnJLNn9kA:4 Received: from [62.113.132.61] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe07.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 1269025837; Sun, 05 Jul 2009 12:36:45 +0200 From: Hans Petter Selasky To: Piotr =?iso-8859-2?q?Zi=EAcik?= Date: Sun, 5 Jul 2009 12:36:21 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-CURRENT; KDE/4.2.4; i386; ; ) References: <200906231035.43096.kosmo@semihalf.com> <200906301037.49367.kosmo@semihalf.com> <200906301128.26046.hselasky@c2i.net> In-Reply-To: <200906301128.26046.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907051236.22783.hselasky@c2i.net> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jul 2009 10:36:48 -0000 On Tuesday 30 June 2009 11:28:23 Hans Petter Selasky wrote: > On Tuesday 30 June 2009 10:37:48 Piotr Zi=EAcik wrote: > > Monday 29 June 2009 15:16:56 Hans Petter Selasky napisa=B3(a): > > > You want to change the flags passed to bus_dmamap_sync() so that the > > > flush/invalidate mapping gets right. I understand why your patch makes > > > it work. That's not the problem. > > > > > > In "src/sys/arm/arm/busdma_machdep.c" there is a function called > > > "_bus_dmamap_sync_bp()". If you look at that function you see that it > > > only triggers on the "BUS_DMASYNC_PREWRITE" and "BUS_DMASYNC_POSTREAD" > > > flags. After your patching only the PREXXXX flags are used, so if bou= ce > > > pages are used on ARM and x86 and amd64 +++, then only > > > BUS_DMASYNC_PREWRITE will do anything. This indicates that your patch > > > is not fully correct. > > Hi, > > > That is true. I have missed "bounce page" case. I can change flags pass= ed > > to bus_dmamap_sync() to fix this on ARM, but this will break MIPS. > > Right, so there is inconsistency among the platforms in how the BUSDMA is > implemented, which should be fixed. > > > This clearly shows the problem - using side effect of busdma to manage > > CPU cache. Current USB implementation relies of this side effect assumi= ng > > that bus_dmamap_sync() with given flags will do cpu cache > > flush/invalidation. This is not true even on i386 ! > > i386 handles this differently. > > > This thread is not about my patch. It is only a fast and dirty hack > > making USB working on platforms without hardware cache coherency > > and showing the problem. I see two proper solutions: > > Ok. > > > 1. Implement usb_pc_cpu_invalidate() / usb_pc_cpu_flush() with > > cpu_*() functions realizing requested operation. > > > > 2. Using busdma in proper way: > > [... prepare data to transfer ...] > > bus_dmamap_sync(..., PREREAD | PREWRITE); > > [... do transfer ...] > > bus_dmamap_sync(..., POSTREAD | POSTWRITE); > > [... check results ...] > > as manpage says: > > > > If read and write operations are not preceded and followed by the > > appropriate synchronization operations, behavior is undefined. > > > > Requirement cited above is currently not met in USB stack. Funtion > > shown in http://fxr.watson.org/fxr/source/dev/usb/controller/ehci.c#L12= 94 > > is just an example of improper usage of bus_dmamap_sync(), which is > > hidden under usb_pc_cpu_invalidate(). > > I'm not violating any rules in busdma. The busdma manual is silent about > doing multiple sync operations of the same kind in a row. If the sheeva > hardware you are using works differently, you have to fix it. > > > This thread started from my question about general usage of > > usb_pc_cpu_*() functions. So I am asking once again - why these functio= ns > > relies on side effect of busdma instead of simply doing cache > > flush/invalidation through cpu_*() functions ? > > Because busdma is the interface for doing these kinds of operations and n= ot > cpu_xxx(). The DMA memory is allocated using busdma and loaded using > busdma, and cpu_xxx() is not a part of the busdma interface. > > > > Grepping through the source code for ARM, I found a line like this: > > > (...) > > > You see that it only performs purge and no prior flush. This is what > > > needs fixing! If semihalf could go through the "arm/arm/cpufunc.c" fi= le > > > and fix those flush and invalidate functions then many people would > > > become happy! > > > > My developement platform is sheeva. CPU functions for this platform are > > implemented correctly. > > > > > Again, it is not a problem in USB firstly, it is a problem in > > > "arm/xxx". > > > > You are suggesting that the problem is in ARM busdma (and in MIPS busdm= a, > > as on this platform USB also does not work). Could you give me example = of > > platform _without_ hardware coherency where new USB stack simply works ? > > If I find time later today I will fix the cache sync operations for > AT91RM9200 which has a built in OHCI, which has been working fine with my > new USB stack. > > Can you please add some prints to the "bus_dmamap_sync_buf()" code on ARM, > and verify which cases the CPU is entering, without your patch and with > your patch. From my analysis your patch will just change the execution > path: > > With your patch it calls: > > cpu_dcache_wb_range((vm_offset_t)buf, len); > cpu_l2cache_wb_range((vm_offset_t)buf, len); > > Without your patch it calls: > > cpu_dcache_wbinv_range((vm_offset_t)buf, len); > cpu_l2cache_wbinv_range((vm_offset_t)buf, len); > > In my view these execution paths are equivalent with regard to clean or > cache flush. They should both perform a cache flush, which is what the USB > code wants to do, but obviously the latter case is not implemented > correctly, most likely because it doesn't flush the buffer like expected! > > Try adding: > > cpu_dcache_wb_range((vm_offset_t)buf, len); > cpu_l2cache_wb_range((vm_offset_t)buf, len); > > Before: > > cpu_dcache_wbinv_range((vm_offset_t)buf, len); > cpu_l2cache_wbinv_range((vm_offset_t)buf, len); > > In: sys/arm/arm/busdma_machdep.c > Hi, I did not find time yet to test on my AT91RM9200 board. I hope to do a test= =20 this week. Did you at semihalf find anything? =2D-HPS From owner-freebsd-arm@FreeBSD.ORG Mon Jul 6 09:29:33 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 069DE1065672 for ; Mon, 6 Jul 2009 09:29:32 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 115918FC1F for ; Mon, 6 Jul 2009 09:29:30 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from [10.0.0.43] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id BEF48C3BB6; Mon, 6 Jul 2009 11:10:52 +0200 (CEST) Message-ID: <4A51BFF6.4090401@semihalf.com> Date: Mon, 06 Jul 2009 11:12:22 +0200 From: Michal Hajduk User-Agent: Mozilla-Thunderbird 2.0.0.19 (X11/20090103) MIME-Version: 1.0 To: tinguely@casselton.net Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-arm@freebsd.org Subject: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2009 09:29:33 -0000 Hello Mark, While testing EHCI (FreeBSD 8-current with new USB-stack) on ARM machine I have found some problem probably related to svn commit *194459* http://svn.freebsd.org/viewvc/base?view=revision&revision=194459 With compiled USB-stack I had vm_fault: ehci0: at mem 0xf1050000-0xf1050fff irq 48,19 on mbus0 lock order reversal: (sleepable after non-sleepable) 1st 0xc0f55a88 256 (UMA zone) @ /home/mih/git/marvell-current/sys/vm/uma_core.c:2036 2nd 0xc0bf2fbc user map (user map) @ /home/mih/git/marvell-current/sys/vm/vm_map.c:3520 KDB: stack backtrace: db_trace_thread() at db_trace_thread+0x10 scp=0xc0b28458 rlv=0xc0916b78 ($a+0x34) rsp=0xc0e46764 rfp=0xc0e46880 r10=0xc36236c0 r9=0x00000dc0 r8=0xc0d3504c r7=0xc3623930 r6=0xffffffff r5=0xc0b52180 r4=0xc0e4676c $a() at $a+0x10 scp=0xc0916b54 rlv=0xc09f6ae0 (kdb_backtrace+0x3c) rsp=0xc0e46884 rfp=0xc0e46894 r4=0xc0c05d60 kdb_backtrace() at kdb_backtrace+0x10 scp=0xc09f6ab4 rlv=0xc0a05e80 (_witness_debugger+0x5c) rsp=0xc0e46898 rfp=0xc0e468ac r4=0x00000001 _witness_debugger() at _witness_debugger+0x14 scp=0xc0a05e38 rlv=0xc0a069a8 (witness_checkorder+0x5f4) rsp=0xc0e468b0 rfp=0xc0e468f8 r5=0xc0bf2fbc r4=0x00000000 witness_checkorder() at witness_checkorder+0x10 scp=0xc0a063c4 rlv=0xc09d5cdc (_sx_slock+0x30) rsp=0xc0e468fc rfp=0xc0e46920 r10=0xc0e46b40 r9=0xc0e46a40 r8=0x00000000 r7=0x00000000 r6=0x00000dc0 r5=0xc0b89084 r4=0xc0bf2fbc _sx_slock() at _sx_slock+0x10 scp=0xc09d5cbc rlv=0xc0b1171c (vm_map_lookup+0x3c) rsp=0xc0e46924 rfp=0xc0e46964 r8=0x00000001 r7=0x00000001 r6=0xc0bf2f74 r5=0x00000000 r4=0xc0e46a68 vm_map_lookup() at vm_map_lookup+0x10 scp=0xc0b116f0 rlv=0xc0b0b61c (vm_fault+0x90) rsp=0xc0e46968 rfp=0xc0e46a98 r10=0xc0e46b40 r9=0xc0e46ef8 r8=0x00000001 r7=0xc0bf2d40 r6=0x00000000 r5=0x00000000 r4=0xc0e46a68 vm_fault() at vm_fault+0x10 scp=0xc0b0b59c rlv=0xc0b37cf0 (data_abort_handler+0x1ec) rsp=0xc0e46a9c rfp=0xc0e46b3c r10=0xc0e46b40 r9=0xc0e46ef8 r8=0x00000001 r7=0xc0bf2d40 r6=0x00000000 r5=0x00000000 r4=0xc0bf2b00 data_abort_handler() at data_abort_handler+0x10 scp=0xc0b37b14 rlv=0xc0b29ee0 (exception_exit) rsp=0xc0e46b40 rfp=0xc0e46bac r10=0xc0f4a1a4 r9=0x0000001f r8=0xc0f54c80 r7=0x00000000 r6=0xc0f54c80 r5=0xffff1004 r4=0xc0f55a80 $a() at $a+0x10 scp=0xc0b08b6c rlv=0xc0b08ef4 ($a+0x3c) rsp=0xc0e46bb0 rfp=0xc0e46bc8 r7=0x0000000f r6=0xc0f54c80 r5=0x00000003 r4=0xc0f55a80 $a() at $a+0x10 scp=0xc0b08ec8 rlv=0xc0b0a13c (uma_zalloc_arg+0x32c) rsp=0xc0e46bcc rfp=0xc0e46c10 r6=0xc3b5efa8 r5=0x0000000f r4=0x0000000f uma_zalloc_arg() at uma_zalloc_arg+0x10 scp=0xc0b09e20 rlv=0xc09bfefc (malloc+0x5c) rsp=0xc0e46c14 rfp=0xc0e46c34 r10=0xc0e46c80 r9=0xc0bebeac r8=0xc0bda080 r7=0x00000002 r6=0x00000100 r5=0xc0f54c80 r4=0x00000004 malloc() at malloc+0x14 scp=0xc09bfeb4 rlv=0xc0b26674 (bus_dmamem_alloc+0x258) rsp=0xc0e46c38 rfp=0xc0e46c70 r8=0x00000004 r7=0x00000002 r6=0xc3621800 r5=0xc0d6a234 r4=0x00000080 bus_dmamem_alloc() at bus_dmamem_alloc+0x10 scp=0xc0b2642c rlv=0xc0966fc4 (usb_pc_alloc_mem+0xa4) rsp=0xc0e46c74 rfp=0xc0e46cb0 r10=0xc3b4de28 r9=0xc0b64844 r8=0xc3b4d808 r7=0xc3b4ddfc r6=0x00000100 r5=0xc3b4bf20 r4=0x00000080 usb_pc_alloc_mem() at usb_pc_alloc_mem+0x10 scp=0xc0966f30 rlv=0xc0962748 ($a+0x2c) rsp=0xc0e46cb4 rfp=0xc0e46cc4 r10=0x00000000 r9=0xc0b64844 r8=0xc095ac30 r7=0xc096271c r6=0xc3b4dc50 r5=0xc3b4a000 r4=0xc3b4dc50 $a() at $a+0x10 scp=0xc096272c rlv=0xc095ad00 (ehci_iterate_hw_softc+0xd0) rsp=0xc0e46cc8 rfp=0xc0e46ce8 r4=0x00000078 ehci_iterate_hw_softc() at ehci_iterate_hw_softc+0x10 scp=0xc095ac40 rlv=0xc09626bc (usb_bus_mem_alloc_all+0xb8) rsp=0xc0e46cec rfp=0xc0e46d18 r7=0xc3b4dc50 r6=0x00000000 r5=0xc3b4ddac r4=0x00000008 usb_bus_mem_alloc_all() at usb_bus_mem_alloc_all+0x10 scp=0xc0962614 rlv=0xc0b49494 ($a+0x60) rsp=0xc0e46d1c rfp=0xc0e46d5c r8=0xc3b4dc50 r7=0xc09f08fc r6=0xc3620000 r5=0xc3b4a000 r4=0xc3620480 $a() at $a+0x10 scp=0xc0b49444 rlv=0xc09f1fc8 (device_attach+0x70) rsp=0xc0e46d60 rfp=0xc0e46da0 r10=0x00000000 r9=0xc0b64844 r8=0xc3620000 r7=0xc09f08fc r6=0xc362004c r5=0xc36bd344 r4=0xc3620480 device_attach() at device_attach+0x10 scp=0xc09f1f68 rlv=0xc09f3060 (bus_generic_attach+0x20) rsp=0xc0e46da4 rfp=0xc0e46db4 r10=0x00000000 r9=0xc0b64844 r8=0xc3620480 r7=0xc0bed508 r6=0xc0bed510 r5=0x00000001 r4=0xc3620000 bus_generic_attach() at bus_generic_attach+0x10 scp=0xc09f3050 rlv=0xc0b40864 ($a+0x1fc) rsp=0xc0e46db8 rfp=0xc0e46de0 r4=0x00000020 $a() at $a+0x10 scp=0xc0b40678 rlv=0xc09f1fc8 (device_attach+0x70) rsp=0xc0e46de4 rfp=0xc0e46e24 r8=0xc3620480 r7=0xc09f08fc r6=0xc36204cc r5=0xc361b204 r4=0xc3620600 device_attach() at device_attach+0x10 scp=0xc09f1f68 rlv=0xc09f3060 (bus_generic_attach+0x20) rsp=0xc0e46e28 rfp=0xc0e46e38 r10=0x00000000 r9=0xc0b64844 r8=0xc3620600 r7=0xc09f08fc r6=0xc362064c r5=0xc361b465 r4=0xc3620480 bus_generic_attach() at bus_generic_attach+0x10 scp=0xc09f3050 rlv=0xc0b2cdcc ($a+0x1c) rsp=0xc0e46e3c rfp=0xc0e46e4c r4=0xc3620600 $a() at $a+0x10 scp=0xc0b2cdc0 rlv=0xc09f1fc8 (device_attach+0x70) rsp=0xc0e46e50 rfp=0xc0e46e90 r4=0x80000000 device_attach() at device_attach+0x10 scp=0xc09f1f68 rlv=0xc09f32a0 (bus_generic_new_pass+0x110) rsp=0xc0e46e94 rfp=0xc0e46eb4 r10=0x00000000 r9=0x0100029c r8=0xc0c05b68 r7=0xc3620a00 r6=0xc0bec458 r5=0xc0bd38d0 r4=0xc3620600 bus_generic_new_pass() at bus_generic_new_pass+0x10 scp=0xc09f31a0 rlv=0xc09efb28 (bus_set_pass+0x98) rsp=0xc0e46eb8 rfp=0xc0e46ed4 r8=0x0103cc50 r7=0xc0c05b68 r6=0x7fffffff r5=0xc3620a00 r4=0xc361c820 bus_set_pass() at bus_set_pass+0x10 scp=0xc09efaa0 rlv=0xc0997a1c (mi_startup+0x8c) rsp=0xc0e46ed8 rfp=0xc0e46ef4 r7=0xc0bf345c r6=0xc0bf3460 r5=0xc0bf3464 r4=0xc0b8f2f4 mi_startup() at mi_startup+0x10 scp=0xc09979a0 rlv=0xc09001e4 (virt_done+0x14) rsp=0xc0e46ef8 rfp=0x00000000 r7=0x00900140 r6=0x0103e020 r5=0x0090014c r4=0x00900224 vm_fault(0xc0bf2f74, 0, 1, 0) -> 1 Fatal kernel mode data abort: 'Translation Fault (S)' trapframe: 0xc0e46b40 FSR=00000005, FAR=00000004, spsr=600000d3 r0 =c0f55a80, r1 =c0f54c80, r2 =00000003, r3 =fffffffb r4 =c0f55a80, r5 =00000003, r6 =c0f54c80, r7 =00000000 r8 =c0f54c80, r9 =0000001f, r10=c0f4a1a4, r11=c0e46bac r12=c0e46bb0, ssp=c0e46b8c, slr=c0b08ef4, pc =c0b08c74 [thread pid 0 tid 100000 ] Stopped at $a+0x118: ldr r2, [r7, #0x004] db> Without new-usb stack with booting from NFS I have a LOR on the line which was added in svn commit *194459* : mge0: link state changed to UP NFS ROOT: 10.0.0.201:/nfsroot/rd6281/ lock order reversal: 1st 0xc36050b0 pmap (pmap) @ /home/mih/git/marvell-current/sys/arm/arm/pmap.c:971 2nd 0xc0d26038 vm object (uma object) @ /home/mih/git/marvell-current/sys/vm/uma_core.c:1011 KDB: stack backtrace: db_trace_thread() at db_trace_thread+0x10 scp=0xc0af6ce8 rlv=0xc0907798 ($a+0x34) rsp=0xcf404880 rfp=0xcf40499c r10=0xc35c84b8 r9=0x000003f3 r8=0xc0cea35c r7=0xc35c8a00 r6=0xffffffff r5=0xc0b1e610 r4=0xcf404888 $a() at $a+0x10 scp=0xc0907774 rlv=0xc09c5370 (kdb_backtrace+0x3c) rsp=0xcf4049a0 rfp=0xcf4049b0 r4=0xc0bbb1d0 kdb_backtrace() at kdb_backtrace+0x10 scp=0xc09c5344 rlv=0xc09d4710 (_witness_debugger+0x5c) rsp=0xcf4049b4 rfp=0xcf4049c8 r4=0x00000001 _witness_debugger() at _witness_debugger+0x14 scp=0xc09d46c8 rlv=0xc09d5238 (witness_checkorder+0x5f4) rsp=0xcf4049cc rfp=0xcf404a14 r5=0xc0d26038 r4=0x00000000 witness_checkorder() at witness_checkorder+0x10 scp=0xc09d4c54 rlv=0xc09912b4 (_mtx_lock_flags+0x34) rsp=0xcf404a18 rfp=0xcf404a40 r10=0xc0d26b80 r9=0xc0ef86e0 r8=0x000003f3 r7=0xc0b46f74 r6=0x00000000 r5=0x00000000 r4=0xc0d26038 _mtx_lock_flags() at _mtx_lock_flags+0x10 scp=0xc0991290 rlv=0xc0ad56b0 ($a+0x3c) rsp=0xcf404a44 rfp=0xcf404a80 r10=0x00001000 r8=0x00000101 r7=0x00000000 r6=0xc0ef86e0 r5=0xc0ef9600 r4=0xc0ef9600 $a() at $a+0x10 scp=0xc0ad5684 rlv=0xc0ad6f44 ($a+0x70) rsp=0xcf404a84 rfp=0xcf404ac0 r10=0xc0ad5674 r9=0xc0ef86e0 r8=0x00000101 r7=0x00000000 r6=0xc0ef86e0 r5=0x00000001 r4=0xc0ef9600 $a() at $a+0x10 scp=0xc0ad6ee4 rlv=0xc0ad74c0 ($a+0xd4) rsp=0xcf404ac4 rfp=0xcf404ae4 r10=0xc0ee4c48 r9=0x00000080 r8=0xc0ef86e0 r7=0x00000000 r6=0xc0ef86e0 r5=0x00000201 r4=0xc0ef9600 $a() at $a+0x10 scp=0xc0ad73fc rlv=0xc0ad7784 ($a+0x3c) rsp=0xcf404ae8 rfp=0xcf404b00 r7=0x0000006f r6=0xc0ef86e0 r5=0x00000001 r4=0xc0ef9600 $a() at $a+0x10 scp=0xc0ad7758 rlv=0xc0ad89cc (uma_zalloc_arg+0x32c) rsp=0xcf404b04 rfp=0xcf404b48 r6=0xc1888de8 r5=0x0000006f r4=0x0000006f uma_zalloc_arg() at uma_zalloc_arg+0x10 scp=0xc0ad86b0 rlv=0xc0b00ca4 ($a+0x870) rsp=0xcf404b4c rfp=0xcf404bd0 r10=0x01f9a55e r9=0x0000003c r8=0xc36050b0 r7=0x00000000 r6=0x00000000 r5=0x01f9a552 r4=0xc3cbe0b8 $a() at $a+0x10 scp=0xc0b00444 rlv=0xc0b0169c (pmap_enter+0x70) rsp=0xcf404bd4 rfp=0xcf404c04 r10=0x00000007 r9=0xc0d1b948 r8=0xc0f951b8 r7=0x7ffff000 r6=0x00000000 r5=0xc36050b0 r4=0xc0b4a4bc pmap_enter() at pmap_enter+0x10 scp=0xc0b0163c rlv=0xc0adaff0 ($a+0x17c) rsp=0xcf404c08 rfp=0xcf404d38 r10=0xcf404de0 r9=0xcf404ef8 r8=0x00000002 r7=0x00000000 r6=0x00000000 r5=0x00000001 r4=0x00000000 vm_fault() at vm_fault+0x10 scp=0xc0ad9e2c rlv=0xc0b065f0 (data_abort_handler+0x1ec) rsp=0xcf404d3c rfp=0xcf404ddc r10=0xcf404de0 r9=0xcf404ef8 r8=0x00000002 r7=0xc3603000 r6=0x00000000 r5=0x7ffff000 r4=0xc3601088 data_abort_handler() at data_abort_handler+0x10 scp=0xc0b06414 rlv=0xc0af8770 (exception_exit) rsp=0xcf404de0 rfp=0xcf404e80 r10=0xc0965f04 r9=0xc3601000 r8=0xc0b892c8 r7=0xc0b892d2 r6=0xcf404eac r5=0xffff1004 r4=0x7fffffff $a() at $a+0x10 scp=0xc0965f14 rlv=0xc097c78c (fork_exit+0x64) rsp=0xcf404e84 rfp=0xcf404ea8 r10=0xc0965f04 r9=0x00000000 r8=0x00000000 r7=0xc3601000 r6=0xcf404eac r5=0xc0d26b80 r4=0xc3603000 fork_exit() at fork_exit+0x10 scp=0xc097c738 rlv=0xc0b05c5c (fork_trampoline+0x14) rsp=0xcf404eac rfp=0x00000000 r10=0x00000000 r8=0x00000000 r7=0xc0b29cb8 r6=0xcf45feac r5=0x00000000 r4=0xc0965f04 lock order reversal: 1st 0xc36050b0 pmap (pmap) @ /home/mih/git/marvell-current/sys/arm/arm/pmap.c:971 2nd 0xc0d1b948 vm page queue mutex (vm page queue mutex) @ /home/mih/git/marvell-current/sys/arm/arm/pmap.c:1696 KDB: stack backtrace: db_trace_thread() at db_trace_thread+0x10 scp=0xc0af6ce8 rlv=0xc0907798 ($a+0x34) rsp=0xcf4047e8 rfp=0xcf404904 r10=0xc35c84b8 r9=0x000006a0 r8=0xc0ce9ee4 r7=0xc35c8178 r6=0xffffffff r5=0xc0b1e610 r4=0xcf4047f0 $a() at $a+0x10 scp=0xc0907774 rlv=0xc09c5370 (kdb_backtrace+0x3c) rsp=0xcf404908 rfp=0xcf404918 r4=0xc0bbb1d0 kdb_backtrace() at kdb_backtrace+0x10 scp=0xc09c5344 rlv=0xc09d4710 (_witness_debugger+0x5c) rsp=0xcf40491c rfp=0xcf404930 r4=0x00000001 _witness_debugger() at _witness_debugger+0x14 scp=0xc09d46c8 rlv=0xc09d5238 (witness_checkorder+0x5f4) rsp=0xcf404934 rfp=0xcf40497c r5=0xc0d1b948 r4=0x00000000 witness_checkorder() at witness_checkorder+0x10 scp=0xc09d4c54 rlv=0xc09912b4 (_mtx_lock_flags+0x34) rsp=0xcf404980 rfp=0xcf4049a8 r10=0xc0d26b80 r9=0x00000012 r8=0x000006a0 r7=0xc0b4a4bc r6=0x00000000 r5=0x00000000 r4=0xc0d1b948 _mtx_lock_flags() at _mtx_lock_flags+0x10 scp=0xc0991290 rlv=0xc0afc600 ($a+0x110) rsp=0xcf4049ac rfp=0xcf4049d8 r10=0x00000000 r8=0xc0f95494 r7=0xc36054dc r6=0xc36050b0 r5=0xc0f95464 r4=0xc1889150 $a() at $a+0x10 scp=0xc0afc500 rlv=0xc0b00f90 ($a+0xb5c) rsp=0xcf4049dc rfp=0xcf404a60 r10=0x01fa300e r9=0x00000000 r8=0xc36050b0 r7=0x00000000 r6=0x00000000 r5=0x01fa3002 r4=0xc3cbde90 $a() at $a+0x10 scp=0xc0b00444 rlv=0xc0b015c8 (pmap_enter_object+0x88) rsp=0xcf404a64 rfp=0xcf404a98 r10=0x00000005 r9=0xc36050b0 r8=0xc0f95464 r7=0x00000000 r6=0x00000097 r5=0xc0f95464 r4=0x00000001 pmap_enter_object() at pmap_enter_object+0x10 scp=0xc0b01550 rlv=0xc0ade254 (vm_map_pmap_enter+0x294) rsp=0xcf404a9c rfp=0xcf404af0 r10=0x00000000 r9=0x00000000 r8=0x00000000 r7=0x00000097 r6=0x00000000 r5=0x00000000 r4=0x0000000f vm_map_pmap_enter() at vm_map_pmap_enter+0x14 scp=0xc0addfd4 rlv=0xc0adef74 (vm_map_insert+0x390) rsp=0xcf404af4 rfp=0xcf404b44 r10=0x00008000 r9=0x00000000 r8=0x00000000 r7=0xc3cc1110 r6=0x00000000 r5=0x00000000 r4=0x00000000 vm_map_insert() at vm_map_insert+0x10 scp=0xc0adebf4 rlv=0xc0963f9c ($a+0x258) rsp=0xcf404b48 rfp=0xcf404b8c r10=0x0009f000 r9=0x00008000 r8=0xc3605000 r7=0x00000000 r6=0x00000000 r5=0x00000005 r4=0x00000000 $a() at $a+0x10 scp=0xc0963d54 rlv=0xc09640b0 ($a+0xe0) rsp=0xcf404b90 rfp=0xcf404bd0 r10=0x00096208 r9=0x00008000 r8=0xc3cc1110 r7=0x00000000 r6=0x00096208 r5=0xfffff000 r4=0x0009f000 $a() at $a+0x10 scp=0xc0963fe0 rlv=0xc0965300 ($a+0x448) rsp=0xcf404bd4 rfp=0xcf404c64 r10=0x00000000 r9=0xc0ba7eb4 r8=0xc0ba19a4 r7=0xcf404d58 r6=0xcf408000 r5=0x00000000 r4=0xcf408034 $a() at $a+0x10 scp=0xc0964ec8 rlv=0xc09799d0 (kern_execve+0x258) rsp=0xcf404c68 rfp=0xcf404de4 r10=0x00000000 r9=0xc3603000 r8=0xcf404d58 r7=0xffffffff r6=0xc0ba8bb4 r5=0x00000000 r4=0x00000002 kern_execve() at kern_execve+0x10 scp=0xc0979788 rlv=0xc097a58c (execve+0x48) rsp=0xcf404de8 rfp=0xcf404e28 r10=0x7ffffffd r9=0xc3601000 r8=0xc0b892c8 r7=0xc0b892d2 r6=0x7ffffff2 r5=0xcf404df0 r4=0xc3603000 execve() at execve+0x10 scp=0xc097a554 rlv=0xc09660f0 ($a+0x1ec) rsp=0xcf404e2c rfp=0xcf404e80 r5=0x7fffffe4 r4=0x7fffffe8 $a() at $a+0x10 scp=0xc0965f14 rlv=0xc097c78c (fork_exit+0x64) rsp=0xcf404e84 rfp=0xcf404ea8 r10=0xc0965f04 r9=0x00000000 r8=0x00000000 r7=0xc3601000 r6=0xcf404eac r5=0xc0d26b80 r4=0xc3603000 fork_exit() at fork_exit+0x10 scp=0xc097c738 rlv=0xc0b05c5c (fork_trampoline+0x14) rsp=0xcf404eac rfp=0x00000000 r10=0x00000000 r8=0x00000000 r7=0xc0b29cb8 r6=0xcf45feac r5=0x00000000 r4=0xc0965f04 When I reverted changes in pmap from this svn commit I didn't have LORs nor panics. I will be grateful for any help or advice. Best regards, MichaÅ‚ Hajduk From owner-freebsd-arm@FreeBSD.ORG Mon Jul 6 09:46:21 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B81591065677 for ; Mon, 6 Jul 2009 09:46:21 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [IPv6:2001:16d8:fffb:4::abba]) by mx1.freebsd.org (Postfix) with ESMTP id 737248FC0C for ; Mon, 6 Jul 2009 09:46:21 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from stasss.yandex.ru (dhcp170-227-red.yandex.net [95.108.170.227]) by mx0.deglitch.com (Postfix) with ESMTPSA id D1B9F8FC27; Mon, 6 Jul 2009 13:46:19 +0400 (MSD) Date: Mon, 6 Jul 2009 13:46:12 +0400 From: Stanislav Sedov To: Michal Hajduk Message-Id: <20090706134612.5cf860e4.stas@FreeBSD.org> In-Reply-To: <4A51BFF6.4090401@semihalf.com> References: <4A51BFF6.4090401@semihalf.com> Organization: The FreeBSD Project X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA1"; boundary="Signature=_Mon__6_Jul_2009_13_46_12_+0400_6BfucpCp+E3UrWB8" Cc: tinguely@casselton.net, freebsd-arm@freebsd.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2009 09:46:22 -0000 --Signature=_Mon__6_Jul_2009_13_46_12_+0400_6BfucpCp+E3UrWB8 Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, 06 Jul 2009 11:12:22 +0200 Michal Hajduk mentioned: > Hello Mark, >=20 > While testing EHCI (FreeBSD 8-current with new USB-stack) on > ARM machine I have found some problem probably related to > svn commit *194459*=20 > http://svn.freebsd.org/viewvc/base?view=3Drevision&revision=3D194459 > With compiled USB-stack I had vm_fault: >=20 Hi, Michal! This is the known problem and we currently working on fixing it. It looks like that r194459, while certainly good, uncovered a memory corruption problem in ARM code. As a workaround you can revert r194459 for now. --=20 Stanislav Sedov ST4096-RIPE --Signature=_Mon__6_Jul_2009_13_46_12_+0400_6BfucpCp+E3UrWB8 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJKUcfqAAoJEKN82nOYvCd0rUQQALFk77UjxaHliEdmkBOeGXUj qpP+3MTmC6RRAzaZqzvF2nf9PMNUSv7pIPbwOLo2m+wtSYzpdpVcyGLXOJR/7mMn Qdwlye8YHd3h4880W47F19R5yzl8S+mHmG913/wjaemiE3f8HL2wpTQmaP2pEvNt T4fYRsXD+oJiXpnnXJWdEk4EE1tCU3hru2mGxxahZP7d+vOx4jr5KAMAaiHO6NDo dX7LJspXXUR5G3TNDdE9OLH/bcM/NDbRtjWWR+JbfJz28m12bcRBBsAD3oBCGYFM q9jyC3Ta3wxYmIKQZX+FV+llWeia/ZNmxocdEfiE0mXW6QR0iMBtp/84h73+faUj +beRFJ+FLaCgjXDRn+Q7V3Jkdd03dyMCgEEg8VzsFM0qmL7ERhtKKlbPm6ZURfEz M+hN55TcM0EzzetXAFPmtZfU5oh1hWpA+gYP0bxx7dgH4dfpkFPAB4yBENihbBgE W74JQXVGPNV0bjo4aBu6veSxGLSwhLKiYu/JTVcZyql3jqTIz2gcD+w4nU2hvKxh M6KZGFAiZvnIjnthX7ZLJSs641slT/eFdlMyuUlvzIf0qipDoLr3GA5MPyDK2b3O UeF5KvfRD0NPJ9CtffwixoFy2oD4UKWv17mMdkR2kPm0wMxngsdS1iOwkM4u3eTR k+8vgY4JgwKL6kR47KIO =vLHp -----END PGP SIGNATURE----- --Signature=_Mon__6_Jul_2009_13_46_12_+0400_6BfucpCp+E3UrWB8-- From owner-freebsd-arm@FreeBSD.ORG Mon Jul 6 10:35:45 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D73341065670 for ; Mon, 6 Jul 2009 10:35:45 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 7BF378FC19 for ; Mon, 6 Jul 2009 10:35:45 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from [10.0.0.43] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 3C160C3AAA; Mon, 6 Jul 2009 12:34:38 +0200 (CEST) Message-ID: <4A51D398.9070104@semihalf.com> Date: Mon, 06 Jul 2009 12:36:08 +0200 From: Michal Hajduk User-Agent: Mozilla-Thunderbird 2.0.0.19 (X11/20090103) MIME-Version: 1.0 To: Stanislav Sedov References: <4A51BFF6.4090401@semihalf.com> <20090706134612.5cf860e4.stas@FreeBSD.org> In-Reply-To: <20090706134612.5cf860e4.stas@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: tinguely@casselton.net, freebsd-arm@freebsd.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2009 10:35:46 -0000 Stanislav Sedov wrote: > On Mon, 06 Jul 2009 11:12:22 +0200 > Michal Hajduk mentioned: > > >> Hello Mark, >> >> While testing EHCI (FreeBSD 8-current with new USB-stack) on >> ARM machine I have found some problem probably related to >> svn commit *194459* >> http://svn.freebsd.org/viewvc/base?view=revision&revision=194459 >> With compiled USB-stack I had vm_fault: >> >> > > Hi, Michal! > > This is the known problem and we currently working on fixing it. It > looks like that r194459, while certainly good, uncovered a memory > corruption problem in ARM code. As a workaround you can revert > r194459 for now. > > Hi, Stanislav! Thank you for answering. Could you tell something more about this memory corruption problem in ARM code? Have you found which part of FreeBSD code is responsible for this, and what do you think about LORs related to r194459? Best regards, MichaÅ‚ Hajduk From owner-freebsd-arm@FreeBSD.ORG Mon Jul 6 11:06:54 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68A3E1065675 for ; Mon, 6 Jul 2009 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 526168FC22 for ; Mon, 6 Jul 2009 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n66B6s8D010695 for ; Mon, 6 Jul 2009 11:06:54 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n66B6rYE010691 for freebsd-arm@FreeBSD.org; Mon, 6 Jul 2009 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 6 Jul 2009 11:06:53 GMT Message-Id: <200907061106.n66B6rYE010691@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-arm@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2009 11:06:55 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o arm/134368 arm [patch] nslu2_led driver for the LEDs on the NSLU2 o arm/134338 arm [patch] Lock GPIO accesses on ixp425 o arm/134092 arm [patch] NSLU.hints contains wrong hints for on board n 3 problems total. From owner-freebsd-arm@FreeBSD.ORG Mon Jul 6 17:14:55 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E81F51065747 for ; Mon, 6 Jul 2009 17:14:55 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 2034D8FC12 for ; Mon, 6 Jul 2009 17:14:54 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n66HEqVX063375; Mon, 6 Jul 2009 12:14:52 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1246900492; bh=OBB4v7sptX8NaGEueuQnzc/3ILTl/JdVyxNUFF8kHJE=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=itw45m+9s70NzDR1JJDh4yhSPPimXhhEKjT3RDabvnNQFFYlMB3hxnrZsjfZfzz7h ajBWC1V1IBfjrXzHSoz3OEet4pIWfnx+TxVtfIADJbWqNhs1N1i7jePg2kNFBUwSTs VwkZqwRXxTajYsoudcs/QtuxhHmBz2TY+2bevcaQ= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n66HEqsV063372; Mon, 6 Jul 2009 12:14:52 -0500 (CDT) (envelope-from tinguely) Date: Mon, 6 Jul 2009 12:14:52 -0500 (CDT) From: Mark Tinguely Message-Id: <200907061714.n66HEqsV063372@casselton.net> To: mih@semihalf.com, stas@FreeBSD.org In-Reply-To: <20090706134612.5cf860e4.stas@FreeBSD.org> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Mon, 06 Jul 2009 12:14:52 -0500 (CDT) Cc: freebsd-arm@FreeBSD.org, tinguely@casselton.net Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2009 17:14:56 -0000 > Michal Hajduk mentioned: > > > Hello Mark, > >=20 > > While testing EHCI (FreeBSD 8-current with new USB-stack) on > > ARM machine I have found some problem probably related to > > svn commit *194459*=20 > > http://svn.freebsd.org/viewvc/base?view=3Drevision&revision=3D194459 > > With compiled USB-stack I had vm_fault: > >=20 > > Hi, Michal! > > This is the known problem and we currently working on fixing it. It > looks like that r194459, while certainly good, uncovered a memory > corruption problem in ARM code. As a workaround you can revert > r194459 for now. > > --=20 > Stanislav Sedov > ST4096-RIPE Sorry about that guys. I will walk through the pmap logic again to see if there is a problem with the patch. --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Mon Jul 6 18:36:29 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69AD41065672; Mon, 6 Jul 2009 18:36:29 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 267098FC0A; Mon, 6 Jul 2009 18:36:28 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n66IaS5p068049; Mon, 6 Jul 2009 13:36:28 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1246905388; bh=XS5k2+K/04smeUgYKX1Nr3MPtodOota1z462PF6tucw=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=iZHyVRHm4KFaJboN9hs73r4aoYoiw8pWMEbOupFdkoWo7F0KwFyHhrVvmvimVhMU3 m3D2snGjGqDRxptTPrSLJfjEl9wDuGykBKaj0ub1p7e/7bBypYusv38+T9ZvKHob/m zD80Lnfp5jY80ZFwQO0V5qINgh5YFcTpaHf1CTKs= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n66IaRSc068048; Mon, 6 Jul 2009 13:36:27 -0500 (CDT) (envelope-from tinguely) Date: Mon, 6 Jul 2009 13:36:27 -0500 (CDT) From: Mark Tinguely Message-Id: <200907061836.n66IaRSc068048@casselton.net> To: mih@semihalf.com, stas@FreeBSD.org In-Reply-To: <20090706134612.5cf860e4.stas@FreeBSD.org> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Mon, 06 Jul 2009 13:36:28 -0500 (CDT) Cc: freebsd-arm@FreeBSD.org, tinguely@casselton.net Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2009 18:36:29 -0000 One quick observation: in pmap_nuke_pv(), lower if statement cleans out the pv_list when there is no other mapping besides the kernel mapping. If I remember correctly, this was put in at the last minute. we had to remove the kernel mapping because the page went back to the one of the queues and the kernel mapping caused a panic. When the kernel mapping is removed from the other-wise empty pv_list, the PG_WRITABLE is turned off like is done when the pv_entry list is normally emptied. But at this point, we know that there still is a kernel mapping and the pmap_enter() should have enabled the PG_WRITABLE flag because kernel mappings should be writable. In my opinion, that flag should be enabled. Does it make sense that we should leave the page modified bit or will it panic on the freeing of the page? if (pv != NULL && (pv->pv_flags & PVF_UNMAN) && TAILQ_NEXT(pv, pv_list) == NULL) { pg->md.pv_kva = pv->pv_va; /* a recursive pmap_nuke_pv */ TAILQ_REMOVE(&pg->md.pv_list, pv, pv_list); TAILQ_REMOVE(&pm->pm_pvlist, pv, pv_plist); if (pv->pv_flags & PVF_WIRED) --pm->pm_stats.wired_count; pg->md.pvh_attrs &= ~PVF_REF; - pg->md.pvh_attrs &= ~PVF_MOD; - vm_page_flag_clear(pg, PG_WRITEABLE); pmap_free_pv_entry(pv); } From owner-freebsd-arm@FreeBSD.ORG Tue Jul 7 13:50:31 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E4D1106564A; Tue, 7 Jul 2009 13:50:31 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id D602C8FC24; Tue, 7 Jul 2009 13:50:29 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from [10.0.0.5] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 74FB5C4021; Tue, 7 Jul 2009 15:49:24 +0200 (CEST) From: Piotr =?iso-8859-2?q?Zi=EAcik?= Organization: Semihalf To: Hans Petter Selasky Date: Tue, 7 Jul 2009 15:50:28 +0200 User-Agent: PLD Linux KMail/1.9.10 References: <200906231035.43096.kosmo@semihalf.com> <200906301128.26046.hselasky@c2i.net> <200907051236.22783.hselasky@c2i.net> In-Reply-To: <200907051236.22783.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907071550.28781.kosmo@semihalf.com> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2009 13:50:31 -0000 > > I did not find time yet to test on my AT91RM9200 board. I hope to do a te= st > this week. Did you at semihalf find anything? > I had Checked USB behaviour on PowerPC without hardware cache coherency. The problem also exists here and patch helps. I also generated some logs which you asked for: =2D--- Original USB code ----- ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 Root mount waiting for: usbus2 usbus1 usbus0 uhub0: 1 port with 1 removable, self powered uhub1: 1 port with 1 removable, self powered uhub2: 1 port with 1 removable, self powered =3D> bus_dmamap_sync(0xC3A9E780, 0xC3B6F540, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B9E168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C40, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B9D168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9D170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9D170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9C170 (size: 0x00000012) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9C170 (size: 0x00000012) =3D> bus_dmamap_sync(0xC3A9E080, 0xC3B6F400, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B9B168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9B170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9B170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E080, 0xC3B6F400, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B9B168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9B170 (size: 0x00000009) =3D> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B9B170 (size: 0x00000009) Root mount waiting for: usbus0 =3D> bus_dmamap_sync(0xC3A9E300, 0xC3AB8D00, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B9A168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E580, 0xC3AB8E40, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B99168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E580, 0xC3B6F380, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B99170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E580, 0xC3B6F380, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B99170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E800, 0xC3B6F480, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B98168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E800, 0xC3AB8E00, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B98170 (size: 0x00000012) =3D> bus_dmamap_sync(0xC3A9E800, 0xC3AB8E00, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B98170 (size: 0x00000012) =3D> bus_dmamap_sync(0xC35DC700, 0xC3AB8B40, [PREREAD PREWRITE ________=20 _________]) -> Writeback Invalidate 0xC3B97168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC35DC700, 0xC3B6F500, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B97170 (size: 0x00000009) =3D> bus_dmamap_sync(0xC35DC700, 0xC3B6F500, [_______ ________ POSTREAD=20 POSTWRITE]) -> Invalidate 0xC3B97170 (size: 0x00000009) usb_alloc_device:1729: Failure selecting configuration index 0: USB_ERR_INV= AL,=20 port 1, addr 2 (ignored) ugen0.2: at usbus0 =2D---- USB Stack with my patch ----- ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 Root mount waiting for: usbus2 usbus1 usbus0 uhub0: 1 port with 1 removable, self powered uhub1: 1 port with 1 removable, self powered uhub2: 1 port with 1 removable, self powered =3D> bus_dmamap_sync(0xC3A9E780, 0xC3B6F540, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9E168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C40, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9D168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9D170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9D170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000012) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000012) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000004) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000004) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) Root mount waiting for: usbus0 =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000009) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000009) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000020) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000020) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) ugen0.2: at usbus0 umass0: on usbus0 umass0: SCSI over Bulk-Only; quirks =3D 0x0000 =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3B9C168 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000001) =3D> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3B9C170 (size: 0x00000001) Root mount waiting for: usbus0 umass0:0:0:-1: Attached to scbus0 =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) Trying to mount root from nfs:10.0.0.201:/nfsroot/mv78100-4/ =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB6098 (size: 0x00000024) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB6098 (size: 0x00000024) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Invalidate 0xC3BC0500 (size: 0x00000100) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Invalidate 0xC3BC0500 (size: 0x00000100) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BBC0A0 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BBC0A0 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) da0 at umass-sim0 bus 0 target 0 lun 0 da0: Removable Direct Access SCSI-0 device da0: 40.000MB/s transfers da0: 1912MB (3915776 512 byte sectors: 255H 63S/T 243C) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) NFS ROOT: 10.0.0.201:/nfsroot/mv78100-4/ =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3AB9C80 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3AB9C80 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Invalidate 0xC3B6D800 (size: 0x00000200) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Invalidate 0xC3B6D800 (size: 0x00000200) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Invalidate 0xC3C2A000 (size: 0x00002000) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Invalidate 0xC3C2A000 (size: 0x00002000) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3AB9B50 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3AB9B50 (size: 0x00000008) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________=20 _________]) -> Writeback 0xC3BB23A8 (size: 0x0000001F) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =3D> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________=20 _________]) -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D) =2D-=20 Best regards, Piotr Zi=EAcik From owner-freebsd-arm@FreeBSD.ORG Tue Jul 7 14:02:34 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00FED1065673; Tue, 7 Jul 2009 14:02:34 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id B11D38FC14; Tue, 7 Jul 2009 14:02:33 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n67E2Wmm010355; Tue, 7 Jul 2009 09:02:32 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1246975352; bh=KMg9yhtXy5UTtOhLkKJhLvthfhASD1AAm1s43XBWIko=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=HUxG/NTTddULYU/QsZiMH8S015FL02VXsuGH7EIqn3vCXAU1ydP3sy6YVbsC6Vwte 6xeGlcJVz9kIZ5Lzg9UNu8S1hgo6ucIcMM8bLmS1scDyt96Frhyvh5ifWnEpKquj0B XDy5LnwNkc+4RyNj86xnMs5lbS80CeASIeuh4Y8Y= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n67E2WgH010354; Tue, 7 Jul 2009 09:02:32 -0500 (CDT) (envelope-from tinguely) Date: Tue, 7 Jul 2009 09:02:32 -0500 (CDT) From: Mark Tinguely Message-Id: <200907071402.n67E2WgH010354@casselton.net> To: mih@semihalf.com, stas@FreeBSD.org, tinguely@casselton.net In-Reply-To: <200907061714.n66HEqsV063372@casselton.net> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Tue, 07 Jul 2009 09:02:32 -0500 (CDT) Cc: freebsd-arm@FreeBSD.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2009 14:02:34 -0000 In arm_remap_nocache()/arm_unmap_nocache() a kernel shadow mapping is made for BUS_DMA_COHERENT pages but is not removed. It is probably one place that causes undeleted mappings required us to add the tail "if" statement in pmap_nuke_pv() that I mentioned yesterday. FYI future ideas: we can get rid of the shadow KVA for the BUS_DMA_COHERENT by setting a bit in the page flags to denote a no_cache situation. The pmap code can see that bit and refraim from turning caching on in pmap_fix_cache(). Speaking of pmap_nuke_pv() and in regards to my yesterday email, the more I think of it, the more I believe the line "pg->md.pvh_attrs &= ~PVF_MOD;" should remain in the code. The line, "vm_page_flag_clear(pg, PG_WRITEABLE);" should be remove to avoid corruption. It would be nice to remove all these unremoved mappings so that "if" statement in pmap_nuke_pv() is unecessary. If we can't find all of these unremoved mappings, we may need to clean out md.pv_kva when a page is freed (or allocated) because we may be tricked into thinking the page is shared when it really is not. This would not cause corruption, but the cache will be turned off unnecessarily on a really unshared page. vm_machdep.c: void arm_unmap_nocache(void *addr, vm_size_t size) { vm_offset_t raddr = (vm_offset_t)addr; int i; size = round_page(size); i = (raddr - arm_nocache_startaddr) / (PAGE_SIZE); - for (; size > 0; size -= PAGE_SIZE, i++) + for (; size > 0; size -= PAGE_SIZE, i++) { arm_nocache_allocated[i / BITS_PER_INT] &= ~(1 << (i % BITS_PER_INT)); + pmap_kremove(raddr); + raddr += PAGE_SIZE; } --Mark Tinguely From owner-freebsd-arm@FreeBSD.ORG Tue Jul 7 14:37:33 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88737106564A; Tue, 7 Jul 2009 14:37:33 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 43C428FC13; Tue, 7 Jul 2009 14:37:32 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n67EbWPC012440; Tue, 7 Jul 2009 09:37:32 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1246977452; bh=tPyaaHTeNmrvluHoaDEjWWr++CmKibKTqqthupyZ+qQ=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=TfggofPU5shilZnFFAfMtKBly/7rR7G4BS0Pv5aOxem3M3+RmEaWaoIdOfZz2ENyO aCxv/6LSKpSUd70O3iD10R+0gJyllKrrO+NBR57+eND3LAIll3eECGVJ3uqxuVjlRm 7ITjE3pTxyY4wQqQF7ewKgz6pwyDTjL3JZ8/zF+c= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n67EbVnX012439; Tue, 7 Jul 2009 09:37:31 -0500 (CDT) (envelope-from tinguely) Date: Tue, 7 Jul 2009 09:37:31 -0500 (CDT) From: Mark Tinguely Message-Id: <200907071437.n67EbVnX012439@casselton.net> To: mih@semihalf.com, stas@FreeBSD.org, tinguely@casselton.net In-Reply-To: <200907071402.n67E2WgH010354@casselton.net> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Tue, 07 Jul 2009 09:37:32 -0500 (CDT) Cc: freebsd-arm@FreeBSD.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2009 14:37:33 -0000 sf_buf_alloc()/sf_buf_free() in vm_machdep.c also leaves dangling kernel mappings. /* * Detatch mapped page and release resources back to the system. */ void sf_buf_free(struct sf_buf *sf) { #ifndef ARM_USE_SMALL_ALLOC mtx_lock(&sf_buf_lock); sf->ref_count--; if (sf->ref_count == 0) { TAILQ_INSERT_TAIL(&sf_buf_freelist, sf, free_entry); nsfbufsused--; + pmap_kremove(sf->kva); if (sf_buf_alloc_want > 0) wakeup_one(&sf_buf_freelist); } mtx_unlock(&sf_buf_lock); #endif --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Tue Jul 7 15:54:51 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB907106567D; Tue, 7 Jul 2009 15:54:51 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 644DC8FC23; Tue, 7 Jul 2009 15:54:51 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n67Fsorh016805; Tue, 7 Jul 2009 10:54:50 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1246982090; bh=uMDDLzSkBaxfiNs0zl3Xckyvafd6ko4EOhhzxYGZiNA=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=rTgUxoh3t3UOz++kzuhVqeFJwYLQpoLZGELo14FOQ00qpz43W5POxL95Qqofa66KA 6zLNoaEPhvQUd3Op/rXYfMf0RHlM+epNrnputMnRww1ikii5L07dP12j7lI/iEFn2D HYnptEI0YDQLg5XrTaSkZnO1rwvESyP57Csi9cq4= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n67Fsouv016804; Tue, 7 Jul 2009 10:54:50 -0500 (CDT) (envelope-from tinguely) Date: Tue, 7 Jul 2009 10:54:50 -0500 (CDT) From: Mark Tinguely Message-Id: <200907071554.n67Fsouv016804@casselton.net> To: mih@semihalf.com, stas@FreeBSD.org, tinguely@casselton.net In-Reply-To: <200907071437.n67EbVnX012439@casselton.net> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Tue, 07 Jul 2009 10:54:50 -0500 (CDT) Cc: freebsd-arm@FreeBSD.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2009 15:54:52 -0000 Apologies for the noise. I am not familiar with the sf_buf_alloc/sf_buf_free code and sent a bad code sequence. If we remove the mapping, it has to come off the active list also: /* * Detatch mapped page and release resources back to the system. */ void sf_buf_free(struct sf_buf *sf) { #ifndef ARM_USE_SMALL_ALLOC mtx_lock(&sf_buf_lock); sf->ref_count--; if (sf->ref_count == 0) { TAILQ_INSERT_TAIL(&sf_buf_freelist, sf, free_entry); nsfbufsused--; + pmap_kremove(sf->kva); + sf->m = NULL; + LIST_REMOVE(sf, list_entry); if (sf_buf_alloc_want > 0) wakeup_one(&sf_buf_freelist); } mtx_unlock(&sf_buf_lock); #endif --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Tue Jul 7 16:47:07 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9F7010656AB; Tue, 7 Jul 2009 16:47:07 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe01.swip.net [212.247.154.1]) by mx1.freebsd.org (Postfix) with ESMTP id D42048FC16; Tue, 7 Jul 2009 16:47:06 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=Hrwt8fWgTlIA:10 a=MXw7gxVQKqGXY79tIT8aFQ==:17 a=6ZsRMLCPI8ooBufYVQoA:9 a=0XZanadGUU8MJNL9l0V2CwABxxkA:4 Received: from [62.113.132.61] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe01.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 219319571; Tue, 07 Jul 2009 18:47:05 +0200 From: Hans Petter Selasky To: Piotr =?iso-8859-2?q?Zi=EAcik?= Date: Tue, 7 Jul 2009 18:46:39 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-CURRENT; KDE/4.2.4; i386; ; ) References: <200906231035.43096.kosmo@semihalf.com> <200907051236.22783.hselasky@c2i.net> <200907071550.28781.kosmo@semihalf.com> In-Reply-To: <200907071550.28781.kosmo@semihalf.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907071846.42086.hselasky@c2i.net> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2009 16:47:08 -0000 On Tuesday 07 July 2009 15:50:28 Piotr Zi=EAcik wrote: > > I did not find time yet to test on my AT91RM9200 board. I hope to do a > > test this week. Did you at semihalf find anything? > Hi, > I had Checked USB behaviour on PowerPC without hardware cache coherency. > The problem also exists here and patch helps. In my source code view the busdma sync function is empty for power-pc. Your= =20 patch should have no effect at all for sync operations? > > I also generated some logs which you asked for: Thanks. I will try to make a build for AT91RM9200 tonight. Did you test my suggested patch: > Try adding: > > cpu_dcache_wb_range((vm_offset_t)buf, len); > cpu_l2cache_wb_range((vm_offset_t)buf, len); > > Before: > > cpu_dcache_wbinv_range((vm_offset_t)buf, len); > cpu_l2cache_wbinv_range((vm_offset_t)buf, len); > > In: sys/arm/arm/busdma_machdep.c Or have you got any comments on that? =2D-HPS From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 09:04:09 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31CF9106568E; Wed, 8 Jul 2009 09:04:09 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe09.tele2.se [212.247.155.1]) by mx1.freebsd.org (Postfix) with ESMTP id 65F2D8FC22; Wed, 8 Jul 2009 09:04:08 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=Hrwt8fWgTlIA:10 a=gg2W7PyvkLb8p4ie143lBA==:17 a=qqY-37dCzyb6F8oYTQwA:9 a=wQ8DsXPsDZOwMyX6hxGZY5avNgEA:4 Received: from [194.248.135.20] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe09.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 928573831; Wed, 08 Jul 2009 11:04:06 +0200 From: Hans Petter Selasky To: Rafal Jaworowski Date: Wed, 8 Jul 2009 11:03:43 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-BETA1; KDE/4.2.4; i386; ; ) References: <200906231035.43096.kosmo@semihalf.com> <200907071846.42086.hselasky@c2i.net> <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com> In-Reply-To: <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200907081103.45388.hselasky@c2i.net> Cc: thompsa@freebsd.org, freebsd-arm@freebsd.org, Piotr =?utf-8?q?Zi=C4=99cik?= , freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 09:04:10 -0000 On Wednesday 08 July 2009 10:58:17 Rafal Jaworowski wrote: > On 2009-07-07, at 18:46, Hans Petter Selasky wrote: > >> I had Checked USB behaviour on PowerPC without hardware cache > >> coherency. > >> The problem also exists here and patch helps. > > > > In my source code view the busdma sync function is empty for power- > > pc. Your > > patch should have no effect at all for sync operations? > > This was about the PPC4xx PowerPC port, not committed to the tree yet. > Unlike most PowerPC this system has a de-coherent DMA and therefore > its busdma sync is non empty, but needs to enforce coherency by > software. > > The point is this is another platform on which USB stack (usb_pc_cpu_* > functions) shows similar issues as reported for ARM and MIPS. And what about my patch suggestion in my previous e-mail having the same subject. Does it work? Regarding my testing on the AT91RM9200 I was short of time yesterday and will try to get it done today. --HPS From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 09:14:27 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A491106566C; Wed, 8 Jul 2009 09:14:27 +0000 (UTC) (envelope-from raj@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 0F3F28FC0A; Wed, 8 Jul 2009 09:14:26 +0000 (UTC) (envelope-from raj@semihalf.com) Received: from [10.0.0.34] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 5B4E9C3AAA; Wed, 8 Jul 2009 10:57:15 +0200 (CEST) Message-Id: <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com> From: Rafal Jaworowski To: Hans Petter Selasky In-Reply-To: <200907071846.42086.hselasky@c2i.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Wed, 8 Jul 2009 10:58:17 +0200 References: <200906231035.43096.kosmo@semihalf.com> <200907051236.22783.hselasky@c2i.net> <200907071550.28781.kosmo@semihalf.com> <200907071846.42086.hselasky@c2i.net> X-Mailer: Apple Mail (2.935.3) Cc: thompsa@freebsd.org, freebsd-arm@freebsd.org, =?ISO-8859-2?Q?Piotr_Zi=EAcik?= , freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 09:14:27 -0000 On 2009-07-07, at 18:46, Hans Petter Selasky wrote: >> I had Checked USB behaviour on PowerPC without hardware cache >> coherency. >> The problem also exists here and patch helps. > > In my source code view the busdma sync function is empty for power- > pc. Your > patch should have no effect at all for sync operations? This was about the PPC4xx PowerPC port, not committed to the tree yet. Unlike most PowerPC this system has a de-coherent DMA and therefore its busdma sync is non empty, but needs to enforce coherency by software. The point is this is another platform on which USB stack (usb_pc_cpu_* functions) shows similar issues as reported for ARM and MIPS. Rafal From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 10:16:42 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6118E1065673; Wed, 8 Jul 2009 10:16:42 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 065DA8FC1D; Wed, 8 Jul 2009 10:16:41 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from [10.0.0.5] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id B1BBEC4021; Wed, 8 Jul 2009 12:15:37 +0200 (CEST) From: Piotr =?iso-8859-2?q?Zi=EAcik?= Organization: Semihalf To: Hans Petter Selasky Date: Wed, 8 Jul 2009 12:16:39 +0200 User-Agent: PLD Linux KMail/1.9.10 References: <200906231035.43096.kosmo@semihalf.com> <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com> <200907081103.45388.hselasky@c2i.net> In-Reply-To: <200907081103.45388.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907081216.40100.kosmo@semihalf.com> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 10:16:43 -0000 Wednesday 08 July 2009 11:03:43 Hans Petter Selasky napisa=B3(a): > > And what about my patch suggestion in my previous e-mail having the same > subject. Does it work? > I have tested it and it does not work. By the way Writeback before Writebac= k=20 Invalidate did not change cache behaviour too much. Writeback invalidate=20 means flush all modified cache lines and then invalidate cache. However looking into logs which I have sent you yesterdat I see one differe= nce=20 which may be significant. My patch changes Invalidate into Writeback=20 Invalidate. In original code if driver write something to memory and then=20 invalidate cache, the write will be lost. With my patch after change will be written to memory and then cache will be invalidated. What do you think ? =2D-=20 Best Regards, Piotr Ziecik From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 10:40:58 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A70161065670; Wed, 8 Jul 2009 10:40:58 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from mail.embedded-brains.de (host-82-135-62-35.customer.m-online.net [82.135.62.35]) by mx1.freebsd.org (Postfix) with ESMTP id 54E6C8FC08; Wed, 8 Jul 2009 10:40:58 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: by mail.embedded-brains.de (Postfix, from userid 65534) id 8B4066F8260; Wed, 8 Jul 2009 12:30:47 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on fidibus X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.3 Received: from [192.168.96.31] (eb0011.eb.z [192.168.96.31]) by mail.embedded-brains.de (Postfix) with ESMTP id 5C83E6F825E; Wed, 8 Jul 2009 12:30:46 +0200 (CEST) Message-ID: <4A547556.4050801@embedded-brains.de> Date: Wed, 08 Jul 2009 12:30:46 +0200 From: Sebastian Huber User-Agent: Thunderbird 2.0.0.19 (X11/20081227) MIME-Version: 1.0 To: =?ISO-8859-2?Q?Piotr_Zi=EAcik?= References: <200906231035.43096.kosmo@semihalf.com> <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com> <200907081103.45388.hselasky@c2i.net> <200907081216.40100.kosmo@semihalf.com> In-Reply-To: <200907081216.40100.kosmo@semihalf.com> Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 8bit Cc: freebsd-arm@freebsd.org, freebsd-usb@freebsd.org, thompsa@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 10:40:58 -0000 Hi! The general approach for systems with no hardware cache coherence protocol should be: Memory to device DMA transfer of a buffer: 1. write all modified cache lines of the buffer back to memory (cache flush for buffer) 2. start DMA transfer Device to memory DMA transfer of a buffer: 1. invalidate the cache of the buffer, here you have to make sure that this buffer does not share a cache line with other buffers (!) 2. start DMA transfer 3. all accesses to the buffer via the cache must wait until the DMA has finished CU -- Sebastian Huber, Embedded Brains GmbH Address : Obere Lagerstr. 30, D-82178 Puchheim, Germany Phone : +49 89 18 90 80 79-6 Fax : +49 89 18 90 80 79-9 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG. From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 10:51:22 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CBB51065674; Wed, 8 Jul 2009 10:51:22 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe09.swipnet.se [212.247.155.1]) by mx1.freebsd.org (Postfix) with ESMTP id 3AFF98FC13; Wed, 8 Jul 2009 10:51:20 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=Hrwt8fWgTlIA:10 a=gg2W7PyvkLb8p4ie143lBA==:17 a=ge6c7HbqCLY6kjjTAcgA:9 a=-Zd0oq5pNAkYy4vOxvgA:7 a=wsCSXzGZHIBTtm1dGkqSUQLDJlkA:4 Received: from [194.248.135.20] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe09.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 928647965; Wed, 08 Jul 2009 12:51:19 +0200 From: Hans Petter Selasky To: Piotr =?iso-8859-2?q?Zi=EAcik?= Date: Wed, 8 Jul 2009 12:50:56 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-BETA1; KDE/4.2.4; i386; ; ) References: <200906231035.43096.kosmo@semihalf.com> <200907081103.45388.hselasky@c2i.net> <200907081216.40100.kosmo@semihalf.com> In-Reply-To: <200907081216.40100.kosmo@semihalf.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907081250.58151.hselasky@c2i.net> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 10:51:22 -0000 Hi, On Wednesday 08 July 2009 12:16:39 Piotr Zi=EAcik wrote: > Wednesday 08 July 2009 11:03:43 Hans Petter Selasky napisa=B3(a): > > And what about my patch suggestion in my previous e-mail having the same > > subject. Does it work? > > I have tested it and it does not work. By the way Writeback before > Writeback Invalidate did not change cache behaviour too much. Writeback > invalidate means flush all modified cache lines and then invalidate cache. By flush you mean write from CPU cache to RAM, right. And nothing else? You= =20 don't mean discard CPU cache by "flush" ??? > > However looking into logs which I have sent you yesterdat I see one > difference which may be significant. My patch changes Invalidate into > Writeback Invalidate. In original code if driver write something to memory > and then invalidate cache If that is the case I'm very surprised. Could you make another printout for= =20 me. Compile the kernel with KDB, and add a call to "kdb_backtrace()" in=20 bus_dmamap_sync() before printing out the writeback and/or invalidate, so t= hat=20 I can see the backtrace. > , the write will be lost. I'm aware about that. > With my patch after > change will be written to memory and then cache will be invalidated. What > do you think ? I still cannot see that I'm using the wrong flags when calling=20 bus_dmamap_sync(). You are absoultely sure that your USB code is indentical to the now stock 8- current USB code, and that there are no integration issues lurking? In the USB "usb_pc_cpu_flush()" is used after CPU write operations to flush= =20 the CPU cache to RAM. "usb_pc_cpu_invalidate()" is used before CPU read=20 operations. So far so good? You are arguing that: BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD flags do not cause a CPU invalidate? Whilst the busdma manual is clear, that these flags are used after "an upda= te=20 of host memory by the device and prior to CPU access to host memory". BUS_DMASYNC_POSTREAD Perform any synchronization required af= ter an update of host memory by the device = and prior to CPU access to host memory. BUS_DMASYNC_POSTWRITE Perform any synchronization required af= ter device access to host memory. And that: BUS_DMASYNC_PREREAD Perform any synchronization required pr= ior to an update of host memory by the devi= ce. BUS_DMASYNC_PREWRITE Perform any synchronization required af= ter an update of host memory by the CPU and prior to device access to host memory. flags do not cause a CPU flush prior to device acccess. /*------------------------------------------------------------------------* * usb_pc_cpu_invalidate - invalidate CPU cache *------------------------------------------------------------------------*/ void usb_pc_cpu_invalidate(struct usb_page_cache *pc) { if (pc->page_offset_end =3D=3D pc->page_offset_buf) { /* nothing has been loaded into this page cache! */ return; } bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD); } /*------------------------------------------------------------------------* * usb_pc_cpu_flush - flush CPU cache *------------------------------------------------------------------------*/ void usb_pc_cpu_flush(struct usb_page_cache *pc) { if (pc->page_offset_end =3D=3D pc->page_offset_buf) { /* nothing has been loaded into this page cache! */ return; } bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE | BUS_DMASYNC_PREREAD); } Thanks for being patient. If there is a bug, it will be fixed. =2D-HPS From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 14:01:01 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40528106564A; Wed, 8 Jul 2009 14:01:01 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 400E68FC19; Wed, 8 Jul 2009 14:01:00 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from [10.0.0.43] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 75989C3AAA; Wed, 8 Jul 2009 15:59:55 +0200 (CEST) Message-ID: <4A54A6B2.1020400@semihalf.com> Date: Wed, 08 Jul 2009 16:01:22 +0200 From: Michal Hajduk User-Agent: Mozilla-Thunderbird 2.0.0.19 (X11/20090103) MIME-Version: 1.0 To: Mark Tinguely References: <200907071554.n67Fsouv016804@casselton.net> In-Reply-To: <200907071554.n67Fsouv016804@casselton.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: stas@FreeBSD.org, freebsd-arm@FreeBSD.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 14:01:01 -0000 Mark Tinguely wrote: > Apologies for the noise. I am not familiar with the sf_buf_alloc/sf_buf_free > code and sent a bad code sequence. If we remove the mapping, it has to > come off the active list also: > > /* > * Detatch mapped page and release resources back to the system. > */ > void > sf_buf_free(struct sf_buf *sf) > { > #ifndef ARM_USE_SMALL_ALLOC > mtx_lock(&sf_buf_lock); > sf->ref_count--; > if (sf->ref_count == 0) { > TAILQ_INSERT_TAIL(&sf_buf_freelist, sf, free_entry); > nsfbufsused--; > + pmap_kremove(sf->kva); > + sf->m = NULL; > + LIST_REMOVE(sf, list_entry); > if (sf_buf_alloc_want > 0) > wakeup_one(&sf_buf_freelist); > } > mtx_unlock(&sf_buf_lock); > #endif > > --Mark Tinguely. > > Hello Mark, Thank you for looking again on r194459. I've tried to compile kernel using your patch with changes regarding PG_WRITABLE and arm_unmap_nocache(), but it didn't help. My problems are rather related to memory corruption, which I've observed in uma_core (pointers had bad values and count of free items in slab had negative value). I am still concerned about LOR, which has appeared in svn commit r194459 (I wrote about it in my first mail). Could you take a look at this LOR? Thanks for your help. Best regards, MichaÅ‚ Hajduk From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 14:29:25 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC6881065676; Wed, 8 Jul 2009 14:29:25 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id A855F8FC15; Wed, 8 Jul 2009 14:29:25 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.3/8.14.1) with ESMTP id n68EN5GH053804; Wed, 8 Jul 2009 08:23:06 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 08 Jul 2009 08:23:43 -0600 (MDT) Message-Id: <20090708.082343.-1343671115.imp@bsdimp.com> To: hselasky@c2i.net From: "M. Warner Losh" In-Reply-To: <200907081103.45388.hselasky@c2i.net> References: <200907071846.42086.hselasky@c2i.net> <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com> <200907081103.45388.hselasky@c2i.net> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-usb@freebsd.org, freebsd-arm@freebsd.org, kosmo@semihalf.com, thompsa@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 14:29:26 -0000 In message: <200907081103.45388.hselasky@c2i.net> Hans Petter Selasky writes: : On Wednesday 08 July 2009 10:58:17 Rafal Jaworowski wrote: : > On 2009-07-07, at 18:46, Hans Petter Selasky wrote: : > >> I had Checked USB behaviour on PowerPC without hardware cache : > >> coherency. : > >> The problem also exists here and patch helps. : > > : > > In my source code view the busdma sync function is empty for power- : > > pc. Your : > > patch should have no effect at all for sync operations? : > : > This was about the PPC4xx PowerPC port, not committed to the tree yet. : > Unlike most PowerPC this system has a de-coherent DMA and therefore : > its busdma sync is non empty, but needs to enforce coherency by : > software. : > : > The point is this is another platform on which USB stack (usb_pc_cpu_* : > functions) shows similar issues as reported for ARM and MIPS. : : And what about my patch suggestion in my previous e-mail having the same : subject. Does it work? : : Regarding my testing on the AT91RM9200 I was short of time yesterday and will : try to get it done today. I think that the root cause of this issue is two fold. First, The 4 operations for busdma are being collapsed to only 2 operations. There are four for a reason (since you have to do different things for each case). Second, and I think this is more important, I think that we're seeing some cache-line poisoning. We're flushing the entire busdma tag rather than just one cache line since we don't have the API to do that. In addition, I don't think that the USB code is being careful enough to ensure that we don't have buffers that live in the same cache line that are simultaneously being used for read and write. The corruption we're seeing is a classic signal that this may be going on, although I must admit that I've not done the detailed analysis to prove it. Warner From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 15:07:32 2009 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92097106567D; Wed, 8 Jul 2009 15:07:32 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 46AAE8FC1A; Wed, 8 Jul 2009 15:07:31 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n68F7VFn070525; Wed, 8 Jul 2009 10:07:31 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1247065651; bh=OYJ2Lcl8qznNEF0P62QsFg/AaEF2g4d54m6T/epMI00=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=MLt+DNs5NdBaswfRCrbvuElvvDk0icwM6D/hWBlBjP7LeBC4ZMZmST7Y+rKmF18jB kl0gfdyPAM0XhAXK+TR233sY+WvEdBgUP44Ca/5FIchTKI18wZhAgW4szkkKE+g1iO ofkAm+CNfz/rz7rgizFd5VwJtbGzzJy1VEyaaYIg= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n68F7Vsu070524; Wed, 8 Jul 2009 10:07:31 -0500 (CDT) (envelope-from tinguely) Date: Wed, 8 Jul 2009 10:07:31 -0500 (CDT) From: Mark Tinguely Message-Id: <200907081507.n68F7Vsu070524@casselton.net> To: mih@semihalf.com, tinguely@casselton.net In-Reply-To: <4A54A6B2.1020400@semihalf.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Wed, 08 Jul 2009 10:07:31 -0500 (CDT) Cc: stas@FreeBSD.org, freebsd-arm@FreeBSD.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 15:07:33 -0000 > Hello Mark, > Thank you for looking again on r194459. > > I've tried to compile kernel using your patch with > changes regarding PG_WRITABLE and arm_unmap_nocache(), > but it didn't help. > > My problems are rather related to memory corruption, which > I've observed in uma_core (pointers had bad values and > count of free items in slab had negative value). > > I am still concerned about LOR, which has appeared in > svn commit r194459 (I wrote about it in my first mail). > Could you take a look at this LOR? > > Thanks for your help. > > Best regards, > MichaÅ‚ Hajduk I don't think this is a corruption issue but an allocation of a l2_bucket issue. The kernel map is locked to do the kenter. A new bucket is needed, UMA is called and the new page is kenter into the map. In pmap_enter_pv() before we allocate a pv_entry, we turn off the kmap lock. I am not sure if the kmap lock should be turned off in the pmap_alloc_l2_bucket(), or the caller, but here is a quick manually made kmap lock disable patch: I still think the last "vm_page_flag_clear(pg, PG_WRITEABLE);" needs to be removed from pmap_nuke_pv(). All dangling kenters need to be kremoved ... ------------------------ /* * Returns a pointer to the L2 bucket associated with the specified pmap * and VA. * * If no L2 bucket exists, perform the necessary allocations to put an L2 * bucket/page table in place. * * Note that if a new L2 bucket/page was allocated, the caller *must* * increment the bucket occupancy counter appropriately *before* * releasing the pmap's lock to ensure no other thread or cpu deallocates * the bucket/page in the meantime. */ static struct l2_bucket * pmap_alloc_l2_bucket(pmap_t pm, vm_offset_t va) { struct l2_dtable *l2; struct l2_bucket *l2b; u_short l1idx; + int km = 0; l1idx = L1_IDX(va); PMAP_ASSERT_LOCKED(pm); + if ((pm != pmap_kernel()) && PMAP_OWNED(pmap_kernel()) { + km = 1; + PMAP_UNLOCK(pmap_kernel()); + } mtx_assert(&vm_page_queue_mtx, MA_OWNED); if ((l2 = pm->pm_l2[L2_IDX(l1idx)]) == NULL) { /* * No mapping at this address, as there is * no entry in the L1 table. * Need to allocate a new l2_dtable. */ again_l2table: PMAP_UNLOCK(pm); vm_page_unlock_queues(); if ((l2 = pmap_alloc_l2_dtable()) == NULL) { vm_page_lock_queues(); PMAP_LOCK(pm); + if (km) + PMAP_LOCK(pmap_kernel()); return (NULL); } vm_page_lock_queues(); PMAP_LOCK(pm); if (pm->pm_l2[L2_IDX(l1idx)] != NULL) { PMAP_UNLOCK(pm); vm_page_unlock_queues(); uma_zfree(l2table_zone, l2); vm_page_lock_queues(); PMAP_LOCK(pm); l2 = pm->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL) goto again_l2table; /* * Someone already allocated the l2_dtable while * we were doing the same. */ } else { bzero(l2, sizeof(*l2)); /* * Link it into the parent pmap */ pm->pm_l2[L2_IDX(l1idx)] = l2; } } l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; /* * Fetch pointer to the L2 page table associated with the address. */ if (l2b->l2b_kva == NULL) { pt_entry_t *ptep; /* * No L2 page table has been allocated. Chances are, this * is because we just allocated the l2_dtable, above. */ again_ptep: PMAP_UNLOCK(pm); vm_page_unlock_queues(); ptep = (void*)uma_zalloc(l2zone, M_NOWAIT|M_USE_RESERVE); vm_page_lock_queues(); PMAP_LOCK(pm); if (l2b->l2b_kva != 0) { /* We lost the race. */ PMAP_UNLOCK(pm); vm_page_unlock_queues(); uma_zfree(l2zone, ptep); vm_page_lock_queues(); PMAP_LOCK(pm); if (l2b->l2b_kva == 0) goto again_ptep; return (l2b); } l2b->l2b_phys = vtophys(ptep); if (ptep == NULL) { /* * Oops, no more L2 page tables available at this * time. We may need to deallocate the l2_dtable * if we allocated a new one above. */ if (l2->l2_occupancy == 0) { pm->pm_l2[L2_IDX(l1idx)] = NULL; pmap_free_l2_dtable(l2); } + if (km) + PMAP_LOCK(pmap_kernel()); return (NULL); } l2->l2_occupancy++; l2b->l2b_kva = ptep; l2b->l2b_l1idx = l1idx; } + if (km) + PMAP_LOCK(pmap_kernel()); return (l2b); } From owner-freebsd-arm@FreeBSD.ORG Wed Jul 8 15:51:17 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81E41106564A; Wed, 8 Jul 2009 15:51:17 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1E08E8FC19; Wed, 8 Jul 2009 15:51:16 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n68FpGCJ073178; Wed, 8 Jul 2009 10:51:16 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1247068276; bh=0bWtS0gVoGm+2SZNa3UUZuyVy2M/yvaa4rsJVO0YkRA=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=GTOxC5j2l2/Fsn2lHnnze16eQXpFP9d2oXekvCbTHZAvf/uoIj6sar61/84YH1Rr+ SuFVWIFkPmy2w4djEu70VRD92gt6gThx81aVUWtZjL1IsR5jSh25Zn4/K762KocLd+ L/KtjtWb42zrcue16PGkIEIXEWM3co2h8eGl0Vos= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n68FpFeM073177; Wed, 8 Jul 2009 10:51:16 -0500 (CDT) (envelope-from tinguely) Date: Wed, 8 Jul 2009 10:51:16 -0500 (CDT) From: Mark Tinguely Message-Id: <200907081551.n68FpFeM073177@casselton.net> To: mih@semihalf.com, tinguely@casselton.net In-Reply-To: <200907081507.n68F7Vsu070524@casselton.net> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Wed, 08 Jul 2009 10:51:16 -0500 (CDT) Cc: freebsd-arm@freebsd.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2009 15:51:17 -0000 I forgot to CC the rest. pmap_get_pv_entry() is called from pmap_enter_locked and can have the same UMA call that can happen in pmap_kenter_internal. move the kmap lock protection from pmap_enter_pv() to pmap_get_pv_entry() static void pmap_enter_pv(struct vm_page *pg, struct pv_entry *pve, pmap_t pm, vm_offset_t va, u_int flags) { - int km; mtx_assert(&vm_page_queue_mtx, MA_OWNED); if (pg->md.pv_kva) { /* PMAP_ASSERT_LOCKED(pmap_kernel()); */ pve->pv_pmap = pmap_kernel(); pve->pv_va = pg->md.pv_kva; pve->pv_flags = PVF_WRITE | PVF_UNMAN; pg->md.pv_kva = 0; TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); TAILQ_INSERT_HEAD(&pm->pm_pvlist, pve, pv_plist); - if ((km = PMAP_OWNED(pmap_kernel()))) - PMAP_UNLOCK(pmap_kernel()); vm_page_unlock_queues(); if ((pve = pmap_get_pv_entry()) == NULL) panic("pmap_kenter_internal: no pv entries"); vm_page_lock_queues(); - if (km) - PMAP_LOCK(pmap_kernel()); } PMAP_ASSERT_LOCKED(pm); pve->pv_pmap = pm; pve->pv_va = va; pve->pv_flags = flags; TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); TAILQ_INSERT_HEAD(&pm->pm_pvlist, pve, pv_plist); pg->md.pvh_attrs |= flags & (PVF_REF | PVF_MOD); if (pve->pv_flags & PVF_WIRED) ++pm->pm_stats.wired_count; vm_page_flag_set(pg, PG_REFERENCED); } static void pmap_free_pv_entry(pv_entry_t pv) { + int km; pv_entry_count--; + if ((km = PMAP_OWNED(pmap_kernel()))) + PMAP_UNLOCK(pmap_kernel()); uma_zfree(pvzone, pv); + if (km) + PMAP_LOCK(pmap_kernel()); } /* * get a new pv_entry, allocating a block from the system * when needed. * the memory allocation is performed bypassing the malloc code * because of the possibility of allocations at interrupt time. */ static pv_entry_t pmap_get_pv_entry(void) { pv_entry_t ret_value; + int km; pv_entry_count++; + if ((km = PMAP_OWNED(pmap_kernel()))) + PMAP_UNLOCK(pmap_kernel()); if (pv_entry_count > pv_entry_high_water) pagedaemon_wakeup(); ret_value = uma_zalloc(pvzone, M_NOWAIT); + if (km) + PMAP_LOCK(pmap_kernel()); return ret_value; } From owner-freebsd-arm@FreeBSD.ORG Thu Jul 9 05:55:37 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF6791065670 for ; Thu, 9 Jul 2009 05:55:36 +0000 (UTC) (envelope-from vassilis.laganakos@yahoo.com) Received: from n7a.bullet.mail.ac4.yahoo.com (n7a.bullet.mail.ac4.yahoo.com [76.13.13.70]) by mx1.freebsd.org (Postfix) with SMTP id A89468FC1F for ; Thu, 9 Jul 2009 05:55:36 +0000 (UTC) (envelope-from vassilis.laganakos@yahoo.com) Received: from [76.13.13.25] by n7.bullet.mail.ac4.yahoo.com with NNFMP; 08 Jul 2009 22:46:33 -0000 Received: from [76.13.10.176] by t4.bullet.mail.ac4.yahoo.com with NNFMP; 08 Jul 2009 22:46:33 -0000 Received: from [127.0.0.1] by omp117.mail.ac4.yahoo.com with NNFMP; 08 Jul 2009 22:46:33 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 843422.25246.bm@omp117.mail.ac4.yahoo.com Received: (qmail 87472 invoked by uid 60001); 8 Jul 2009 22:46:33 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1247093193; bh=86+uUragVqUpK8kb+TGByhrDUdn0YZGCyT+ZaQqr8AA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=tOMXlxmpb44+jYUmDxmrYm7vHkiHFPi1YRZAB00E9Z0rjbg+o3xTDjQ0QrrY3CZWfi7vQVuy1UxNait7x4nMCUVucnxgVFLAyi3/li6XRcaRPBC9FmsUYgF+fglOcs0U/oc9K6rEZioIg/KkcV43jQbFWqUUJUIyN8TOfG5Rabo= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=drv9P9F8RnjsDHZ2Aco3hmmPVoMw3bJ77ej2QvesKBAZzo6ydCuMoz21mSnvy/bt61xlsgih5p/Fiqpiqn7mJoyedg+4lTtPT5a/jZQf6t3RS5eOJYD+48CA0eFNEDJ+d78z/nt++TAcbo0Ej9mhC1Snb0leSvPp9dCOWkCJAhI=; Message-ID: <462282.87449.qm@web59413.mail.ac4.yahoo.com> X-YMail-OSG: RvAUHpQVM1lGpXdkBT5Vvimz.EVuayQqyb9jhEF5IK2lNExjQ6l8hJ3lj_UNSPixPzzaVxFj4cslD1wHkH7omLcdoCQmmSLm.45N1CiDmEIx7h3heGQHUy.7hkFlLSxZOuFe3R19V5WCXf1JXyfZBhwNHuZ8O7gTrFwS2utxiDAkaSrwWKABcABKzMEPNnNJvf_wHyMrgR7X2Ch.SVYimw.nPNpRpJSpOejPlny_wWO2I7AaQ8N5 Received: from [82.6.108.148] by web59413.mail.ac4.yahoo.com via HTTP; Wed, 08 Jul 2009 15:46:33 PDT X-Mailer: YahooMailRC/1358.21 YahooMailWebService/0.7.289.15 References: <200906180305.n5I35qci045136@casselton.net> Date: Wed, 8 Jul 2009 15:46:33 -0700 (PDT) From: Vassilis Laganakos To: Mark Tinguely In-Reply-To: <200906180305.n5I35qci045136@casselton.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-arm@freebsd.org Subject: Re: ARMv7 - EABI - Cross Compiler X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2009 05:55:37 -0000 Hi Mark, Sorry for the delayed reply, I was in transitional period, moving things around. > > I don't know what are the changes from previous versions of GCC with the trunk. > > Maybe the release notes have more info on this.... > > FreeBSD compiler has a variable defined into it that the kernel sources > expect to find, it has extended format commands, and it modifies > gcc/gcc.c to look for the support programs in normal places. > Oh ok, I assume the info is in the handbook (or in the sources :) ) > > > I made changes to the gcc-4.5-20090528 directory and the build directory. > > > I will look at the libraries too. > > > > > Cool! > > I am still using modified GNU make file. Static libraries can cross > compile an application to ARM but I gave up on configure/gmake on > libmudflap. My library compile process became a hacking in the configure > files. I was just walking around the problems as they popped up. Had > I read and understood all the configure options, maybe it would have > gone smoother. The "-Werror" complaints and "xgcc -E" as a pre-processor > command are the two biggest problem in the library compilation process. > libmudflap gave me trouble too. Went through quite a few web pages to get my head around, still I'm not sure what to do to config things the way I want... I guess the -Werror would be kind of expected in such transitions. > Once we have a cross compiler that can make a kernel, it seems a waste > of time to keep going with the configure/gmake route. Your idea of > moving the sources over to the head/contrib/gcc and editing the BSD > make files in head/gnu/usr.bin/cc is the way to go because making > the compiler from sources is part of the cross build world. > Thanks, let's hope it works, and it is not crazy difficult to do... > I diff-ed the sources from the GNU original (4.2.1 if memory serves me > right) sources and filled in the missing pieces into the GCC 4.5 files. > Some things have already been put into GCC 4.5, some areas seemed re-written. > I was mostly interested in getting the FreeBSD extensions in. I can give you > diff of my sources against GCC 4.5 original code or tar up both the source > and build directory. The changes would be simular for GCC 4.4. > That would be very helpful! Is the number of missing pieces huge? > There is nothing special about GCC 4.5, I just took the most recent source > that is available. Also, I just blindly copied the freebsd* header files > from head source into GCC 4.5. The real way of doing this is to edit up > the existing GCC 4.5 header files and add the new items. I did that with > a couple that are used in the ARM build. > True, I think we have to merge the current header files with the new header files to have the new items in. Also, some things in the 4.4 header arm files seem to be doing some things in a way different than the linux-gnueabi way that I'm more familiar with. Trying to understand :) (once I have my stuff working again I will *finally* put the info in a wiki...) > The sources in the gcc support library files are not changed; it would > just take some BSD make file editing to get them to compile under the > head directory. > Ok. > --Mark. Once I get my hands back on this, I'll post here the progress and keep you informed. Many thanks, Vassilis L. From owner-freebsd-arm@FreeBSD.ORG Thu Jul 9 15:21:02 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E4B81065674; Thu, 9 Jul 2009 15:21:02 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 40FC88FC1A; Thu, 9 Jul 2009 15:21:02 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from [10.0.0.5] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 92FACC3AAA; Thu, 9 Jul 2009 17:20:00 +0200 (CEST) From: Piotr =?iso-8859-2?q?Zi=EAcik?= Organization: Semihalf To: Hans Petter Selasky Date: Thu, 9 Jul 2009 17:21:00 +0200 User-Agent: PLD Linux KMail/1.9.10 References: <200906231035.43096.kosmo@semihalf.com> <200907081216.40100.kosmo@semihalf.com> <200907081250.58151.hselasky@c2i.net> In-Reply-To: <200907081250.58151.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907091721.01272.kosmo@semihalf.com> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2009 15:21:03 -0000 Wednesday 08 July 2009 12:50:56 Hans Petter Selasky napisa=B3(a): > By flush you mean write from CPU cache to RAM, right. And nothing else? Y= ou > don't mean discard CPU cache by "flush" ??? Yes. By flush I mean write all valid and modified cache lines to RAM. > > However looking into logs which I have sent you yesterdat I see one > > difference which may be significant. My patch changes Invalidate into > > Writeback Invalidate. In original code if driver write something to > > memory and then invalidate cache > > If that is the case I'm very surprised. Could you make another printout f= or > me. Compile the kernel with KDB, and add a call to "kdb_backtrace()" in > bus_dmamap_sync() before printing out the writeback and/or invalidate, so > that I can see the backtrace. > > > , the write will be lost. > > I'm aware about that. =46ull log with backtraces is here:=20 http://people.freebsd.org/~raj/logs/usb-cache.log > > With my patch after > > change will be written to memory and then cache will be invalidated. Wh= at > > do you think ? > > You are absoultely sure that your USB code is indentical to the now stock > 8- current USB code, and that there are no integration issues lurking? > I am using current from 2009.06.15. In later versions is bug in ARM pmap causing segmentation fault in ehci atatch code (for more details loot at http://www.nabble.com/pmap-problem-in-FreeBSD-current-td24352351.html). =2D-=20 Pozdrawiam. Piotr Zi=EAcik From owner-freebsd-arm@FreeBSD.ORG Thu Jul 9 15:23:14 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36F9F106566C; Thu, 9 Jul 2009 15:23:14 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id E0BA48FC1D; Thu, 9 Jul 2009 15:23:13 +0000 (UTC) (envelope-from kosmo@semihalf.com) Received: from [10.0.0.5] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 27497C3AAA; Thu, 9 Jul 2009 17:22:12 +0200 (CEST) From: Piotr =?iso-8859-2?q?Zi=EAcik?= Organization: Semihalf To: Sebastian Huber Date: Thu, 9 Jul 2009 17:23:12 +0200 User-Agent: PLD Linux KMail/1.9.10 References: <200906231035.43096.kosmo@semihalf.com> <200907081216.40100.kosmo@semihalf.com> <4A547556.4050801@embedded-brains.de> In-Reply-To: <4A547556.4050801@embedded-brains.de> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907091723.13121.kosmo@semihalf.com> Cc: freebsd-arm@freebsd.org, freebsd-usb@freebsd.org, thompsa@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2009 15:23:14 -0000 Wednesday 08 July 2009 12:30:46 Sebastian Huber napisa=B3(a): > Device to memory DMA transfer of a buffer: > (...) > 3. all accesses to the buffer via the cache must wait until the DMA has > finished And this is the problem in my oppinion - USB stack does not wait. =2D-=20 Best Regards. Piotr Zi=EAcik From owner-freebsd-arm@FreeBSD.ORG Thu Jul 9 16:35:07 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0E121065676; Thu, 9 Jul 2009 16:35:07 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe02.swip.net [212.247.154.33]) by mx1.freebsd.org (Postfix) with ESMTP id A70B58FC19; Thu, 9 Jul 2009 16:35:06 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=Hrwt8fWgTlIA:10 a=MXw7gxVQKqGXY79tIT8aFQ==:17 a=6I5d2MoRAAAA:8 a=9I5xiGouAAAA:8 a=3Fu2VmpGvCISSvz8iQMA:9 a=XGy9aEl_9vEWPm5gYIwA:7 a=LQzljaXy9EPa0d3N-Vp1QT03x7AA:4 Received: from [62.113.132.61] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe02.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 1283422939; Thu, 09 Jul 2009 18:35:04 +0200 From: Hans Petter Selasky To: Piotr =?iso-8859-2?q?Zi=EAcik?= Date: Thu, 9 Jul 2009 18:34:40 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-BETA1; KDE/4.2.4; i386; ; ) References: <200906231035.43096.kosmo@semihalf.com> <200907081250.58151.hselasky@c2i.net> <200907091721.01272.kosmo@semihalf.com> In-Reply-To: <200907091721.01272.kosmo@semihalf.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907091834.42462.hselasky@c2i.net> Cc: freebsd-arm@freebsd.org, thompsa@freebsd.org, freebsd-usb@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2009 16:35:08 -0000 Hi Piotr, On Thursday 09 July 2009 17:21:00 Piotr Zi=EAcik wrote: > Wednesday 08 July 2009 12:50:56 Hans Petter Selasky napisa=B3(a): > > By flush you mean write from CPU cache to RAM, right. And nothing else? > > You don't mean discard CPU cache by "flush" ??? > > Yes. By flush I mean write all valid and modified cache lines to RAM. > > > > However looking into logs which I have sent you yesterdat I see one > > > difference which may be significant. My patch changes Invalidate into > > > Writeback Invalidate. In original code if driver write something to > > > memory and then invalidate cache > > > > If that is the case I'm very surprised. Could you make another printout > > for me. Compile the kernel with KDB, and add a call to "kdb_backtrace()" > > in bus_dmamap_sync() before printing out the writeback and/or invalidat= e, > > so that I can see the backtrace. > > > > > , the write will be lost. > > > > I'm aware about that. > > Full log with backtraces is here: > http://people.freebsd.org/~raj/logs/usb-cache.log 1) My analysis: Only the data areas are being flushed/invalidated. No trans= fer=20 descriptors are flushed/invalidated. I see no cache operations happening on= =20 any DMA control structures, even though there are calls from EHCI to=20 xxx_pc_flush() and xxx_pc_invalidate(). One patch you can try is to add an additional unload call to=20 "usbd_transfer_setup_sub_malloc()": =3D=3D=3D=3D //depot/projects/usb/src/sys/dev/usb/usb_transfer.c#159 -=20 /home/hans_other/usb.p4/src/sys/dev/usb/usb_transfer.c =3D=3D=3D=3D @@ -268,6 +268,10 @@ pg, z, align)) { return (1); /* failure */ } + + bus_dmamap_unload(parm->dma_page_cache_ptr->tag, + parm->dma_page_cache_ptr->map); + /* Set beginning of current buffer */ buf =3D parm->dma_page_cache_ptr->buffer; /* Make room for one DMA page cache and one page */ This will avoid the same memory area being loaded twice. Not sure if there = is=20 still a bug in pmap about this! 2) You can also try to remove the "BUS_DMA_COHERENT" flag from "usb_busdma.= c". My time is limited today, because I have to work on libusb v1.0 support. Could you provide a new trace, showing an enumeration failure. In your=20 previous trace there was no error, because the printouts probably caused to= =20 CPU to flush out its cache. Debug details: /boot/loader.conf hw.usb.debug=3D15 hw.usb.ehci.debug=3D15 hw.usb.ohci.debug=3D15 hw.usb.uhci.debug=3D15 > I am using current from 2009.06.15. In later versions is bug in ARM pmap > causing segmentation fault in ehci atatch code (for more details loot at > http://www.nabble.com/pmap-problem-in-FreeBSD-current-td24352351.html). Ok. I have a feeling we will soon figure out the problem. =2D-HPS From owner-freebsd-arm@FreeBSD.ORG Thu Jul 9 16:46:03 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A06DE1065672; Thu, 9 Jul 2009 16:46:03 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe10.tele2.se [212.247.155.33]) by mx1.freebsd.org (Postfix) with ESMTP id D09278FC0A; Thu, 9 Jul 2009 16:46:02 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=Hrwt8fWgTlIA:10 a=MXw7gxVQKqGXY79tIT8aFQ==:17 a=6I5d2MoRAAAA:8 a=CB5tFlUBj5lX0kOYFHwA:9 a=lPvmI1w4oMshsF84DFMlSaN-tz0A:4 Received: from [62.113.132.61] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe10.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 1105609322; Thu, 09 Jul 2009 18:46:01 +0200 From: Hans Petter Selasky To: freebsd-usb@freebsd.org Date: Thu, 9 Jul 2009 18:45:39 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-BETA1; KDE/4.2.4; i386; ; ) References: <200906231035.43096.kosmo@semihalf.com> <200907091721.01272.kosmo@semihalf.com> <200907091834.42462.hselasky@c2i.net> In-Reply-To: <200907091834.42462.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-2" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200907091845.40515.hselasky@c2i.net> Cc: freebsd-arm@freebsd.org, Piotr =?iso-8859-2?q?Zi=EAcik?= , thompsa@freebsd.org Subject: Re: CPU Cache and busdma usage in USB X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2009 16:46:04 -0000 On Thursday 09 July 2009 18:34:40 Hans Petter Selasky wrote: > > Full log with backtraces is here: > > http://people.freebsd.org/~raj/logs/usb-cache.log > > 1) My analysis: Only the data areas are being flushed/invalidated. I also see that the flushing/invalidating is being done correctly. --HPS From owner-freebsd-arm@FreeBSD.ORG Fri Jul 10 08:27:07 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42B741065672; Fri, 10 Jul 2009 08:27:07 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id EF5AD8FC0A; Fri, 10 Jul 2009 08:27:06 +0000 (UTC) (envelope-from mih@semihalf.com) Received: from [10.0.0.43] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPSA id 4BC4FC3BB0; Fri, 10 Jul 2009 10:23:54 +0200 (CEST) Message-ID: <4A56FB72.9040807@semihalf.com> Date: Fri, 10 Jul 2009 10:27:30 +0200 From: Michal Hajduk User-Agent: Mozilla-Thunderbird 2.0.0.19 (X11/20090103) MIME-Version: 1.0 To: Mark Tinguely References: <200907081551.n68FpFeM073177@casselton.net> In-Reply-To: <200907081551.n68FpFeM073177@casselton.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-arm@freebsd.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jul 2009 08:27:07 -0000 Hi Mark, We've diagnosed the problem and it was related to arm_nocache_remap(). When the page was remaped into non-cacheable virtual memory region there was no dcache write-back operation, so count of free items in slab had differ value then should had. This write-back operation should have been done in pmap_set_cache_entry() but PVF_REF flag was missing so this part of code was omitted. I think that we should enable PVF_REF flag in pmap_enter_pv(). ================================================ diff --git a/sys/arm/arm/pmap.c b/sys/arm/arm/pmap.c index 3cdab65..133dc6d 100644 --- a/sys/arm/arm/pmap.c +++ b/sys/arm/arm/pmap.c @@ -1683,7 +1683,7 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry *pve, pmap_t pm, /* PMAP_ASSERT_LOCKED(pmap_kernel()); */ pve->pv_pmap = pmap_kernel(); pve->pv_va = pg->md.pv_kva; - pve->pv_flags = PVF_WRITE | PVF_UNMAN; + pve->pv_flags = PVF_WRITE | PVF_UNMAN | PVF_REF; pg->md.pv_kva = 0; TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); ================================================= With this patch I've had no panics. Many thanks, MichaÅ‚ Hajduk From owner-freebsd-arm@FreeBSD.ORG Fri Jul 10 13:07:49 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3441106567C; Fri, 10 Jul 2009 13:07:49 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 45D6A8FC0C; Fri, 10 Jul 2009 13:07:48 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n6AD7mut081111; Fri, 10 Jul 2009 08:07:48 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1247231268; bh=j09QZcQKlHRE/PSz0jQ6l6XZj1R8R7zQzMkVcH+IZTU=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=g7+8aOGwCBJJICnRxiEqp8g3KZ41146uliTIqT5aM6UrhbWBiuYMTYBiqda8UXZ5u 1pV7lS1mCh5YiuK07KhyI3nnW9mkN6Ltvxdp1qTfAFxjOsnwccUFEYYDQW7g8cA52R +XMTg7rxywHOzaXuX6KdW10Ry/mONwbBvK/0g1F8= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n6AD7lZT081110; Fri, 10 Jul 2009 08:07:47 -0500 (CDT) (envelope-from tinguely) Date: Fri, 10 Jul 2009 08:07:47 -0500 (CDT) From: Mark Tinguely Message-Id: <200907101307.n6AD7lZT081110@casselton.net> To: mih@semihalf.com, tinguely@casselton.net In-Reply-To: <4A56FB72.9040807@semihalf.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Fri, 10 Jul 2009 08:07:48 -0500 (CDT) Cc: freebsd-arm@freebsd.org Subject: Re: pmap problem in FreeBSD current X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jul 2009 13:07:50 -0000 > Hi Mark, > We've diagnosed the problem and it was related to arm_nocache_remap(). > When the page was remaped into non-cacheable virtual memory region > there was no dcache write-back operation, so count of free items in > slab had differ value then should had. > This write-back operation should have been done in pmap_set_cache_entry() > but PVF_REF flag was missing so this part of code was omitted. > > I think that we should enable PVF_REF flag in pmap_enter_pv(). > > ================================================ > > diff --git a/sys/arm/arm/pmap.c b/sys/arm/arm/pmap.c > index 3cdab65..133dc6d 100644 > --- a/sys/arm/arm/pmap.c > +++ b/sys/arm/arm/pmap.c > @@ -1683,7 +1683,7 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry > *pve, pmap_t pm, > /* PMAP_ASSERT_LOCKED(pmap_kernel()); */ > pve->pv_pmap = pmap_kernel(); > pve->pv_va = pg->md.pv_kva; > - pve->pv_flags = PVF_WRITE | PVF_UNMAN; > + pve->pv_flags = PVF_WRITE | PVF_UNMAN | PVF_REF; > pg->md.pv_kva = 0; > > TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); > > ================================================= > > With this patch I've had no panics. > > Many thanks, > MichaÅ‚ Hajduk > Good job. This strange, your inclusion writes back the whole page of the original buffer. This writeback of just the original buffer size should be done in bus_dmamem_alloc() in busdma_machdep.c after the arm_remap_nocache() with the line "cpu_idcache_wbinv_range(vm_offset_t)*vaddr, dmat->maxsize)". I wonder if your cpu_idcache_wbinv_range() implementation correctly wb/invalidates when the buffer is not cache line aligned or is smaller than a cache-line? The other situation that would do this is if your cache line larger than you think. I know the ARMv7 use 64 byte caches instead of the more traditional 32 bytes. This should not be your case because last week I looked up your processor information and thought it said it used a 32 byte cache line. Did http://www.casselton.net/~tinguely/arm_pmap_unmanaged.diff fix the lock problems? Thank-you for the test information and the code scouting. --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Fri Jul 10 14:54:08 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A02F106566B; Fri, 10 Jul 2009 14:54:08 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 476678FC0C; Fri, 10 Jul 2009 14:54:08 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.3/8.14.3) with ESMTP id n6AEs7v6087495; Fri, 10 Jul 2009 09:54:07 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1247237647; bh=IKFxBuXtrwC/tmtdThv1T+QAvi2noRt5xGSWIPAiZ44=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=lXCRrrubGEtzQWNfBMVOtUh8LkAz2Ro54vNBqggEpGSns9luyQ3RkZ3LmemawKANG B6nOAV/CmssxUqRh7Xsqy0/z2ZG0q1Gula7+PXTr2Z2X4dii+EJtJfQPSeqp8dfG5z zmWpkdD/eTS1PkpFwWlENZmOMUH3QXcyINqT+mPI= Received: (from tinguely@localhost) by casselton.net (8.14.3/8.14.2/Submit) id n6AEs7nJ087492; Fri, 10 Jul 2009 09:54:07 -0500 (CDT) (envelope-from tinguely) Date: Fri, 10 Jul 2009 09:54:07 -0500 (CDT) From: Mark Tinguely Message-Id: <200907101454.n6AEs7nJ087492@casselton.net> To: mih@semihalf.com, tinguely@casselton.net In-Reply-To: <200907101307.n6AD7lZT081110@casselton.net> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.2 (casselton.net [127.0.0.1]); Fri, 10 Jul 2009 09:54:07 -0500 (CDT) Cc: freebsd-arm@freebsd.org Subject: Re: pmap problem in FreeBSD current - PS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jul 2009 14:54:08 -0000 PS. sorry to even question your cpu_(l2)idcache_wbinv_range() implemetation. the cpu_idcache_wbinv_range() in bus_dmamem_alloc() is wrong. arm_remap_nocache() should wb/inv the entire page especially if the buffer that uses the page is less than a page size or non-page aligned. The cache lines NOT in the buffer is the data that can be lost if used while the nocache shadow mapping is in effect. As an open question to the group: should we add an explict page cpu_(l2)idcache_wbinv_range() in the arm_remap_nocache() routine, so we know that it happens rather than a side effect of the new managing of kernel pages? For freebsd-9, I will rewrite arm_remap_nocache() to get rid of the shadow map. Did your lock problems go away? Good work. --Mark Tinguely From owner-freebsd-arm@FreeBSD.ORG Sat Jul 11 08:52:56 2009 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF4BD1065678 for ; Sat, 11 Jul 2009 08:52:56 +0000 (UTC) (envelope-from toby@iacmusic.com) Received: from 4-George.m6.net (4-George.m6.net [70.84.97.170]) by mx1.freebsd.org (Postfix) with ESMTP id 42F3A8FC21 for ; Sat, 11 Jul 2009 08:52:55 +0000 (UTC) (envelope-from toby@iacmusic.com) Received: from pool-70-106-84-225.hag.east.verizon.net [70.106.84.225] by 4-George.m6.net with SMTP; Sat, 11 Jul 2009 09:08:18 +0100 X-Unsent: 1 Date: Sat, 11 Jul 2009 04:22:17 -0400 Content-Transfer-Encoding: quoted-printable To: freebsd-arm@freebsd.org From: Toby@IACmusic.com Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Finally, a song contest that is free to enter. $27, 000+ in prizes too! X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jul 2009 08:52:59 -0000 =0A =0A Hi, =0A =0A IACmusic.com has started a major song contest and this one you can enter for free! &n= bsp; Thanks to a major new =0Asponsorship, we're throwing a party that = will make major waves in the Indie =0AWorld. Got a =0Asong= you know is good ? You could win, there are going to be a lot of = =0Awinners.. =0A =0A It's our [1]YEAR =0AOF THE I= NDIE Celebration and it started "Indiependents" Day July =0A4th!&nb= sp; =0AThere are 16 gen= re categories to choose from to enter your song in, =0Aincluding Songwri= ting. The Grand Prize is a huge package that =0Aincludes $1000 wor= th of musical equipment (whatever you need), 2 =0Aweeks stay in a c= ondo suite at your choice of a number of US =0Avacation spots, an i= Pod Shuffle, and a IAC Prime Perpetual Lifetime =0Amembership. But= there are also 3 nice prizes in each of 16 =0Acategories and you can en= ter any original song. You will get a =0Alot of additional exposur= e even by advancing to the later rounds of the =0Acompetition. He= re's a [2]direct =0Alink to enter your song, if not logged in you will hit = login page to do so. =0A =0A Go [3]here =0A= for the details. =0A =0A Good luc= k, =0A =0A The Staff at IACmusic.com (the Ind= ie Capitol of the World) =0A =0A =0A References 1. file://localhost/tmp/3D"htt= 2. 3D"http://iacmusic.com/quickSignup.aspx" 3. file://localhost/tmp/3D=