From owner-svn-src-all@FreeBSD.ORG Tue May 27 16:17:26 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BB4BAB3; Tue, 27 May 2014 16:17:26 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27BF62A9A; Tue, 27 May 2014 16:17:26 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.8/8.14.8) with ESMTP id s4RGHQLH090470; Tue, 27 May 2014 16:17:26 GMT (envelope-from ian@svn.freebsd.org) Received: (from ian@localhost) by svn.freebsd.org (8.14.8/8.14.8/Submit) id s4RGHPN3090463; Tue, 27 May 2014 16:17:25 GMT (envelope-from ian@svn.freebsd.org) Message-Id: <201405271617.s4RGHPN3090463@svn.freebsd.org> From: Ian Lepore Date: Tue, 27 May 2014 16:17:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r266755 - in stable/10/sys/arm: arm include ti X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2014 16:17:26 -0000 Author: ian Date: Tue May 27 16:17:25 2014 New Revision: 266755 URL: http://svnweb.freebsd.org/changeset/base/266755 Log: MFC 266621: Eliminte spurious interrupts caused by ARM weak memory ordering. Modified: stable/10/sys/arm/arm/gic.c stable/10/sys/arm/arm/intr.c stable/10/sys/arm/include/intr.h stable/10/sys/arm/ti/aintc.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/arm/arm/gic.c ============================================================================== --- stable/10/sys/arm/arm/gic.c Tue May 27 16:13:01 2014 (r266754) +++ stable/10/sys/arm/arm/gic.c Tue May 27 16:17:25 2014 (r266755) @@ -83,6 +83,8 @@ __FBSDID("$FreeBSD$"); #define GICC_ABPR 0x001C /* v1 ICCABPR */ #define GICC_IIDR 0x00FC /* v1 ICCIIDR*/ +#define GIC_LAST_IPI 15 /* Irqs 0-15 are IPIs. */ + /* First bit is a polarity bit (0 - low, 1 - high) */ #define GICD_ICFGR_POL_LOW (0 << 0) #define GICD_ICFGR_POL_HIGH (1 << 0) @@ -268,6 +270,8 @@ gic_post_filter(void *arg) { uintptr_t irq = (uintptr_t) arg; + if (irq > GIC_LAST_IPI) + arm_irq_memory_barrier(irq); gic_c_write_4(GICC_EOIR, irq); } @@ -284,13 +288,13 @@ arm_get_next_irq(int last_irq) * have this information later. */ - if ((active_irq & 0x3ff) < 16) + if ((active_irq & 0x3ff) <= GIC_LAST_IPI) gic_c_write_4(GICC_EOIR, active_irq); active_irq &= 0x3FF; if (active_irq == 0x3FF) { if (last_irq == -1) - printf("Spurious interrupt detected [0x%08x]\n", active_irq); + printf("Spurious interrupt detected\n"); return -1; } @@ -309,6 +313,8 @@ void arm_unmask_irq(uintptr_t nb) { + if (nb > GIC_LAST_IPI) + arm_irq_memory_barrier(nb); gic_d_write_4(GICD_ISENABLER(nb >> 5), (1UL << (nb & 0x1F))); } Modified: stable/10/sys/arm/arm/intr.c ============================================================================== --- stable/10/sys/arm/arm/intr.c Tue May 27 16:13:01 2014 (r266754) +++ stable/10/sys/arm/arm/intr.c Tue May 27 16:17:25 2014 (r266755) @@ -149,3 +149,67 @@ arm_irq_handler(struct trapframe *frame) } } } + +/* + * arm_irq_memory_barrier() + * + * Ensure all writes to device memory have reached devices before proceeding. + * + * This is intended to be called from the post-filter and post-thread routines + * of an interrupt controller implementation. A peripheral device driver should + * use bus_space_barrier() if it needs to ensure a write has reached the + * hardware for some reason other than clearing interrupt conditions. + * + * The need for this function arises from the ARM weak memory ordering model. + * Writes to locations mapped with the Device attribute bypass any caches, but + * are buffered. Multiple writes to the same device will be observed by that + * device in the order issued by the cpu. Writes to different devices may + * appear at those devices in a different order than issued by the cpu. That + * is, if the cpu writes to device A then device B, the write to device B could + * complete before the write to device A. + * + * Consider a typical device interrupt handler which services the interrupt and + * writes to a device status-acknowledge register to clear the interrupt before + * returning. That write is posted to the L2 controller which "immediately" + * places it in a store buffer and automatically drains that buffer. This can + * be less immediate than you'd think... There may be no free slots in the store + * buffers, so an existing buffer has to be drained first to make room. The + * target bus may be busy with other traffic (such as DMA for various devices), + * delaying the drain of the store buffer for some indeterminate time. While + * all this delay is happening, execution proceeds on the CPU, unwinding its way + * out of the interrupt call stack to the point where the interrupt driver code + * is ready to EOI and unmask the interrupt. The interrupt controller may be + * accessed via a faster bus than the hardware whose handler just ran; the write + * to unmask and EOI the interrupt may complete quickly while the device write + * to ack and clear the interrupt source is still lingering in a store buffer + * waiting for access to a slower bus. With the interrupt unmasked at the + * interrupt controller but still active at the device, as soon as interrupts + * are enabled on the core the device re-interrupts immediately: now you've got + * a spurious interrupt on your hands. + * + * The right way to fix this problem is for every device driver to use the + * proper bus_space_barrier() calls in its interrupt handler. For ARM a single + * barrier call at the end of the handler would work. This would have to be + * done to every driver in the system, not just arm-specific drivers. + * + * Another potential fix is to map all device memory as Strongly-Ordered rather + * than Device memory, which takes the store buffers out of the picture. This + * has a pretty big impact on overall system performance, because each strongly + * ordered memory access causes all L2 store buffers to be drained. + * + * A compromise solution is to have the interrupt controller implementation call + * this function to establish a barrier between writes to the interrupt-source + * device and writes to the interrupt controller device. + * + * This takes the interrupt number as an argument, and currently doesn't use it. + * The plan is that maybe some day there is a way to flag certain interrupts as + * "memory barrier safe" and we can avoid this overhead with them. + */ +void +arm_irq_memory_barrier(uintptr_t irq) +{ + + dsb(); + cpu_l2cache_drain_writebuf(); +} + Modified: stable/10/sys/arm/include/intr.h ============================================================================== --- stable/10/sys/arm/include/intr.h Tue May 27 16:13:01 2014 (r266754) +++ stable/10/sys/arm/include/intr.h Tue May 27 16:17:25 2014 (r266755) @@ -79,6 +79,8 @@ extern void (*arm_post_filter)(void *); extern int (*arm_config_irq)(int irq, enum intr_trigger trig, enum intr_polarity pol); +void arm_irq_memory_barrier(uintptr_t); + void gic_init_secondary(void); #endif /* _MACHINE_INTR_H */ Modified: stable/10/sys/arm/ti/aintc.c ============================================================================== --- stable/10/sys/arm/ti/aintc.c Tue May 27 16:13:01 2014 (r266754) +++ stable/10/sys/arm/ti/aintc.c Tue May 27 16:17:25 2014 (r266755) @@ -180,5 +180,7 @@ arm_mask_irq(uintptr_t nb) void arm_unmask_irq(uintptr_t nb) { + + arm_irq_memory_barrier(nb); aintc_write_4(INTC_MIR_CLEAR(nb >> 5), (1UL << (nb & 0x1F))); }