From owner-svn-src-head@freebsd.org Thu Apr 25 20:37:46 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8E5315A1DA3; Thu, 25 Apr 2019 20:37:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 903886AA86; Thu, 25 Apr 2019 20:37:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-3.local (ralph.baldwin.cx [66.234.199.215]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id B03898C37; Thu, 25 Apr 2019 20:37:44 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: svn commit: r346386 - in head/sys: dev/bge dev/pci dev/twa x86/iommu To: Ryan Stone , Tycho Nightingale Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, Ed Maste References: <201904191343.x3JDhYVF010453@repo.freebsd.org> From: John Baldwin Openpgp: preference=signencrypt Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: <5c43013c-1fc6-57c2-6dec-1fdfc5213fb3@FreeBSD.org> Date: Thu, 25 Apr 2019 13:37:43 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 903886AA86 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.98)[-0.977,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Apr 2019 20:37:46 -0000 I had looked for the aac change, but wasn't able to find it, perhaps because I looked at tags created in aac.c rather than aac_pci.c. I agree aac will need to be re-patched. I'm not really certain how many other devices are actually broken. They would all be due to a firmware bug, nothing inherent in PCI. I believe twa(4) and bge(4) issues predated aac(4) FWIW. Unfortunately, the main bit of discussion about moving the limit into the PCI bus itself seems to be an IRC discussion on 2/28/12 that resulted in revision r232267 as a quick MFC'able fix, but I don't have a log of that conversation. :( I couldn't find anything in e-mail either that was definitive for why this might have been inherent in PCI-e vs a few firmware writers having similar bugs. On 4/25/19 12:20 PM, Ryan Stone wrote: > Following up, this is what will have to be re-instated in the aac driver: > > http://svn.freebsd.org/changeset/base/232260 > > However, my biggest concern is that we have no idea how many new > devices with the broken behaviour might have been introduced since we > fixed the problem in general. How does Linux handle the issue? > > On Thu, Apr 25, 2019 at 3:17 PM Ryan Stone wrote: >> >> This change makes me *very* uncomfortable. It was originally brought >> in due to issues with Adaptec RAID cards using the aac(9) driver. The >> symptoms of the bug included silent corruption of data as it was >> written to disk. Are we sure that this change is a good idea, given >> how catastrophic it is when a device gets this wrong? >> >> On Fri, Apr 19, 2019 at 9:43 AM Tycho Nightingale wrote: >>> >>> Author: tychon >>> Date: Fri Apr 19 13:43:33 2019 >>> New Revision: 346386 >>> URL: https://svnweb.freebsd.org/changeset/base/346386 >>> >>> Log: >>> remove the 4GB boundary requirement on PCI DMA segments >>> >>> Reviewed by: kib >>> Discussed with: jhb >>> Sponsored by: Dell EMC Isilon >>> Differential Revision: https://reviews.freebsd.org/D19867 >>> >>> Modified: >>> head/sys/dev/bge/if_bgereg.h >>> head/sys/dev/pci/pci.c >>> head/sys/dev/pci/pcivar.h >>> head/sys/dev/twa/tw_osl.h >>> head/sys/dev/twa/tw_osl_freebsd.c >>> head/sys/x86/iommu/intel_ctx.c >>> >>> Modified: head/sys/dev/bge/if_bgereg.h >>> ============================================================================== >>> --- head/sys/dev/bge/if_bgereg.h Fri Apr 19 13:23:41 2019 (r346385) >>> +++ head/sys/dev/bge/if_bgereg.h Fri Apr 19 13:43:33 2019 (r346386) >>> @@ -3067,3 +3067,11 @@ struct bge_softc { >>> #define BGE_LOCK_ASSERT(_sc) mtx_assert(&(_sc)->bge_mtx, MA_OWNED) >>> #define BGE_UNLOCK(_sc) mtx_unlock(&(_sc)->bge_mtx) >>> #define BGE_LOCK_DESTROY(_sc) mtx_destroy(&(_sc)->bge_mtx) >>> + >>> +#ifdef BUS_SPACE_MAXADDR >>> +#if (BUS_SPACE_MAXADDR > 0xFFFFFFFF) >>> +#define BGE_DMA_BOUNDARY (0x100000000) >>> +#else >>> +#define BGE_DMA_BOUNDARY 0 >>> +#endif >>> +#endif >>> >>> Modified: head/sys/dev/pci/pci.c >>> ============================================================================== >>> --- head/sys/dev/pci/pci.c Fri Apr 19 13:23:41 2019 (r346385) >>> +++ head/sys/dev/pci/pci.c Fri Apr 19 13:43:33 2019 (r346386) >>> @@ -4343,9 +4343,6 @@ pci_attach_common(device_t dev) >>> { >>> struct pci_softc *sc; >>> int busno, domain; >>> -#ifdef PCI_DMA_BOUNDARY >>> - int error, tag_valid; >>> -#endif >>> #ifdef PCI_RES_BUS >>> int rid; >>> #endif >>> @@ -4365,23 +4362,7 @@ pci_attach_common(device_t dev) >>> if (bootverbose) >>> device_printf(dev, "domain=%d, physical bus=%d\n", >>> domain, busno); >>> -#ifdef PCI_DMA_BOUNDARY >>> - tag_valid = 0; >>> - if (device_get_devclass(device_get_parent(device_get_parent(dev))) != >>> - devclass_find("pci")) { >>> - error = bus_dma_tag_create(bus_get_dma_tag(dev), 1, >>> - PCI_DMA_BOUNDARY, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, >>> - NULL, NULL, BUS_SPACE_MAXSIZE, BUS_SPACE_UNRESTRICTED, >>> - BUS_SPACE_MAXSIZE, 0, NULL, NULL, &sc->sc_dma_tag); >>> - if (error) >>> - device_printf(dev, "Failed to create DMA tag: %d\n", >>> - error); >>> - else >>> - tag_valid = 1; >>> - } >>> - if (!tag_valid) >>> -#endif >>> - sc->sc_dma_tag = bus_get_dma_tag(dev); >>> + sc->sc_dma_tag = bus_get_dma_tag(dev); >>> return (0); >>> } >>> >>> >>> Modified: head/sys/dev/pci/pcivar.h >>> ============================================================================== >>> --- head/sys/dev/pci/pcivar.h Fri Apr 19 13:23:41 2019 (r346385) >>> +++ head/sys/dev/pci/pcivar.h Fri Apr 19 13:43:33 2019 (r346386) >>> @@ -693,14 +693,6 @@ int pcie_link_reset(device_t port, int pcie_location); >>> >>> void pci_print_faulted_dev(void); >>> >>> -#ifdef BUS_SPACE_MAXADDR >>> -#if (BUS_SPACE_MAXADDR > 0xFFFFFFFF) >>> -#define PCI_DMA_BOUNDARY 0x100000000 >>> -#else >>> -#define PCI_DMA_BOUNDARY 0 >>> -#endif >>> -#endif >>> - >>> #endif /* _SYS_BUS_H_ */ >>> >>> /* >>> >>> Modified: head/sys/dev/twa/tw_osl.h >>> ============================================================================== >>> --- head/sys/dev/twa/tw_osl.h Fri Apr 19 13:23:41 2019 (r346385) >>> +++ head/sys/dev/twa/tw_osl.h Fri Apr 19 13:43:33 2019 (r346386) >>> @@ -57,6 +57,12 @@ >>> #define TW_OSLI_MAX_NUM_IOS (TW_OSLI_MAX_NUM_REQUESTS - 2) >>> #define TW_OSLI_MAX_NUM_AENS 0x100 >>> >>> +#ifdef PAE >>> +#define TW_OSLI_DMA_BOUNDARY (1u << 31) >>> +#else >>> +#define TW_OSLI_DMA_BOUNDARY ((bus_size_t)((uint64_t)1 << 32)) >>> +#endif >>> + >>> /* Possible values of req->state. */ >>> #define TW_OSLI_REQ_STATE_INIT 0x0 /* being initialized */ >>> #define TW_OSLI_REQ_STATE_BUSY 0x1 /* submitted to CL */ >>> >>> Modified: head/sys/dev/twa/tw_osl_freebsd.c >>> ============================================================================== >>> --- head/sys/dev/twa/tw_osl_freebsd.c Fri Apr 19 13:23:41 2019 (r346385) >>> +++ head/sys/dev/twa/tw_osl_freebsd.c Fri Apr 19 13:43:33 2019 (r346386) >>> @@ -551,7 +551,7 @@ tw_osli_alloc_mem(struct twa_softc *sc) >>> /* Create the parent dma tag. */ >>> if (bus_dma_tag_create(bus_get_dma_tag(sc->bus_dev), /* parent */ >>> sc->alignment, /* alignment */ >>> - 0, /* boundary */ >>> + TW_OSLI_DMA_BOUNDARY, /* boundary */ >>> BUS_SPACE_MAXADDR, /* lowaddr */ >>> BUS_SPACE_MAXADDR, /* highaddr */ >>> NULL, NULL, /* filter, filterarg */ >>> >>> Modified: head/sys/x86/iommu/intel_ctx.c >>> ============================================================================== >>> --- head/sys/x86/iommu/intel_ctx.c Fri Apr 19 13:23:41 2019 (r346385) >>> +++ head/sys/x86/iommu/intel_ctx.c Fri Apr 19 13:43:33 2019 (r346386) >>> @@ -130,7 +130,7 @@ ctx_tag_init(struct dmar_ctx *ctx, device_t dev) >>> maxaddr = MIN(ctx->domain->end, BUS_SPACE_MAXADDR); >>> ctx->ctx_tag.common.ref_count = 1; /* Prevent free */ >>> ctx->ctx_tag.common.impl = &bus_dma_dmar_impl; >>> - ctx->ctx_tag.common.boundary = PCI_DMA_BOUNDARY; >>> + ctx->ctx_tag.common.boundary = 0; >>> ctx->ctx_tag.common.lowaddr = maxaddr; >>> ctx->ctx_tag.common.highaddr = maxaddr; >>> ctx->ctx_tag.common.maxsize = maxaddr; >>> > -- John Baldwin