From owner-freebsd-arm@freebsd.org Wed Oct 7 04:50:46 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1F45D3FADD2 for ; Wed, 7 Oct 2020 04:50:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-19.consmr.mail.gq1.yahoo.com (sonic305-19.consmr.mail.gq1.yahoo.com [98.137.64.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C5hjd1bT7z4KGS for ; Wed, 7 Oct 2020 04:50:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: p8a2FC0VM1nf_iIm7nn56a6WyjODa6a7J1isb4Vt7cYost4n0uCyucL9OF1pKAz FhChw17huD2oFzUEu0y1XPvcbfoWP.4t1dTFtc7o56TbD8uazzRUCmvyFvRn3x8nXyI2tvBazoFw P0GHw0rlF.Fn1i9Uq2ORGQ47lvreHuN2pKU_wnZo4y8Jpzu.4a0e.Zwh5VkpVWEQC24ZttL_JfIe 6Jlev.kFGvNRnLgmVnFtz5eFR4hgw.sWBEh5WGu8uF0LwlmqKXSHlJG0Ik8WLy79EgGM_s6IzPBW DD.PjCooPce8BLO7uBxXJf3A6hmF9UiqkPvyh9Xfxsp6abg8jE0tbUNnG9LaFLu5plY4IbWZJUYe EgR0NfGnPi7ZGtefrFrKG1cSMXv7VEADnHl22y1Ua.VMFnaSGb6ZyV2G2yvqua73bD0Mb2Jj1X4w AW3cXrryz5TLTtdvUIvDdm0.e_2LdAxYTK8BFPirCpM2uwlmFDbg3ruLPcrrQCMoJ0oLFk.cEdBt Vey.JedjJbSEoFohkyPgsbL83lNdTgRfjGgdsWGewBSmWguzZkEnoYDZMJuLm4Jbe1UBfG4acsWG Xo.jH2.1UWOVDv1rujSsSEl4wuWWrQB6KkojZAl1cNtA_qUzOhaNhWTt847zbQhnuPlmMv_hQ1wa Ikp6XpdXR0qCRnhP6rKCNJrfROZpukmYDJJV0BN4RHKET.N3XTjbmTnReZMxlsfYGxjL4oMPiw95 Jywl0LA0YPUU4hvgBpAcdz.tO5TdzSB1tEkmrtdgCn_D6jzL_bmvGgn4XbV64YFlTG5mBNSzv9Sd JiX_SnH5msQeFsS..VBbmOp6VBWXisUkYymCCnsafTTOO4OIWk_cw_yrSYXdpEklO9Any3saXRT9 3bU5unhjMmuPybeNrBt2TfP3bqYfC0p3QS8BdrQywGO9GGZbyFTzaCqEHYoGYlk649Dk6c6CbqIn ZClq9jIYtsAfZJ2.K0rQPfUTCz56QB7LqBsiLu0_FcrYUs16AX3OlXr0mURfNSSPtiFXxgIvsGv0 QBH8vxRca.GwLDeLH6jaxglmaqYQdm659.dKSbVNaStr1IX4C11WRdGZ7WklAe_bx4IgP.7HTwHR nXYOlu4_5GFRiZoXi3QsyXd7ufo5vPMGNln1oF0_sBid6AUfbyipI5stGeuWBz.mVosRc7BIdiFF T4aaReDdHW4uyjKxFf56UN70Sfl9rhFJ8E4yBxaR7MtaO2Qmvem_9MlCRE2sys2b6uWhNYkvK0G0 FwAYqUIEsAk2yewmG.Ou2Gwehc_ZzwN8CP1q_PnfrBtJoT9lOhKDQFkRMeMHvjlDRgMW6JkauhxO R_ZAw6uLD4keAOum5V0MrfpfUsb3Pv8MUSr4oTgP_InzGKjH.h2rVJHPyXYJam3GLE82RbXos5m. dVLsIB0.YDw7NGb52h0R2AEK98.eY62rwnVPS.DGxLjXTTyhmGo99r6yMbP0pcIcGWaI9Yf6g96M uui5uJ5F2L2zbtFER3kBNuWz.Bzu4QYmhwLxhXg30afoMpsXV8nJdCR6v1PHAMHSFuvjtAkKjo5N ql2n_S_IWPig8nKSkEOfbRKvgR_BVXV_zkMA- Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Wed, 7 Oct 2020 04:50:42 +0000 Received: by smtp418.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 2ae6e81db8dd70c0fe6ae8b3696829a6; Wed, 07 Oct 2020 04:50:38 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: A basis for a possible update to the pcie based xhci support? It survived huge-file duplicate-then-diff testing so far. Date: Tue, 6 Oct 2020 21:50:36 -0700 References: <31C1F4F8-6727-4EBE-9D20-39F5B2DA89A5@yahoo.com> To: Robert Crowston , freebsd-arm In-Reply-To: <31C1F4F8-6727-4EBE-9D20-39F5B2DA89A5@yahoo.com> Message-Id: <0A440C23-7D21-4515-B872-1C64FF80A873@yahoo.com> X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Rspamd-Queue-Id: 4C5hjd1bT7z4KGS X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.73 / 15.00]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.19)[-1.191]; FREEMAIL_TO(0.00)[protonmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.02)[-1.022]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.019]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.82:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.82:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2020 04:50:46 -0000 On 2020-Oct-6, at 21:43, Mark Millard wrote: > Note: based on a head -r363932 context, not more recent. >=20 > First off, a note about lowaddr values. What sysctl showed > me were the likes of (prior to the changes that this note > is about): >=20 > . . . > hw.busdma.zone2.lowaddr: 0x3c000fff > . . . > hw.busdma.zone1.lowaddr: 0x3fffffff > . . . > hw.busdma.zone0.lowaddr: 0xffffffff > . . . >=20 > So I've guessed that lowaddr should identify the > end page of the possibly-use-it region, not the > first do-not-use-it page. > If wrong, at most it > should avoid bouncing one page that it could > avoid. That was a wonderfully messed up sentence. Trying again: "If wrong, at most it would bounce one page that it could avoid bouncing." > But, if correct, it might bounce a page > that it should instead of not doing so. >=20 > Otherwise what I've done is put back some of your old > bcm2838_pci.c code and removed the sc->sc_bus.dma_bits > adjustment from the bcm2838_xhci.c code. Be warned > that I copied the likes of REG_VALUE_4GB_WINDOW and > REG_VALUE_4GB_CONFIG without understanding the values > or encoding. (I'm not pcie knowledgable.) >=20 > # svnlite diff /usr/src/sys/arm/broadcom/ > Index: /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c (revision = 365932) > +++ /usr/src/sys/arm/broadcom/bcm2835/bcm2838_pci.c (working copy) > @@ -91,27 +91,22 @@ > #define REG_EP_CONFIG_CHOICE 0x9000 > #define REG_EP_CONFIG_DATA 0x8000 >=20 > +#define REG_VALUE_4GB_WINDOW 0x11 > +#define REG_VALUE_4GB_CONFIG 0x88003000 > + > /* > * The system memory controller can address up to 16 GiB of physical = memory > * (although at time of writing the largest memory size available for = purchase > - * is 8 GiB). However, the system DMA controller is capable of = accessing only a > - * limited portion of the address space. Worse, the PCI-e controller = has further > - * constraints for DMA, and those limitations are not wholly clear to = the > - * author. NetBSD and Linux allow DMA on the lower 3 GiB of the = physical memory, > - * but experimentation shows DMA performed above 960 MiB results in = data > - * corruption with this driver. The limit of 960 MiB is taken from = OpenBSD, but > + * is 8 GiB). However, the system DMA controller in early enough = boards is > + * capable of accessing only a limited portion of the address space = (3 GiByte). > + * Worse, the PCI-e controller has further constraints for DMA, and = those > + * limitations are not wholly clear to the author. NetBSD and Linux = allow > + * DMA on the lower 3 GiB of the physical memory. OpenBSD used 960 = MiByte but > * apparently that value was chosen for satisfying a constraint of an = unrelated > * peripheral. > - * > - * Whatever the true maximum address, 960 MiB works. > */ > -#define DMA_HIGH_LIMIT 0x3c000000 > -#define MAX_MEMORY_LOG2 0x21 > -#define REG_VALUE_DMA_WINDOW_LOW (MAX_MEMORY_LOG2 - 0xf) > +#define DMA_HIGH_LIMIT = ((bus_addr_t)0xc0000000u-1) > #define REG_VALUE_DMA_WINDOW_HIGH 0x0 > -#define DMA_WINDOW_ENABLE 0x3000 > -#define REG_VALUE_DMA_WINDOW_CONFIG \ > - (((MAX_MEMORY_LOG2 - 0xf) << 0x1b) | DMA_WINDOW_ENABLE) >=20 > #define REG_VALUE_MSI_CONFIG 0xffe06540 >=20 > @@ -645,9 +640,9 @@ > DMA_HIGH_LIMIT, /* lowaddr */ > BUS_SPACE_MAXADDR, /* highaddr */ > NULL, NULL, /* filter, filterarg */ > - DMA_HIGH_LIMIT, /* maxsize */ > + BUS_SPACE_MAXSIZE, /* maxsize */ > BUS_SPACE_UNRESTRICTED, /* nsegments */ > - DMA_HIGH_LIMIT, /* maxsegsize */ > + BUS_SPACE_MAXSIZE, /* maxsegsize */ > 0, /* flags */ > NULL, NULL, /* lockfunc, lockarg */ > &sc->dmat); > @@ -674,9 +669,9 @@ > * Set PCI->CPU memory window. This encodes the inbound window = showing > * the system memory to the controller. > */ > - bcm_pcib_set_reg(sc, REG_DMA_WINDOW_LOW, = REG_VALUE_DMA_WINDOW_LOW); > + bcm_pcib_set_reg(sc, REG_DMA_WINDOW_LOW, REG_VALUE_4GB_WINDOW); > bcm_pcib_set_reg(sc, REG_DMA_WINDOW_HIGH, = REG_VALUE_DMA_WINDOW_HIGH); > - bcm_pcib_set_reg(sc, REG_DMA_CONFIG, = REG_VALUE_DMA_WINDOW_CONFIG); > + bcm_pcib_set_reg(sc, REG_DMA_CONFIG, REG_VALUE_4GB_CONFIG); >=20 > bcm_pcib_set_reg(sc, REG_BRIDGE_GISB_WINDOW, 0); > bcm_pcib_set_reg(sc, REG_DMA_WINDOW_1, 0); > Index: /usr/src/sys/arm/broadcom/bcm2835/bcm2838_xhci.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/arm/broadcom/bcm2835/bcm2838_xhci.c (revision = 365932) > +++ /usr/src/sys/arm/broadcom/bcm2835/bcm2838_xhci.c (working copy) > @@ -189,15 +189,7 @@ > bcm_xhci_install_xhci_firmware(dev); >=20 > error =3D xhci_pci_attach(dev); > - if (error) > - return (error); > - > - /* 32 bit DMA is a limitation of the PCI-e controller, not the = VL805. */ > - sc->sc_bus.dma_bits =3D 32; > - if (bootverbose) > - device_printf(dev, "note: switched to 32-bit DMA.\n"); > - > - return (0); > + return (error); > } >=20 > /* >=20 > I've concluded from what I've seen in the code that lowaddr > should be based on the pcie properties and should not worry > about the maxsize and maxseg size figures being possibly > smaller: that is a dma engine use worry, not a pci one. (Not > that I could get that from the documentation that I quoted in > the review.) Thus I put back the 2 BUS_SPACE_MAXSIZE uses. >=20 > After the first huge-file duplicate-then-diff test sysctl > reported lots of bounced transfers: >=20 > # sysctl hw.busdma > hw.busdma.zone1.alignment: 4096 > hw.busdma.zone1.lowaddr: 0x3fffffff > hw.busdma.zone1.total_deferred: 0 > hw.busdma.zone1.total_bounced: 755770 > hw.busdma.zone1.active_bpages: 0 > hw.busdma.zone1.reserved_bpages: 0 > hw.busdma.zone1.free_bpages: 838 > hw.busdma.zone1.total_bpages: 838 > hw.busdma.zone0.alignment: 4096 > hw.busdma.zone0.lowaddr: 0xffffffff > hw.busdma.zone0.total_deferred: 0 > hw.busdma.zone0.total_bounced: 0 > hw.busdma.zone0.active_bpages: 256 > hw.busdma.zone0.reserved_bpages: 0 > hw.busdma.zone0.free_bpages: 257 > hw.busdma.zone0.total_bpages: 513 > hw.busdma.total_bpages: 1351 >=20 > For the non-power-of-2 boundary (0xc0000000-1), it > appears to use the next smaller power of 2 for the > boundary (0x40000000-1), without having to explicitly > code both types of values specially for the RPi4B. > (Of course, it also avoids using 2 GiBytes to > potentially avoid more bouncing.) >=20 > I'll note that, prior to the change, there > was after an example first test: >=20 > hw.busdma.zone2.total_bounced: 1091942 >=20 > and 174 in zone 1. So the bounce count has > decreased. >=20 > I'll note that "total_bounced" need not be the > a page count: it is incremented by 1 after > the loop for a bounce, not inside the loop. > Lots of pages of data were bounced. >=20 > For reference (the test as of a gpu_mem_1024=3D32 > context): >=20 > Physical memory chunk(s): > 0x00000000002000 - 0x00000007ef0fff, 133099520 bytes (32495 pages) > 0x00000007f0f000 - 0x00000034bfffff, 751767552 bytes (183537 pages) > 0x00000036052000 - 0x0000003cb2efff, 112054272 bytes (27357 pages) > 0x0000003cb36000 - 0x0000003cb36fff, 4096 bytes (1 pages) > 0x0000003cb38000 - 0x0000003cb39fff, 8192 bytes (2 pages) > 0x0000003cb3b000 - 0x0000003cb3cfff, 8192 bytes (2 pages) > 0x0000003cb40000 - 0x0000003cb40fff, 4096 bytes (1 pages) > 0x0000003cb42000 - 0x0000003cb43fff, 8192 bytes (2 pages) > 0x0000003cb45000 - 0x0000003df4ffff, 21016576 bytes (5131 pages) > 0x0000003df60000 - 0x0000003dffffff, 655360 bytes (160 pages) > 0x00000040000000 - 0x000000fbffffff, 3154116608 bytes (770048 pages) > 0x00000100000000 - 0x000001f372afff, 4084379648 bytes (997163 pages) >=20 >=20 > FYI, before the huge-file duplicate-and-diff test: >=20 > # sysctl hw.busdma > hw.busdma.zone1.alignment: 4096 > hw.busdma.zone1.lowaddr: 0x3fffffff > hw.busdma.zone1.total_deferred: 0 > hw.busdma.zone1.total_bounced: 866 > hw.busdma.zone1.active_bpages: 2 > hw.busdma.zone1.reserved_bpages: 0 > hw.busdma.zone1.free_bpages: 836 > hw.busdma.zone1.total_bpages: 838 > hw.busdma.zone0.alignment: 4096 > hw.busdma.zone0.lowaddr: 0xffffffff > hw.busdma.zone0.total_deferred: 0 > hw.busdma.zone0.total_bounced: 0 > hw.busdma.zone0.active_bpages: 256 > hw.busdma.zone0.reserved_bpages: 0 > hw.busdma.zone0.free_bpages: 257 > hw.busdma.zone0.total_bpages: 513 > hw.busdma.total_bpages: 1351 >=20 > After the duplicate but before the diff: >=20 > # sysctl hw.busdma > hw.busdma.zone1.alignment: 4096 > hw.busdma.zone1.lowaddr: 0x3fffffff > hw.busdma.zone1.total_deferred: 0 > hw.busdma.zone1.total_bounced: 513604 > hw.busdma.zone1.active_bpages: 8 > hw.busdma.zone1.reserved_bpages: 0 > hw.busdma.zone1.free_bpages: 830 > hw.busdma.zone1.total_bpages: 838 > hw.busdma.zone0.alignment: 4096 > hw.busdma.zone0.lowaddr: 0xffffffff > hw.busdma.zone0.total_deferred: 0 > hw.busdma.zone0.total_bounced: 0 > hw.busdma.zone0.active_bpages: 256 > hw.busdma.zone0.reserved_bpages: 0 > hw.busdma.zone0.free_bpages: 257 > hw.busdma.zone0.total_bpages: 513 > hw.busdma.total_bpages: 1351 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)