From owner-svn-src-head@freebsd.org Sat Sep 19 15:22:05 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A03BE3E4439; Sat, 19 Sep 2020 15:22:05 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BtvZP3nM2z415W; Sat, 19 Sep 2020 15:22:05 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 66674DD5A; Sat, 19 Sep 2020 15:22:05 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 08JFM50q076058; Sat, 19 Sep 2020 15:22:05 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 08JFM5ix076057; Sat, 19 Sep 2020 15:22:05 GMT (envelope-from markj@FreeBSD.org) Message-Id: <202009191522.08JFM5ix076057@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Sat, 19 Sep 2020 15:22:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r365906 - head/sys/amd64/amd64 X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: head/sys/amd64/amd64 X-SVN-Commit-Revision: 365906 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Sep 2020 15:22:05 -0000 Author: markj Date: Sat Sep 19 15:22:04 2020 New Revision: 365906 URL: https://svnweb.freebsd.org/changeset/base/365906 Log: Fix some nits in 1G page support in the amd64 pmap. - Move assertions out of the main loop to avoid duplicate conditional expressions, and improve assertion messages. - Fix va_next updates. In some cases we were not doing the wraparound check before continuing the loop. - Use the right va_next. In pmap_advise() and pmap_copy() we would step through 1G pages 2M at a time. - Copy 1G mappings in pmap_copy(). Reviewed by: alc, kib MFC with: r365518 Sponsored by: Juniper Networks, Inc., Klara, Inc. Differential Revision: https://reviews.freebsd.org/D26463 Modified: head/sys/amd64/amd64/pmap.c Modified: head/sys/amd64/amd64/pmap.c ============================================================================== --- head/sys/amd64/amd64/pmap.c Sat Sep 19 15:10:17 2020 (r365905) +++ head/sys/amd64/amd64/pmap.c Sat Sep 19 15:22:04 2020 (r365906) @@ -5935,22 +5935,20 @@ pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t continue; } - pdpe = pmap_pml4e_to_pdpe(pml4e, sva); va_next = (sva + NBPDP) & ~PDPMASK; - if ((*pdpe & PG_V) == 0) { - if (va_next < sva) - va_next = eva; + if (va_next < sva) + va_next = eva; + pdpe = pmap_pml4e_to_pdpe(pml4e, sva); + if ((*pdpe & PG_V) == 0) continue; - } - - KASSERT((*pdpe & PG_PS) == 0 || va_next <= eva, - ("pmap_remove of non-transient 1G page " - "pdpe %#lx sva %#lx eva %#lx va_next %#lx", - *pdpe, sva, eva, va_next)); if ((*pdpe & PG_PS) != 0) { + KASSERT(va_next <= eva, + ("partial update of non-transparent 1G mapping " + "pdpe %#lx sva %#lx eva %#lx va_next %#lx", + *pdpe, sva, eva, va_next)); MPASS(pmap != kernel_pmap); /* XXXKIB */ MPASS((*pdpe & (PG_MANAGED | PG_G)) == 0); - anyvalid = 1; + anyvalid = 1; *pdpe = 0; pmap_resident_count_dec(pmap, NBPDP / PAGE_SIZE); mt = PHYS_TO_VM_PAGE(*pmap_pml4e(pmap, sva) & PG_FRAME); @@ -6228,19 +6226,17 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t continue; } - pdpe = pmap_pml4e_to_pdpe(pml4e, sva); va_next = (sva + NBPDP) & ~PDPMASK; - if ((*pdpe & PG_V) == 0) { - if (va_next < sva) - va_next = eva; + if (va_next < sva) + va_next = eva; + pdpe = pmap_pml4e_to_pdpe(pml4e, sva); + if ((*pdpe & PG_V) == 0) continue; - } - - KASSERT((*pdpe & PG_PS) == 0 || va_next <= eva, - ("pmap_remove of non-transient 1G page " - "pdpe %#lx sva %#lx eva %#lx va_next %#lx", - *pdpe, sva, eva, va_next)); if ((*pdpe & PG_PS) != 0) { + KASSERT(va_next <= eva, + ("partial update of non-transparent 1G mapping " + "pdpe %#lx sva %#lx eva %#lx va_next %#lx", + *pdpe, sva, eva, va_next)); retry_pdpe: obits = pbits = *pdpe; MPASS((pbits & (PG_MANAGED | PG_G)) == 0); @@ -7373,17 +7369,18 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t va_next = eva; continue; } - pdpe = pmap_pml4e_to_pdpe(pml4e, sva); + va_next = (sva + NBPDP) & ~PDPMASK; if (va_next < sva) va_next = eva; + pdpe = pmap_pml4e_to_pdpe(pml4e, sva); if ((*pdpe & PG_V) == 0) continue; - KASSERT((*pdpe & PG_PS) == 0 || va_next <= eva, - ("pmap_unwire of non-transient 1G page " - "pdpe %#lx sva %#lx eva %#lx va_next %#lx", - *pdpe, sva, eva, va_next)); if ((*pdpe & PG_PS) != 0) { + KASSERT(va_next <= eva, + ("partial update of non-transparent 1G mapping " + "pdpe %#lx sva %#lx eva %#lx va_next %#lx", + *pdpe, sva, eva, va_next)); MPASS(pmap != kernel_pmap); /* XXXKIB */ MPASS((*pdpe & (PG_MANAGED | PG_G)) == 0); atomic_clear_long(pdpe, PG_W); @@ -7498,21 +7495,42 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_ continue; } + va_next = (addr + NBPDP) & ~PDPMASK; + if (va_next < addr) + va_next = end_addr; pdpe = pmap_pml4e_to_pdpe(pml4e, addr); - if ((*pdpe & PG_V) == 0) { - va_next = (addr + NBPDP) & ~PDPMASK; - if (va_next < addr) - va_next = end_addr; + if ((*pdpe & PG_V) == 0) continue; + if ((*pdpe & PG_PS) != 0) { + KASSERT(va_next <= end_addr, + ("partial update of non-transparent 1G mapping " + "pdpe %#lx sva %#lx eva %#lx va_next %#lx", + *pdpe, addr, end_addr, va_next)); + MPASS((addr & PDPMASK) == 0); + MPASS((*pdpe & PG_MANAGED) == 0); + srcptepaddr = *pdpe; + pdpe = pmap_pdpe(dst_pmap, addr); + if (pdpe == NULL) { + if (_pmap_allocpte(dst_pmap, + pmap_pml4e_pindex(addr), NULL, addr) == + NULL) + break; + pdpe = pmap_pdpe(dst_pmap, addr); + } else { + pml4e = pmap_pml4e(dst_pmap, addr); + dst_pdpg = PHYS_TO_VM_PAGE(*pml4e & PG_FRAME); + dst_pdpg->ref_count++; + } + KASSERT(*pdpe == 0, + ("1G mapping present in dst pmap " + "pdpe %#lx sva %#lx eva %#lx va_next %#lx", + *pdpe, addr, end_addr, va_next)); + *pdpe = srcptepaddr & ~PG_W; + pmap_resident_count_inc(dst_pmap, NBPDP / PAGE_SIZE); + continue; } va_next = (addr + NBPDR) & ~PDRMASK; - KASSERT((*pdpe & PG_PS) == 0 || va_next <= end_addr, - ("pmap_copy of partial non-transient 1G page " - "pdpe %#lx sva %#lx eva %#lx va_next %#lx", - *pdpe, addr, end_addr, va_next)); - if ((*pdpe & PG_PS) != 0) - continue; if (va_next < addr) va_next = end_addr; @@ -8559,22 +8577,24 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t va_next = eva; continue; } + + va_next = (sva + NBPDP) & ~PDPMASK; + if (va_next < sva) + va_next = eva; pdpe = pmap_pml4e_to_pdpe(pml4e, sva); - if ((*pdpe & PG_V) == 0) { - va_next = (sva + NBPDP) & ~PDPMASK; - if (va_next < sva) - va_next = eva; + if ((*pdpe & PG_V) == 0) continue; + if ((*pdpe & PG_PS) != 0) { + KASSERT(va_next <= eva, + ("partial update of non-transparent 1G mapping " + "pdpe %#lx sva %#lx eva %#lx va_next %#lx", + *pdpe, sva, eva, va_next)); + continue; } + va_next = (sva + NBPDR) & ~PDRMASK; if (va_next < sva) va_next = eva; - KASSERT((*pdpe & PG_PS) == 0 || va_next <= eva, - ("pmap_advise of non-transient 1G page " - "pdpe %#lx sva %#lx eva %#lx va_next %#lx", - *pdpe, sva, eva, va_next)); - if ((*pdpe & PG_PS) != 0) - continue; pde = pmap_pdpe_to_pde(pdpe, sva); oldpde = *pde; if ((oldpde & PG_V) == 0)