From owner-svn-src-all@FreeBSD.ORG Fri May 15 07:57:48 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 48E1E518; Fri, 15 May 2015 07:57:48 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 377BA162A; Fri, 15 May 2015 07:57:48 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id t4F7vm9O078110; Fri, 15 May 2015 07:57:48 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by svn.freebsd.org (8.14.9/8.14.9/Submit) id t4F7vm4r078109; Fri, 15 May 2015 07:57:48 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201505150757.t4F7vm4r078109@svn.freebsd.org> X-Authentication-Warning: svn.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Fri, 15 May 2015 07:57:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r282945 - head/sys/amd64/amd64 X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 May 2015 07:57:48 -0000 Author: kib Date: Fri May 15 07:57:47 2015 New Revision: 282945 URL: https://svnweb.freebsd.org/changeset/base/282945 Log: Implement the support for PCID in UP kernels. Requested by: alc Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 3 weeks Modified: head/sys/amd64/amd64/pmap.c Modified: head/sys/amd64/amd64/pmap.c ============================================================================== --- head/sys/amd64/amd64/pmap.c Fri May 15 07:54:31 2015 (r282944) +++ head/sys/amd64/amd64/pmap.c Fri May 15 07:57:47 2015 (r282945) @@ -909,11 +909,7 @@ pmap_bootstrap(vm_paddr_t *firstaddr) * pc_pcid_next and pc_pcid_gen are initialized by AP * during pcpu setup. */ -#ifdef SMP load_cr4(rcr4() | CR4_PCIDE); -#else - pmap_pcid_enabled = 0; -#endif } else { pmap_pcid_enabled = 0; } @@ -1559,61 +1555,79 @@ pmap_update_pde(pmap_t pmap, vm_offset_t #else /* !SMP */ /* * Normal, non-SMP, invalidation functions. - * We inline these within pmap.c for speed. */ -PMAP_INLINE void +void pmap_invalidate_page(pmap_t pmap, vm_offset_t va) { - switch (pmap->pm_type) { - case PT_X86: - if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) - invlpg(va); - break; - case PT_RVI: - case PT_EPT: + if (pmap->pm_type == PT_RVI || pmap->pm_type == PT_EPT) { pmap->pm_eptgen++; - break; - default: - panic("pmap_invalidate_page: unknown type: %d", pmap->pm_type); + return; } + KASSERT(pmap->pm_type == PT_X86, + ("pmap_invalidate_range: unknown type %d", pmap->pm_type)); + + if (pmap == kernel_pmap || pmap == PCPU_GET(curpmap)) + invlpg(va); + else if (pmap_pcid_enabled) + pmap->pm_pcids[0].pm_gen = 0; } -PMAP_INLINE void +void pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { vm_offset_t addr; - switch (pmap->pm_type) { - case PT_X86: - if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) - for (addr = sva; addr < eva; addr += PAGE_SIZE) - invlpg(addr); - break; - case PT_RVI: - case PT_EPT: + if (pmap->pm_type == PT_RVI || pmap->pm_type == PT_EPT) { pmap->pm_eptgen++; - break; - default: - panic("pmap_invalidate_range: unknown type: %d", pmap->pm_type); + return; + } + KASSERT(pmap->pm_type == PT_X86, + ("pmap_invalidate_range: unknown type %d", pmap->pm_type)); + + if (pmap == kernel_pmap || pmap == PCPU_GET(curpmap)) { + for (addr = sva; addr < eva; addr += PAGE_SIZE) + invlpg(addr); + } else if (pmap_pcid_enabled) { + pmap->pm_pcids[0].pm_gen = 0; } } -PMAP_INLINE void +void pmap_invalidate_all(pmap_t pmap) { + struct invpcid_descr d; - switch (pmap->pm_type) { - case PT_X86: - if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) - invltlb(); - break; - case PT_RVI: - case PT_EPT: + if (pmap->pm_type == PT_RVI || pmap->pm_type == PT_EPT) { pmap->pm_eptgen++; - break; - default: - panic("pmap_invalidate_all: unknown type %d", pmap->pm_type); + return; + } + KASSERT(pmap->pm_type == PT_X86, + ("pmap_invalidate_all: unknown type %d", pmap->pm_type)); + + if (pmap == kernel_pmap) { + if (pmap_pcid_enabled && invpcid_works) { + bzero(&d, sizeof(d)); + invpcid(&d, INVPCID_CTXGLOB); + } else { + invltlb_globpcid(); + } + } else if (pmap == PCPU_GET(curpmap)) { + if (pmap_pcid_enabled) { + if (invpcid_works) { + d.pcid = pmap->pm_pcids[0].pm_pcid; + d.pad = 0; + d.addr = 0; + invpcid(&d, INVPCID_CTX); + } else { + load_cr3(pmap->pm_cr3 | pmap->pm_pcids[0]. + pm_pcid); + } + } else { + invltlb(); + } + } else if (pmap_pcid_enabled) { + pmap->pm_pcids[0].pm_gen = 0; } } @@ -1629,10 +1643,10 @@ pmap_update_pde(pmap_t pmap, vm_offset_t { pmap_update_pde_store(pmap, pde, newpde); - if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) + if (pmap == kernel_pmap || pmap == PCPU_GET(curpmap)) pmap_update_pde_invalidate(pmap, va, newpde); else - CPU_ZERO(&pmap->pm_save); + pmap->pm_pcids[0].pm_gen = 0; } #endif /* !SMP */