From owner-svn-src-user@FreeBSD.ORG  Fri Feb 15 14:48:07 2013
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
 by hub.freebsd.org (Postfix) with ESMTP id 2749D856;
 Fri, 15 Feb 2013 14:48:07 +0000 (UTC)
 (envelope-from attilio@FreeBSD.org)
Received: from svn.freebsd.org (svn.freebsd.org
 [IPv6:2001:1900:2254:2068::e6a:0])
 by mx1.freebsd.org (Postfix) with ESMTP id 01806111;
 Fri, 15 Feb 2013 14:48:07 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
 by svn.freebsd.org (8.14.5/8.14.5) with ESMTP id r1FEm6N1083096;
 Fri, 15 Feb 2013 14:48:06 GMT (envelope-from attilio@svn.freebsd.org)
Received: (from attilio@localhost)
 by svn.freebsd.org (8.14.5/8.14.5/Submit) id r1FEm6oW083095;
 Fri, 15 Feb 2013 14:48:06 GMT (envelope-from attilio@svn.freebsd.org)
Message-Id: <201302151448.r1FEm6oW083095@svn.freebsd.org>
From: Attilio Rao <attilio@FreeBSD.org>
Date: Fri, 15 Feb 2013 14:48:06 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r246834 - user/attilio/vmc-playground/sys/vm
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "SVN commit messages for the experimental &quot; user&quot;
 src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
 <mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
 <mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Feb 2013 14:48:07 -0000

Author: attilio
Date: Fri Feb 15 14:48:06 2013
New Revision: 246834
URL: http://svnweb.freebsd.org/changeset/base/246834

Log:
  Improve dynamic branch prediction and i-cache utilization:
  - Use predict_false() to tag boot-time cache decisions
  - Compact boot-time cache allocation into a separate, non-inline,
    function that won't be called most of the times.
  
  Sponsored by:	EMC / Isilon storage division

Modified:
  user/attilio/vmc-playground/sys/vm/vm_radix.c

Modified: user/attilio/vmc-playground/sys/vm/vm_radix.c
==============================================================================
--- user/attilio/vmc-playground/sys/vm/vm_radix.c	Fri Feb 15 14:22:23 2013	(r246833)
+++ user/attilio/vmc-playground/sys/vm/vm_radix.c	Fri Feb 15 14:48:06 2013	(r246834)
@@ -128,6 +128,18 @@ vm_radix_node_zone_dtor(void *mem, int s
 }
 #endif
 
+static struct vm_radix_node *
+vm_radix_carve_bootcache(void)
+{
+	struct vm_radix_node *rnode;
+
+	if (boot_cache_cnt == VM_RADIX_BOOT_CACHE)
+		panic("%s: Increase VM_RADIX_BOOT_CACHE", __func__);
+	rnode = &boot_cache[boot_cache_cnt];
+	boot_cache_cnt++;
+	return (rnode);
+}
+
 /*
  * Allocate a radix node.  Pre-allocation ensures that the request will be
  * always successfully satisfied.
@@ -137,12 +149,9 @@ vm_radix_node_get(vm_pindex_t owner, uin
 {
 	struct vm_radix_node *rnode;
 
-	if (boot_cache_cnt <= VM_RADIX_BOOT_CACHE) {
-		if (boot_cache_cnt == VM_RADIX_BOOT_CACHE)
-			panic("%s: Increase VM_RADIX_BOOT_CACHE", __func__);
-		rnode = &boot_cache[boot_cache_cnt];
-		boot_cache_cnt++;
-	} else {
+	if (__predict_false(boot_cache_cnt <= VM_RADIX_BOOT_CACHE))
+		rnode = vm_radix_carve_bootcache();
+	else {
 		rnode = uma_zalloc(vm_radix_node_zone, M_NOWAIT | M_ZERO);
 
 		/*
@@ -173,7 +182,8 @@ static __inline void
 vm_radix_node_put(struct vm_radix_node *rnode)
 {
 
-	if (rnode > boot_cache && rnode <= &boot_cache[VM_RADIX_BOOT_CACHE])
+	if (__predict_false(rnode > boot_cache &&
+	    rnode <= &boot_cache[VM_RADIX_BOOT_CACHE]))
 		return;
 	uma_zfree(vm_radix_node_zone, rnode);
 }