From owner-freebsd-current@FreeBSD.ORG  Tue Apr 24 19:13:52 2012
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 34ABC1065676;
	Tue, 24 Apr 2012 19:13:52 +0000 (UTC) (envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
	by mx1.freebsd.org (Postfix) with ESMTP id E190F8FC0A;
	Tue, 24 Apr 2012 19:13:51 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD))
	(envelope-from <slw@zxy.spb.ru>)
	id 1SMlBn-0008db-UK; Tue, 24 Apr 2012 23:14:15 +0400
Date: Tue, 24 Apr 2012 23:14:15 +0400
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: freebsd-current@freebsd.org, freebsd-performance@freebsd.org
Message-ID: <20120424191415.GE76983@zxy.spb.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: 
Subject: pmap and mtx scalability problem
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 19:13:52 -0000

I treid make -j 30 build{world,kernel} (latest -CURRENT) on 24-core machine and see poor
scalability of pmap/mtx -- more then 50% cpu spend on system time.

pmcstat:

@ CPU_CLK_UNHALTED_CORE [194841 samples]

42.65%  [83102]    _mtx_lock_sleep @ /boot/kernel/kernel
 40.97%  [34051]     pmap_enter
  100.0%  [34051]      vm_fault_hold
   100.0%  [34051]       trap_pfault
 30.40%  [25262]     vm_page_activate
  100.0%  [25262]      vm_fault_hold
   100.0%  [25262]       trap_pfault
 18.41%  [15300]     vm_pageq_remove
  73.63%  [11266]      vm_page_free_toq
   70.69%  [7964]        vm_object_terminate
   29.31%  [3302]        vm_object_page_remove
  26.37%  [4034]       vm_fault_hold
   100.0%  [4034]        trap_pfault

make -j 8:

15.44%  [10740]    _mtx_lock_sleep @ /boot/kernel/kernel
 38.77%  [4164]      pmap_enter
  99.93%  [4161]       vm_fault_hold
   100.0%  [4161]        trap_pfault
  00.07%  [3]          kmem_back
   100.0%  [3]           kmem_malloc
 27.98%  [3005]      vm_page_activate
  100.0%  [3005]       vm_fault_hold
   100.0%  [3005]        trap_pfault
 20.64%  [2217]      vm_pageq_remove
  64.73%  [1435]       vm_page_free_toq
   63.41%  [910]         vm_object_terminate

make -j 4

06.86%  [4222]     pagezero @ /boot/kernel/kernel
 98.39%  [4154]      trap_pfault
  100.0%  [4154]       trap
 01.11%  [47]        vm_fault_hold