From owner-freebsd-virtualization@FreeBSD.ORG Wed Apr 1 00:06:27 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 043958DA for ; Wed, 1 Apr 2015 00:06:27 +0000 (UTC) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B29ADDBA for ; Wed, 1 Apr 2015 00:06:26 +0000 (UTC) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 4AFAE280F84 for ; Wed, 1 Apr 2015 10:06:23 +1000 (EST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au X-Virus-Scanned: amavisd-new at iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8JNnR6fgVsn4 for ; Wed, 1 Apr 2015 10:06:23 +1000 (EST) Received: from Peters-MacBook-Pro.local (c-76-126-65-88.hsd1.ca.comcast.net [76.126.65.88]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 858FD280F74; Wed, 1 Apr 2015 10:06:20 +1000 (EST) Message-ID: <551B367B.4020104@freebsd.org> Date: Tue, 31 Mar 2015 17:06:19 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Stefan Andritoiu Subject: Re: Bhyve: Investigating poor guest performance when host is busy References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 00:06:27 -0000 > 1. Can anyone tell me what the cause might be? What may be happening? > 2. Do you know if there is currently any work in investigation this > problem? Or anything related? > 3. Is Gang Scheduling or Coscheduling implemented in FreeBSD? > 4. Do you know of any other solution to this kind of problem? > 5. Can you recommend me any papers/videos/links in anyway related to this? I answered these in the FreeBSD forums post, but reproduced again here for the list: 1. The main issue is 'lock holder preemption', where a vCPU that is holding a spinlock has been pre-empted by the host scheduler, resulting in other vCPUs that are trying to acquire that lock to spin for full quantums. Booting is a variant of this for FreeBSD since the AP spins on a memory location waiting for a BSP to start up. 2. There's some minor investigation going on. 3. No. 4. I don't know that 'classic' gang scheduling is the answer (see 5). What has been thought of for bhyve at least is to a) have the concept of vCPU 'groups' in the scheduler, b) provide metrics to assist the scheduler in trying to spread out threads associated with a vCPU group so they don't end up on the same physical CPU (avoidance of lock-holder preemption), and c) implement pause-loop exits (see the Intel SDM, 24.6.13) in the hypervisor and provide that information to the scheduler so it can give a temporary priority boost to vCPUs that have been preempted but aren't currently running. 5. The classic reference on this is VMWare's scheduler paper: www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf later, Peter.