From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 06:47:05 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9023EB94 for ; Tue, 23 Jun 2015 06:47:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CE7DFAE6 for ; Tue, 23 Jun 2015 06:47:04 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA11222; Tue, 23 Jun 2015 09:46:56 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z7Hyy-0006Ax-Cu; Tue, 23 Jun 2015 09:46:56 +0300 Message-ID: <558900A7.40609@FreeBSD.org> Date: Tue, 23 Jun 2015 09:45:59 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: centos 7.1 with multiple virtual processors References: <5587EE05.2020001@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 06:47:05 -0000 On 23/06/2015 05:37, Neel Natu wrote: > Hi Andriy, > > FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus > fine on my host with 8 physical cores. > > I have some questions about your setup inline. > > On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: >> >> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >> hang on startup and bhyve would start spinning. >> >> The following are the last messages seen in the VM: >> >> Switching to clocksource hpet >> ------------[ cut here ]------------ >> WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() >> Modules linked in: >> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 >> Hardware name: BHYVE, BIOS 1.00 03/14/2014 >> 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa >> ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f >> ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 >> Call Trace: >> [] dump_stack+0x19/0x1b >> [] warn_slowpath_common+0x6b/0xb0 >> [] warn_slowpath_null+0x1a/0x20 >> [] clockevents_program_event+0xdb/0xf0 >> [] tick_handle_periodic_broadcast+0x41/0x50 >> [] timer_interrupt+0x15/0x20 >> [] handle_irq_event_percpu+0x3e/0x1e0 >> [] handle_irq_event+0x3d/0x60 >> [] handle_edge_irq+0x77/0x130 >> [] handle_irq+0xbf/0x150 >> [] ? irq_enter+0x17/0xa0 >> [] do_IRQ+0x4f/0xf0 >> [] common_interrupt+0x6d/0x6d >> [] ? selinux_inode_alloc_security+0x59/0xa0 >> [] ? __d_instantiate+0xbf/0x100 >> [] ? __d_instantiate+0x9f/0x100 >> [] d_instantiate+0x3d/0x70 >> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 >> [] __create_file+0x1c2/0x2c0 >> [] ? set_graph_function+0x1f/0x1f >> [] debugfs_create_dir+0x1b/0x20 >> [] tracing_init_dentry_tr+0x7e/0x90 >> [] tracing_init_dentry+0x10/0x20 >> [] ftrace_init_debugfs+0x13/0x1fd >> [] ? set_graph_function+0x1f/0x1f >> [] do_one_initcall+0xb8/0x230 >> [] kernel_init_freeable+0x18b/0x22a >> [] ? initcall_blacklist+0xb0/0xb0 >> [] ? rest_init+0x80/0x80 >> [] kernel_init+0xe/0xf0 >> [] ret_from_fork+0x7c/0xb0 >> [] ? rest_init+0x80/0x80 >> ---[ end trace d5caa1cab8e7e98d ]--- >> > > A few questions to narrow this down: > - Is the host very busy when the VM is started (or what is the host > doing when this happened)? The host typically is not heavily loaded. There is X server running and some applications. I'd imagine that those could cause some additional latency but not CPU starvation. > - How many vcpus are you giving to the VM? > - How many cores on the host? I tried only 2 / 2. >> >> At the same time sometimes there is one or more of spurious NMIs on the _host_ >> system: >> NMI ISA c, EISA ff >> NMI ... going to debugger >> > > Hmm, that's interesting. Are you using hwpmc to do instruction sampling? hwpmc driver is in the kernel, but it was not used. >> bhyve seems to spin here: >> vmm.ko`svm_vmrun+0x894 >> vmm.ko`vm_run+0xbb7 >> vmm.ko`vmmdev_ioctl+0x5a4 >> kernel`devfs_ioctl_f+0x13b >> kernel`kern_ioctl+0x1e1 >> kernel`sys_ioctl+0x16a >> kernel`amd64_syscall+0x3ca >> kernel`0xffffffff8088997b >> >> (kgdb) list *svm_vmrun+0x894 >> 0xffffffff813c9194 is in svm_vmrun >> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). >> 1890 >> 1891 static __inline void >> 1892 enable_gintr(void) >> 1893 { >> 1894 >> 1895 __asm __volatile("stgi"); >> 1896 } >> 1897 >> 1898 /* >> 1899 * Start vcpu with specified RIP. >> > > Yeah, that's not surprising because host interrupts are blocked when > the cpu is executing in guest context. The 'enable_gintr()' enables > interrupts so it gets blamed by the interrupt-based sampling. > > In this case it just means that the cpu was in guest context when a > host-interrupt fired. I see. FWIW, that was captured with DTrace. -- Andriy Gapon