From owner-freebsd-virtualization@FreeBSD.ORG Sun May 18 14:44:32 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B2C338F for ; Sun, 18 May 2014 14:44:32 +0000 (UTC) Received: from mail-wg0-x234.google.com (mail-wg0-x234.google.com [IPv6:2a00:1450:400c:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ACBBA2FC7 for ; Sun, 18 May 2014 14:44:31 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id l18so6713423wgh.23 for ; Sun, 18 May 2014 07:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Sm5Ofj8N2Tc0l33Xg+LzjNGy22KAZR04VJk0h8k/n7M=; b=VQ/arsrieqTUUXl8PEVgPTm5W9JMnoFJWUiWc90YKd9GRRDVDRsI0UGv5ezxUbi6J0 C9/Hx7RCLBLAglwhsDmeQSc7yvOFxI8k/cunXfF+6akH/niwThx9Yb4iY8aOVtOOpjTd btDWSgdD0XnizVhmjYX2cvZEQHMe4jQSaF9k/D1Tq2P+GHBYTvCno/nUP6j4ek03BKK5 +8R6DDEQC+Me0B0DXmTcxr7miVI+m8uSentgGOjWiIWsmLPqXPbPwWoi9ra4qVENtXHF dARNk5W4poU/iBHuyC7qrLhYK4SUsz6kuNrKDm64QAVExSuItRd9jT1lXlh1KcqG0LKt IDYg== MIME-Version: 1.0 X-Received: by 10.180.82.7 with SMTP id e7mr8076641wiy.6.1400424269886; Sun, 18 May 2014 07:44:29 -0700 (PDT) Received: by 10.194.135.49 with HTTP; Sun, 18 May 2014 07:44:29 -0700 (PDT) In-Reply-To: <045ce77ed17da4bd515bcc3cafe9c7f8@webmail.renzel.net.local> References: <045ce77ed17da4bd515bcc3cafe9c7f8@webmail.renzel.net.local> Date: Sun, 18 May 2014 07:44:29 -0700 Message-ID: Subject: Re: bhyve: svm (amd-v) update From: Anish To: Nils Beyer Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 May 2014 14:44:32 -0000 Thanks for testing it. >Your patch applied cleanly to the working copy of the "bhyve_svm"-project. I was then able to merge with HEAD (using "theirs-full" on one file) and compile the kernel. So, to me it looks OK to commit. Yes, that's correct. You have to retain changes in sys/amd64/vmm/amd/amdv.c from bhyve_svm branch. >Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom 1055T. It produces 200% load on the host CPU, and the emulated machine generates endlessly: Its 200% load because of 2 vcpus to guest. It stuck in loop even with single processor(1 vcpu) after PCI probing[debug messages with linux .....earlyprintk=serial debug] [ 3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 3.691987] NET: Registered protocol family 1 [ 3.693382] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 3.695214] PCI: CLS 64 bytes, default 64 [ 3.698176] Trying to unpack rootfs image as initramfs... [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] [ 3.505631] pnp: PnP ACPI: found 5 devices [ 3.506417] ACPI: bus type PNP unregistered [ 3.635781] pci 0000:00:06.0: no compatible bridge window for [mem 0xfe440000 -0xfe45ffff pref] [ 3.637555] pci 0000:00:06.0: BAR 6: assigned [mem 0x80000000-0x8001ffff pref ] [ 3.638986] pci 0000:00:01.0: BAR 6: assigned [mem 0x80020000-0x800207ff pref ] [ 3.640416] pci 0000:00:04.0: BAR 6: assigned [mem 0x80020800-0x80020fff pref ] [ 3.641864] pci 0000:00:05.0: BAR 6: assigned [mem 0x80021000-0x800217ff pref ] [ 3.643259] pci 0000:00:00.0: not setting up bridge for bus 0000:01 [ 3.644550] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7] [ 3.645670] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff] [ 3.646795] pci_bus 0000:00: resource 6 [mem 0x80000000-0xdfffffff] [ 3.648031] pci_bus 0000:00: resource 7 [mem 0xd000000000-0xfcffffffff] [ 3.650970] NET: Registered protocol family 2 [ 3.661491] TCP established hash table entries: 16384 (order: 6, 262144 bytes ) [ 3.671854] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 3.681116] TCP: Hash tables configured (established 16384 bind 16384) [ 3.683335] TCP: reno registered [ 3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 3.691987] NET: Registered protocol family 1 [ 3.693382] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 3.695214] PCI: CLS 64 bytes, default 64 [ 3.698176] Trying to unpack rootfs image as initramfs... [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] [ 30.596366] Modules linked in: >Additionally, It produces a lot of MSR requests: Yes, on AMD Linux is touching more MSRs( AMD specific -address 0xC00XXXX) compared to Intel. Thanks and regards, Anish On Fri, May 16, 2014 at 2:17 PM, Nils Beyer wrote: > Hi Anish, > > Anish wrote: > > If patches looks good to you, we can submit it. I have been testing it on > > Phenom box which lacks some of newer SVM features. > > Your patch applied cleanly to the working copy of the "bhyve_svm"-project. > I was then able to merge with HEAD > (using "theirs-full" on one file) and compile the kernel. So, to me it > looks OK to commit. > > Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom > 1055T. It produces 200% load on the > host CPU, and the emulated machine generates endlessly: > > ======================================================================================= > BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1] > Modules linked in: > CPU 0 > Modules linked in: > > Pid: 1, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 BHYVE > RIP: 0010:[] [] rc_is_bit_0+0x3a/0x69 > RSP: 0018:ffff88013e79dca0 EFLAGS: 00000a96 > RAX: 000000000000009c RBX: ffff88013e79dcc0 RCX: ffff880004bdcc7c > RDX: 00000000002f9dee RSI: ffffc9000402c538 RDI: ffff88013e79ddb0 > RBP: ffffffff8100b9ce R08: ffffc9000402c788 R09: ffffffff81de32b8 > R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000003 > R13: ffffffff81157602 R14: ffff88013e79dc20 R15: 00000000000000d2 > FS: 0000000000000000(0000) GS:ffff880028200000(0000) > knlGS:0000000000000000 > CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > CR2: 0000000000000000 CR3: 0000000001a85000 CR4: 00000000000007b0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process swapper (pid: 1, threadinfo ffff88013e79c000, task > ffff88013e79b500) > Stack: > ffffc9000402c644 00000000000000e5 00000000000003b1 ffff88013e79ddb0 > ffff88013e79dcf0 ffffffff81c549b7 ffffffffffffffb6 0000000000000000 > 0000000000000000 ffffc9000402c000 ffff88013e79de30 ffffffff81c554e2 > Call Trace: > [] ? rc_get_bit+0x1b/0x79 > [] ? unlzma+0xa42/0xc67 > [] ? flush_buffer+0x0/0xa3 > [] ? do_utimes+0xdb/0x170 > [] ? nofill+0x0/0x10 > [] ? unpack_to_rootfs+0x167/0x27a > [] ? error+0x0/0x17 > [] ? pci_get_subsys+0x35/0x40 > [] ? populate_rootfs+0x0/0xd3 > [] ? populate_rootfs+0x5b/0xd3 > [] ? do_one_initcall+0x3c/0x1d0 > [] ? kernel_init+0x29b/0x2f7 > [] ? child_rip+0xa/0x20 > [] ? kernel_init+0x0/0x2f7 > [] ? child_rip+0x0/0x20 > Code: ff ff ff 00 77 35 48 8b 47 18 48 39 47 08 72 0d 48 89 75 e8 e8 95 ff > ff ff 48 8b 75 e8 48 8b 4b 08 c1 63 28 08 8b 53 24 0f b6 01 <48> 83 c1 01 > c1 e2 08 48 89 4b 08 09 d0 89 43 24 0f b7 06 8b 53 > Call Trace: > [] ? rc_get_bit+0x1b/0x79 > [] ? unlzma+0xa42/0xc67 > [] ? flush_buffer+0x0/0xa3 > [] ? do_utimes+0xdb/0x170 > [] ? nofill+0x0/0x10 > [] ? unpack_to_rootfs+0x167/0x27a > [] ? error+0x0/0x17 > [] ? pci_get_subsys+0x35/0x40 > [] ? populate_rootfs+0x0/0xd3 > [] ? populate_rootfs+0x5b/0xd3 > [] ? do_one_initcall+0x3c/0x1d0 > [] ? kernel_init+0x29b/0x2f7 > [] ? child_rip+0xa/0x20 > [] ? kernel_init+0x0/0x2f7 > [] ? child_rip+0x0/0x20 > > ======================================================================================= > > > Additionally, It produces a lot of MSR requests: > > ======================================================================================= > May 16 09:32:03 10.255.255.96 kernel: emulate_rdmsr 0xc0010015 > May 16 09:32:18 10.255.255.96 kernel: emulate_rdmsr 0x1b > May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0xc0010112 > May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0xc0010048 > May 16 09:32:23 10.255.255.96 kernel: emulate_wrmsr 0xc0010048 > May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0x8b > May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0xc0010140 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010000 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010001 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010002 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010003 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010004 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010004 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010000 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0x1b > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010048 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010048 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0x8b > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010000 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:25 10.255.255.96 last message repeated 314 times > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0x1b > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010048 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010048 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0x8b > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:25 10.255.255.96 last message repeated 49 times > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 > May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010000 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:25 10.255.255.96 last message repeated 885 times > May 16 09:32:25 10.255.255.96 kernel: 010055 > May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:25 10.255.255.96 last message repeated 4820 times > May 16 09:32:26 10.255.255.96 kernel: 010055 > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:26 10.255.255.96 last message repeated 4364 times > May 16 09:32:26 10.255.255.96 kernel: 010055 > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:26 10.255.255.96 last message repeated 25 times > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc001001f > May 16 09:32:26 10.255.255.96 kernel: emulate_wrmsr 0xc001001f > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc001001f > May 16 09:32:26 10.255.255.96 kernel: emulate_wrmsr 0xc001001f > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:26 10.255.255.96 last message repeated 391 times > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc001001f > May 16 09:32:26 10.255.255.96 kernel: emulate_wrmsr 0xc001001f > May 16 09:32:26 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:34 10.255.255.96 last message repeated 73074 times > May 16 09:32:34 10.255.255.96 kernel: 010055 > May 16 09:32:34 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:35 10.255.255.96 last message repeated 14648 times > May 16 09:32:35 10.255.255.96 kernel: 010055 > May 16 09:32:35 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:35 10.255.255.96 last message repeated 8098 times > May 16 09:32:36 10.255.255.96 kernel: 010055 > May 16 09:32:36 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:36 10.255.255.96 last message repeated 7895 times > May 16 09:32:36 10.255.255.96 kernel: 010055 > May 16 09:32:36 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:36 10.255.255.96 last message repeated 8272 times > May 16 09:32:36 10.255.255.96 kernel: 010055 > May 16 09:32:36 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:37 10.255.255.96 last message repeated 8696 times > May 16 09:32:37 10.255.255.96 kernel: 010055 > May 16 09:32:37 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:37 10.255.255.96 last message repeated 12333 times > May 16 09:32:38 10.255.255.96 kernel: 010055 > May 16 09:32:38 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:41 10.255.255.96 last message repeated 30370 times > May 16 09:32:41 10.255.255.96 kernel: emulate_rdmsr 0xc001103a > May 16 09:32:41 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:49 10.255.255.96 last message repeated 85577 times > May 16 09:32:49 10.255.255.96 kernel: 010055 > May 16 09:32:49 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:56 10.255.255.96 last message repeated 9534 times > May 16 09:32:57 10.255.255.96 kernel: 010055 > May 16 09:32:57 10.255.255.96 kernel: emulate_rdmsr 0xc0010055 > May 16 09:32:58 10.255.255.96 last message repeated 17524 times > [and so on] > > ======================================================================================= > > I'd love to see CentOS perfectly running on my Phenom as it runs perfectly > on an Intel i3. > > If you need any further information/debug, please let me know... > > > > TIA and regards, > Nils > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" >