From owner-freebsd-virtualization@FreeBSD.ORG Thu Jul 17 18:15:46 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 872019B4 for ; Thu, 17 Jul 2014 18:15:46 +0000 (UTC) Received: from mail-vc0-x22e.google.com (mail-vc0-x22e.google.com [IPv6:2607:f8b0:400c:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46AF2217B for ; Thu, 17 Jul 2014 18:15:46 +0000 (UTC) Received: by mail-vc0-f174.google.com with SMTP id la4so5369366vcb.19 for ; Thu, 17 Jul 2014 11:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=NY/ZZwgB4eox7nUlGS6PIjYn+Y2Q+lbEoE0JTV/OfDI=; b=fTRKod9b0do86QJt/4rVOFLHuM9aizBNnEcceWNIpWUKrPYDYZZmgRYzaKEHyBLMVH ciD/byo0sl3B/SJ1WX+BdcIpvqc3WnzZlclOQybQrZApMlkSDGA382LUt6UU81XX+U+r 39YtmuvwSYe9Osfvkbcnkexu/AEi/H0s1JBrmej93xgO7WcELLCXlztwe5bN8Oo0j2ez 6K/Y/nP7fbhRPLIO6+CPQN4IZgNJRVBuo3udiSlkuWdcd5EAc2+/fv9worw9c54Kyr2B /nfney9U24yx/SKjQTWvP0GyaO6XcO80jwWDmrESu84wJ9s6NKydVrl9icUkYrmiAkAj xUDA== X-Received: by 10.53.13.200 with SMTP id fa8mr9063676vdd.57.1405620943501; Thu, 17 Jul 2014 11:15:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.221.61.131 with HTTP; Thu, 17 Jul 2014 11:15:23 -0700 (PDT) From: Prakhar Deep Date: Thu, 17 Jul 2014 23:45:23 +0530 Message-ID: Subject: Panic: 32-bit PAE enabled OS with 4GB RAM and 2 vCPUs on Bhyve To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 18:15:46 -0000 Hi, I am facing kernel panic when 2nd processor is executing it's reset code while running a 32-bit OS as a VM (with PAE enabled) on =E2=80=98bhyve= =E2=80=99 with following parameters: 1. Number of vCPUs 2 or more, and 2. Memory 4GB or more When number of vCPU =3D1 and memory > 4GB i.e. PAE mode with only one CPU o= r vCPU =3D 2 or more and memory < 4GB i.e non-PAE mode then VM comes up successfully. The OS is able to bring up the boot processor but it panics when AP is executing it's reset code. The OS uses fixed kernel virtual addresses which are mapped by a level 2 page table which is replicated for each CPU to allow some of these mappings to be different on a per-CPU basis. These pages act as storage for many global variables which are used when processor is executing reset code. When a second (non-boot) CPU comes up, we are seeing that per-CPU pages for this CPU are reset to all zeroes when the second CPU sets the PGE flag in CR4. This then leads to incorrect code paths. I am not able to understand why the pages mapped to AP are getting set to 0 when PGE flag is set in CR4 as this does not happen for boot processor. If i do not set PGE flag then pages are not set to 0 but the code panics some where else but not due to zeroed out global values. I have tried FreeBSD i386 with PAE support compiled into the kernel on Bhyve with >4GB RAM and 2 vCPUS and VM came up successfully. Also, the PGE bit is set in the CR4 register for FreeBSD. It will be helpful if anyone could throw some light on following points: 1. Why are the pages mapped to AP are being zeroed when PGE flag is set for AP in it's CR4 ? 2. Are there are any changes required in a 32-bit OS to boot it in PAE mode with 4GB or more RAM and 2 or more vCPUS on Bhyve ? 3. Was FreeBSD (i386) kernel updated to boot on Bhyve with 4GB or more RAM and 2 or more vCPUS ? Thanks. Prakhar From owner-freebsd-virtualization@FreeBSD.ORG Fri Jul 18 07:08:07 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 63DA8449 for ; Fri, 18 Jul 2014 07:08:07 +0000 (UTC) Received: from mail-qc0-x230.google.com (mail-qc0-x230.google.com [IPv6:2607:f8b0:400d:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 251E32FC3 for ; Fri, 18 Jul 2014 07:08:07 +0000 (UTC) Received: by mail-qc0-f176.google.com with SMTP id i17so3009662qcy.7 for ; Fri, 18 Jul 2014 00:08:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=dXZizUTDtENRTGu3QY+9t9opobBZ6roVinpTXODKHt8=; b=ici6CWA02f7o96tXauezUtyl9KYVPe7+Xfl3Wm9Qf3hPd87jUjuEcGPyJNCWv8OqlZ ECFaTCa62pxbhHyzXcfL26InsCCp9c6JPD1cmIu93xRf2ZvLvmn2Do4m8C1PKfiMtj+g R+d5fQNJj2mRnLSS1C7bargSy1f6co25akLIKM/ufhKmaxBf8ASI6NCm8s+Dx/o4qu6B Hm2Q8xsGw1lcWuNEhMB/+1+JgY0bYWERv5g4HQ+BUI5WzoaXMVPrs67Nr1ekgv8RY0I2 XShKh/SXssz+p3rCEXWRe7xwdW/OeOC3msgVGLXA7YfHhHjqlsx2dfMcrn8hG4g17suG GNeA== MIME-Version: 1.0 X-Received: by 10.140.87.229 with SMTP id r92mr4416568qgd.43.1405667286216; Fri, 18 Jul 2014 00:08:06 -0700 (PDT) Received: by 10.140.48.37 with HTTP; Fri, 18 Jul 2014 00:08:06 -0700 (PDT) In-Reply-To: References: Date: Fri, 18 Jul 2014 00:08:06 -0700 Message-ID: Subject: Re: Panic: 32-bit PAE enabled OS with 4GB RAM and 2 vCPUs on Bhyve From: Neel Natu To: Prakhar Deep Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jul 2014 07:08:07 -0000 Hi Prakhar, On Thu, Jul 17, 2014 at 11:15 AM, Prakhar Deep wrot= e: > Hi, > > > I am facing kernel panic when 2nd processor is executing it's reset > code while running a 32-bit OS as a VM (with PAE enabled) on =E2=80=98bhy= ve=E2=80=99 with > following parameters: > > 1. Number of vCPUs 2 or more, and > > 2. Memory 4GB or more > > When number of vCPU =3D1 and memory > 4GB i.e. PAE mode with only one CPU= or > vCPU =3D 2 or more and memory < 4GB i.e non-PAE mode then VM comes up > successfully. The OS is able to bring up the boot processor but it panics > when AP is executing it's reset code. > > > The OS uses fixed kernel virtual addresses which are mapped by a level 2 > page table which is replicated for each CPU to allow some of these mappin= gs > to be different on a > per-CPU basis. These pages act as storage for many global variables which > are used when processor is executing reset code. When a second (non-boot) > CPU comes up, we are seeing that per-CPU pages for this CPU are reset to > all zeroes when the second CPU sets the PGE flag in CR4. This then leads = to > incorrect code paths. > > I am not able to understand why the pages mapped to AP are getting set to= 0 > when PGE flag is set in CR4 as this does not happen for boot processor. I= f > i do not set PGE flag then pages are not set to 0 but the code panics som= e > where else but not due to zeroed out global values. > > I have tried FreeBSD i386 with PAE support compiled into the kernel on > Bhyve with >4GB RAM and 2 vCPUS and VM came up successfully. Also, the PG= E > bit is set in the CR4 register for FreeBSD. > > > It will be helpful if anyone could throw some light on following points: > > > > 1. Why are the pages mapped to AP are being zeroed when PGE flag is set f= or > AP in it's CR4 ? > It is hard to say but my guess would be that the 4 PDPTE registers in the VMCS are not being properly updated during AP spinup. It would be helpful to see the assembly instructions in the AP spinup code that modify %cr0, %cr4 and %cr3 to know for sure. > 2. Are there are any changes required in a 32-bit OS to boot it in PAE mo= de > with 4GB or more RAM and 2 or more vCPUS on Bhyve ? > No. > 3. Was FreeBSD (i386) kernel updated to boot on Bhyve with 4GB or more RA= M > and 2 or more vCPUS ? > No. best Neel > Thanks. > > Prakhar > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org"