From owner-freebsd-virtualization@FreeBSD.ORG Fri Jul 18 07:08:07 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 63DA8449 for ; Fri, 18 Jul 2014 07:08:07 +0000 (UTC) Received: from mail-qc0-x230.google.com (mail-qc0-x230.google.com [IPv6:2607:f8b0:400d:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 251E32FC3 for ; Fri, 18 Jul 2014 07:08:07 +0000 (UTC) Received: by mail-qc0-f176.google.com with SMTP id i17so3009662qcy.7 for ; Fri, 18 Jul 2014 00:08:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=dXZizUTDtENRTGu3QY+9t9opobBZ6roVinpTXODKHt8=; b=ici6CWA02f7o96tXauezUtyl9KYVPe7+Xfl3Wm9Qf3hPd87jUjuEcGPyJNCWv8OqlZ ECFaTCa62pxbhHyzXcfL26InsCCp9c6JPD1cmIu93xRf2ZvLvmn2Do4m8C1PKfiMtj+g R+d5fQNJj2mRnLSS1C7bargSy1f6co25akLIKM/ufhKmaxBf8ASI6NCm8s+Dx/o4qu6B Hm2Q8xsGw1lcWuNEhMB/+1+JgY0bYWERv5g4HQ+BUI5WzoaXMVPrs67Nr1ekgv8RY0I2 XShKh/SXssz+p3rCEXWRe7xwdW/OeOC3msgVGLXA7YfHhHjqlsx2dfMcrn8hG4g17suG GNeA== MIME-Version: 1.0 X-Received: by 10.140.87.229 with SMTP id r92mr4416568qgd.43.1405667286216; Fri, 18 Jul 2014 00:08:06 -0700 (PDT) Received: by 10.140.48.37 with HTTP; Fri, 18 Jul 2014 00:08:06 -0700 (PDT) In-Reply-To: References: Date: Fri, 18 Jul 2014 00:08:06 -0700 Message-ID: Subject: Re: Panic: 32-bit PAE enabled OS with 4GB RAM and 2 vCPUs on Bhyve From: Neel Natu To: Prakhar Deep Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jul 2014 07:08:07 -0000 Hi Prakhar, On Thu, Jul 17, 2014 at 11:15 AM, Prakhar Deep wrot= e: > Hi, > > > I am facing kernel panic when 2nd processor is executing it's reset > code while running a 32-bit OS as a VM (with PAE enabled) on =E2=80=98bhy= ve=E2=80=99 with > following parameters: > > 1. Number of vCPUs 2 or more, and > > 2. Memory 4GB or more > > When number of vCPU =3D1 and memory > 4GB i.e. PAE mode with only one CPU= or > vCPU =3D 2 or more and memory < 4GB i.e non-PAE mode then VM comes up > successfully. The OS is able to bring up the boot processor but it panics > when AP is executing it's reset code. > > > The OS uses fixed kernel virtual addresses which are mapped by a level 2 > page table which is replicated for each CPU to allow some of these mappin= gs > to be different on a > per-CPU basis. These pages act as storage for many global variables which > are used when processor is executing reset code. When a second (non-boot) > CPU comes up, we are seeing that per-CPU pages for this CPU are reset to > all zeroes when the second CPU sets the PGE flag in CR4. This then leads = to > incorrect code paths. > > I am not able to understand why the pages mapped to AP are getting set to= 0 > when PGE flag is set in CR4 as this does not happen for boot processor. I= f > i do not set PGE flag then pages are not set to 0 but the code panics som= e > where else but not due to zeroed out global values. > > I have tried FreeBSD i386 with PAE support compiled into the kernel on > Bhyve with >4GB RAM and 2 vCPUS and VM came up successfully. Also, the PG= E > bit is set in the CR4 register for FreeBSD. > > > It will be helpful if anyone could throw some light on following points: > > > > 1. Why are the pages mapped to AP are being zeroed when PGE flag is set f= or > AP in it's CR4 ? > It is hard to say but my guess would be that the 4 PDPTE registers in the VMCS are not being properly updated during AP spinup. It would be helpful to see the assembly instructions in the AP spinup code that modify %cr0, %cr4 and %cr3 to know for sure. > 2. Are there are any changes required in a 32-bit OS to boot it in PAE mo= de > with 4GB or more RAM and 2 or more vCPUS on Bhyve ? > No. > 3. Was FreeBSD (i386) kernel updated to boot on Bhyve with 4GB or more RA= M > and 2 or more vCPUS ? > No. best Neel > Thanks. > > Prakhar > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org"