Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 10 Feb 2010 08:43:14 -0500
From:      John Baldwin <jhb@freebsd.org>
To:        Tom McLaughlin <tmclaugh@sdf.lonestar.org>
Cc:        kib@freebsd.org, freebsd-stable@freebsd.org
Subject:   Re: Recent MFC to 7 causes crash on VMware ESXi
Message-ID:  <201002100843.14432.jhb@freebsd.org>
In-Reply-To: <4B71FC5A.9080601@sdf.lonestar.org>
References:  <4B6B89E7.8030002@sdf.lonestar.org> <201002091723.44625.jhb@freebsd.org> <4B71FC5A.9080601@sdf.lonestar.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday 09 February 2010 7:22:50 pm Tom McLaughlin wrote:
> John Baldwin wrote, On 02/09/2010 05:23 PM:
> > On Tuesday 09 February 2010 5:17:32 pm Tom McLaughlin wrote:
> >> John Baldwin wrote, On 02/09/2010 01:52 PM:
> >>> On Monday 08 February 2010 9:49:00 am John Baldwin wrote:
> >>>> On Saturday 06 February 2010 4:47:16 pm Tom McLaughlin wrote:
> >>>>> John Baldwin wrote, On 02/05/2010 08:27 AM:
> >>>>>> On Thursday 04 February 2010 10:00:55 pm Tom McLaughlin wrote:
> >>>>>>> Hi all, a recent MFC to 7-STABLE has started to cause issues for my 
> > VMs
> >>>>>>> on VMware ESXi 3.5u4.  After loading the mpt driver for the LSI disk
> >>>>>>> controller the VM just shuts off.  The workaround is to change the 
> > disk
> >>>>>>> controller to the BusLogic type.  Still, it used to work up until 
last
> >>>>>>> week.  The change was made around January 26th and based on the 
> > commits
> >>>>>>> that day I'm guessing it's either r203047 or r203073
> >>>>>>>
> >>>>>>> I have the same issue with both amd64 and i386 VMs.  This affects 
HEAD
> >>>>>>> and 8-STABLE as well and first affected HEAD over the summer.  (I 
just
> >>>>>>> worked around it and went about my business at the time. :-/)  I've
> >>>>>>> attached a dmesg from a kernel before the problem and one from after 
> > it
> >>>>>>> started.
> >>>>>>
> >>>>>> What if you set 'hw.clfush_disable=1' from the loader?
> >>>>>>
> >>>>>
> >>>>> Yes, that corrected it on all my VMs.  I've talked to people on ESXi 4
> >>>>> and they do not see the problem.  I have yet to try 3.5u5 to see if 
this
> >>>>> is a non-issue.  3.5 will be supported for awhile longer from VMware.
> >>>>> I'm going to try upgrading the box during the week.
> >>>>
> >>>> I believe folks had to do this on HEAD/8.x as well.  Perhaps we can 
> >>>> automatically disable clflush if we are executing under VMware or Xen:
> >>>
> >>> Tom, were you able to verify that this patch fixes the problem for you
> >>> without requiring you to set the hw.clflush_disable tunable?
> >>>
> >>
> >> John, I'm getting the following build error on all branches:
> >>
> >> /usr/src/sys/amd64/amd64/initcpu.c: In function 'initializecpucache':
> >> /usr/src/sys/amd64/amd64/initcpu.c:184: error: 'vm_guest' undeclared
> >> (first use in this function)
> >> /usr/src/sys/amd64/amd64/initcpu.c:184: error: (Each undeclared
> >> identifier is reported only once
> >> /usr/src/sys/amd64/amd64/initcpu.c:184: error: for each function it
> >> appears in.)
> > 
> > Oh foo.  Can you add 'extern int vm_guest;' to that file near the top?
> 
> Adding that to the top fixed the build on 8.x and HEAD.  Both of those
> are also now working just fine.  However, 7.x still does not build.

Ok, I will work on a more formal 7.x patch.

> cc -c -O2 -frename-registers -pipe -fno-strict-aliasing  -std=c99 -g
> -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes
> -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef
> -Wno-pointer-sign -fformat-extensions -nostdinc  -I. -I/usr/src/sys
> -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS
> -include opt_global.h -fno-common -finline-limit=8000 --param
> inline-unit-growth=100 --param large-function-growth=1000
> -mcmodel=kernel -mno-red-zone  -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx
> -mno-3dnow  -msoft-float -fno-asynchronous-unwind-tables -ffreestanding
> -Werror  vers.c
> linking kernel.debug
> initcpu.o(.text+0x272): In function `initializecpucache':
> /usr/src/sys/amd64/amd64/initcpu.c:186: undefined reference to `vm_guest'
> *** Error code 1
> 
> 
> > 
> >> *** Error code 1
> >>
> >>
> >> tom
> >>
> >>>> Index: amd64/amd64/initcpu.c
> >>>> ===================================================================
> >>>> --- amd64/amd64/initcpu.c	(revision 203430)
> >>>> +++ amd64/amd64/initcpu.c	(working copy)
> >>>> @@ -177,17 +177,16 @@
> >>>>  	if ((cpu_feature & CPUID_CLFSH) != 0)
> >>>>  		cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8;
> >>>>  	/*
> >>>> -	 * XXXKIB: (temporary) hack to work around traps generated when
> >>>> -	 * CLFLUSHing APIC registers window.
> >>>> +	 * XXXKIB: (temporary) hack to work around traps generated
> >>>> +	 * when CLFLUSHing APIC registers window under virtualization
> >>>> +	 * environments.
> >>>>  	 */
> >>>>  	TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable);
> >>>> -	if (cpu_vendor_id == CPU_VENDOR_INTEL && !(cpu_feature & CPUID_SS) &&
> >>>> -	    hw_clflush_disable == -1)
> >>>> +	if (vm_guest != 0 /* VM_GUEST_NO */ && hw_clflush_disable == -1)
> >>>>  		cpu_feature &= ~CPUID_CLFSH;
> >>>>  	/*
> >>>>  	 * Allow to disable CLFLUSH feature manually by
> >>>> -	 * hw.clflush_disable tunable.  This may help Xen guest on some AMD
> >>>> -	 * CPUs.
> >>>> +	 * hw.clflush_disable tunable.
> >>>>  	 */
> >>>>  	if (hw_clflush_disable == 1)
> >>>>  		cpu_feature &= ~CPUID_CLFSH;
> >>>> Index: i386/i386/initcpu.c
> >>>> ===================================================================
> >>>> --- i386/i386/initcpu.c	(revision 203430)
> >>>> +++ i386/i386/initcpu.c	(working copy)
> >>>> @@ -724,17 +724,16 @@
> >>>>  	if ((cpu_feature & CPUID_CLFSH) != 0)
> >>>>  		cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8;
> >>>>  	/*
> >>>> -	 * XXXKIB: (temporary) hack to work around traps generated when
> >>>> -	 * CLFLUSHing APIC registers window.
> >>>> +	 * XXXKIB: (temporary) hack to work around traps generated
> >>>> +	 * when CLFLUSHing APIC registers window under virtualization
> >>>> +	 * environments.
> >>>>  	 */
> >>>>  	TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable);
> >>>> -	if (cpu_vendor_id == CPU_VENDOR_INTEL && !(cpu_feature & CPUID_SS) &&
> >>>> -	    hw_clflush_disable == -1)
> >>>> +	if (vm_guest != 0 /* VM_GUEST_NO */ && hw_clflush_disable == -1)
> >>>>  		cpu_feature &= ~CPUID_CLFSH;
> >>>>  	/*
> >>>>  	 * Allow to disable CLFLUSH feature manually by
> >>>> -	 * hw.clflush_disable tunable.  This may help Xen guest on some AMD
> >>>> -	 * CPUs.
> >>>> +	 * hw.clflush_disable tunable.
> >>>>  	 */
> >>>>  	if (hw_clflush_disable == 1)
> >>>>  		cpu_feature &= ~CPUID_CLFSH;
> >>>>
> >>>> -- 
> >>>> John Baldwin
> >>>> _______________________________________________
> >>>> freebsd-stable@freebsd.org mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>>> To unsubscribe, send any mail to "freebsd-stable-
unsubscribe@freebsd.org"
> >>>>
> >>>
> >>
> >>
> >> -- 
> >> | tmclaugh at sdf.lonestar.org                 tmclaugh at FreeBSD.org |
> >> | FreeBSD                                       http://www.FreeBSD.org |
> >>
> >>
> > 
> 
> 
> -- 
> | tmclaugh at sdf.lonestar.org                 tmclaugh at FreeBSD.org |
> | FreeBSD                                       http://www.FreeBSD.org |
> 
> 

-- 
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201002100843.14432.jhb>