From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 3 21:53:21 2005 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 139CA16A41C for ; Fri, 3 Jun 2005 21:53:21 +0000 (GMT) (envelope-from ups@tree.com) Received: from smtp.speedfactory.net (talon.speedfactory.net [66.23.216.215]) by mx1.FreeBSD.org (Postfix) with ESMTP id 991E743D1F for ; Fri, 3 Jun 2005 21:53:20 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 29509 invoked by uid 210); 3 Jun 2005 22:48:56 +0000 Received: from 66.23.216.49 by talon (envelope-from , uid 201) with qmail-scanner-1.25st (clamdscan: 0.85.1/908. spamassassin: 3.0.2. perlscan: 1.25st. Clear:RC:1(66.23.216.49):. Processed in 0.043527 secs); 03 Jun 2005 22:48:56 -0000 X-Qmail-Scanner-Mail-From: ups@tree.com via talon X-Qmail-Scanner: 1.25st (Clear:RC:1(66.23.216.49):. Processed in 0.043527 secs Process 29503) Received: from 66-23-216-49.clients.speedfactory.net (HELO palm.tree.com) (66.23.216.49) by smtp.speedfactory.net with AES256-SHA encrypted SMTP; 3 Jun 2005 22:48:56 +0000 Received: from [127.0.0.1] (ups@localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id j53LrIpP055774; Fri, 3 Jun 2005 17:53:18 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: Matthew Dillon In-Reply-To: <200506032057.j53KvOFw062012@apollo.backplane.com> References: <200506032057.j53KvOFw062012@apollo.backplane.com> Content-Type: text/plain Message-Id: <1117835598.27369.12036.camel@palm> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Fri, 03 Jun 2005 17:53:18 -0400 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Possible instruction pipelining problem between HT's on the same die ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 21:53:21 -0000 On Fri, 2005-06-03 at 16:57, Matthew Dillon wrote: > I've been tracking down a crash one of our users gets occassionally. > He has a quad Intel(R) XEON(TM) CPU 2.00GHz (1996.61-MHz 686-class CPU) > system. > > After getting a few of these crashes he pulled three of the four cpus > out. But with just one physical cpu, with HTT turned on (so two > logical cpus), he is still getting these crashes. > > This is the sequence that causes the bad data: > > cpu #0 write A > write B > > (HT)cpu #1 read B > if (B) > read A <---- gets OLD data in A, not new data > > Now I was depending on the presumed write ordering, so if a foreign > cpu sees that B is updated it can assume that A has also been updated. > > But I'm beginning to think that it isn't working as advertised. I've > read the manuals over and over again and they seem to only guarentee > write ordering between physical cpus, not between logical HT cpus, and > even then it appears that a cpu can do a speculative read and > thus get an old value for A even after getting a new value for B. > > I looked at the various SFENCE/LFENCE/MFENCE instructions and they > do not seem to guarentee ordering for speculative accesses at all. > They all say that they do not protect against speculative reads. > Bus-locked instructions don't seem to avoid speculative reads either. > > I'm even more confused because this bug is occuring between two logical > cpus on the same physical die. Is write ordering not guarenteed with > respect to the other logical cpu? Can one logical cpu prefetch data > early then then becomes obsolete by the time the instruction is actually > run? Or perhaps its a pipeline bug... I just don't know. But it's > damn annoying. > > The only solution I see is to use an actual serializing instruction > like cpuid. I really do not want to have to use cpuid :-(. > > So, has anyone seen anything similar? This is normal behaviour. Take a look at IA-32 Intel Developers ... Vol 3, Section: 7.2.2 for details + solutions. Stephan