From owner-freebsd-hardware@FreeBSD.ORG Tue Apr 27 02:10:25 2010 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E33EB1065670; Tue, 27 Apr 2010 02:10:24 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8864A8FC0A; Tue, 27 Apr 2010 02:10:24 +0000 (UTC) Received: by gwj18 with SMTP id 18so3968720gwj.13 for ; Mon, 26 Apr 2010 19:10:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=YR0jzaNWhlvslyP55RyUO4mqSdq3/S8rxnV8hG/QCnU=; b=ZrVpEN87SVnwp5t8H36BgTxmHBM7PvQIwR7rZcgz7maBE0hAbw1nDpfxvjCBenDAEa jZr7guWt/wi/Svxlt11fgtpTpo17jyRvoJvHVl09Fy9VQXGdFVVU3AaDhsPZNNWLhPeJ kTsm0qNya7EcLYi6Yg1ueU1GpuT+s8/tteWEQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=utyrFN+wHRAH2E/p9l21IJ321IkTbIol9AZUX2440XyTBg9CUbQMI3LeiD0tvbXDJI uFK2ycqZeUhG1B+90AOE5FiluMbjUm1RfOPbxBqbHyePr3J3sfcY+wM4uRSucqOnmFhD zEtk9Ba0eFrgkxp82XVEixrORqA1Qb8wjlCag= MIME-Version: 1.0 Received: by 10.101.149.17 with SMTP id b17mr515180ano.80.1272334217642; Mon, 26 Apr 2010 19:10:17 -0700 (PDT) Received: by 10.100.194.19 with HTTP; Mon, 26 Apr 2010 19:10:17 -0700 (PDT) In-Reply-To: <201004201202.56275.jhb@freebsd.org> References: <4B5478D1.4000900@greatbaysoftware.com> <201004200913.05426.jhb@freebsd.org> <4BCDCEB2.8040407@greatbaysoftware.com> <201004201202.56275.jhb@freebsd.org> Date: Mon, 26 Apr 2010 22:10:17 -0400 Message-ID: From: Alexander Sack To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Charles Owens , freebsd-hardware@freebsd.org Subject: Re: Minute+ delay between kernel load and initialization (solved!) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Apr 2010 02:10:25 -0000 On Tue, Apr 20, 2010 at 12:02 PM, John Baldwin wrote: > On Tuesday 20 April 2010 11:56:34 am Charles Owens wrote: >> On 4/20/2010 9:13 AM, John Baldwin wrote: >> > On Monday 19 April 2010 6:05:06 pm Charles Owens wrote: >> > >> >> On 1/18/2010 10:05 AM, Charles Owens wrote: >> >> >> >>> Hello, >> >>> >> >>> We have a new system based on the Intel S5520 motherboard that seems= to >> >>> work fine except during _every_ boot it pauses for about one minute = 15 >> >>> seconds just before any kernel initialization messages appear. =A0 W= e see >> >>> the loader appearing to complete its work (the kernel is loaded) and >> >>> then... it sits. =A0Once it starts up again (with usual kernel-boot >> >>> messages) it appears to boot normally (no error messages). >> >>> >> >>> This has been seen with both FreeBSD 8.0 and 7.1, using GENERIC kern= els. >> >>> =A0I'd appreciate any and all assistance in figuring this out. >> >>> >> >>> Boot output follows (happens to be from a PAE-enabled kernel) >> >>> >> >>> Thank you, >> >>> >> >>> Charles >> >>> >> >>> >> >>> >> >> [truncated] >> >> >> >> An answer to this was graciously provided by Titus Manea. =A0 For det= ails, >> >> see >> >> >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=3Di386/144956 >> >> >> > Interesting. =A0I do not see the long delay on Nehalem machines here. = =A0Would >> > either of you be able to debug this further? =A0You could maybe grab T= SC > values >> > at various points during the early console probe and print out the > relevant >> > deltas after cninit() returns. =A0You could then move the TSC probe po= ints >> > around to pinpoint which operations are taking a long time. >> > >> >> John, >> >> I'm not going to have the cycles to look at this any time soon, >> unfortunately. =A0 But, just in case, how does one display TSC values? = =A0We >> did try to enable the kernel debugger, but the delay happens _before_ >> the debugger prompt is available. >> >> We now think of this more as an issue seen with some particular >> motherboard/BIOS combinations .... initially the only thing we really >> had to go on was the fact that the behavior was seen on new >> Nehalem-based systems. > > You would have to patch the source to add various calls to rdtsc() that w= ere > saved in some sort of global array. =A0Then, once the console was fully > initialized you could print out the TSC deltas by walking the array and > printing out the delta between each pair of entries. =A0You could then mo= ve the > rdtsc() calls around on subsequent tests to narrow down where the pause i= s > happening. Sorry John, I completely missed this. If you haven't done it already, I can look into it for sure. I have several Nehalem machines that do this. Its on my TODO list though I thought the kdb probe was the fix? Is that NOT the fix? Just let me know, guys, -aps From owner-freebsd-hardware@FreeBSD.ORG Tue Apr 27 15:24:42 2010 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BFCB106564A; Tue, 27 Apr 2010 15:24:42 +0000 (UTC) (envelope-from cowens@greatbaysoftware.com) Received: from portcityhosting.com (edge.tidalhosting.net [64.140.243.92]) by mx1.freebsd.org (Postfix) with ESMTP id 007F48FC29; Tue, 27 Apr 2010 15:24:41 +0000 (UTC) Received: from Inbox ([75.202.235.205]) by portcityhosting.com with MailEnable ESMTP; Tue, 27 Apr 2010 11:24:36 -0400 X-WatchGuard-Mail-Exception: Allow MIME-Version: 1.0 From: Charles Owens Date: Tue, 27 Apr 2010 11:24:50 -0400 To: Alexander Sack , John Baldwin Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" X-WatchGuard-AntiVirus: part scanned. clean action=allow X-ME-Bayesian: 0.000000 Message-Id: <20100427152442.6BFCB106564A@hub.freebsd.org> Cc: freebsd-hardware@freebsd.org Subject: RE: Minute+ delay between kernel load and initialization (solved!) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Apr 2010 15:24:42 -0000 The device.hints tweak is a fine workaround. If problem is at all widespre= ad, though, I'd argue that it is insufficient.=20 ---- Charles Owens Great Bay Software, Inc. sent from my phone -----Original Message----- From: Alexander Sack Sent: Monday, April 26, 2010 10:10 PM To: John Baldwin Cc: Charles Owens ; freebsd-hardware@freebsd.o= rg Subject: Re: Minute+ delay between kernel load and initialization (solved!) On Tue, Apr 20, 2010 at 12:02 PM, John Baldwin wrote: > On Tuesday 20 April 2010 11:56:34 am Charles Owens wrote: >> On 4/20/2010 9:13 AM, John Baldwin wrote: >> > On Monday 19 April 2010 6:05:06 pm Charles Owens wrote: >> > >> >> On 1/18/2010 10:05 AM, Charles Owens wrote: >> >> >> >>> Hello, >> >>> >> >>> We have a new system based on the Intel S5520 motherboard that seems= to >> >>> work fine except during _every_ boot it pauses for about one minute = 15 >> >>> seconds just before any kernel initialization messages appear. =A0 W= e see >> >>> the loader appearing to complete its work (the kernel is loaded) and >> >>> then... it sits. =A0Once it starts up again (with usual kernel-boot >> >>> messages) it appears to boot normally (no error messages). >> >>> >> >>> This has been seen with both FreeBSD 8.0 and 7.1, using GENERIC kern= els. >> >>> =A0I'd appreciate any and all assistance in figuring this out. >> >>> >> >>> Boot output follows (happens to be from a PAE-enabled kernel) >> >>> >> >>> Thank you, >> >>> >> >>> Charles >> >>> >> >>> >> >>> >> >> [truncated] >> >> >> >> An answer to this was graciously provided by Titus Manea. =A0 For det= ails, >> >> see >> >> >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=3Di386/144956 >> >> >> > Interesting. =A0I do not see the long delay on Nehalem machines here. = =A0Would >> > either of you be able to debug this further? =A0You could maybe grab T= SC > values >> > at various points during the early console probe and print out the > relevant >> > deltas after cninit() returns. =A0You could then move the TSC probe po= ints >> > around to pinpoint which operations are taking a long time. >> > >> >> John, >> >> I'm not going to have the cycles to look at this any time soon, >> unfortunately. =A0 But, just in case, how does one display TSC values? = =A0We >> did try to enable the kernel debugger, but the delay happens _before_ >> the debugger prompt is available. >> >> We now think of this more as an issue seen with some particular >> motherboard/BIOS combinations .... initially the only thing we really >> had to go on was the fact that the behavior was seen on new >> Nehalem-based systems. > > You would have to patch the source to add various calls to rdtsc() that w= ere > saved in some sort of global array. =A0Then, once the console was fully > initialized you could print out the TSC deltas by walking the array and > printing out the delta between each pair of entries. =A0You could then mo= ve the > rdtsc() calls around on subsequent tests to narrow down where the pause i= s > happening. Sorry John, I completely missed this. If you haven't done it already, I can look into it for sure. I have several Nehalem machines that do this. Its on my TODO list though I thought the kdb probe was the fix? Is that NOT the fix? Just let me know, guys, -aps From owner-freebsd-hardware@FreeBSD.ORG Tue Apr 27 16:51:42 2010 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2F001065672 for ; Tue, 27 Apr 2010 16:51:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6E0418FC22 for ; Tue, 27 Apr 2010 16:51:42 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 0735146B2C; Tue, 27 Apr 2010 12:51:42 -0400 (EDT) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 6EFC58A025; Tue, 27 Apr 2010 12:51:32 -0400 (EDT) From: John Baldwin To: Alexander Sack Date: Tue, 27 Apr 2010 09:45:33 -0400 User-Agent: KMail/1.12.1 (FreeBSD/7.3-CBSD-20100217; KDE/4.3.1; amd64; ; ) References: <4B5478D1.4000900@greatbaysoftware.com> <201004201202.56275.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201004270945.33722.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Tue, 27 Apr 2010 12:51:32 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00, DATE_IN_PAST_03_06 autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Charles Owens , freebsd-hardware@freebsd.org Subject: Re: Minute+ delay between kernel load and initialization (solved!) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Apr 2010 16:51:42 -0000 On Monday 26 April 2010 10:10:17 pm Alexander Sack wrote: > On Tue, Apr 20, 2010 at 12:02 PM, John Baldwin wrote: > > On Tuesday 20 April 2010 11:56:34 am Charles Owens wrote: > >> On 4/20/2010 9:13 AM, John Baldwin wrote: > >> > On Monday 19 April 2010 6:05:06 pm Charles Owens wrote: > >> > > >> >> On 1/18/2010 10:05 AM, Charles Owens wrote: > >> >> > >> >>> Hello, > >> >>> > >> >>> We have a new system based on the Intel S5520 motherboard that seems to > >> >>> work fine except during _every_ boot it pauses for about one minute 15 > >> >>> seconds just before any kernel initialization messages appear. We see > >> >>> the loader appearing to complete its work (the kernel is loaded) and > >> >>> then... it sits. Once it starts up again (with usual kernel-boot > >> >>> messages) it appears to boot normally (no error messages). > >> >>> > >> >>> This has been seen with both FreeBSD 8.0 and 7.1, using GENERIC kernels. > >> >>> I'd appreciate any and all assistance in figuring this out. > >> >>> > >> >>> Boot output follows (happens to be from a PAE-enabled kernel) > >> >>> > >> >>> Thank you, > >> >>> > >> >>> Charles > >> >>> > >> >>> > >> >>> > >> >> [truncated] > >> >> > >> >> An answer to this was graciously provided by Titus Manea. For details, > >> >> see > >> >> > >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=i386/144956 > >> >> > >> > Interesting. I do not see the long delay on Nehalem machines here. Would > >> > either of you be able to debug this further? You could maybe grab TSC > > values > >> > at various points during the early console probe and print out the > > relevant > >> > deltas after cninit() returns. You could then move the TSC probe points > >> > around to pinpoint which operations are taking a long time. > >> > > >> > >> John, > >> > >> I'm not going to have the cycles to look at this any time soon, > >> unfortunately. But, just in case, how does one display TSC values? We > >> did try to enable the kernel debugger, but the delay happens _before_ > >> the debugger prompt is available. > >> > >> We now think of this more as an issue seen with some particular > >> motherboard/BIOS combinations .... initially the only thing we really > >> had to go on was the fact that the behavior was seen on new > >> Nehalem-based systems. > > > > You would have to patch the source to add various calls to rdtsc() that were > > saved in some sort of global array. Then, once the console was fully > > initialized you could print out the TSC deltas by walking the array and > > printing out the delta between each pair of entries. You could then move the > > rdtsc() calls around on subsequent tests to narrow down where the pause is > > happening. > > Sorry John, I completely missed this. If you haven't done it already, > I can look into it for sure. I have several Nehalem machines that do > this. Its on my TODO list though I thought the kdb probe was the fix? > Is that NOT the fix? > > Just let me know, guys, Maxim has a patch for the atkbd driver that works in his testing. I think it is on another thread however. -- John Baldwin From owner-freebsd-hardware@FreeBSD.ORG Tue Apr 27 16:54:07 2010 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EBA6106566B; Tue, 27 Apr 2010 16:54:07 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-gx0-f211.google.com (mail-gx0-f211.google.com [209.85.217.211]) by mx1.freebsd.org (Postfix) with ESMTP id C46FD8FC14; Tue, 27 Apr 2010 16:54:06 +0000 (UTC) Received: by gxk3 with SMTP id 3so6458783gxk.13 for ; Tue, 27 Apr 2010 09:53:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=xdcvL4b99ei2zNP1L9aFHsEGD0Lv+teXU/PEWOOXLSk=; b=VRa819eilCDHdooOTHyTLqNm+NcZ6xmP7DRPigy0qjw5urr2prtTR9fqagzPZk3G4g +eR6oZF+C+HVH2Dg33YHbFvgYty5DOu9kUVYb2HGc8Qz5WWU8MmrAWLBdt/SYUIWr5qr hqO2W/aVRm9qfFWLcPEejmdvgeHpVnSPREAGg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=aLV7LKZMqdmv3xk+OtHZdC0rnqBRIiH/XXerWomxSB1Fn4EoFUp++YYCG312dfY3Y0 LXCyTXy1vzUHXx4rqGMYUZZLpMaMVNkVpilSmvRwEeqH8YKlEfeoTxUecnl9n0cUOhUR P6+qv8l0epIercN/5p+HfiFJ6gxerlUDmDzbc= MIME-Version: 1.0 Received: by 10.101.199.13 with SMTP id b13mr1409390anq.58.1272387237212; Tue, 27 Apr 2010 09:53:57 -0700 (PDT) Received: by 10.100.194.19 with HTTP; Tue, 27 Apr 2010 09:53:57 -0700 (PDT) In-Reply-To: <201004270945.33722.jhb@freebsd.org> References: <4B5478D1.4000900@greatbaysoftware.com> <201004201202.56275.jhb@freebsd.org> <201004270945.33722.jhb@freebsd.org> Date: Tue, 27 Apr 2010 12:53:57 -0400 Message-ID: From: Alexander Sack To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Charles Owens , freebsd-hardware@freebsd.org Subject: Re: Minute+ delay between kernel load and initialization (solved!) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Apr 2010 16:54:07 -0000 On Tue, Apr 27, 2010 at 9:45 AM, John Baldwin wrote: > On Monday 26 April 2010 10:10:17 pm Alexander Sack wrote: >> On Tue, Apr 20, 2010 at 12:02 PM, John Baldwin wrote: >> > On Tuesday 20 April 2010 11:56:34 am Charles Owens wrote: >> >> On 4/20/2010 9:13 AM, John Baldwin wrote: >> >> > On Monday 19 April 2010 6:05:06 pm Charles Owens wrote: >> >> > >> >> >> On 1/18/2010 10:05 AM, Charles Owens wrote: >> >> >> >> >> >>> Hello, >> >> >>> >> >> >>> We have a new system based on the Intel S5520 motherboard that se= ems > to >> >> >>> work fine except during _every_ boot it pauses for about one minu= te > 15 >> >> >>> seconds just before any kernel initialization messages appear. = =A0 We > see >> >> >>> the loader appearing to complete its work (the kernel is loaded) = and >> >> >>> then... it sits. =A0Once it starts up again (with usual kernel-bo= ot >> >> >>> messages) it appears to boot normally (no error messages). >> >> >>> >> >> >>> This has been seen with both FreeBSD 8.0 and 7.1, using GENERIC > kernels. >> >> >>> =A0I'd appreciate any and all assistance in figuring this out. >> >> >>> >> >> >>> Boot output follows (happens to be from a PAE-enabled kernel) >> >> >>> >> >> >>> Thank you, >> >> >>> >> >> >>> Charles >> >> >>> >> >> >>> >> >> >>> >> >> >> [truncated] >> >> >> >> >> >> An answer to this was graciously provided by Titus Manea. =A0 For > details, >> >> >> see >> >> >> >> >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=3Di386/144956 >> >> >> >> >> > Interesting. =A0I do not see the long delay on Nehalem machines her= e. > =A0Would >> >> > either of you be able to debug this further? =A0You could maybe gra= b TSC >> > values >> >> > at various points during the early console probe and print out the >> > relevant >> >> > deltas after cninit() returns. =A0You could then move the TSC probe > points >> >> > around to pinpoint which operations are taking a long time. >> >> > >> >> >> >> John, >> >> >> >> I'm not going to have the cycles to look at this any time soon, >> >> unfortunately. =A0 But, just in case, how does one display TSC values= ? =A0We >> >> did try to enable the kernel debugger, but the delay happens _before_ >> >> the debugger prompt is available. >> >> >> >> We now think of this more as an issue seen with some particular >> >> motherboard/BIOS combinations .... initially the only thing we really >> >> had to go on was the fact that the behavior was seen on new >> >> Nehalem-based systems. >> > >> > You would have to patch the source to add various calls to rdtsc() tha= t > were >> > saved in some sort of global array. =A0Then, once the console was full= y >> > initialized you could print out the TSC deltas by walking the array an= d >> > printing out the delta between each pair of entries. =A0You could then= move > the >> > rdtsc() calls around on subsequent tests to narrow down where the paus= e is >> > happening. >> >> Sorry John, I completely missed this. =A0If you haven't done it already, >> I can look into it for sure. =A0I have several Nehalem machines that do >> this. =A0Its on my TODO list though I thought the kdb probe was the fix? >> =A0Is that NOT the fix? >> >> Just let me know, guys, > > Maxim has a patch for the atkbd driver that works in his testing. =A0I th= ink it > is on another thread however. Alright, I will take a look and do the same on my end as another datapoint. I have been waiting all day to get my hands on this box! :) -aps