From owner-freebsd-stable@FreeBSD.ORG Sun Aug 7 09:20:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF65E1065672 for ; Sun, 7 Aug 2011 09:20:21 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4F9B28FC15 for ; Sun, 7 Aug 2011 09:20:20 +0000 (UTC) Received: by wyh21 with SMTP id 21so342986wyh.13 for ; Sun, 07 Aug 2011 02:20:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=kn8hsmznXiXc6+JFpmxGmDfaWMRRO5A8b5LOpQfBKZ4=; b=PUO6ghOMNKBhrx9wMCeUpmiw9GyIO5Ss36+MLPsYrShUC75Fxs1pUddFnO28CeV3od h3ffuSDmz8/MoCLy1UUOXbHgYgM9P8ooEKlPGmMZqFKhMJVsxP9QvdNev92z2gYfP781 Jc4Xj0lYW3OKCPhKT/ojRII4HWMN+IzBJBW+k= Received: by 10.216.170.8 with SMTP id o8mr899217wel.101.1312708819803; Sun, 07 Aug 2011 02:20:19 -0700 (PDT) Received: from nox.fritz.box (87-198-213-102.static.ptr.magnet.ie [87.198.213.102]) by mx.google.com with ESMTPS id m38sm2831386weq.21.2011.08.07.02.20.16 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 07 Aug 2011 02:20:17 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: text/plain; charset=us-ascii From: Sean Rees In-Reply-To: <20110806062415.GB88904@in-addr.com> Date: Sun, 7 Aug 2011 10:20:15 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <010350C0-B3B3-44FC-8D94-A111C579860C@gmail.com> References: <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> <6E45CE57-491E-4077-B14C-751C73647EFC@gsoft.com.au> <4E3CBB74.9020208@FreeBSD.org> <20110806062415.GB88904@in-addr.com> To: Gary Palmer X-Mailer: Apple Mail (2.1244.3) Cc: Doug Barton , freebsd-stable@freebsd.org Subject: Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2011 09:20:21 -0000 On Aug 6, 2011, at 07:24, Gary Palmer wrote: > On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: >> On 08/05/2011 20:38, Daniel O'Connor wrote: >>=20 >>> Ahh, but OP had moved these files away and performance was still = poor.. _that_ is the bug. >>=20 >> I'm no file system expert, but it seems to me the key questions are; = how >> long does it take the system to recover from this condition, and if = it's >> more than N $periods is that a problem? We can't stop users from = doing >> wacky stuff, but the system should be robust in the face of this. >=20 > Its been quite a while since I worked on the filesystem stuff in any > detail but I believe, at least for UFS, it doesn't GC the directory, > just truncate it if enough of the entries at the end are deleted > to free up at least one fragment or block. If you create N files and > then a directory and move the N files into the directory, the = directory > entry will still be N+1 records into the directory and the only way to > "recover" is to recreate the directory that formerly contained the N > files. It is theoretically possible to compat the directory but since=20= > the code to do that wasn't written when I last worked with UFS I = suspect > its non trivial. >=20 > I don't know what ZFS does in this situation It sounds like it does something similar. I re-ran the experiment to see if I could narrow down the problem. % mkdir foo % cd foo && for i in {1..1000}; do touch $i; done % ls > list % for file in $(cat list); do rm -f $file; done % time ls (slow!) % rm -f list % time ls (slow!) I would like to dig into this a bit more, I suppose it's probably a good = enough reason to explore how DTrace works :) Sean= From owner-freebsd-stable@FreeBSD.ORG Sun Aug 7 09:22:45 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E75CA1065679 for ; Sun, 7 Aug 2011 09:22:45 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id A1DE78FC14 for ; Sun, 7 Aug 2011 09:22:45 +0000 (UTC) Received: by vws18 with SMTP id 18so1854370vws.13 for ; Sun, 07 Aug 2011 02:22:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=InoLCgKbn09IRaXCGS1mfrBUsEWJI2loGpc9Ffj8SOg=; b=Rgktw4npEMdITLBKJueTMcENNiKE6NdLpwn2Ln/6E4sqBrdTVeUyyuQ9eA3suKc6Bt u0F/duAjNOqKZL2hqZrfToeUYWGJFv8iQ4+afvnG1MQZeJAM9An+/PAwthWWkEZ5yHQ5 siGX09DGQ9P41FGFuy4ny4IUGW6+3f1Lj/Z3M= MIME-Version: 1.0 Received: by 10.52.172.228 with SMTP id bf4mr986274vdc.264.1312708964490; Sun, 07 Aug 2011 02:22:44 -0700 (PDT) Received: by 10.52.169.6 with HTTP; Sun, 7 Aug 2011 02:22:44 -0700 (PDT) In-Reply-To: <010350C0-B3B3-44FC-8D94-A111C579860C@gmail.com> References: <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> <6E45CE57-491E-4077-B14C-751C73647EFC@gsoft.com.au> <4E3CBB74.9020208@FreeBSD.org> <20110806062415.GB88904@in-addr.com> <010350C0-B3B3-44FC-8D94-A111C579860C@gmail.com> Date: Sun, 7 Aug 2011 10:22:44 +0100 Message-ID: From: "seanrees@gmail.com" To: Gary Palmer Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Doug Barton , freebsd-stable@freebsd.org Subject: Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2011 09:22:46 -0000 On Sun, Aug 7, 2011 at 10:20 AM, Sean Rees wrote: > > On Aug 6, 2011, at 07:24, Gary Palmer wrote: > >> On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: >>> On 08/05/2011 20:38, Daniel O'Connor wrote: >>> >>>> Ahh, but OP had moved these files away and performance was still poor.= . _that_ is the bug. >>> >>> I'm no file system expert, but it seems to me the key questions are; ho= w >>> long does it take the system to recover from this condition, and if it'= s >>> more than N $periods is that a problem? We can't stop users from doing >>> wacky stuff, but the system should be robust in the face of this. >> >> Its been quite a while since I worked on the filesystem stuff in any >> detail but I believe, at least for UFS, it doesn't GC the directory, >> just truncate it if enough of the entries at the end are deleted >> to free up at least one fragment or block. =A0If you create N files and >> then a directory and move the N files into the directory, the directory >> entry will still be N+1 records into the directory and the only way to >> "recover" is to recreate the directory that formerly contained the N >> files. =A0It is theoretically possible to compat the directory but since >> the code to do that wasn't written when I last worked with UFS I suspect >> its non trivial. >> >> I don't know what ZFS does in this situation > > It sounds like it does something similar. > > I re-ran the experiment to see if I could narrow down the problem. > > % mkdir foo > % cd foo && for i in {1..1000}; do touch $i; done Self-pedant mode enabled: for i in {1..1000000} :) I truncated the zeros in correcting the copy/paste from my shell :) Sean From owner-freebsd-stable@FreeBSD.ORG Mon Aug 8 11:11:29 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41C14106567B for ; Mon, 8 Aug 2011 11:11:29 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id BE6C38FC14 for ; Mon, 8 Aug 2011 11:11:28 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p78BBIUb038662 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 8 Aug 2011 14:11:24 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E3FC456.8040508@digsys.bg> Date: Mon, 08 Aug 2011 14:11:18 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110720 Thunderbird/5.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> <6E45CE57-491E-4077-B14C-751C73647EFC@gsoft.com.au> <4E3CBB74.9020208@FreeBSD.org> <20110806062415.GB88904@in-addr.com> In-Reply-To: <20110806062415.GB88904@in-addr.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Aug 2011 11:11:29 -0000 On 06.08.11 09:24, Gary Palmer wrote: > Its been quite a while since I worked on the filesystem stuff in any > detail but I believe, at least for UFS, it doesn't GC the directory, > just truncate it if enough of the entries at the end are deleted to > free up at least one fragment or block. This was my point indeed. If you empty a directory or remove files form the end of the directory is it truncated, this is not really a GC, but rather a shortcut. I guess the reason why it does not use GC is because of concurrency/locking reasons. Or maybe the code was just not written yet. But with ZFS this should be much easier to implement. If it is the same in Solaris, then it is not done so far... But then, the promise made by ZFS is to provide constant directory access timing. I am just wondering.. does implementing such garbage collection merit a new ZFS filesystem version? Daniel From owner-freebsd-stable@FreeBSD.ORG Mon Aug 8 16:30:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27DCF106564A for ; Mon, 8 Aug 2011 16:30:37 +0000 (UTC) (envelope-from zkolic@sbb.rs) Received: from smtp4.sbb.rs (smtp4.sbb.rs [89.216.2.36]) by mx1.freebsd.org (Postfix) with ESMTP id 936D58FC12 for ; Mon, 8 Aug 2011 16:30:36 +0000 (UTC) Received: from faust (cable-94-189-184-240.dynamic.sbb.rs [94.189.184.240]) by smtp4.sbb.rs (8.14.0/8.14.0) with ESMTP id p78GUXwO007250 for ; Mon, 8 Aug 2011 18:30:33 +0200 Received: by faust (Postfix, from userid 1001) id 582071701D; Mon, 8 Aug 2011 18:30:58 +0200 (CEST) Date: Mon, 8 Aug 2011 18:30:58 +0200 From: Zoran Kolic To: freebsd-stable@freebsd.org Message-ID: <20110808163058.GA1095@faust> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-SMTP-Vilter-Version: 1.3.2 X-SBB-Virus-Status: clean X-SBB-Spam-Score: -1.8 Subject: Re: happy hacker lite 2 keyboard X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Aug 2011 16:30:37 -0000 Finally got the kb. As almost everyone anticipated, it refused to work connected to ps/2 adapter with usb -> ps/2. I had to recompile the kernel and put usb code back. So far, so good. Thanks all for support. Zoran From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 08:38:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 852D7106566B for ; Tue, 9 Aug 2011 08:38:02 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 36E178FC0C for ; Tue, 9 Aug 2011 08:38:01 +0000 (UTC) Received: by vws18 with SMTP id 18so3567305vws.13 for ; Tue, 09 Aug 2011 01:38:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=25AFkSBYEzJ7oIz/h2LFrTSX9p/savH/IZaTB58V+no=; b=l4nhCOhF5M3lsbcSv6BXZRDEyirCeCG6QU6EJkr1QSEj5zXoxy44qSJCMBj4ywJdTQ XGbHSuMEM3yHQKxrJoPJRMNC2rYG2ulEjUnPJvcXMAtvYVR9TjRUC0nzVo6KvLS8GrBm NdnUdYOvYLRAho/R0eqBEhrprRDPiFS/W+Nbs= MIME-Version: 1.0 Received: by 10.52.22.201 with SMTP id g9mr6572497vdf.331.1312879081151; Tue, 09 Aug 2011 01:38:01 -0700 (PDT) Received: by 10.52.168.229 with HTTP; Tue, 9 Aug 2011 01:38:01 -0700 (PDT) In-Reply-To: References: <20110802090830.GA92646@icarus.home.lan> <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> Date: Tue, 9 Aug 2011 09:38:01 +0100 Message-ID: From: "seanrees@gmail.com" To: Ronald Klop Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: zpool doesn't upgrade - Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 08:38:02 -0000 On Tue, Aug 2, 2011 at 8:59 PM, Ronald Klop w= rote: > On Tue, 02 Aug 2011 12:55:43 +0200, seanrees@gmail.com > wrote: > I think this zpool upgrade thing is weird. Can you try 'zpool upgrade -a'= ? > > Mine says: > zpool get version zroot > NAME =A0 PROPERTY =A0VALUE =A0 =A0SOURCE > zroot =A0version =A0 28 =A0 =A0 =A0 default > > Mind the SOURCE=3Ddefault vs. SOURCE=3Dlocal. > Is it possible you did 'zpool set version=3D15 tank' in the past? You can > check that with 'zpool history'. > > NB: if you upgrade the boot pool, don't forget to upgrade to boot loader. > (See UPDATING) % sudo zpool upgrade -a Password: This system is currently running ZFS pool version 15. All pools are formatted using this version. I checked zpool history and I never set the version explicitly. My 'world' is from the 8th of March; it's possible my tree is sufficiently old (my kernel was built on the 12th of June; I'm fairly sure its from the same tree as the world, but it's also possible my kernel and userland have been out of sync for 2 months). I'll upgrade this machine sometime soon and see if that fixes the issue. Sean From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 09:16:29 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95B9C106566B for ; Tue, 9 Aug 2011 09:16:29 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 329DE8FC12 for ; Tue, 9 Aug 2011 09:16:28 +0000 (UTC) Received: by wwi36 with SMTP id 36so4455870wwi.31 for ; Tue, 09 Aug 2011 02:16:28 -0700 (PDT) Received: by 10.216.176.17 with SMTP id a17mr2533464wem.72.1312879924722; Tue, 09 Aug 2011 01:52:04 -0700 (PDT) Received: from dfleuriot-at-hi-media.com ([83.167.62.196]) by mx.google.com with ESMTPS id ff6sm5139938wbb.15.2011.08.09.01.52.01 (version=SSLv3 cipher=OTHER); Tue, 09 Aug 2011 01:52:02 -0700 (PDT) Message-ID: <4E40F530.8020507@my.gd> Date: Tue, 09 Aug 2011 10:52:00 +0200 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 09:16:29 -0000 On 8/2/11 9:39 AM, seanrees@gmail.com wrote: > Hi there, > > I Googled around and checked the PRs and wasn't successful in finding > any reports of what I'm seeing. I'm hoping someone here can help me > debug what's going on. > > On my FreeBSD 8.2-S machine (built circa 12th June), I created a > directory and populated it over the course of 3 weeks with about 2 > million individual files. As you might imagine, a 'ls' of this > directory took quite some time. > > The files were conveniently named with a timestamp in the filename > (still images from a security camera, once per second) so I've since > moved them all to timestamped directories (yyyy/MM/dd/hh/mm). What I > found though was the original directory the images were in is still > very slow to ls -- and it only has 1 file in it, another directory. > While not addressing your original question, which many people have already, I'll toss in the following: I do hope you've disabled access times on your ZFS dataset ? zfs set atime=off YOUR_DATASET/supercamera/captures From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 13:18:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48F0A1065672 for ; Tue, 9 Aug 2011 13:18:07 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smtp-out0.tiscali.nl (smtp-out0.tiscali.nl [195.241.79.175]) by mx1.freebsd.org (Postfix) with ESMTP id F1C578FC1B for ; Tue, 9 Aug 2011 13:18:06 +0000 (UTC) Received: from [212.182.167.131] (helo=sjakie.klop.ws) by smtp-out0.tiscali.nl with esmtp (Exim) (envelope-from ) id 1QqmC5-00033F-Sy for freebsd-stable@freebsd.org; Tue, 09 Aug 2011 15:18:05 +0200 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id E9B9AF6 for ; Tue, 9 Aug 2011 15:18:01 +0200 (CEST) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-stable@freebsd.org References: <20110802090830.GA92646@icarus.home.lan> <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> Date: Tue, 09 Aug 2011 15:18:01 +0200 MIME-Version: 1.0 From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/11.50 (FreeBSD) Content-Transfer-Encoding: quoted-printable Subject: Re: zpool doesn't upgrade - Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 13:18:07 -0000 On Tue, 09 Aug 2011 10:38:01 +0200, seanrees@gmail.com =20 wrote: > On Tue, Aug 2, 2011 at 8:59 PM, Ronald Klop =20 > wrote: >> On Tue, 02 Aug 2011 12:55:43 +0200, seanrees@gmail.com =20 >> >> wrote: >> I think this zpool upgrade thing is weird. Can you try 'zpool upgrade = =20 >> -a'? >> >> Mine says: >> zpool get version zroot >> NAME PROPERTY VALUE SOURCE >> zroot version 28 default >> >> Mind the SOURCE=3Ddefault vs. SOURCE=3Dlocal. >> Is it possible you did 'zpool set version=3D15 tank' in the past? You = can >> check that with 'zpool history'. >> >> NB: if you upgrade the boot pool, don't forget to upgrade to boot =20 >> loader. >> (See UPDATING) > > % sudo zpool upgrade -a > Password: > This system is currently running ZFS pool version 15. > > All pools are formatted using this version. > > I checked zpool history and I never set the version explicitly. My > 'world' is from the 8th of March; it's possible my tree is > sufficiently old (my kernel was built on the 12th of June; I'm fairly > sure its from the same tree as the world, but it's also possible my > kernel and userland have been out of sync for 2 months). > > I'll upgrade this machine sometime soon and see if that fixes the issue= . > > Sean You can set the property to 28 and upgrade after that. zpool set version=3D28 zroot From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 14:26:58 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 314531065675 for ; Tue, 9 Aug 2011 14:26:58 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 9B92F8FC16 for ; Tue, 9 Aug 2011 14:26:57 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p79EQklr046125 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 9 Aug 2011 17:26:52 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E4143A6.6030307@digsys.bg> Date: Tue, 09 Aug 2011 17:26:46 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110720 Thunderbird/5.0 MIME-Version: 1.0 To: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 14:26:58 -0000 I am trying to set up 64GB partitions for swap for a system that has 64GB of RAM (with the idea to dump kernel core etc). But, on 8-stable as of today I get: WARNING: reducing size to maximum of 67108864 blocks per swap unit Is there workaround for this limitation? Daniel From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 14:37:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B58E106566C for ; Tue, 9 Aug 2011 14:37:30 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from asmtpout024.mac.com (asmtpout024.mac.com [17.148.16.99]) by mx1.freebsd.org (Postfix) with ESMTP id 52BE58FC0A for ; Tue, 9 Aug 2011 14:37:30 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from [10.1.2.70] ([173.200.178.70]) by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01 64bit (built Jul 15 2010)) with ESMTPSA id <0LPO00L8D0LKEP40@asmtp024.mac.com> for freebsd-stable@freebsd.org; Tue, 09 Aug 2011 07:36:57 -0700 (PDT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.4.6813,1.0.211,0.0.0000 definitions=2011-08-09_06:2011-08-09, 2011-08-08, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=2 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1108090116 From: Chuck Swiger In-reply-to: <4E4143A6.6030307@digsys.bg> Date: Tue, 09 Aug 2011 07:36:56 -0700 Message-id: <935F8EC2-88E0-45A3-BE8B-7210BE223BC5@mac.com> References: <4E4143A6.6030307@digsys.bg> To: Daniel Kalchev X-Mailer: Apple Mail (2.1084) Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 14:37:30 -0000 On Aug 9, 2011, at 7:26 AM, Daniel Kalchev wrote: > I am trying to set up 64GB partitions for swap for a system that has 64GB of RAM (with the idea to dump kernel core etc). But, on 8-stable as of today I get: > > WARNING: reducing size to maximum of 67108864 blocks per swap unit > > Is there workaround for this limitation? Apparently, the 32GB swapspace limit is per swap area; you can add up to 4 swap areas so create two or three 32GB swap partitions. Regards, -- -Chuck PS: Older thread with some specific details: Begin forwarded message: > From: Matthew Dillon > Date: August 5, 2008 11:29:21 AM PDT > To: Max Laier > Cc: freebsd-stable@freebsd.org, Lin Jui-Nan Eric > Subject: Re: Max size of one swap slice > > :> Recently we found that we can only allocate 32GB for one swap slice. > :> Does there is any sysctl oid or any kernel option to increase it? Why > :> we have this restriction? > : > :this is a consequence of the data structure used to manage swap space. See > :sys/blist.h for details. It *seems* that you *might* be able to increase the > :coverage by decreasing BLIST_META_RADIX, but that's from a quick glance and > :most certainly not a good idea. > : > :However, the blist is a abstract enough API so that you can likely replace it > :with something that supports 64bit addresses (and thus 512*2^64 bytes of swap > :space per device) ... but I don't see why you'd want to do something like > :this. Remember that you need memory to manage your swap space as well! > : > :-- > :/"\ Best regards, | mlaier@freebsd.org > :\ / Max Laier | ICQ #67774661 > > The core structures can handle 2 billion swap pages == 2TB of swap, > but the blist code hits arithmatic overflows if a single blist has > more then (0x40000000 / BLIST_META_RADIX) = 1G/16 = 64M swap blocks, > or 256GB. > > I think the VM/BIO system had additional overflow issues due to > conversions back and forth between PAGE_SIZE and DEV_BSIZE which > further restricted the limit to 32GB. Those restrictions may be gone > now that FreeBSD is using 64 bit block numbers, so you may be able to > pop it up to 256GB with virtually no effort (but you need to test it > significantly!). > > With some work on the blist code only (not its structures) the arithmatic > overflow issues could also be resolved, increasing the swap capability > to 2TB. > > I do not recommend changing any of the core blist structure, particularly > not BLIST_META_RADIX. Just don't try :-). You do NOT want to bump > the swap block number fields to 64 bits. > > Also note that significant memory is used to manage that much swap. It's > a factor of 1:16384 or so for the blist structures and probably about > the same amount for the vm_object tracking structures. 32G of swap needs > around 2-4MB of wired ram. > > -Matt > Matthew Dillon > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 15:46:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7EB1E1065670 for ; Tue, 9 Aug 2011 15:46:48 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (m209-73.dsl.rawbw.com [198.144.209.73]) by mx1.freebsd.org (Postfix) with ESMTP id 57CCD8FC1B for ; Tue, 9 Aug 2011 15:46:48 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.5/8.14.5) with ESMTP id p79FGkDJ042114; Tue, 9 Aug 2011 08:16:46 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.5/8.14.5/Submit) id p79FGkWH042113; Tue, 9 Aug 2011 08:16:46 -0700 (PDT) (envelope-from david) Date: Tue, 9 Aug 2011 08:16:46 -0700 From: David Wolfskill To: Daniel Kalchev Message-ID: <20110809151646.GF1814@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , Daniel Kalchev , FreeBSD-STABLE Mailing List References: <4E4143A6.6030307@digsys.bg> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1Y7d0dPL928TPQbc" Content-Disposition: inline In-Reply-To: <4E4143A6.6030307@digsys.bg> User-Agent: Mutt/1.4.2.3i Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 15:46:48 -0000 --1Y7d0dPL928TPQbc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 09, 2011 at 05:26:46PM +0300, Daniel Kalchev wrote: > I am trying to set up 64GB partitions for swap for a system that has=20 > 64GB of RAM (with the idea to dump kernel core etc). But, on 8-stable as= =20 > of today I get: >=20 > WARNING: reducing size to maximum of 67108864 blocks per swap unit >=20 > Is there workaround for this limitation? While FreeBSD cannot address more than 32GB per swap space, it permits as many as 32 swap spaces to be active concurrently. (Sorry; I'm not finding the reference for this just now.) Peace, david --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --1Y7d0dPL928TPQbc Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk5BT10ACgkQmprOCmdXAD1S7QCghA6T9MIr6fJGUetaumSirY26 dDMAn3y1tQDXOBzQLFfZWNArnfDbogFg =/oMI -----END PGP SIGNATURE----- --1Y7d0dPL928TPQbc-- From owner-freebsd-stable@FreeBSD.ORG Tue Aug 9 18:07:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 700941065673 for ; Tue, 9 Aug 2011 18:07:16 +0000 (UTC) (envelope-from cpghost@cordula.ws) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 302758FC17 for ; Tue, 9 Aug 2011 18:07:15 +0000 (UTC) Received: by vws18 with SMTP id 18so295688vws.13 for ; Tue, 09 Aug 2011 11:07:15 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.20.232 with SMTP id q8mr7568865vde.125.1312913235184; Tue, 09 Aug 2011 11:07:15 -0700 (PDT) Received: by 10.220.85.129 with HTTP; Tue, 9 Aug 2011 11:07:15 -0700 (PDT) X-Originating-IP: [93.221.185.184] In-Reply-To: References: <4DF3913F.1000108@lazlarlyricon.com> <4DFB516B.30006@it4pro.pl> <4DFB6EEC.20206@lazlarlyricon.com> Date: Tue, 9 Aug 2011 20:07:15 +0200 Message-ID: From: "C. P. Ghost" To: Olivier Smedts Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: "log_sysevent: type 19 is not implemented" messages during boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 18:07:16 -0000 On Fri, Jun 17, 2011 at 5:28 PM, Olivier Smedts wrote: > 2011/6/17 Rolf Nielsen : >> 2011-06-17 16:29, Artem Belevich skrev: >> <...> >>>> >>>> Are you sure that it's harmless? It appeared for me as an evidence of >>>> pool >>>> breakage. I had these messages when I ran any zpool command on broken >>>> pool. >>>> I do't havesingle one after pool is fixed. Here's my thread on freebsd= -fs >>>> : >>>> http://lists.freebsd.org/pipermail/freebsd-fs/2011-June/011639.html >>> >>> Indeed. Same story here. Last week I've got my pool corrupted due to a >>> bad memory stick. =A0Then I've got tons of thse "log_sysevent: type >>> 19..." messages. After re-importing the pool with -F the messages went >>> away. So, from where I stand, those messages do seem to correlate with >>> a problem and should not be hushed by default. > > On mine, it pops up at boot on the console if the cache device is > inaccessible (usb key). No real problem on the pool, but an > information with more meaning would be useful. On one of my test systems, I have a zpool consisting of 3 HDDs in external enclosures attached via eSATA ports. If the enclosures are not powered (i.e. the disks are off), and the system boots, then this message appears. Of course, the zpool isn't available then, not being available physically. OTOH, when the enclosures are on, and the system boots, this message doesn't show up, and the zpool is available and healthy alright. This message appeared since ZFSv28, and on this system and in this specific scenario, it is absolutely harmless. >> /Rolf > > Olivier Smedts=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0 _ -cpghost. --=20 Cordula's Web. http://www.cordula.ws/ From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 01:52:46 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4EBE1065675 for ; Wed, 10 Aug 2011 01:52:46 +0000 (UTC) (envelope-from lichray@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8638E8FC1C for ; Wed, 10 Aug 2011 01:52:46 +0000 (UTC) Received: by gwb15 with SMTP id 15so484381gwb.13 for ; Tue, 09 Aug 2011 18:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=1Kdiv7JRKgWIW9fERkIspBnwN1OsDMeIJ6nd5ifiN24=; b=fLcw5yz/37H5n2+zjApHbnvYSefshL0/doxGwy2fEe0IRABmtk/zQtjoUTbMQLDdkQ iKcRhZScjAtOMuoTTlcBMOo4ckentwRuDsr3IL4ZYuxZCDzsd/5x8uhGT4b8liaOMtVc k+mAyopKLnmmbnM7oHdRbidZo6a+BGqWGFrIw= MIME-Version: 1.0 Received: by 10.42.147.72 with SMTP id m8mr4956659icv.86.1312939429649; Tue, 09 Aug 2011 18:23:49 -0700 (PDT) Received: by 10.231.33.205 with HTTP; Tue, 9 Aug 2011 18:23:49 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 20:23:49 -0500 Message-ID: From: Zhihao Yuan To: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Fwd: Again, snd_hda headphones problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 01:52:46 -0000 Hi, Sorry about to post this on two freebsd- mailing lists. This problem makes me unable to use external headphones/speakers, which is a very serious problem. I pasted a full verbose boot log here: http://lists.freebsd.org/pipermail/freebsd-multimedia/2011-August/012389.ht= ml ---------- Forwarded message ---------- From: Zhihao Yuan Date: Wed, Aug 3, 2011 at 1:43 PM Subject: Again, snd_hda headphones problem To: freebsd-multimedia@freebsd.org Hi, I posted this message on FreeBSD forum, but got no response. So I'm seeking some help here. I bought a laptop, HP Elitebook 8540w. ~> uname -a FreeBSD compaq.yuetime 8.2-STABLE FreeBSD 8.2-STABLE #4 r224501: Fri Jul 29 22:28:38 CDT 2011 =C2=A0 =C2=A0 lichray@compaq.yuetime:/usr/obj/home/lichray/devel/freebsd-st= able/sys/HOUKAGO =C2=A0amd64 I have already set up the default_unit: ~> cat /dev/sndstat FreeBSD Audio Driver (newpcm: 64bit 2009061500/amd64) Installed devices: pcm0: (play) pcm1: (play) pcm2: (play) pcm3: (play) pcm4: (play/rec) default pcm5: (play) pindump: hdac1: Dumping AFG cad=3D0 nid=3D1 pins: hdac1: =C2=A0nid 10 0x2121101f as =C2=A01 seq 15 =C2=A0 =C2=A0Headphones = =C2=A0Jack jack =C2=A01 loc 33 color =C2=A0 Black misc 0 hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: =C2=A0 =C2=A0OUT HP =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Sense: 0x00000000 hdac1: =C2=A0nid 11 0x03a1102e as =C2=A02 seq 14 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Mic =C2=A0Jack jack =C2=A01 loc =C2=A03 color =C2=A0 Black misc 0 hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: IN =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 VREF Sense: 0x00000000 hdac1: =C2=A0nid 12 0x90a70120 as =C2=A02 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Mic Fixed jack =C2=A07 loc 16 color Unknown misc 1 hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: IN =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 VREF Sense: 0x00000000 hdac1: =C2=A0nid 13 0x90170110 as =C2=A01 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 = Speaker Fixed jack =C2=A07 loc 16 color Unknown misc 1 hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: =C2=A0 =C2=A0OUT =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sense: 0x00000000 hdac1: =C2=A0nid 14 0x21811040 as =C2=A04 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 = Line-in =C2=A0Jack jack =C2=A01 loc 33 color =C2=A0 Black misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: IN =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 VREF Sense: 0x00000000 hdac1: =C2=A0nid 15 0x03211030 as =C2=A03 seq =C2=A00 =C2=A0 =C2=A0Headphon= es =C2=A0Jack jack =C2=A01 loc =C2=A03 color =C2=A0 Black misc 0 hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: =C2=A0 =C2=A0OUT =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sense: 0x00000000 hdac1: =C2=A0nid 20 0x40f000f0 as 15 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Other =C2=A0None jack =C2=A00 loc =C2=A00 color Unknown misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: IN OUT hdac1: =C2=A0nid 24 0x40f000f0 as 15 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Other =C2=A0None jack =C2=A00 loc =C2=A00 color Unknown misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: IN hdac1: =C2=A0nid 25 0x40f000f0 as 15 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Other =C2=A0None jack =C2=A00 loc =C2=A00 color Unknown misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: IN hdac1: =C2=A0nid 30 0x40f000f0 as 15 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Other =C2=A0None jack =C2=A00 loc =C2=A00 color Unknown misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: =C2=A0 =C2=A0OUT hdac1: =C2=A0nid 31 0x40f000f0 as 15 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Other =C2=A0None jack =C2=A00 loc =C2=A00 color Unknown misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: =C2=A0 =C2=A0OUT =C2=A0 =C2=A0EAPD hdac1: =C2=A0nid 32 0x40f000f0 as 15 seq =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Other =C2=A0None jack =C2=A00 loc =C2=A00 color Unknown misc 0 [DISABLED] hdac1: =C2=A0 =C2=A0 =C2=A0 =C2=A0Caps: =C2=A0 =C2=A0OUT hdac1: NumGPIO=3D8 NumGPO=3D0 NumGPI=3D0 GPIWake=3D1 GPIUnsol=3D1 hdac1: GPIO: data=3D0x00000000 enable=3D0x00000000 direction=3D0x00000000 hdac1: =C2=A0 =C2=A0 =C2=A0 wake=3D0x00000000 =C2=A0unsol=3D0x00000000 =C2= =A0 =C2=A0sticky=3D0x00000000 And the sound comes from the internal speaker, and the internal record works. But the sound does not work on the headphones. I read many posts and the snd_hda(4), and I know I need to bind the headphones and the internal speaker to the same as, and I need to do the same on output. But I don't know which device is which, and I just don't know how to deal with the seq. Please help me if you are pro on this. -- Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___________________________________________________ 4BSD -- http://4bsd.biz/ --=20 Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___________________________________________________ 4BSD -- http://4bsd.biz/ From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 03:12:11 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BD151065680 for ; Wed, 10 Aug 2011 03:12:11 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id ED67F8FC19 for ; Wed, 10 Aug 2011 03:12:10 +0000 (UTC) Received: by qwc9 with SMTP id 9so430975qwc.13 for ; Tue, 09 Aug 2011 20:12:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=OyJ8KUz0eE0jqz7YHDk43Fxet/NwLAtgUkg55dYjp0Q=; b=AlyEJ0Zins22S+JQSw7Us0wSffo8ejcqgc2UCN0TvGXviafIzvysnjXV1lFmqBlZQ+ gO4gNYNfOlklzjn1zI/yxefjnL7TzBdhstW44Qk3oAhYzIbnxRsR3W5Ht+oEpA6BQsyW oq0RsJiXiWepEr9wX9ET8h5Vjr8AId0FLufyg= MIME-Version: 1.0 Received: by 10.229.8.138 with SMTP id h10mr5658682qch.105.1312945930137; Tue, 09 Aug 2011 20:12:10 -0700 (PDT) Received: by 10.229.185.85 with HTTP; Tue, 9 Aug 2011 20:12:10 -0700 (PDT) Date: Tue, 9 Aug 2011 20:12:10 -0700 Message-ID: From: Kevin Oberman To: "freebsd-stable@freebsd.org Stable" Content-Type: text/plain; charset=ISO-8859-1 Subject: GPT boot blocks, booting and booteasy X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 03:12:11 -0000 Running FreeBSD 8-Stable of July 30 and I am trying to figure out a bit about how GPT works. I am unsure of what I can do with boot blocks. 000000000011111111112222222222333333333344444444445555555555666666666677 Here is what I've done so far. I have just created my first GPT structured FreeBSD system and it's all working fine, but I don't have any ability to select booting from another partition or drive. I boot the system and FreeBSD just boots with no option to boot from another disk. I have /boot/pmbr loaded into the PMBR and gptboot into the freebsd-boot partition. I'll admit that I did this by rote and don't understand how these two files interact with the UEFI BIOS to get the loader started. I'm not really certain that I even need both. Is it possible to build a "custom" booteasy boot system with boot0cfg or some other tool so I can select d ifferent bootable partition or my other disk which is sliced in the traditional fashion? Can anyone point me to any information on how the boot process works with GPT? Thanks! -- R. Kevin Oberman, Network Engineer - Retired E-mail: kob6558@gmail.com From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 07:13:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53A25106566B for ; Wed, 10 Aug 2011 07:13:35 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id BDCA08FC08 for ; Wed, 10 Aug 2011 07:13:34 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p7A7DFum049078 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 10 Aug 2011 10:13:22 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E422F8A.1070508@digsys.bg> Date: Wed, 10 Aug 2011 10:13:14 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110720 Thunderbird/5.0 MIME-Version: 1.0 To: David Wolfskill , FreeBSD-STABLE Mailing List References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> In-Reply-To: <20110809151646.GF1814@albert.catwhisker.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 07:13:35 -0000 On 09.08.11 18:16, David Wolfskill wrote: > While FreeBSD cannot address more than 32GB per swap space, it permits > as many as 32 swap spaces to be active concurrently. I am more concerned that with 32GB of swap in single device I could not dump kernel core, with 64GB of RAM. Daniel From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 07:48:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76A94106566B for ; Wed, 10 Aug 2011 07:48:01 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 5E9438FC0A for ; Wed, 10 Aug 2011 07:48:01 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta06.emeryville.ca.mail.comcast.net with comcast id JXlw1h0031vN32cA6XnxFW; Wed, 10 Aug 2011 07:47:57 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta22.emeryville.ca.mail.comcast.net with comcast id JXnz1h0021t3BNj8iXnz81; Wed, 10 Aug 2011 07:48:00 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 30D67102C19; Wed, 10 Aug 2011 00:47:59 -0700 (PDT) Date: Wed, 10 Aug 2011 00:47:59 -0700 From: Jeremy Chadwick To: Daniel Kalchev Message-ID: <20110810074759.GA30254@icarus.home.lan> References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E422F8A.1070508@digsys.bg> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 07:48:01 -0000 On Wed, Aug 10, 2011 at 10:13:14AM +0300, Daniel Kalchev wrote: > On 09.08.11 18:16, David Wolfskill wrote: > >While FreeBSD cannot address more than 32GB per swap space, it > >permits as many as 32 swap spaces to be active concurrently. > > I am more concerned that with 32GB of swap in single device I could > not dump kernel core, with 64GB of RAM. My apologies if I've misunderstood something, but why does this of any concern? Machine has 64GB RAM. You have a single swap slice that's effectively 32GB. How is a kernel panic worth of 64GB RAM going to fit into a 32GB swap slice? I think what folks are saying is that if you use multiple swap slices (e.g. two of 32GB each), you can achieve what you need. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 08:09:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65C37106566B for ; Wed, 10 Aug 2011 08:09:27 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id E47A38FC08 for ; Wed, 10 Aug 2011 08:09:26 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p7A89G2O049693 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 10 Aug 2011 11:09:22 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E423CAC.20008@digsys.bg> Date: Wed, 10 Aug 2011 11:09:16 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110720 Thunderbird/5.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> In-Reply-To: <20110810074759.GA30254@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 08:09:27 -0000 On 10.08.11 10:47, Jeremy Chadwick wrote: > On Wed, Aug 10, 2011 at 10:13:14AM +0300, Daniel Kalchev wrote: >> I am more concerned that with 32GB of swap in single device I could >> not dump kernel core, with 64GB of RAM. > My apologies if I've misunderstood something, but why does this of any > concern? Machine has 64GB RAM. You have a single swap slice that's > effectively 32GB. How is a kernel panic worth of 64GB RAM going to fit > into a 32GB swap slice? > The swap partitions are 64GB, it is only that FreeBSD refuses to use more than 32GB of each for swap. But.. it might happily dump core to the whole partition, tests will show. Daniel From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 08:21:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F8791065676 for ; Wed, 10 Aug 2011 08:21:39 +0000 (UTC) (envelope-from perryh@pluto.rain.com) Received: from agora.rdrop.com (agora.rdrop.com [IPv6:2607:f678:1010::34]) by mx1.freebsd.org (Postfix) with ESMTP id 7061F8FC18 for ; Wed, 10 Aug 2011 08:21:39 +0000 (UTC) Received: from agora.rdrop.com (66@localhost [127.0.0.1]) by agora.rdrop.com (8.13.1/8.12.7) with ESMTP id p7A8LUda085554 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 10 Aug 2011 01:21:30 -0700 (PDT) (envelope-from perryh@pluto.rain.com) Received: (from uucp@localhost) by agora.rdrop.com (8.13.1/8.12.9/Submit) with UUCP id p7A8LUmm085553; Wed, 10 Aug 2011 01:21:30 -0700 (PDT) Received: from fbsd81 ([192.168.200.81]) by pluto.rain.com (4.1/SMI-4.1-pluto-M2060407) id AA08126; Wed, 10 Aug 11 01:16:21 PDT Date: Wed, 10 Aug 2011 08:16:16 -0700 From: perryh@pluto.rain.com To: cswiger@mac.com Message-Id: <4e42a0c0.e2t/9MF98O3HFjb1%perryh@pluto.rain.com> References: <4E4143A6.6030307@digsys.bg> <935F8EC2-88E0-45A3-BE8B-7210BE223BC5@mac.com> In-Reply-To: <935F8EC2-88E0-45A3-BE8B-7210BE223BC5@mac.com> User-Agent: nail 11.25 7/29/05 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, daniel@digsys.bg Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 08:21:39 -0000 Chuck Swiger wrote: > On Aug 9, 2011, at 7:26 AM, Daniel Kalchev wrote: > > I am trying to set up 64GB partitions for swap for a system that > > has 64GB of RAM (with the idea to dump kernel core etc). But, on > > 8-stable as of today I get: > > > > WARNING: reducing size to maximum of 67108864 blocks per swap unit > > > > Is there workaround for this limitation? > > Apparently, the 32GB swapspace limit is per swap area; you can add > up to 4 swap areas so create two or three 32GB swap partitions. Will that enable a 64GB dump? In 8.1, dumpon(8) says: The dumpon utility is used to specify a device where the kernel can save a crash dump in the case of a panic. ... For most systems the size of the specified dump device must be at least the size of physical memory. ... The dumpon utility will refuse to enable a dump device which is smaller than the total amount of physical memory as reported by the hw.physmem sysctl(8) variable. Note the use of the singluar: "a device" and "the specified device". From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 08:38:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B42E81065673 for ; Wed, 10 Aug 2011 08:38:37 +0000 (UTC) (envelope-from Holger.Kipp@alogis.com) Received: from alogis.com (firewall.solit-ag.de [212.184.102.1]) by mx1.freebsd.org (Postfix) with ESMTP id 3D0938FC0A for ; Wed, 10 Aug 2011 08:38:36 +0000 (UTC) Received: from msx3.exchange.alogis.com (msx3exchange.alogis.com [10.1.1.6] (may be forged)) by alogis.com (8.13.4/8.13.1) with ESMTP id p7A8QGqp016144; Wed, 10 Aug 2011 10:26:16 +0200 (CEST) (envelope-from Holger.Kipp@alogis.com) Received: from MSX3.exchange.alogis.com ([fe80::c8ed:428a:a157:b61]) by msx3.exchange.alogis.com ([fe80::c8ed:428a:a157:b61%13]) with mapi id 14.01.0255.000; Wed, 10 Aug 2011 10:27:28 +0200 From: Holger Kipp To: Daniel Kalchev Thread-Topic: 32GB limit per swap device? Thread-Index: AQHMVqDZ7UbCWbbykkWQgMEFR68dspUUf+sAgAELPACAAAm2gIAABfIAgAAEtAA= Date: Wed, 10 Aug 2011 08:27:27 +0000 Message-ID: References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> In-Reply-To: <4E423CAC.20008@digsys.bg> Accept-Language: en-GB, de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.1.1.4] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: FreeBSD-STABLE Mailing List , Jeremy Chadwick Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 08:38:37 -0000 Am 10.08.2011 um 10:09 schrieb Daniel Kalchev: > On 10.08.11 10:47, Jeremy Chadwick wrote: >> On Wed, Aug 10, 2011 at 10:13:14AM +0300, Daniel Kalchev wrote: >>> I am more concerned that with 32GB of swap in single device I could not= dump kernel core, with 64GB of RAM. >> My apologies if I've misunderstood something, but why does this of any >> concern? Machine has 64GB RAM. You have a single swap slice that's >> effectively 32GB. How is a kernel panic worth of 64GB RAM going to fit >> into a 32GB swap slice? >> > The swap partitions are 64GB, it is only that FreeBSD refuses to use more= than 32GB of each for swap. But.. it might happily dump core to the whole = partition, tests will show. I doubt it. Have you tried increasing kern.maxswzone? It is the size in KB = (for 32GB it is set to 33554432). kern.maxswzone: Maximum memory for swap metadata Best regards, Holger -- Holger Kipp Diplom-Mathematiker Senior Consultant Tel. : +49 30 436 58 114 Fax. : +49 30 436 58 214 Mobil: +49 178 36 58 114 Email: holger.kipp@alogis.com alogis AG Alt-Moabit 90b D-10559 Berlin web : http://www.alogis.com ---------------------------------------------------------- alogis AG Sitz/Registergericht: Berlin/AG Charlottenburg, HRB 71484 Vorstand: Arne Friedrichs, Joern Samuelson Aufsichtsratsvorsitzender: Reinhard Mielke From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 08:48:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A15131065676 for ; Wed, 10 Aug 2011 08:48:02 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.westchester.pa.mail.comcast.net (qmta13.westchester.pa.mail.comcast.net [76.96.59.243]) by mx1.freebsd.org (Postfix) with ESMTP id 4108F8FC12 for ; Wed, 10 Aug 2011 08:48:01 +0000 (UTC) Received: from omta19.westchester.pa.mail.comcast.net ([76.96.62.98]) by qmta13.westchester.pa.mail.comcast.net with comcast id JYkP1h00227AodY5DYo2xZ; Wed, 10 Aug 2011 08:48:02 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta19.westchester.pa.mail.comcast.net with comcast id JYo01h00S1t3BNj3fYo1rm; Wed, 10 Aug 2011 08:48:02 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6E493102C19; Wed, 10 Aug 2011 01:47:59 -0700 (PDT) Date: Wed, 10 Aug 2011 01:47:59 -0700 From: Jeremy Chadwick To: Holger Kipp Message-ID: <20110810084759.GA32346@icarus.home.lan> References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD-STABLE Mailing List , Daniel Kalchev Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 08:48:02 -0000 On Wed, Aug 10, 2011 at 08:27:27AM +0000, Holger Kipp wrote: > > Am 10.08.2011 um 10:09 schrieb Daniel Kalchev: > > > On 10.08.11 10:47, Jeremy Chadwick wrote: > >> On Wed, Aug 10, 2011 at 10:13:14AM +0300, Daniel Kalchev wrote: > >>> I am more concerned that with 32GB of swap in single device I could not dump kernel core, with 64GB of RAM. > >> My apologies if I've misunderstood something, but why does this of any > >> concern? Machine has 64GB RAM. You have a single swap slice that's > >> effectively 32GB. How is a kernel panic worth of 64GB RAM going to fit > >> into a 32GB swap slice? > >> > > The swap partitions are 64GB, it is only that FreeBSD refuses to use more than 32GB of each for swap. But.. it might happily dump core to the whole partition, tests will show. > > I doubt it. Have you tried increasing kern.maxswzone? It is the size in KB (for 32GB it is set to 33554432). > kern.maxswzone: Maximum memory for swap metadata The variable is set to 32MBytes, not 32GBytes. The default size is defined as variable VM_SWZONE_SIZE_MAX, which is 32*1024*1024 per sys/i386/include/param.h and sys/amd64/include/param.h. Furthermore, this variable does not do what you're implying. It's used within swap_pager_swap_init() in sys/vm/swap_pager.c, which is what's used for initialising a process that's being paged out (swapped out). That doesn't appear to have anything to do with actual memory dumps. So we're back to where we started: swap slices/partitions can be greater than 32GBytes in size, but "something" is limiting the maximum amount of memory which can be dumped to a single swap swap to 32GBytes. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 09:37:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 806AE1065672 for ; Wed, 10 Aug 2011 09:37:36 +0000 (UTC) (envelope-from Holger.Kipp@alogis.com) Received: from alogis.com (firewall.solit-ag.de [212.184.102.1]) by mx1.freebsd.org (Postfix) with ESMTP id 2C94E8FC17 for ; Wed, 10 Aug 2011 09:37:35 +0000 (UTC) Received: from msx3.exchange.alogis.com (msx3exchange.alogis.com [10.1.1.6] (may be forged)) by alogis.com (8.13.4/8.13.1) with ESMTP id p7A9bPVp017697; Wed, 10 Aug 2011 11:37:25 +0200 (CEST) (envelope-from Holger.Kipp@alogis.com) Received: from MSX3.exchange.alogis.com ([fe80::c8ed:428a:a157:b61]) by msx3.exchange.alogis.com ([fe80::c8ed:428a:a157:b61%13]) with mapi id 14.01.0255.000; Wed, 10 Aug 2011 11:38:37 +0200 From: Holger Kipp To: Jeremy Chadwick Thread-Topic: 32GB limit per swap device? Thread-Index: AQHMVqDZ7UbCWbbykkWQgMEFR68dspUUf+sAgAELPACAAAm2gIAABfIAgAAEtACAAAYdgIAADcSA Date: Wed, 10 Aug 2011 09:38:36 +0000 Message-ID: <84407276-7546-4ABE-A0EE-A5E82E1CC358@alogis.com> References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> <20110810084759.GA32346@icarus.home.lan> In-Reply-To: <20110810084759.GA32346@icarus.home.lan> Accept-Language: en-GB, de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.1.1.4] Content-Type: text/plain; charset="us-ascii" Content-ID: <18A3B64EA73E454098033816C76734ED@exchange.alogis.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: FreeBSD-STABLE Mailing List , Daniel Kalchev Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 09:37:36 -0000 Am 10.08.2011 um 10:47 schrieb Jeremy Chadwick: > On Wed, Aug 10, 2011 at 08:27:27AM +0000, Holger Kipp wrote: >> >> Am 10.08.2011 um 10:09 schrieb Daniel Kalchev: >> >>> On 10.08.11 10:47, Jeremy Chadwick wrote: >>>> On Wed, Aug 10, 2011 at 10:13:14AM +0300, Daniel Kalchev wrote: >>>>> I am more concerned that with 32GB of swap in single device I could n= ot dump kernel core, with 64GB of RAM. >>>> My apologies if I've misunderstood something, but why does this of any >>>> concern? Machine has 64GB RAM. You have a single swap slice that's >>>> effectively 32GB. How is a kernel panic worth of 64GB RAM going to fi= t >>>> into a 32GB swap slice? >>>> >>> The swap partitions are 64GB, it is only that FreeBSD refuses to use mo= re than 32GB of each for swap. But.. it might happily dump core to the whol= e partition, tests will show. >> >> I doubt it. Have you tried increasing kern.maxswzone? It is the size in = KB (for 32GB it is set to 33554432). >> kern.maxswzone: Maximum memory for swap metadata > > The variable is set to 32MBytes, not 32GBytes. The default size is > defined as variable VM_SWZONE_SIZE_MAX, which is 32*1024*1024 per > sys/i386/include/param.h and sys/amd64/include/param.h. > > Furthermore, this variable does not do what you're implying. Ah, then I misinterpreted the output of "man loader" which states: kern.maxswzone Limits the amount of KVM to be used to hold swap meta information, which directly governs the maximum amount o= f swap the system can support. > So we're back to where we started: swap slices/partitions can be greater > than 32GBytes in size, but "something" is limiting the maximum amount of > memory which can be dumped to a single swap swap to 32GBytes. Anyway, there is one thing in /usr/src/sys/vm/swap_pager.c (so if you get t= his warning (reducing size to maximum of... per swap unit)), this might be the reason: /* * If we go beyond this, we get overflows in the radix * tree bitmap code. */ mblocks =3D 0x40000000 / BLIST_META_RADIX; if (nblks > mblocks) { printf("WARNING: reducing size to maximum of %lu blocks per= swap unit\n", mblocks); nblks =3D mblocks; } Hope this is of some help. Best regards, Holger -- Holger Kipp Diplom-Mathematiker Senior Consultant Tel. : +49 30 436 58 114 Fax. : +49 30 436 58 214 Mobil: +49 178 36 58 114 Email: holger.kipp@alogis.com alogis AG Alt-Moabit 90b D-10559 Berlin web : http://www.alogis.com ---------------------------------------------------------- alogis AG Sitz/Registergericht: Berlin/AG Charlottenburg, HRB 71484 Vorstand: Arne Friedrichs, Joern Samuelson Aufsichtsratsvorsitzender: Reinhard Mielke From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 10:12:13 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76AA41065670 for ; Wed, 10 Aug 2011 10:12:13 +0000 (UTC) (envelope-from goran.lowkrantz@ismobile.com) Received: from mail.ismobile.com (mail.ismobile.com [62.119.44.68]) by mx1.freebsd.org (Postfix) with ESMTP id 25BDE8FC08 for ; Wed, 10 Aug 2011 10:12:12 +0000 (UTC) Received: from mail.ismobile.com (mail.ismobile.com [62.119.44.68]) by dkim.mail.arcticgroup.se (Postfix) with ESMTP id AFCA11CE5C; Wed, 10 Aug 2011 11:55:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=ismobile.com; h=date:from :to:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=UZM4Hby GyeUTb9pGPDTm/HiaXJA=; b=LpuIZq2tE6pM80KvXpL/aB2tEGKUKKYV8klMvHj j1681vGx+BCusijHuBHvxyH6iCCIdcD8SxSQxlsZ6Le7sRwQkeYuZZcCs2zqSMmQ HT8XI/CN1thRHzqTiHUwxUgcerOZJhY7rgvtKnBJq2waAxafy25DiW1x6AoIgc+m LdKc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=ismobile.com; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=selector1; b=K 2d41jfaybF5Oyc6T1vOfOIOvOJseh+h2xv/9pb8h0opyjauaoD/bE7v9Uwo9ZBS+ KFCdFL/yfopZHl+EXwaHjnXh7P4YHe56lV2oYF8sbUQijWGDFo6bXg8QU8ydkgbv nTz+wnkb3JmkSW0OMWiWiZ40z4Xs3JyqUSeGmjuehQ= Received: from [172.16.2.60] (syn.hq.ismobile.com [172.16.2.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.ismobile.com (Postfix) with ESMTPSA id 2C03C1CE01; Wed, 10 Aug 2011 11:55:10 +0200 (CEST) Date: Wed, 10 Aug 2011 11:55:08 +0200 From: Goran Lowkrantz To: Daniel Kalchev , FreeBSD-STABLE Mailing List Message-ID: <09FA79CBCA4E1ED3C7385539@[172.16.2.60]> In-Reply-To: <4E4143A6.6030307@digsys.bg> References: <4E4143A6.6030307@digsys.bg> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 10:12:13 -0000 Hi Daniel, Just a stupid question, as I have done something different. Can't you use a = different device or slice for the dump? In that case there is no limitation = on the size of the dump device, as far as I know. My setup: 96GB, dump device local 160G disc, slice for swap, slice for=20 dump, system boots from nanobsd usb-stick. All other disk ZFS filesystems=20 using LSI HBA for NFS and Samba clients. MVH G=F6ran --On August 9, 2011 17:26:46 +0300 Daniel Kalchev wrote: > I am trying to set up 64GB partitions for swap for a system that has 64GB > of RAM (with the idea to dump kernel core etc). But, on 8-stable as of > today I get: > > WARNING: reducing size to maximum of 67108864 blocks per swap unit > > Is there workaround for this limitation? > > Daniel > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" ................................................... the future isMobile Goran Lowkrantz System Architect, isMobile AB Sandviksgatan 81, PO Box 58, S-971 03 Lule=E5, Sweden Mobile: +46(0)70-587 87 82 http://www.ismobile.com ............................................... From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 10:44:05 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9416B106566B for ; Wed, 10 Aug 2011 10:44:05 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 1B6398FC16 for ; Wed, 10 Aug 2011 10:44:04 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p7AAgBII050421 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 10 Aug 2011 13:42:17 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E426083.1020207@digsys.bg> Date: Wed, 10 Aug 2011 13:42:11 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110720 Thunderbird/5.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> <20110810084759.GA32346@icarus.home.lan> In-Reply-To: <20110810084759.GA32346@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 10:44:05 -0000 On 10.08.11 11:47, Jeremy Chadwick wrote: > So we're back to where we started: swap slices/partitions can be > greater than 32GBytes in size, but "something" is limiting the maximum > amount of memory which can be dumped to a single swap swap to 32GBytes. It seems there is still some confusion. Partition size(s) is 64GB, but FreeBSD will use only 32GB of that for swap. Trying: sysctl debug.minidump=0 sysctl debug.kdb.panic=1 Produces 64GB dump, but.. I can't see the dump on the swap device (savecore doesn't find the magic number). My swap device is gmirror or two slices (/dev/mirror/swap). Removing the gmirror and listing both slices for swap .. works. I could use separate device for dumps, but the disk real estate on this particular blade is limited... Well, I really asked two questions: Q1: Is FreeBSD still limited to 32GB per swap slice? A1: yes. There is limit set in /usr/src/sys/vm/swap_pager.c Q2: If FreeBSD will only use 32GB of the slice for swap, will it dump larger (64GB in my case) core in there? A2a: yes. A2b: If the swap slice is gmirrored, you don't get any dump back. I believe the gmirror bug might exist in smaller partitions as well, but haven't tested it yet (have few such systems that never duped core). It does not matter if I do full dump or minidump: on gmirrored 64GB partittion savecore does not find anything. Daniel From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 11:19:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B60E6106564A for ; Wed, 10 Aug 2011 11:19:49 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (unknown [IPv6:2a03:3100:c:13::5]) by mx1.freebsd.org (Postfix) with ESMTP id 200A58FC18 for ; Wed, 10 Aug 2011 11:19:48 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.5/8.14.5) with ESMTP id p7ABJYes017652; Wed, 10 Aug 2011 18:19:34 +0700 (NOVST) (envelope-from egrosbein@rdtc.ru) Message-ID: <4E426941.3040208@rdtc.ru> Date: Wed, 10 Aug 2011 18:19:29 +0700 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: Daniel Kalchev References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> <20110810084759.GA32346@icarus.home.lan> <4E426083.1020207@digsys.bg> In-Reply-To: <4E426083.1020207@digsys.bg> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 11:19:49 -0000 10.08.2011 17:42, Daniel Kalchev writes: > I believe the gmirror bug might exist in smaller partitions as well, but > haven't tested it yet (have few such systems that never duped core). It > does not matter if I do full dump or minidump: on gmirrored 64GB > partittion savecore does not find anything. You should read gmirror(8) manual page about "Doing kernel dumps to gmirror providers". Eugene Grosbein From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 12:34:33 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 163F0106564A for ; Wed, 10 Aug 2011 12:34:33 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 933C18FC0C for ; Wed, 10 Aug 2011 12:34:32 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p7ACYMBr050968 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 10 Aug 2011 15:34:28 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E427ACE.1040804@digsys.bg> Date: Wed, 10 Aug 2011 15:34:22 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110720 Thunderbird/5.0 MIME-Version: 1.0 To: Eugene Grosbein References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> <20110810084759.GA32346@icarus.home.lan> <4E426083.1020207@digsys.bg> <4E426941.3040208@rdtc.ru> In-Reply-To: <4E426941.3040208@rdtc.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 12:34:33 -0000 On 10.08.11 14:19, Eugene Grosbein wrote: > You should read gmirror(8) manual page about "Doing kernel dumps to > gmirror providers". Thanks, I totally forgot about the gmirror limitations. When using the default minidump, the result is: savecore: first and last dump headers disagree on /dev/mirror/swap There seems to be no problem when a full dump is performed. This is probably an entirely unrelated issue however. Daniel From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 13:21:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 170241065677 for ; Wed, 10 Aug 2011 13:21:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id B53BC8FC16 for ; Wed, 10 Aug 2011 13:21:50 +0000 (UTC) Received: from omta23.westchester.pa.mail.comcast.net ([76.96.62.74]) by qmta03.westchester.pa.mail.comcast.net with comcast id JdH71h0091c6gX853dMqW0; Wed, 10 Aug 2011 13:21:50 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta23.westchester.pa.mail.comcast.net with comcast id JdMp1h0041t3BNj3jdMqlC; Wed, 10 Aug 2011 13:21:50 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 2F2F6102C19; Wed, 10 Aug 2011 06:21:48 -0700 (PDT) Date: Wed, 10 Aug 2011 06:21:48 -0700 From: Jeremy Chadwick To: Daniel Kalchev Message-ID: <20110810132148.GA37186@icarus.home.lan> References: <4E4143A6.6030307@digsys.bg> <20110809151646.GF1814@albert.catwhisker.org> <4E422F8A.1070508@digsys.bg> <20110810074759.GA30254@icarus.home.lan> <4E423CAC.20008@digsys.bg> <20110810084759.GA32346@icarus.home.lan> <4E426083.1020207@digsys.bg> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E426083.1020207@digsys.bg> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD-STABLE Mailing List Subject: Re: 32GB limit per swap device? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 13:21:51 -0000 On Wed, Aug 10, 2011 at 01:42:11PM +0300, Daniel Kalchev wrote: > > > On 10.08.11 11:47, Jeremy Chadwick wrote: > >So we're back to where we started: swap slices/partitions can be > >greater than 32GBytes in size, but "something" is limiting the > >maximum amount of memory which can be dumped to a single swap swap > >to 32GBytes. > It seems there is still some confusion. Partition size(s) is 64GB, > but FreeBSD will use only 32GB of that for swap. > > Trying: > > sysctl debug.minidump=0 > sysctl debug.kdb.panic=1 > > Produces 64GB dump, but.. I can't see the dump on the swap device > (savecore doesn't find the magic number). My swap device is gmirror > or two slices (/dev/mirror/swap). > > Removing the gmirror and listing both slices for swap .. works. > > > I could use separate device for dumps, but the disk real estate on > this particular blade is limited... > > Well, I really asked two questions: > Q1: Is FreeBSD still limited to 32GB per swap slice? > A1: yes. There is limit set in /usr/src/sys/vm/swap_pager.c > > Q2: If FreeBSD will only use 32GB of the slice for swap, will it > dump larger (64GB in my case) core in there? > A2a: yes. > A2b: If the swap slice is gmirrored, you don't get any dump back. > > I believe the gmirror bug might exist in smaller partitions as well, > but haven't tested it yet (have few such systems that never duped > core). It does not matter if I do full dump or minidump: on > gmirrored 64GB partittion savecore does not find anything. Is this somehow related to the NOTES section of gmirror(1), which talks about caveats/problems with generating kernel dumps on gmirrored swap? It seems that as the more technical details of your system start to present themselves, the answers start to present themselves as well. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 14:33:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 313E8106564A for ; Wed, 10 Aug 2011 14:33:50 +0000 (UTC) (envelope-from prvs=1203777b38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id B01F68FC1D for ; Wed, 10 Aug 2011 14:33:49 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 15:22:25 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 15:22:25 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014544638.msg for ; Wed, 10 Aug 2011 15:22:24 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1203777b38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> From: "Steven Hartland" To: Date: Wed, 10 Aug 2011 15:22:52 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Subject: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 14:33:50 -0000 We're currently experiencing a large number of kernel panics on FreeBSD 8.2-RELEASE across a large number of machines here. The base stack reported is a double fault with no additional details and CTRL+ALT+ESC fails to break to the debugger as does and NMI, even though it at least tries printing the following many times some quite jumbled:- NMI ... going to debugger We've configured the dump device but that also seems to fail to capture any details just sitting there after panic with Dumping 4465MB: The machines are single disk ZFS root install and the dump device is configured using the gptid, could this be what's preventing the dump happening? The kernel is compiled with:- options KDB # Kernel debugger related code options KDB_TRACE # Print a stack trace for a panic We have remove KVM but not remote serial on the most of the machines. Any advice on how to debug this issue? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 14:44:03 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD197106566B; Wed, 10 Aug 2011 14:44:03 +0000 (UTC) (envelope-from VaNs9@yandex.ru) Received: from forward11.mail.yandex.net (forward11.mail.yandex.net [IPv6:2a02:6b8:0:801::1]) by mx1.freebsd.org (Postfix) with ESMTP id 6EE0F8FC16; Wed, 10 Aug 2011 14:44:02 +0000 (UTC) Received: from web150.yandex.ru (web150.yandex.ru [95.108.130.108]) by forward11.mail.yandex.net (Yandex) with ESMTP id BD9FFE859A1; Wed, 10 Aug 2011 18:44:00 +0400 (MSD) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1312987440; bh=sw/IOv57EJzbj8ANrFhne79DtT/Bu/vo3IEEPaeTlOs=; h=From:To:Cc:Subject:MIME-Version:Message-Id:Date:Content-Type; b=vQrBEO2Pi8C/WlfAAvKukywok8A32crqz610GiZtbO68r1oS0iQwLvwTpymKfNGDw xL4Ivbrdj2TG+3O0T4Z5dmvJSmQwGknXMYR2pJhlmaR26y2reK8poP5VRaBAuVq4oQ 20IKRwIw8CY6JFL662mhDhyv6wEQrRvnhOmshHYU= Received: from localhost (localhost.localdomain [127.0.0.1]) by web150.yandex.ru (Yandex) with ESMTP id A8BBC6570377; Wed, 10 Aug 2011 18:44:00 +0400 (MSD) X-Yandex-Spam: 1 Received: from [83.102.175.242] ([83.102.175.242]) by web150.yandex.ru with HTTP; Wed, 10 Aug 2011 18:44:00 +0400 From: N V To: current@freebsd.org MIME-Version: 1.0 Message-Id: <491711312987440@web150.yandex.ru> Date: Wed, 10 Aug 2011 18:44:00 +0400 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Content-Type: multipart/mixed; boundary="----==--bound.49172.web150.yandex.ru" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: stable@freebsd.org Subject: FreeBSD-9.0-BETA1-i386-bootonly X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 14:44:03 -0000 ------==--bound.49172.web150.yandex.ru Content-Transfer-Encoding: 7bit Content-Type: text/plain Hi. Tried to use FreeBSD-9.0-BETA1-i386-bootonly.iso in VirtualBox to test. Installation stops after trying to fetch files from ftp. Attached screenshot is informative, I think. Seems to use i386/ twice for some reason. Regards, Vans. ------==--bound.49172.web150.yandex.ru-- From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 15:12:58 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 262141065670 for ; Wed, 10 Aug 2011 15:12:58 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.emeryville.ca.mail.comcast.net (qmta14.emeryville.ca.mail.comcast.net [76.96.27.212]) by mx1.freebsd.org (Postfix) with ESMTP id 0E2BD8FC14 for ; Wed, 10 Aug 2011 15:12:57 +0000 (UTC) Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87]) by qmta14.emeryville.ca.mail.comcast.net with comcast id JfBu1h0021smiN4AEfCtzW; Wed, 10 Aug 2011 15:12:53 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta20.emeryville.ca.mail.comcast.net with comcast id JfDB1h00u1t3BNj8gfDB8X; Wed, 10 Aug 2011 15:13:12 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 8958D102C19; Wed, 10 Aug 2011 08:12:56 -0700 (PDT) Date: Wed, 10 Aug 2011 08:12:56 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20110810151256.GA38601@icarus.home.lan> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 15:12:58 -0000 On Wed, Aug 10, 2011 at 03:22:52PM +0100, Steven Hartland wrote: > The base stack reported is a double fault with no additional > details and CTRL+ALT+ESC fails to break to the debugger as > does and NMI, even though it at least tries printing the > following many times some quite jumbled:- > NMI ... going to debugger You may be interested in these system tunables (not sysctls). These come from sys/amd64/amd64/trap.c (i386 has the same): machdep.kdb_on_nmi (defaults to 1) machdep.panic_on_nmi (defaults to 1) If what you're seeing is a hardware NMI that fires, followed by the machine panicing, the above tunables are probably doing that. A hardware NMI could indicate an actual hardware issue of sorts, depending on how the motherboard vendor implemented what they did. For example, on a series of mainboards we have at my workplace, the BIOS can be configured to generate either an NMI or SMI# when different kinds of ECC RAM errors happen (either single-bit or multi-bit parity errors). I don't know if that's what you're seeing. If you're generating the NMI yourself (possibly via the KVM, etc.) then okay, that's different. I'm trying to discern whether or not *you're* generating the NMI, or if the NMI just happens and causes a panic for you and that's what you're worried about. Now to discuss the "jumbled console output": The interspersing of kernel text output has plagued FreeBSD for a very long time (since approximately 6.x). There have been statements from kernel coders that you can decrease the likelihood of it happening by increasing the PRINTF_BUFR_SIZE (not a typo) option in your kernel configuration. The issue is exacerbated by use of SMP (either multi-core or multi-CPU). The default (assuming your kernel configs are based off of GENERIC within the past 4-5 years) is 128. However, the same developers stated that they have great reservations over increasing this number dramatically (meaning, something like 256 will probably work, but larger "may have repercussions which are unknown at this time"). I have stated publicly then, and will do so again now, that this option does not solve the problem. I acknowledge it may make it "less likely to happen" or may decrease the amount of interspersed output, but in my experience neither of those prove true; and more importantly, said option does not solve the problem. I've talked on-list with John Baldwin about this problem in the past, who had some pretty good ideas of how to solve it. I should point out that Solaris 10 and OpenSolaris (not sure about present-day releases) both have this problem as well, especially during kernel panics or MCEs. Linux addressed this issue by implementing a ring-based cyclic buffer for its kernel messages (syslog/klogd), and the model is extremely well-documented (quite clever too): http://www.mjmwired.net/kernel/Documentation/trace/ring-buffer-design.txt I'm still surprised not a single GSoC project has attempted to solve this for FreeBSD. It really is a serious matter, as it makes getting kernel backtraces and crash data a serious pain in the butt. It can also impact real-time debugging. These are the *worst* times to have to tolerate something like this. I can point you to old threads about this, and my old FreeBSD wiki page ("Commonly reported issues") touches on this as well. The point I want to get across is that PRINTF_BUFR_SIZE does not solve the problem. > We've configured the dump device but that also seems to fail > to capture any details just sitting there after panic with > Dumping 4465MB: > > The machines are single disk ZFS root install and the dump > device is configured using the gptid, could this be what's > preventing the dump happening? I can tell you that others have reported this problem where the kernel panic/dump begins but either locks up after showing the first progress metre/amount, or during the dumping itself. I give everyone the same advice: please make sure that you have a swap partition that's large enough to fit your entire memory contents (preferably a swap that's 2x or 1.5x the amount of physical RAM), and please make sure it's on a dedicated slice (e.g. ada0s1b). I do not advise any sort of "abstraction" layer between swap and the rest of the system. It might seem like a great/fun/awesome idea followed by "whatever jdc, it works!" but when a crash happens -- which is when you need it most -- and it doesn't work, I won't sympathise. :-) As for the GPT aspects of things: I'm still not familiar with GPT (as a technology I am, but when it comes to actual usability I am not). > The kernel is compiled with:- > options KDB # Kernel debugger related code > options KDB_TRACE # Print a stack trace for a panic > > We have remove KVM but not remote serial on the most of the > machines. As long as remote KVM provides actual VGA-level redirection, then that's sufficient (though makes copy-pasting output basically impossible). We use serial console and tend to use these options; the DDB and GDB options may be helpful for you, but not if the system is behaving the way you describe. # Debugging options options BREAK_TO_DEBUGGER # Sending a serial BREAK drops to DDB options ALT_BREAK_TO_DEBUGGER # Permit ~ to drop to DDB options KDB # Enable kernel debugger support options KDB_TRACE # Print stack trace automatically on panic options DDB # Support DDB options GDB # Support remote GDB In combination with this, we use the following in /etc/rc.conf (the dumpdev line is important, else savecore won't pick up anything): dumpdev="auto" ddb_enable="yes" But we do not use any ddb scripts. I keep it in there Just In Case. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 15:24:32 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A7CB106566C for ; Wed, 10 Aug 2011 15:24:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DD59B8FC08 for ; Wed, 10 Aug 2011 15:24:31 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA23719; Wed, 10 Aug 2011 18:24:28 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E42A2AC.3030604@FreeBSD.org> Date: Wed, 10 Aug 2011 18:24:28 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> In-Reply-To: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 15:24:32 -0000 on 10/08/2011 17:22 Steven Hartland said the following: > The kernel is compiled with:- > options KDB # Kernel debugger related code > options KDB_TRACE # Print a stack trace for a panic You also have to provide an actual debugger backend like built-in DDB or a stub for remote GDB to get online debugging. No guarantees that that would help you to get the debugging information, but without that the chances are even slimmer. You may also try this patch and see if it provides any improvements for post-panic environment (dumping etc): http://people.freebsd.org/~avg/stop_scheduler_on_panic.8.x.diff It might also be a good idea to at least capture a screenshot of whatever information you get on console when the panic happens. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 15:36:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D149106564A for ; Wed, 10 Aug 2011 15:36:53 +0000 (UTC) (envelope-from prvs=1203777b38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id EB2F48FC13 for ; Wed, 10 Aug 2011 15:36:52 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 16:35:24 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 16:35:24 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014545369.msg for ; Wed, 10 Aug 2011 16:35:23 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1203777b38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: From: "Steven Hartland" To: References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> Date: Wed, 10 Aug 2011 16:35:51 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 15:36:53 -0000 ----- Original Message ----- From: "Steven Hartland" To: Sent: Wednesday, August 10, 2011 3:22 PM Subject: debugging frequent kernel panics on 8.2-RELEASE > We're currently experiencing a large number of kernel panics > on FreeBSD 8.2-RELEASE across a large number of machines here. > > The base stack reported is a double fault with no additional > details and CTRL+ALT+ESC fails to break to the debugger as > does and NMI, even though it at least tries printing the > following many times some quite jumbled:- > NMI ... going to debugger > > We've configured the dump device but that also seems to fail > to capture any details just sitting there after panic with > Dumping 4465MB: > > The machines are single disk ZFS root install and the dump > device is configured using the gptid, could this be what's > preventing the dump happening? > > The kernel is compiled with:- > options KDB # Kernel debugger related code > options KDB_TRACE # Print a stack trace for a panic > > We have remove KVM but not remote serial on the most of the > machines. > > Any advice on how to debug this issue? ldn32.multiplay.co.uk dumped core - see /var/crash/vmcore.0 Wed Aug 10 14:02:07 UTC 2011 FreeBSD crash 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Jul 21 11:05:52 BST 2011 root@crash:/usr/obj/usr/src/sys/MULTIPLAY amd64 panic: double fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal double fault rip = 0xffffffff8052f6f1 rsp = 0xffffff86ce600fb0 rbp = 0xffffff86ce601210 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff803af91e at kdb_backtrace+0x5e #1 0xffffffff8037d817 at panic+0x187 #2 0xffffffff80574316 at dblfault_handler+0x96 #3 0xffffffff8055d06d at Xdblfault+0xad Uptime: 13d20h53m31s Physical memory: 24555 MB Dumping 3283 MB: 3268 3252 3236 3220 3204 3188 3172 3156 3140 3124 3108 3092 3076 3060 3044 3028 3012 2996 2980 2964 2948 2932 2916 2900 2884 2868 2852 2836 2820 2804 2788 2772 2756 2740 272 4 2708 2692 2676 2660 2644 2628 2612 2596 2580 2564 2548 2532 2516 2500 2484 2468 2452 2436 2420 2404 2388 2372 2356 2340 2324 2308 2292 2276 2260 2244 2228 2212 2196 2180 2164 2148 2132 211 6 2100 2084 2068 2052 2036 2020 2004 1988 1972 1956 1940 1924 1908 1892 1876 1860 1844 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652 1636 1620 1604 1588 1572 1556 1540 1524 150 8 1492 1476 1460 1444 1428 1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204 1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 8 68 852 836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 68 52 36 20 4 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/linprocfs.ko Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/nullfs.ko One of the machines has managed to dump where all the others have failed to do so here's the stack from core.txt.0 #0 sched_switch (td=0xffffffff80830bc0, newtd=0xffffff000a73f8c0, flags=Variable "flags" is not available.) at /usr/src/sys/kern/sched_ule.c:1858 1858 cpuid = PCPU_GET(cpuid); (kgdb) #0 sched_switch (td=0xffffffff80830bc0, newtd=0xffffff000a73f8c0, flags=Variable "flags" is not available.) at /usr/src/sys/kern/sched_ule.c:1858 #1 0xffffffff80385c86 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:449 #2 0xffffffff803b92d2 in sleepq_timedwait (wchan=0xffffffff80830760, pri=68) at /usr/src/sys/kern/subr_sleepqueue.c:644 #3 0xffffffff803861e1 in _sleep (ident=0xffffffff80830760, lock=0x0, priority=Variable "priority" is not available. ) at /usr/src/sys/kern/kern_synch.c:230 #4 0xffffffff80532c29 in scheduler (dummy=Variable "dummy" is not available. ) at /usr/src/sys/vm/vm_glue.c:807 #5 0xffffffff80335d67 in mi_startup () at /usr/src/sys/kern/init_main.c:254 #6 0xffffffff8016efac in btext () at /usr/src/sys/amd64/amd64/locore.S:81 #7 0xffffffff808556e0 in sleepq_chains () #8 0xffffffff8083b1e0 in cpu_top () #9 0x0000000000000000 in ?? () #10 0xffffffff80830bc0 in proc0 () #11 0xffffffff80ba4b90 in ?? () #12 0xffffffff80ba4b38 in ?? () #13 0xffffff000a73f8c0 in ?? () #14 0xffffffff803a2cc9 in sched_switch (td=0x0, newtd=0x0, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1852 Previous frame inner to this frame (corrupt stack?) (kgdb) Not sure this really points to the cause, but we have the crash dump so can do more digging if someone would point me in the correct direction. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 15:46:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F5E9106564A for ; Wed, 10 Aug 2011 15:46:39 +0000 (UTC) (envelope-from prvs=1203777b38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 0A4D08FC19 for ; Wed, 10 Aug 2011 15:46:38 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 16:46:00 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 16:45:57 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014545481.msg for ; Wed, 10 Aug 2011 16:45:57 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1203777b38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: From: "Steven Hartland" To: "Jeremy Chadwick" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <20110810151256.GA38601@icarus.home.lan> Date: Wed, 10 Aug 2011 16:46:17 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 15:46:39 -0000 ----- Original Message ----- From: "Jeremy Chadwick" > On Wed, Aug 10, 2011 at 03:22:52PM +0100, Steven Hartland wrote: >> The base stack reported is a double fault with no additional >> details and CTRL+ALT+ESC fails to break to the debugger as >> does and NMI, even though it at least tries printing the >> following many times some quite jumbled:- >> NMI ... going to debugger > If you're generating the NMI yourself (possibly via the KVM, etc.) then > okay, that's different. I'm trying to discern whether or not *you're* > generating the NMI, or if the NMI just happens and causes a panic for > you and that's what you're worried about. Yer generating it after panic in order to try and get to the debugger :) > Now to discuss the "jumbled console output": ... > The default (assuming your kernel configs are based off of GENERIC > within the past 4-5 years) is 128. However, the same developers stated > that they have great reservations over increasing this number > dramatically (meaning, something like 256 will probably work, but larger > "may have repercussions which are unknown at this time"). Might try that if it will help but with so many production machines to action I'd like to try and avoid if possible. >> The machines are single disk ZFS root install and the dump >> device is configured using the gptid, could this be what's >> preventing the dump happening? > > I can tell you that others have reported this problem where the kernel > panic/dump begins but either locks up after showing the first progress > metre/amount, or during the dumping itself. Ahh, so possibly not a gptid issue > I give everyone the same advice: please make sure that you have a swap > partition that's large enough to fit your entire memory contents > (preferably a swap that's 2x or 1.5x the amount of physical RAM), and > please make sure it's on a dedicated slice (e.g. ada0s1b). I do not > advise any sort of "abstraction" layer between swap and the rest of the > system. It might seem like a great/fun/awesome idea followed by > "whatever jdc, it works!" but when a crash happens -- which is when you > need it most -- and it doesn't work, I won't sympathise. :-) > > As for the GPT aspects of things: I'm still not familiar with GPT (as a > technology I am, but when it comes to actual usability I am not). Just managed to get a crash dump from one machine so hopefully will be able to make some progress is someone can point me in the right direction. > # Debugging options > options BREAK_TO_DEBUGGER # Sending a serial BREAK drops to DDB > options ALT_BREAK_TO_DEBUGGER # Permit ~ to drop to DDB > options KDB # Enable kernel debugger support > options KDB_TRACE # Print stack trace automatically on panic > options DDB # Support DDB > options GDB # Support remote GDB Cheers > In combination with this, we use the following in /etc/rc.conf (the > dumpdev line is important, else savecore won't pick up anything): > > dumpdev="auto" I thought this was ment to be the default from back in the 6.x days but it didnt seem to work, so I added the gptid device from /etc/fstab > ddb_enable="yes" Thanks :) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 16:00:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACA07106564A for ; Wed, 10 Aug 2011 16:00:21 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.westchester.pa.mail.comcast.net (qmta06.westchester.pa.mail.comcast.net [76.96.62.56]) by mx1.freebsd.org (Postfix) with ESMTP id 5E8218FC22 for ; Wed, 10 Aug 2011 16:00:21 +0000 (UTC) Received: from omta11.westchester.pa.mail.comcast.net ([76.96.62.36]) by qmta06.westchester.pa.mail.comcast.net with comcast id Jfv81h0020mv7h056g0MbW; Wed, 10 Aug 2011 16:00:21 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta11.westchester.pa.mail.comcast.net with comcast id Jg0L1h00N1t3BNj3Xg0Lpk; Wed, 10 Aug 2011 16:00:21 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 050B0102C19; Wed, 10 Aug 2011 09:00:19 -0700 (PDT) Date: Wed, 10 Aug 2011 09:00:19 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20110810160018.GA40279@icarus.home.lan> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <20110810151256.GA38601@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:00:21 -0000 On Wed, Aug 10, 2011 at 04:46:17PM +0100, Steven Hartland wrote: > >On Wed, Aug 10, 2011 at 03:22:52PM +0100, Steven Hartland wrote: > >>The base stack reported is a double fault with no additional > >>details and CTRL+ALT+ESC fails to break to the debugger as > >>does and NMI, even though it at least tries printing the > >>following many times some quite jumbled:- > >>NMI ... going to debugger > > >If you're generating the NMI yourself (possibly via the KVM, etc.) then > >okay, that's different. I'm trying to discern whether or not *you're* > >generating the NMI, or if the NMI just happens and causes a panic for > >you and that's what you're worried about. > > Yer generating it after panic in order to try and get to the debugger :) Understood, thanks for clarifying. > >Now to discuss the "jumbled console output": > ... > >The default (assuming your kernel configs are based off of GENERIC > >within the past 4-5 years) is 128. However, the same developers stated > >that they have great reservations over increasing this number > >dramatically (meaning, something like 256 will probably work, but larger > >"may have repercussions which are unknown at this time"). > > Might try that if it will help but with so many production machines to > action I'd like to try and avoid if possible. I've used PRINTF_BUFR_SIZE=256 with success on our systems, but since it doesn't actually *solve* the problem, I just use the default 128 and just grit my teeth when we experience it. It's larger values (e.g. 512/1024, etc.) which there is concern over. > >In combination with this, we use the following in /etc/rc.conf (the > >dumpdev line is important, else savecore won't pick up anything): > > > >dumpdev="auto" > > I thought this was ment to be the default from back in the 6.x days but > it didnt seem to work, so I added the gptid device from /etc/fstab /etc/defaults/rc.conf has dumpdev="NO", which affects two things: both /etc/rc.d/dumpon (this script is a little tricky, you really have to read it slowly/pay close attention to what's going on), and /etc/rc.d/savecore. I've always wondered why dumpdev="NO" is the default, not "auto", since on a system with no swap devices in /etc/fstab dumpdev="auto" should behave the same. Possibly the idea of the default is to ensure that savecore(8) never gets run (e.g. there's no guarantee someone has /var/crash, or a /var that's big enough to hold a crash dump; possibly embedded systems or NFS-only systems, for example). Touchy subject I guess. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 16:26:54 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6C4D106564A for ; Wed, 10 Aug 2011 16:26:54 +0000 (UTC) (envelope-from prvs=1203777b38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5A3968FC08 for ; Wed, 10 Aug 2011 16:26:54 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 17:25:55 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 17:25:55 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014546027.msg for ; Wed, 10 Aug 2011 17:25:55 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1203777b38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <8F68A154264C4AF199B7540C48CA9A40@multiplay.co.uk> From: "Steven Hartland" To: "Jeremy Chadwick" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <20110810151256.GA38601@icarus.home.lan> <20110810160018.GA40279@icarus.home.lan> Date: Wed, 10 Aug 2011 17:26:27 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:26:54 -0000 ----- Original Message ----- From: "Jeremy Chadwick" freebsd@jdc.parodius.com >> >In combination with this, we use the following in /etc/rc.conf (the >> >dumpdev line is important, else savecore won't pick up anything): >> > >> >dumpdev="auto" >> >> I thought this was ment to be the default from back in the 6.x days but >> it didnt seem to work, so I added the gptid device from /etc/fstab > > /etc/defaults/rc.conf has dumpdev="NO", which affects two things: both > /etc/rc.d/dumpon (this script is a little tricky, you really have to > read it slowly/pay close attention to what's going on), and > /etc/rc.d/savecore. Hmm, someone might want to correct the docs then:- http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html "AUTO is the default as of FreeBSD 6.0" Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 16:57:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DECE2106564A for ; Wed, 10 Aug 2011 16:57:07 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.westchester.pa.mail.comcast.net (qmta13.westchester.pa.mail.comcast.net [76.96.59.243]) by mx1.freebsd.org (Postfix) with ESMTP id 8A7538FC17 for ; Wed, 10 Aug 2011 16:57:07 +0000 (UTC) Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73]) by qmta13.westchester.pa.mail.comcast.net with comcast id JgUM1h0071ap0As5Dgx7cv; Wed, 10 Aug 2011 16:57:07 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta22.westchester.pa.mail.comcast.net with comcast id Jgx61h00x1t3BNj3igx7YP; Wed, 10 Aug 2011 16:57:07 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4B754102C19; Wed, 10 Aug 2011 09:57:05 -0700 (PDT) Date: Wed, 10 Aug 2011 09:57:05 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20110810165705.GA41272@icarus.home.lan> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <20110810151256.GA38601@icarus.home.lan> <20110810160018.GA40279@icarus.home.lan> <8F68A154264C4AF199B7540C48CA9A40@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8F68A154264C4AF199B7540C48CA9A40@multiplay.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:57:07 -0000 On Wed, Aug 10, 2011 at 05:26:27PM +0100, Steven Hartland wrote: > ----- Original Message ----- From: "Jeremy Chadwick" > freebsd@jdc.parodius.com > > >>>In combination with this, we use the following in /etc/rc.conf (the > >>>dumpdev line is important, else savecore won't pick up anything): > >>> > >>>dumpdev="auto" > >> > >>I thought this was ment to be the default from back in the 6.x days but > >>it didnt seem to work, so I added the gptid device from /etc/fstab > > > >/etc/defaults/rc.conf has dumpdev="NO", which affects two things: both > >/etc/rc.d/dumpon (this script is a little tricky, you really have to > >read it slowly/pay close attention to what's going on), and > >/etc/rc.d/savecore. > > Hmm, someone might want to correct the docs then:- > http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html > > "AUTO is the default as of FreeBSD 6.0" It used to be "auto", and was changed to "no" in this commit back in September 2009, and was reviewed by two separate people: http://www.freebsd.org/cgi/cvsweb.cgi/src/etc/defaults/rc.conf#rev1.358.2.2 Prior to that, it was "auto", as confirmed here (circa June 2005): http://www.freebsd.org/cgi/cvsweb.cgi/src/etc/defaults/rc.conf#rev1.250 So basically the documentation is both correct and incorrect. For anyone running FreeBSD later than September 2009 (I would need to spend some time figuring out what releases that was), dumpdev will not be enabled by default. Prior to that (which includes 6.x), it will be. The documentation needs to be updated to reflect reality (specifically the commit that was done in September 2009). I'll file a PR for this, but won't have the PR number until later today. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 19:28:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8B7C106566B for ; Wed, 10 Aug 2011 19:28:14 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 632918FC15 for ; Wed, 10 Aug 2011 19:28:14 +0000 (UTC) Received: by fxe4 with SMTP id 4so1598976fxe.13 for ; Wed, 10 Aug 2011 12:28:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=181EbQ/Qh0hSVldoB/kUIXoBhYDdhiNOQG7dpNf7UyI=; b=oUPJjZJ3QLiH8H90E3WpNMf6v9Yjd1qdJKVugX8brs3I9ZtZSXFxmNjscrlXIRkQS5 weyDV/kBvyO7TkD5izllBAc17Q608Jz0Kxc7FxgyNg2KUxRU3xt3GOET6VWupZslPOnV XQcv3eas/iT3Fwhr15tkyRJGObK8zCiMnviUI= MIME-Version: 1.0 Received: by 10.223.149.207 with SMTP id u15mr5761042fav.69.1313003044465; Wed, 10 Aug 2011 12:04:04 -0700 (PDT) Received: by 10.223.58.137 with HTTP; Wed, 10 Aug 2011 12:04:04 -0700 (PDT) Date: Wed, 10 Aug 2011 14:04:04 -0500 Message-ID: From: Adam Vande More To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Recent STABLE unable to start process in background X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 19:28:14 -0000 I am unable to start processes in the background after a recent upgrade to stable from 8.1R. I get: suspended (tty output) when trying to start a process like python /usr/home/adam/randr/manage.py runserver 0.0.0.0:18080 & my shell is zsh 4.3.12 -- Adam Vande More From owner-freebsd-stable@FreeBSD.ORG Wed Aug 10 23:55:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE45B1065670 for ; Wed, 10 Aug 2011 23:55:18 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 539468FC08 for ; Wed, 10 Aug 2011 23:55:17 +0000 (UTC) Received: (qmail 66794 invoked by uid 907); 10 Aug 2011 23:55:16 -0000 Received: from b13FC.static.pacific.net.au (HELO [192.168.1.158]) (202.7.88.252) (smtp-auth username janm, mechanism plain) by midgard.transactionware.com (qpsmtpd/0.84) with (AES128-SHA encrypted) ESMTPSA; Thu, 11 Aug 2011 09:55:16 +1000 From: Jan Mikkelsen Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Date: Thu, 11 Aug 2011 09:55:18 +1000 Message-Id: <4846F699-215D-4408-BD3C-4860305BF6B8@transactionware.com> To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Apple Message framework v1244.3) X-Mailer: Apple Mail (2.1244.3) Subject: Patch to puc(4) to support Moxa CP-112UL board X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 23:55:18 -0000 Hi, I have added these device IDs to pucdata.c to support the Moxa CP-112UL = board family. Should I submit a problem report, or is there an easier way to get the = patch merged? (I care about 8-STABLE at the moment =85) Thanks, Jan Mikkelsen =3D=3D=3D=3D //depot/vendor/freebsd/8.2/src/sys/dev/puc/pucdata.c#4 = (text) =3D=3D=3D=3D @@ -523,6 +523,12 @@ PUC_PORT_4S, 0x18, 0, 8, }, =20 + { 0x1393, 0x1120, 0xffff, 0, + "Moxa Technologies, CP-112UL family", + DEFAULT_RCLK * 8, + PUC_PORT_2S, 0x18, 0, 8, + }, + { 0x1393, 0x1141, 0xffff, 0, "Moxa Technologies, Industio CP-114", DEFAULT_RCLK * 8, From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 00:03:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE0CB106564A for ; Thu, 11 Aug 2011 00:03:24 +0000 (UTC) (envelope-from rs@mytum.de) Received: from vr1.steg0.de (vr1.steg0.eu [IPv6:2607:f740:0:3f::13d]) by mx1.freebsd.org (Postfix) with ESMTP id D2AE38FC18 for ; Thu, 11 Aug 2011 00:03:24 +0000 (UTC) Received: from sun2.hq.steg0.eu ([2a01:170:1046:0:221:28ff:fe14:b097]) by vr1.steg0.de with esmtpa (Exim 4.69) (envelope-from ) id 1QrIjy-0006RH-3K; Thu, 11 Aug 2011 00:03:23 +0000 Message-ID: <4E431C3A.6080707@mytum.de> Date: Thu, 11 Aug 2011 02:03:06 +0200 From: Raimund Steger User-Agent: Thunderbird 2.0.0.23 (X11/20091118) MIME-Version: 1.0 To: Adam Vande More References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score-2: -2.8 (--) Cc: freebsd-stable@freebsd.org Subject: Re: Recent STABLE unable to start process in background X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 00:03:25 -0000 Adam Vande More wrote: > I am unable to start processes in the background after a recent upgrade to > stable from 8.1R. > > I get: > > suspended (tty output) > > when trying to start a process like > > python /usr/home/adam/randr/manage.py runserver 0.0.0.0:18080 & > > my shell is zsh 4.3.12 > Looks like SIGTTOU (output from background process)? This should be controllable with stty -tostop. (But why has it changed...?) Raimund From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 01:22:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24E391065670 for ; Thu, 11 Aug 2011 01:22:10 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.westchester.pa.mail.comcast.net (qmta09.westchester.pa.mail.comcast.net [76.96.62.96]) by mx1.freebsd.org (Postfix) with ESMTP id C5AC08FC18 for ; Thu, 11 Aug 2011 01:22:09 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta09.westchester.pa.mail.comcast.net with comcast id Jp9E1h0051ei1Bg59pN9gR; Thu, 11 Aug 2011 01:22:09 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.westchester.pa.mail.comcast.net with comcast id JpN81h0091t3BNj3kpN9jG; Thu, 11 Aug 2011 01:22:09 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 3B16A102C1A; Wed, 10 Aug 2011 18:22:07 -0700 (PDT) Date: Wed, 10 Aug 2011 18:22:07 -0700 From: Jeremy Chadwick To: Raimund Steger Message-ID: <20110811012207.GA86377@icarus.home.lan> References: <4E431C3A.6080707@mytum.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E431C3A.6080707@mytum.de> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Adam Vande More , freebsd-stable@freebsd.org Subject: Re: Recent STABLE unable to start process in background X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 01:22:10 -0000 On Thu, Aug 11, 2011 at 02:03:06AM +0200, Raimund Steger wrote: > Adam Vande More wrote: > >I am unable to start processes in the background after a recent upgrade to > >stable from 8.1R. > > > >I get: > > > >suspended (tty output) > > > >when trying to start a process like > > > >python /usr/home/adam/randr/manage.py runserver 0.0.0.0:18080 & > > > >my shell is zsh 4.3.12 > > > > Looks like SIGTTOU (output from background process)? > This should be controllable with stty -tostop. > (But why has it changed...?) On all our RELENG_8 systems (though I use bash), -tostop is default. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 02:51:34 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57923106566C for ; Thu, 11 Aug 2011 02:51:34 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id D82F08FC0A for ; Thu, 11 Aug 2011 02:51:33 +0000 (UTC) Received: by bkat8 with SMTP id t8so869079bka.13 for ; Wed, 10 Aug 2011 19:51:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=d/c4qSw/CGNXKUsRZomS0H135e1CCqMTBFCoh5QulT0=; b=t4aVOoEGFvnU/vjiZDyjpa+SAWUledXkoo/2dMUgj17q7Uk/GebZXCc6znp93UVrpa 91kJnnEvWFgnysD4a9BMhGbF7pl8+1WuJHmeXGw4/E6RdAvn2em3eI9HFwLhloHFXjpi CsFSpYS6KajuZtbHXfJHAVs0NqJ5ufhnzT+QU= MIME-Version: 1.0 Received: by 10.204.7.201 with SMTP id e9mr2706555bke.126.1313031092490; Wed, 10 Aug 2011 19:51:32 -0700 (PDT) Received: by 10.204.35.148 with HTTP; Wed, 10 Aug 2011 19:51:32 -0700 (PDT) In-Reply-To: <20110811012207.GA86377@icarus.home.lan> References: <4E431C3A.6080707@mytum.de> <20110811012207.GA86377@icarus.home.lan> Date: Wed, 10 Aug 2011 21:51:32 -0500 Message-ID: From: Adam Vande More To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Raimund Steger , freebsd-stable@freebsd.org Subject: Re: Recent STABLE unable to start process in background X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 02:51:34 -0000 On Wed, Aug 10, 2011 at 8:22 PM, Jeremy Chadwick wrote: > > Looks like SIGTTOU (output from background process)? > > This should be controllable with stty -tostop. > > (But why has it changed...?) > > On all our RELENG_8 systems (though I use bash), -tostop is default. > Hm, it seems there might be something wrong with zsh. stty -a on an old and new setup produces identical output with -tostop set. The old setup runs zsh-4.3.10_3 which works correctly, but zsh-4.3.12 doesn't work on the new. The latest bash works fine on the new. I can file a bug report on zsh, but could someone confirm that it's the likely candidate for a problem so I don't send anyone on a wild goose chase? Thanks, -- Adam Vande More From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 03:24:04 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 4FF5D1065676 for ; Thu, 11 Aug 2011 03:24:04 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-199-253.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 03344150CBB; Thu, 11 Aug 2011 03:24:03 +0000 (UTC) Message-ID: <4E434B53.4090202@FreeBSD.org> Date: Wed, 10 Aug 2011 20:24:03 -0700 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110706 Thunderbird/5.0 MIME-Version: 1.0 To: Adam Vande More References: <4E431C3A.6080707@mytum.de> <20110811012207.GA86377@icarus.home.lan> In-Reply-To: X-Enigmail-Version: 1.2.1 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Raimund Steger , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: Recent STABLE unable to start process in background X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 03:24:04 -0000 On 08/10/2011 19:51, Adam Vande More wrote: > On Wed, Aug 10, 2011 at 8:22 PM, Jeremy Chadwick > wrote: > >>> Looks like SIGTTOU (output from background process)? >>> This should be controllable with stty -tostop. >>> (But why has it changed...?) >> >> On all our RELENG_8 systems (though I use bash), -tostop is default. >> > > Hm, it seems there might be something wrong with zsh. > > stty -a on an old and new setup produces identical output with -tostop set. > The old setup runs zsh-4.3.10_3 which works correctly, but zsh-4.3.12 > doesn't work on the new. The latest bash works fine on the new. I can file > a bug report on zsh, but could someone confirm that it's the likely > candidate for a problem so I don't send anyone on a wild goose chase? Back up the old zsh on the working system, install the new one, test. -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 04:15:54 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47D7B1065670 for ; Thu, 11 Aug 2011 04:15:54 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id CC3408FC08 for ; Thu, 11 Aug 2011 04:15:53 +0000 (UTC) Received: by bkat8 with SMTP id t8so910193bka.13 for ; Wed, 10 Aug 2011 21:15:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=mQMJIlq7OJUo2Ku8AKAOlsvhSaADICJ2GirlW5wm/q8=; b=Da5Rmw5zvJRxBkNZhdr3r5SfwNRrZpHIIzirVMpJla9d1g3K2ReZKQEXkpYvUf2gc/ N4HFzHUjBZC2tp3b/we+Ten47X7J2NNjlFPwm+CcwbIde7254Hg8LKCic+p062rtzzch o1lr/8Q9jUfuuYmwctBcn7u+cpH78dUojaxyA= MIME-Version: 1.0 Received: by 10.205.81.8 with SMTP id zw8mr674098bkb.282.1313036152414; Wed, 10 Aug 2011 21:15:52 -0700 (PDT) Received: by 10.204.35.148 with HTTP; Wed, 10 Aug 2011 21:15:52 -0700 (PDT) In-Reply-To: <4E434B53.4090202@FreeBSD.org> References: <4E431C3A.6080707@mytum.de> <20110811012207.GA86377@icarus.home.lan> <4E434B53.4090202@FreeBSD.org> Date: Wed, 10 Aug 2011 23:15:52 -0500 Message-ID: From: Adam Vande More To: Doug Barton Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Raimund Steger , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: Recent STABLE unable to start process in background X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 04:15:54 -0000 On Wed, Aug 10, 2011 at 10:24 PM, Doug Barton wrote: > Back up the old zsh on the working system, install the new one, test. > Sorry for the noise, it was a bug in Django 1.3. I had multiple versions installed and it was picking up the wrong one on the effected machine. -- Adam Vande More From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 04:25:46 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0387106564A for ; Thu, 11 Aug 2011 04:25:46 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6A3448FC0C for ; Thu, 11 Aug 2011 04:25:46 +0000 (UTC) Received: by gwb15 with SMTP id 15so1315347gwb.13 for ; Wed, 10 Aug 2011 21:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=2tfRgaUbVoFUS6/6DWxN5sec9sUwV/k3NI6DszsOD+8=; b=iPZdRN/lfLb/oIChoDNDH2bxdp7DKKbxlnYS8KJxivTlnZMamHTe3fR+rmDAyHEDwT 1QD+q8YlaJMv0T4tEU3PpoIwgh7RQqMJq+Ck53YiIsk9t2fptkWt0ZPvgkPeCWa3xheU l0ad5NIjUGYsHV3LMICe8cU3orD+PcKYXWUjU= MIME-Version: 1.0 Received: by 10.151.118.1 with SMTP id v1mr122805ybm.386.1313036745528; Wed, 10 Aug 2011 21:25:45 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.151.26.14 with HTTP; Wed, 10 Aug 2011 21:25:45 -0700 (PDT) In-Reply-To: <4846F699-215D-4408-BD3C-4860305BF6B8@transactionware.com> References: <4846F699-215D-4408-BD3C-4860305BF6B8@transactionware.com> Date: Thu, 11 Aug 2011 12:25:45 +0800 X-Google-Sender-Auth: 5gZBI8GvsJR9nUUiXqCRiUsN8H4 Message-ID: From: Adrian Chadd To: Jan Mikkelsen Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-stable@freebsd.org Subject: Re: Patch to puc(4) to support Moxa CP-112UL board X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 04:25:46 -0000 On 11 August 2011 07:55, Jan Mikkelsen wrote: > Hi, > > I have added these device IDs to pucdata.c to support the Moxa CP-112UL board family. > > Should I submit a problem report, or is there an easier way to get the patch merged? The "right" way is to get a PR submitted, then chase it up with the maintainer of that subsystem. Adrian From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 07:08:31 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C153106566B; Thu, 11 Aug 2011 07:08:31 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 777CE8FC08; Thu, 11 Aug 2011 07:08:30 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA06362; Thu, 11 Aug 2011 10:08:28 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E437FEB.1050102@FreeBSD.org> Date: Thu, 11 Aug 2011 10:08:27 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Adrian Chadd References: <4846F699-215D-4408-BD3C-4860305BF6B8@transactionware.com> In-Reply-To: X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Jan Mikkelsen , freebsd-stable@FreeBSD.org Subject: Re: Patch to puc(4) to support Moxa CP-112UL board X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 07:08:31 -0000 on 11/08/2011 07:25 Adrian Chadd said the following: > On 11 August 2011 07:55, Jan Mikkelsen wrote: >> Hi, >> >> I have added these device IDs to pucdata.c to support the Moxa CP-112UL board family. >> >> Should I submit a problem report, or is there an easier way to get the patch merged? > > The "right" way is to get a PR submitted, then chase it up with the > maintainer of that subsystem. I usually did it in the opposite order. No need for a PR if a patch is sufficiently trivial and a developer is sufficiently responsive at the moment. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 07:12:04 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5ED1106566B for ; Thu, 11 Aug 2011 07:12:04 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3C98B8FC14 for ; Thu, 11 Aug 2011 07:12:03 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA06402; Thu, 11 Aug 2011 10:12:01 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E4380C0.7070908@FreeBSD.org> Date: Thu, 11 Aug 2011 10:12:00 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> In-Reply-To: X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 07:12:05 -0000 on 10/08/2011 18:35 Steven Hartland said the following: > Fatal double fault > rip = 0xffffffff8052f6f1 > rsp = 0xffffff86ce600fb0 > rbp = 0xffffff86ce601210 > cpuid = 0; apic id = 00 > panic: double fault > cpuid = 0 > KDB: stack backtrace: > #0 0xffffffff803af91e at kdb_backtrace+0x5e > #1 0xffffffff8037d817 at panic+0x187 > #2 0xffffffff80574316 at dblfault_handler+0x96 > #3 0xffffffff8055d06d at Xdblfault+0xad [snip] > #0 sched_switch (td=0xffffffff80830bc0, newtd=0xffffff000a73f8c0, flags=Variable > "flags" is not available.) > at /usr/src/sys/kern/sched_ule.c:1858 > 1858 cpuid = PCPU_GET(cpuid); > (kgdb) > #0 sched_switch (td=0xffffffff80830bc0, newtd=0xffffff000a73f8c0, flags=Variable > "flags" is not available.) > at /usr/src/sys/kern/sched_ule.c:1858 > #1 0xffffffff80385c86 in mi_switch (flags=260, newtd=0x0) > at /usr/src/sys/kern/kern_synch.c:449 > #2 0xffffffff803b92d2 in sleepq_timedwait (wchan=0xffffffff80830760, pri=68) > at /usr/src/sys/kern/subr_sleepqueue.c:644 > #3 0xffffffff803861e1 in _sleep (ident=0xffffffff80830760, lock=0x0, > priority=Variable "priority" is not available. > ) at /usr/src/sys/kern/kern_synch.c:230 > #4 0xffffffff80532c29 in scheduler (dummy=Variable "dummy" is not available. > ) at /usr/src/sys/vm/vm_glue.c:807 > #5 0xffffffff80335d67 in mi_startup () at /usr/src/sys/kern/init_main.c:254 > #6 0xffffffff8016efac in btext () at /usr/src/sys/amd64/amd64/locore.S:81 > #7 0xffffffff808556e0 in sleepq_chains () > #8 0xffffffff8083b1e0 in cpu_top () > #9 0x0000000000000000 in ?? () > #10 0xffffffff80830bc0 in proc0 () > #11 0xffffffff80ba4b90 in ?? () > #12 0xffffffff80ba4b38 in ?? () > #13 0xffffff000a73f8c0 in ?? () > #14 0xffffffff803a2cc9 in sched_switch (td=0x0, newtd=0x0, flags=Variable "flags" > is not available. > ) > at /usr/src/sys/kern/sched_ule.c:1852 > Previous frame inner to this frame (corrupt stack?) > (kgdb) Looks like this is just the first thread in the kernel. Perhaps 'thread apply all bt' could help to find the culprit. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 08:41:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75A9F1065670 for ; Thu, 11 Aug 2011 08:41:08 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 32D5F8FC08 for ; Thu, 11 Aug 2011 08:41:07 +0000 (UTC) Received: by gyd10 with SMTP id 10so1394223gyd.13 for ; Thu, 11 Aug 2011 01:41:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=lmfbtR5XXkBn7fCdtJ7HEa0DP6wWRSjyyCjBCWBv+x4=; b=TUjxrO6PgdKVpQvkYXZiLQ5AaBIEMERLgHd0jdmV39aEqgMFSXC4c4nPJyVo4G2gC/ btCmgI6FEiIikqzwAGyOCaVigFPXWOyjy+UkYuEdg8mr0cX5dePCCVNW7ro+b2OHRTBn BvQjpFP2QFxAPpa6jUDW+4iCJ+C1cVFdOn0r0= MIME-Version: 1.0 Received: by 10.236.143.40 with SMTP id k28mr5146883yhj.34.1313050544176; Thu, 11 Aug 2011 01:15:44 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.236.108.33 with HTTP; Thu, 11 Aug 2011 01:15:44 -0700 (PDT) In-Reply-To: <4E4380C0.7070908@FreeBSD.org> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <4E4380C0.7070908@FreeBSD.org> Date: Thu, 11 Aug 2011 10:15:44 +0200 X-Google-Sender-Auth: FfMILqTqsuJKAIg-jcU0Gf2z_kM Message-ID: From: Attilio Rao To: Andriy Gapon Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, Steven Hartland Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 08:41:08 -0000 I'd really point the finger to faulty hw. Please run all the necessary diagnostic tools for catching it. Attilio 2011/8/11 Andriy Gapon : > on 10/08/2011 18:35 Steven Hartland said the following: >> Fatal double fault >> rip =3D 0xffffffff8052f6f1 >> rsp =3D 0xffffff86ce600fb0 >> rbp =3D 0xffffff86ce601210 >> cpuid =3D 0; apic id =3D 00 >> panic: double fault >> cpuid =3D 0 >> KDB: stack backtrace: >> #0 0xffffffff803af91e at kdb_backtrace+0x5e >> #1 0xffffffff8037d817 at panic+0x187 >> #2 0xffffffff80574316 at dblfault_handler+0x96 >> #3 0xffffffff8055d06d at Xdblfault+0xad > [snip] >> #0 =C2=A0sched_switch (td=3D0xffffffff80830bc0, newtd=3D0xffffff000a73f8= c0, flags=3DVariable >> "flags" is not available.) >> =C2=A0 =C2=A0at /usr/src/sys/kern/sched_ule.c:1858 >> 1858 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0cpuid =3D PCPU_GET(cpuid); >> (kgdb) >> #0 =C2=A0sched_switch (td=3D0xffffffff80830bc0, newtd=3D0xffffff000a73f8= c0, flags=3DVariable >> "flags" is not available.) >> =C2=A0 =C2=A0at /usr/src/sys/kern/sched_ule.c:1858 >> #1 =C2=A00xffffffff80385c86 in mi_switch (flags=3D260, newtd=3D0x0) >> =C2=A0 =C2=A0at /usr/src/sys/kern/kern_synch.c:449 >> #2 =C2=A00xffffffff803b92d2 in sleepq_timedwait (wchan=3D0xffffffff80830= 760, pri=3D68) >> =C2=A0 =C2=A0at /usr/src/sys/kern/subr_sleepqueue.c:644 >> #3 =C2=A00xffffffff803861e1 in _sleep (ident=3D0xffffffff80830760, lock= =3D0x0, >> =C2=A0 =C2=A0priority=3DVariable "priority" is not available. >> ) at /usr/src/sys/kern/kern_synch.c:230 >> #4 =C2=A00xffffffff80532c29 in scheduler (dummy=3DVariable "dummy" is no= t available. >> ) at /usr/src/sys/vm/vm_glue.c:807 >> #5 =C2=A00xffffffff80335d67 in mi_startup () at /usr/src/sys/kern/init_m= ain.c:254 >> #6 =C2=A00xffffffff8016efac in btext () at /usr/src/sys/amd64/amd64/loco= re.S:81 >> #7 =C2=A00xffffffff808556e0 in sleepq_chains () >> #8 =C2=A00xffffffff8083b1e0 in cpu_top () >> #9 =C2=A00x0000000000000000 in ?? () >> #10 0xffffffff80830bc0 in proc0 () >> #11 0xffffffff80ba4b90 in ?? () >> #12 0xffffffff80ba4b38 in ?? () >> #13 0xffffff000a73f8c0 in ?? () >> #14 0xffffffff803a2cc9 in sched_switch (td=3D0x0, newtd=3D0x0, flags=3DV= ariable "flags" >> is not available. >> ) >> =C2=A0 =C2=A0at /usr/src/sys/kern/sched_ule.c:1852 >> Previous frame inner to this frame (corrupt stack?) >> (kgdb) > > Looks like this is just the first thread in the kernel. > Perhaps 'thread apply all bt' could help to find the culprit. > > -- > Andriy Gapon > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 09:11:40 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40FF3106564A; Thu, 11 Aug 2011 09:11:40 +0000 (UTC) (envelope-from prvs=1204ca57bc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5F8B78FC13; Thu, 11 Aug 2011 09:11:39 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 10:00:22 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 10:00:22 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014557844.msg; Thu, 11 Aug 2011 10:00:22 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1204ca57bc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <44DD20E1CFA949E8A1B15B3847769DCB@multiplay.co.uk> From: "Steven Hartland" To: "Attilio Rao" , "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> Date: Thu, 11 Aug 2011 09:59:36 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 09:11:40 -0000 That's not the issue as its happening across board over 130 machines :( Regards Steve ----- Original Message ----- From: "Attilio Rao" I'd really point the finger to faulty hw. Please run all the necessary diagnostic tools for catching it. Attilio ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 09:29:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3D65106567C for ; Thu, 11 Aug 2011 09:29:00 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.westchester.pa.mail.comcast.net (qmta12.westchester.pa.mail.comcast.net [76.96.59.227]) by mx1.freebsd.org (Postfix) with ESMTP id 717E48FC13 for ; Thu, 11 Aug 2011 09:29:00 +0000 (UTC) Received: from omta12.westchester.pa.mail.comcast.net ([76.96.62.44]) by qmta12.westchester.pa.mail.comcast.net with comcast id JxKm1h0020xGWP85CxV0ms; Thu, 11 Aug 2011 09:29:00 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta12.westchester.pa.mail.comcast.net with comcast id JxUz1h00N1t3BNj3YxUz6R; Thu, 11 Aug 2011 09:29:00 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 2A11F102C1A; Thu, 11 Aug 2011 02:28:58 -0700 (PDT) Date: Thu, 11 Aug 2011 02:28:58 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20110811092858.GA94514@icarus.home.lan> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <4E4380C0.7070908@FreeBSD.org> <44DD20E1CFA949E8A1B15B3847769DCB@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44DD20E1CFA949E8A1B15B3847769DCB@multiplay.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Attilio Rao , freebsd-stable@freebsd.org, Andriy Gapon Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 09:29:00 -0000 On Thu, Aug 11, 2011 at 09:59:36AM +0100, Steven Hartland wrote: > That's not the issue as its happening across board over 130 machines :( Agreed, bad hardware sounds unlikely here. I could believe some strange incompatibility (e.g. BIOS quirk or the like[1]) that might cause problems en masse across many servers, but hardware issues are unlikely in this situation. [1]: I mention this because we had something similar happen at my workplace. For months we used a specific model of system from our vendor which worked reliably, zero issues. Then we got a new shipment of boxes (same model as prior) which started acting very odd (often AHCI timeout issues or MCEs which when decoded would usually turn out to be nonsensical). It took weeks to determine the cause given how slow the vendor was to respond: root cause turned out to be that the vendor decided, on a whim, to start shipping a newer BIOS version which wasn't "as compatible" with Solaris as previous BIOSes. Downgrading all the systems to the older BIOS fixed the problem. In Steve's case this is unlikely to be the situation, but I thought I'd share the story anyway. "SKU ABCXYZ-1" from August 2009 is not necessarily the same thing as "SKU ABCXYZ-1" from May 2010. ;-) This is also why I prefer to buy/build my own systems, since I cannot trust vendors to not mess about with settings w/out changing SKUs, P/Ns, or revision numbers. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 09:43:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1005D106566B; Thu, 11 Aug 2011 09:43:27 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id AB1C48FC0C; Thu, 11 Aug 2011 09:43:26 +0000 (UTC) Received: by yxl31 with SMTP id 31so1410923yxl.13 for ; Thu, 11 Aug 2011 02:43:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=PKtPQIWrVjkNrpcvkQulBIRfLOVajtCZ/GGcfZKxnOk=; b=Fyc1awJpwgbym5KdaTdFBycUDD8gs5/zyFNcc4O5X21rgAwqBiaaEIZd1gUwM96+g1 vIs0FXLbyF6MYGHUzUzeX+gBDIJijzvHhiJ3qk+8Mk3pMZID76l8+QFZ1nHhf8zsaGsB WtcgEkLDSoSTiAPRnhZJQp7+aY3UerLQzauhE= MIME-Version: 1.0 Received: by 10.236.181.6 with SMTP id k6mr6817260yhm.102.1313055805946; Thu, 11 Aug 2011 02:43:25 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.236.108.33 with HTTP; Thu, 11 Aug 2011 02:43:25 -0700 (PDT) In-Reply-To: <20110811092858.GA94514@icarus.home.lan> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <4E4380C0.7070908@FreeBSD.org> <44DD20E1CFA949E8A1B15B3847769DCB@multiplay.co.uk> <20110811092858.GA94514@icarus.home.lan> Date: Thu, 11 Aug 2011 11:43:25 +0200 X-Google-Sender-Auth: P0srNRwDoN4pWyh4w3YeKXSZdR0 Message-ID: From: Attilio Rao To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, Steven Hartland , Andriy Gapon Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 09:43:27 -0000 2011/8/11 Jeremy Chadwick : > On Thu, Aug 11, 2011 at 09:59:36AM +0100, Steven Hartland wrote: >> That's not the issue as its happening across board over 130 machines :( > > Agreed, bad hardware sounds unlikely here. =C2=A0I could believe some str= ange > incompatibility (e.g. BIOS quirk or the like[1]) that might cause problem= s > en masse across many servers, but hardware issues are unlikely in this > situation. > > [1]: I mention this because we had something similar happen at my > workplace. =C2=A0For months we used a specific model of system from our > vendor which worked reliably, zero issues. =C2=A0Then we got a new shipme= nt > of boxes (same model as prior) which started acting very odd (often AHCI > timeout issues or MCEs which when decoded would usually turn out to be > nonsensical). =C2=A0It took weeks to determine the cause given how slow t= he > vendor was to respond: root cause turned out to be that the vendor > decided, on a whim, to start shipping a newer BIOS version which wasn't > "as compatible" with Solaris as previous BIOSes. =C2=A0Downgrading all th= e > systems to the older BIOS fixed the problem. That falls in the "hw problem" category for me. Anyway, we really would need much more information in order to take a proactive action. Would it be possible to access to one of the panic'ing machine? Is it always the same panic which is happening or it is variadic (like: once page fault, once fatal double fault, once fatal trap, etc.). Whatever informations you can provide may be valuable here. Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 10:38:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0471E106566C; Thu, 11 Aug 2011 10:38:44 +0000 (UTC) (envelope-from prvs=1204ca57bc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 20A298FC1D; Thu, 11 Aug 2011 10:38:42 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 11:37:46 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 11:37:46 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014558581.msg; Thu, 11 Aug 2011 11:37:46 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1204ca57bc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Jeremy Chadwick" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <4E4380C0.7070908@FreeBSD.org> <44DD20E1CFA949E8A1B15B3847769DCB@multiplay.co.uk> <20110811092858.GA94514@icarus.home.lan> Date: Thu, 11 Aug 2011 11:38:17 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: Attilio Rao , freebsd-stable@freebsd.org, Andriy Gapon Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 10:38:44 -0000 ----- Original Message ----- From: "Jeremy Chadwick" > On Thu, Aug 11, 2011 at 09:59:36AM +0100, Steven Hartland wrote: >> That's not the issue as its happening across board over 130 machines :( > > Agreed, bad hardware sounds unlikely here. I could believe some strange > incompatibility (e.g. BIOS quirk or the like[1]) that might cause problems > en masse across many servers, but hardware issues are unlikely in this > situation. Its affecting a range of hardware from supermicro blades / 2u's & dell blades. So it seems more like a software bug. > [1]: I mention this because we had something similar happen at my > workplace. For months we used a specific model of system from our > vendor which worked reliably, zero issues. Then we got a new shipment > of boxes (same model as prior) which started acting very odd (often AHCI > timeout issues or MCEs which when decoded would usually turn out to be > nonsensical). It took weeks to determine the cause given how slow the > vendor was to respond: root cause turned out to be that the vendor > decided, on a whim, to start shipping a newer BIOS version which wasn't > "as compatible" with Solaris as previous BIOSes. Downgrading all the > systems to the older BIOS fixed the problem. The machines have been working for months fine, the panics only started last week. We've been looking at the changes made last week to see if we can identify the cause. The only change made in that time frame was the rollout of the change to kern.ipc.nmbclusters to workaround the tcp re-assembly issue. In this case we raised the value from the default of 25600 to 262144. We've used this value for a long time on our core webservers, which are also running 8.2 so I'd be very surprised if this was the cause. That said we're looking to roll out kern.ipc.nmbclusters=51200 to try and rule it out. Prior to this, 1-2 weeks previous, we rolled out a significant update which included:- 1. Adding IPv6 to the kernel (although no machines are configued with it yet) 2. Adding ipmi module to the kernel, although not loaded. 3. Rebuilding ALL ports to the latest version 4. Restructuring the server layout to be one jail per java server (~60 servers per machine) 5. Restructing the filesystem to be a base nullfs mount + devfs + zfs volume per server This update had been testing for 2 weeks prior to that, so in total 3-4 weeks before any panics where seen but that doesn't mean the issue didnt exist at that time. Currently we're seeing 1-4 panics a day across all machines. So currently the most likely suspects are:- 1. kern.ipc.nmbclusters 2. nullfs 3. ipv6 4. a package update, most likely being openjdk6-b23 5. jail > In Steve's case this is unlikely to be the situation, but I thought I'd > share the story anyway. "SKU ABCXYZ-1" from August 2009 is not > necessarily the same thing as "SKU ABCXYZ-1" from May 2010. ;-) This > is also why I prefer to buy/build my own systems, since I cannot trust > vendors to not mess about with settings w/out changing SKUs, P/Ns, or > revision numbers. This caused us much scratching of heads when looking for that tcp issue the other day. As it seemed to effecting the newer machines more than the old, we even found two machines with the same "version" of the bios but that's clearly a different build as the date and available options where different, quite frustrating! Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 11:40:33 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B11D21065672; Thu, 11 Aug 2011 11:40:33 +0000 (UTC) (envelope-from prvs=1204ca57bc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 106598FC1E; Thu, 11 Aug 2011 11:40:32 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 12:39:23 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 12:39:23 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014559134.msg; Thu, 11 Aug 2011 12:39:21 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1204ca57bc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <4E4380C0.7070908@FreeBSD.org> Date: Thu, 11 Aug 2011 12:39:52 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 11:40:33 -0000 ----- Original Message ----- From: "Andriy Gapon" > on 10/08/2011 18:35 Steven Hartland said the following: >> Fatal double fault ... >> #14 0xffffffff803a2cc9 in sched_switch (td=0x0, newtd=0x0, flags=Variable "flags" >> is not available. >> ) >> at /usr/src/sys/kern/sched_ule.c:1852 >> Previous frame inner to this frame (corrupt stack?) >> (kgdb) > > Looks like this is just the first thread in the kernel. > Perhaps 'thread apply all bt' could help to find the culprit. The trimmed down output, removed the 10,000's of ?? lines here:- http://blog.multiplay.co.uk/dropzone/freebsd/panic-2011-08-11-1402.txt The raw output is here:- http://blog.multiplay.co.uk/dropzone/freebsd/panic-full-2011-08-11-1402.txt.bz2 I'm not sure how useful its going to be as pretty much all of it seems to be just:- #0 sched_tswitch (td=0xffffff00194d4460, newtd=0xffffff000a74a000, flags=Variable "flags" is not available. #1 0xffffffff80385c86 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:449 #2 0xffffffff803b8a0c in sleepq_catch_signals (wchan=0xffffff02f27c48c0, pri=92) at /usr/src/sys/kern/subr_sleepqueue.c:418 #3 0xffffffff803b9326 in sleepq_wait_sig (wchan=Variable "wchan" is not available. #4 0xffffffff80386149 in _sleep (ident=0xffffff02f27c48c0, lock=0xffffff02f27c49b8, priority=Variable "priority" is not available. #5 0xffffffff8035079d in kern_wait (td=0xffffff00194d4460, pid=91362, status=0xffffff86cdbffabc, options=Variable "options" is not available. #6 0xffffffff80350e95 in wait4 (td=Variable "td" is not available. #7 0xffffffff803bb8e5 in syscallenter (td=0xffffff00194d4460, sa=0xffffff86cdbffba0) at /usr/src/sys/kern/subr_trap.c:315 #8 0xffffffff80574a0b in syscall (frame=0xffffff86cdbffc40) at /usr/src/sys/amd64/amd64/trap.c:888 #9 0xffffffff8055d242 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:377 On one machine we had a little more info on console which may indicate java as the problem. http://blog.multiplay.co.uk/dropzone/freebsd/panic-java.jpg Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 14:08:56 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F227C106566C for ; Thu, 11 Aug 2011 14:08:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 48AE28FC19 for ; Thu, 11 Aug 2011 14:08:55 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA15604; Thu, 11 Aug 2011 17:08:51 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E43E272.1060204@FreeBSD.org> Date: Thu, 11 Aug 2011 17:08:50 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk> <4E4380C0.7070908@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 14:08:56 -0000 on 11/08/2011 14:39 Steven Hartland said the following: > The trimmed down output, removed the 10,000's of ?? lines here:- > http://blog.multiplay.co.uk/dropzone/freebsd/panic-2011-08-11-1402.txt > > The raw output is here:- > http://blog.multiplay.co.uk/dropzone/freebsd/panic-full-2011-08-11-1402.txt.bz2 > > I'm not sure how useful its going to be as pretty much all of it seems > to be just:- > #0 sched_tswitch (td=0xffffff00194d4460, newtd=0xffffff000a74a000, flags=Variable > "flags" is not available. > #1 0xffffffff80385c86 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:449 > #2 0xffffffff803b8a0c in sleepq_catch_signals (wchan=0xffffff02f27c48c0, pri=92) > at /usr/src/sys/kern/subr_sleepqueue.c:418 > #3 0xffffffff803b9326 in sleepq_wait_sig (wchan=Variable "wchan" is not available. > #4 0xffffffff80386149 in _sleep (ident=0xffffff02f27c48c0, > lock=0xffffff02f27c49b8, priority=Variable "priority" is not available. > #5 0xffffffff8035079d in kern_wait (td=0xffffff00194d4460, pid=91362, > status=0xffffff86cdbffabc, options=Variable "options" is not available. > #6 0xffffffff80350e95 in wait4 (td=Variable "td" is not available. > #7 0xffffffff803bb8e5 in syscallenter (td=0xffffff00194d4460, > sa=0xffffff86cdbffba0) at /usr/src/sys/kern/subr_trap.c:315 > #8 0xffffffff80574a0b in syscall (frame=0xffffff86cdbffc40) at > /usr/src/sys/amd64/amd64/trap.c:888 > #9 0xffffffff8055d242 in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:377 > > On one machine we had a little more info on console which may indicate > java as the problem. > > http://blog.multiplay.co.uk/dropzone/freebsd/panic-java.jpg I would really appreciate if you could try to reproduce the problem with the patch that I sent earlier. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 14:16:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F333106566B for ; Thu, 11 Aug 2011 14:16:18 +0000 (UTC) (envelope-from szem@augenstein.net) Received: from www33.your-server.de (www33.your-server.de [213.133.104.33]) by mx1.freebsd.org (Postfix) with ESMTP id 4F3DB8FC20 for ; Thu, 11 Aug 2011 14:16:18 +0000 (UTC) Received: from [188.174.5.44] (helo=callisto.augenstein.ten) by www33.your-server.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1QrVlk-0002Gw-NX for freebsd-stable@freebsd.org; Thu, 11 Aug 2011 15:57:56 +0200 Received: from auge (uid 666) (envelope-from szem@augenstein.net) id 4ac31 by callisto.augenstein.ten (DragonFly Mail Agent) Thu, 11 Aug 2011 15:59:19 +0200 Date: Thu, 11 Aug 2011 15:59:19 +0200 From: Norbert Augenstein To: freebsd-stable@freebsd.org, stable@freebsd.org Message-ID: <20110811135917.GA1586@callisto.augenstein.ten> Mail-Followup-To: freebsd-stable@freebsd.org, stable@freebsd.org References: <491711312987440@web150.yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <491711312987440@web150.yandex.ru> X-Operating-System: FreeBSD 8.2-RELEASE amd64 X-Copyright: (c) auge -> Norbert Augenstein, Kissing, Germany User-Agent: Mutt/1.5.21 (2010-09-15) X-Authenticated-Sender: es11arm@augenstein.net X-Virus-Scanned: Clear (ClamAV 0.97.2/13427/Thu Aug 11 01:31:52 2011) Cc: Subject: Re: FreeBSD-9.0-BETA1-i386-bootonly X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 14:16:18 -0000 On Wed, Aug 10, 2011 at 06:44:00PM +0400, N V wrote: > Hi. > > Tried to use FreeBSD-9.0-BETA1-i386-bootonly.iso in VirtualBox to test. Installation stops after trying to fetch files from ftp. Attached screenshot is informative, I think. Seems to use i386/ twice for some reason. > same here for amd64, but the disc1.iso worked fine From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 14:16:24 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6F8D1065674 for ; Thu, 11 Aug 2011 14:16:24 +0000 (UTC) (envelope-from szem@augenstein.net) Received: from www33.your-server.de (www33.your-server.de [213.133.104.33]) by mx1.freebsd.org (Postfix) with ESMTP id 723728FC21 for ; Thu, 11 Aug 2011 14:16:24 +0000 (UTC) Received: from [188.174.5.44] (helo=callisto.augenstein.ten) by www33.your-server.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1QrVlk-0002Gx-TV for stable@freebsd.org; Thu, 11 Aug 2011 15:57:57 +0200 Received: from auge (uid 666) (envelope-from szem@augenstein.net) id 4ac31 by callisto.augenstein.ten (DragonFly Mail Agent) Thu, 11 Aug 2011 15:59:19 +0200 Date: Thu, 11 Aug 2011 15:59:19 +0200 From: Norbert Augenstein To: freebsd-stable@freebsd.org, stable@freebsd.org Message-ID: <20110811135917.GA1586@callisto.augenstein.ten> Mail-Followup-To: freebsd-stable@freebsd.org, stable@freebsd.org References: <491711312987440@web150.yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <491711312987440@web150.yandex.ru> X-Operating-System: FreeBSD 8.2-RELEASE amd64 X-Copyright: (c) auge -> Norbert Augenstein, Kissing, Germany User-Agent: Mutt/1.5.21 (2010-09-15) X-Authenticated-Sender: es11arm@augenstein.net X-Virus-Scanned: Clear (ClamAV 0.97.2/13427/Thu Aug 11 01:31:52 2011) Cc: Subject: Re: FreeBSD-9.0-BETA1-i386-bootonly X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 14:16:24 -0000 On Wed, Aug 10, 2011 at 06:44:00PM +0400, N V wrote: > Hi. > > Tried to use FreeBSD-9.0-BETA1-i386-bootonly.iso in VirtualBox to test. Installation stops after trying to fetch files from ftp. Attached screenshot is informative, I think. Seems to use i386/ twice for some reason. > same here for amd64, but the disc1.iso worked fine From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 16:37:15 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70CC91065672; Thu, 11 Aug 2011 16:37:15 +0000 (UTC) (envelope-from prvs=1204ca57bc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id BE4BA8FC18; Thu, 11 Aug 2011 16:37:14 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 17:36:41 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 17:36:36 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014562257.msg; Thu, 11 Aug 2011 17:36:30 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1204ca57bc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> Date: Thu, 11 Aug 2011 17:37:02 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 16:37:15 -0000 ----- Original Message ----- From: "Andriy Gapon" > > I would really appreciate if you could try to reproduce the problem with the patch > that I sent earlier. Hi Andriy, what's the risk of this patch causing other issues? I ask as to get results from this we've going to have to roll it out to over 130+ production machines, so I'd like to be clear on the risks before I sign that off. Regard Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 16:50:49 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 531B91065675 for ; Thu, 11 Aug 2011 16:50:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id A10FF8FC18 for ; Thu, 11 Aug 2011 16:50:48 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA18270; Thu, 11 Aug 2011 19:50:45 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E440865.1040500@FreeBSD.org> Date: Thu, 11 Aug 2011 19:50:45 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> In-Reply-To: <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 16:50:49 -0000 on 11/08/2011 19:37 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" > >> >> I would really appreciate if you could try to reproduce the problem with the patch >> that I sent earlier. > > Hi Andriy, what's the risk of this patch causing other issues? I can not estimate. The code is supposed to affect only things that happen after panic, so make your guess. > I ask as to get results from this we've going to have to roll it > out to over 130+ production machines, so I'd like to be clear on > the risks before I sign that off. I will be happy if you try the patch on a single machine provided the problem is that reproducible. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 17:15:10 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F25ED1065672; Thu, 11 Aug 2011 17:15:10 +0000 (UTC) (envelope-from prvs=1204ca57bc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id EFD838FC13; Thu, 11 Aug 2011 17:15:09 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 18:14:35 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 18:14:34 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014562616.msg; Thu, 11 Aug 2011 18:14:34 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1204ca57bc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> <4E440865.1040500@FreeBSD.org> Date: Thu, 11 Aug 2011 18:14:05 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 17:15:11 -0000 ----- Original Message ----- From: "Andriy Gapon" >>> I would really appreciate if you could try to reproduce the >>> problem with the patch that I sent earlier. >> >> Hi Andriy, what's the risk of this patch causing other issues? > > I can not estimate. > The code is supposed to affect only things that happen after panic, > so make your guess. So in theory should be good. >> I ask as to get results from this we've going to have to roll it >> out to over 130+ production machines, so I'd like to be clear on >> the risks before I sign that off. > > I will be happy if you try the patch on a single machine > provided the problem is that reproducible. Unfortunately although its happening a lot its taking the large numbers of machines to make it that way. Over the 130+ machines we're seeing between 3 and 8 panics a day, so based on that we could be waiting quite some time for a specific machine to panic :( Don't think we're going to make any progress on this in the current state so I think we'll give it a shot. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 17:36:23 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA449106566B for ; Thu, 11 Aug 2011 17:36:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 402E98FC1B for ; Thu, 11 Aug 2011 17:36:23 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA18950; Thu, 11 Aug 2011 20:36:20 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QrZB6-000KYv-GH; Thu, 11 Aug 2011 20:36:20 +0300 Message-ID: <4E441314.6060606@FreeBSD.org> Date: Thu, 11 Aug 2011 20:36:20 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110706 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> <4E440865.1040500@FreeBSD.org> <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> In-Reply-To: <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 17:36:24 -0000 on 11/08/2011 20:14 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" > >>>> I would really appreciate if you could try to reproduce the >>>> problem with the patch that I sent earlier. >>> >>> Hi Andriy, what's the risk of this patch causing other issues? >> >> I can not estimate. >> The code is supposed to affect only things that happen after panic, >> so make your guess. > > So in theory should be good. > >>> I ask as to get results from this we've going to have to roll it >>> out to over 130+ production machines, so I'd like to be clear on >>> the risks before I sign that off. >> >> I will be happy if you try the patch on a single machine >> provided the problem is that reproducible. > > Unfortunately although its happening a lot its taking the > large numbers of machines to make it that way. > > Over the 130+ machines we're seeing between 3 and 8 panics > a day, so based on that we could be waiting quite some time > for a specific machine to panic :( > > Don't think we're going to make any progress on this in the current > state so I think we'll give it a shot. Maybe test it on couple of machines first just in case I overlooked something essential, although I have a report from another use that the patch didn't break anything for him (it was tested for an unrelated issue). -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 20:13:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 077F51065672; Thu, 11 Aug 2011 20:13:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 88AA08FC08; Thu, 11 Aug 2011 20:13:17 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEALI3RE6DaFvO/2dsb2JhbAA+A4RIpAeBQAEBAQEDAQEBICsgCwwPDgcDAgINGQIpAQkmBggHBAEcBIdSrC2ROIEsghSBd4EQBJEBgg2RCw X-IronPort-AV: E=Sophos;i="4.67,358,1309752000"; d="scan'208";a="134148111" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 11 Aug 2011 16:12:59 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id B66FFB3F2C; Thu, 11 Aug 2011 16:12:59 -0400 (EDT) Date: Thu, 11 Aug 2011 16:12:59 -0400 (EDT) From: Rick Macklem To: Steven Hartland Message-ID: <1030083399.38743.1313093579690.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-stable@FreeBSD.org, Andriy Gapon Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 20:13:18 -0000 Steven Hartland wrote: > ----- Original Message ----- > From: "Andriy Gapon" > > >>> I would really appreciate if you could try to reproduce the > >>> problem with the patch that I sent earlier. > >> > >> Hi Andriy, what's the risk of this patch causing other issues? > > > > I can not estimate. > > The code is supposed to affect only things that happen after panic, > > so make your guess. > > So in theory should be good. > > >> I ask as to get results from this we've going to have to roll it > >> out to over 130+ production machines, so I'd like to be clear on > >> the risks before I sign that off. > > > > I will be happy if you try the patch on a single machine > > provided the problem is that reproducible. > > Unfortunately although its happening a lot its taking the > large numbers of machines to make it that way. > > Over the 130+ machines we're seeing between 3 and 8 panics > a day, so based on that we could be waiting quite some time > for a specific machine to panic :( > > Don't think we're going to make any progress on this in the current > state so I think we'll give it a shot. > Just a random thought that is probably not relevent, but... Is it possible that some change for the upgrade is making the machines run hotter and they're failing when they overhead? rick > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Thu Aug 11 22:05:20 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FCC4106566B; Thu, 11 Aug 2011 22:05:20 +0000 (UTC) (envelope-from prvs=1204ca57bc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 899528FC18; Thu, 11 Aug 2011 22:05:19 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 23:04:29 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 11 Aug 2011 23:04:28 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014565635.msg; Thu, 11 Aug 2011 23:04:28 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1204ca57bc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Rick Macklem" References: <1030083399.38743.1313093579690.JavaMail.root@erie.cs.uoguelph.ca> Date: Thu, 11 Aug 2011 23:04:59 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org, Andriy Gapon Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 22:05:20 -0000 ----- Original Message ----- From: "Rick Macklem" > Just a random thought that is probably not relevent, but... > Is it possible that some change for the upgrade is making the machines > run hotter and they're failing when they overhead? The machines have full HW monitoring and we've not seen reports of temperature issues, add to that quite a few are L series so run really cool anyway, I very much doubt it. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Fri Aug 12 17:10:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8996F106564A for ; Fri, 12 Aug 2011 17:10:44 +0000 (UTC) (envelope-from lamarw@metrocast.net) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 6634E8FC13 for ; Fri, 12 Aug 2011 17:10:43 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1Qrv0r-0000BD-O5 for freebsd-stable@freebsd.org; Fri, 12 Aug 2011 09:55:13 -0700 Date: Fri, 12 Aug 2011 09:55:13 -0700 (PDT) From: lamarw To: freebsd-stable@freebsd.org Message-ID: <1313168113739-4693999.post@n5.nabble.com> In-Reply-To: <4D306D96.7080002@raveisking.de> References: <20101209213556.GA3322@pollux.local.net> <4D306D96.7080002@raveisking.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: umass: AutoSense failed X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 17:10:44 -0000 I was experiencing this same issue on my FreeBSD 8.2-RELEASE-p2, FreeNAS-8.0.1-BETA4-i386 box. What fixed it for me was moving my bootable usb drive to the lower usb port (apparently da0). Without thinking I had put my console cable on that port and the bootable drive on the next port. After reading several postings I tried putting the usb drive in the other port and it came up without the error message: (da0:umass-sim0:0:0:0): AutoSense failed. Apparently it was attemting to seek response from the usb port (da0?) and got none. Also, apparently it didn't seek response's from any other ports. So, for now I'm up and runing. Hope this helps others. Incidentally, if anyone would like to chime in on defining da0 I would appreciate that. Lamar -- View this message in context: http://freebsd.1045724.n5.nabble.com/umass-AutoSense-failed-tp3972016p4693999.html Sent from the freebsd-stable mailing list archive at Nabble.com. From owner-freebsd-stable@FreeBSD.ORG Fri Aug 12 20:17:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63A6A1065676; Fri, 12 Aug 2011 20:17:50 +0000 (UTC) (envelope-from aboyer@averesystems.com) Received: from zimbra.averesystems.com (75-149-8-245-Pennsylvania.hfc.comcastbusiness.net [75.149.8.245]) by mx1.freebsd.org (Postfix) with ESMTP id 16C0B8FC14; Fri, 12 Aug 2011 20:17:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.averesystems.com (Postfix) with ESMTP id 3BA61446005; Fri, 12 Aug 2011 15:59:35 -0400 (EDT) X-Virus-Scanned: amavisd-new at averesystems.com Received: from zimbra.averesystems.com ([127.0.0.1]) by localhost (zimbra.averesystems.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TsXUPV3Qnm3Z; Fri, 12 Aug 2011 15:59:33 -0400 (EDT) Received: from riven.arriad.com (fw.arriad.com [10.0.0.16]) by zimbra.averesystems.com (Postfix) with ESMTPSA id 5B8D58BC001; Fri, 12 Aug 2011 15:59:33 -0400 (EDT) From: Andrew Boyer Date: Fri, 12 Aug 2011 15:59:21 -0400 Message-Id: To: Andriy Gapon , Hans Petter Selasky Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Vishal.Shah@netapp.com, freebsd-stable@freebsd.org, Steven Hartland , Eugene Grosbein , Jeremiah Lott Subject: USB/coredump hangs in 8 and 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 20:17:50 -0000 Re: panic: bufwrite: buffer is not busy??? (originally on freebsd-net) Re: debugging frequent kernel panics on 8.2-RELEASE (originally on = freebsd-stable) Re: System hang in USB umass module while processing panic (originally = on freebsd-usb) Hello Andriy and Hans, Sorry for tying in so many discussions on this topic, but I think I have = an explanation for the problems we have been reporting* with hanging = coredumps on multicore systems on 8.2-RELEASE, and it has implications = for Andriy's proposed scheduler patch** and for USB. In today's 8.X and 9.X branches, nothing that I can find stops the other = CPUs when the kernel panics, but many parts of the locking code get = disabled (grep on 'panicstr'). The 'bufwrite: buffer is not busy???' = panic is caused by the syncer encountering an error. If that happens = when it's on the dumping CPU everything hangs. If it's running on a = different CPU, it will be blocked and hidden by the panic_cpu spinlock = in panic(), and the dump continues, polling every attached keyboard for = a Ctl-C. But, the new 8.X USB stack relies on multithreading. (The new stack is = the variable that broke coredumps for us in the 7.1->8.2 transition, I = think.) SVN 224223 fixes a hang that would happen when dumpsys() polls = the USB keyboard (IPMI KVM, in our case). That helps, but it only gets = as far as usb_process(), where it hangs in a loop around a cv_wait() = call. This is easy to reproduce by adding code to the watchdog to break = into the debugger if panicstr is set. I am experimenting with Andriy's patch** to stop the scheduler and it = seems to be most of the way there, stopping the CPUs and disabling the = rest of locking. There are a few places that still reference panicstr, = but that's minor. These are the changes I made to the patch: * Changed ukbd_do_poll() to return immediately if SCHEDULER_STOPPED() = is true, so that we don't hang up in USB. ukbd_yield() locks up in = DROP_GIANT(), and if you skip ukbd_yield(), usbd_transfer_poll() locks = up trying to drop mutexes. * Changed the call to spinlock_enter() back to critical_enter(), so = that interrupts stay enabled and the hardclock still functions. * Added code in the beginning of panic() to switch to CPU 0, so that = we're able to service the hardclock interrupts and so that watchdog = panics get through. This has worked 100% for me so far, although anyone using a USB keyboard = or dump device would still be out of luck. Thoughts? It seems like stopping all of the other CPUs is the right = thing to do on a panic (what are they doing otherwise?). Are the USB = issues fixable? If Andriy's patch get committed it might just involve = short-circuiting all of the locking in the polling path, but I haven't = gotten that far yet. I bet dumping to NFS will have the same problem. Thanks, Andrew * - http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/155421 ** - http://people.freebsd.org/~avg/stop_scheduler_on_panic.8.x.diff -------------------------------------------------- Andrew Boyer aboyer@averesystems.com From owner-freebsd-stable@FreeBSD.ORG Fri Aug 12 21:39:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B754C106564A for ; Fri, 12 Aug 2011 21:39:15 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe05.c2i.net [212.247.154.130]) by mx1.freebsd.org (Postfix) with ESMTP id 232458FC08 for ; Fri, 12 Aug 2011 21:39:14 +0000 (UTC) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.1 cv=b20eX9+WwT+rchjRp65q2r9rWd5NV3fsbYwDDw8k8E8= c=1 sm=1 a=SvYTsOw2Z4kA:10 a=EPV5yV1zpIAA:10 a=WQU8e4WWZSUA:10 a=Q9fys5e9bTEA:10 a=CL8lFSKtTFcA:10 a=i9M/sDlu2rpZ9XS819oYzg==:17 a=EHQJzY0G4o8SrkulcNEA:9 a=re_1W8GF1jsLQfJYVUgA:7 a=PUjeQqilurYA:10 a=i9M/sDlu2rpZ9XS819oYzg==:117 Received: from [188.126.198.129] (account mc467741@c2i.net HELO laptop002.hselasky.homeunix.org) by mailfe05.swip.net (CommuniGate Pro SMTP 5.2.19) with ESMTPA id 163138349; Fri, 12 Aug 2011 23:29:10 +0200 From: Hans Petter Selasky To: Andrew Boyer Date: Fri, 12 Aug 2011 23:26:49 +0200 User-Agent: KMail/1.13.5 (FreeBSD/8.2-STABLE; KDE/4.4.5; amd64; ; ) References: In-Reply-To: X-Face: *nPdTl_}RuAI6^PVpA02T?$%Xa^>@hE0uyUIoiha$pC:9TVgl.Oq, NwSZ4V"|LR.+tj}g5 %V,x^qOs~mnU3]Gn; cQLv&.N>TrxmSFf+p6(30a/{)KUU!s}w\IhQBj}[g}bj0I3^glmC( :AuzV9:.hESm-x4h240C`9=w MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201108122326.49597.hselasky@c2i.net> Cc: freebsd-stable@freebsd.org, Eugene Grosbein , Vishal.Shah@netapp.com, Andriy Gapon , Jeremiah Lott , Steven Hartland Subject: Re: USB/coredump hangs in 8 and 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 21:39:15 -0000 On Friday 12 August 2011 21:59:21 Andrew Boyer wrote: > Re: panic: bufwrite: buffer is not busy??? (originally on freebsd-net) > Re: debugging frequent kernel panics on 8.2-RELEASE (originally on > freebsd-stable) Re: System hang in USB umass module while processing panic > (originally on freebsd-usb) > > Hello Andriy and Hans, > > Sorry for tying in so many discussions on this topic, but I think I have an > explanation for the problems we have been reporting* with hanging > coredumps on multicore systems on 8.2-RELEASE, and it has implications for > Andriy's proposed scheduler patch** and for USB. > > In today's 8.X and 9.X branches, nothing that I can find stops the other > CPUs when the kernel panics, but many parts of the locking code get > disabled (grep on 'panicstr'). The 'bufwrite: buffer is not busy???' > panic is caused by the syncer encountering an error. If that happens when > it's on the dumping CPU everything hangs. If it's running on a different > CPU, it will be blocked and hidden by the panic_cpu spinlock in panic(), > and the dump continues, polling every attached keyboard for a Ctl-C. > > But, the new 8.X USB stack relies on multithreading. (The new stack is the > variable that broke coredumps for us in the 7.1->8.2 transition, I think.) > SVN 224223 fixes a hang that would happen when dumpsys() polls the USB > keyboard (IPMI KVM, in our case). That helps, but it only gets as far as > usb_process(), where it hangs in a loop around a cv_wait() call. This is > easy to reproduce by adding code to the watchdog to break into the > debugger if panicstr is set. > > I am experimenting with Andriy's patch** to stop the scheduler and it seems > to be most of the way there, stopping the CPUs and disabling the rest of > locking. There are a few places that still reference panicstr, but that's > minor. These are the changes I made to the patch: * Changed > ukbd_do_poll() to return immediately if SCHEDULER_STOPPED() is true, so > that we don't hang up in USB. ukbd_yield() locks up in DROP_GIANT(), and > if you skip ukbd_yield(), usbd_transfer_poll() locks up trying to drop > mutexes. * Changed the call to spinlock_enter() back to critical_enter(), > so that interrupts stay enabled and the hardclock still functions. * Added > code in the beginning of panic() to switch to CPU 0, so that we're able to > service the hardclock interrupts and so that watchdog panics get through. > > This has worked 100% for me so far, although anyone using a USB keyboard or > dump device would still be out of luck. > > Thoughts? It seems like stopping all of the other CPUs is the right thing > to do on a panic (what are they doing otherwise?). Are the USB issues > fixable? If Andriy's patch get committed it might just involve > short-circuiting all of the locking in the polling path, but I haven't > gotten that far yet. I bet dumping to NFS will have the same problem. Hi. USB does not rely on multithreading when doing polling. It bypasses the processing thread and calls the function directly. Also I can add the USB has recursive checking flags, so that if important functions are already called, the code will simply return. USB does not rely on locking after panic, except maybe mtx_owned() returning the correct value. Your approaching having the mtx_lock() / mtx_unlock() functions simply do nothing will affect the USB polling ability if mtx_owned() does not return true when the lock is locked. So maybe in case of SCHEDULER stopped we should just steal the lock instead of just returning. Also I sssume that all interrupts and all other processes are blocked at the moment of panic or dump. --HPS