From owner-freebsd-stable@FreeBSD.ORG Tue Dec 22 14:00:20 2009 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF814106568B for ; Tue, 22 Dec 2009 14:00:20 +0000 (UTC) (envelope-from flo@smeets.im) Received: from mail.solomo.de (mail.solomo.de [85.214.124.163]) by mx1.freebsd.org (Postfix) with ESMTP id 76F168FC08 for ; Tue, 22 Dec 2009 14:00:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.solomo.de (Postfix) with ESMTP id 1CBAC6229D; Tue, 22 Dec 2009 14:50:17 +0100 (CET) X-Virus-Scanned: amavisd-new at vistream.de Received: from mail.solomo.de ([127.0.0.1]) by localhost (db1.solomo.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id sXKYtISah0aG; Tue, 22 Dec 2009 14:49:55 +0100 (CET) Received: from nibbler.vistream.local (relay3.vistream.de [87.139.10.28]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.solomo.de (Postfix) with ESMTPSA id EC88E62298; Tue, 22 Dec 2009 14:49:54 +0100 (CET) Message-ID: <4B30CE81.7030303@smeets.im> Date: Tue, 22 Dec 2009 14:49:53 +0100 From: Florian Smeets User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091220 Shredder/3.0.1pre MIME-Version: 1.0 To: Pete French References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: Disc lock up on 8.0-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2009 14:00:21 -0000 On 12/22/09 1:16 PM, Pete French wrote: > I've been gradually testing 8.0 on several machines propr to deploying it > live, but I currently have a machine which appears to lock-up at 3am > every day. The symptoms are that the machine is still pingable, but doing > anything which requires access to the disc just freezes (so you cant login for > example). I've seen simiilar behaviour behore on machines when the disc > syste has locked up for some reason, so am ttentatively guessing that > this is the cause. > > The machine is an HP DL360 G5 with a ciss0 controller for the drives. > I have upgraded to the latest STABLE but the freeze still happens. > Am including a dmesg below, and will compile it with KDB, DDB to > see what happens. > > The machine is booting from a UFS partition, but is using ZFS for everything > else. The fcat it deadlocks at 3am makes me thing this is something to > do with scheduled jobs maybe ? Then again, I have an almost identical DL360 > which is running 8.0 and is rock solid. > Hi Pete, i'm trying to track down the same problem. The box in question has everything on UFS (mirrored ataraid) and a backup disk with ZFS on it attached to USB. The freeze happens at 3am too, i have these log messages: Dec 20 03:00:00 XXX newsyslog[2810]: logfile turned over due to size>100K Dec 20 03:03:21 XXX kernel: Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl. Dec 20 03:03:22 XXX kernel: maxproc limit exceeded by uid 0, please see tuning(7) and login.conf(5). Dec 20 03:03:53 XXX last message repeated 31 times I had increased vm.pmap.pv_entry_max and vm.pmap.shpgperproc then only the maxproc limit exceeded message remained. As the box is remote with only ssh access, it's a little difficult to debug this. During the weekend i waited till 3 o'clock with a top running, and saw that hundreds/thousands of /bin/sh processes were started. After that i commented out periodic daily in /etc/crontab, that "solved" the problem for me. I was not able to debug this any further yet, i have one other box with all UFS and a ZFS backup disc also running latest 8-STABLE but it does not exhibit the problem. Cheers, Florian