From owner-freebsd-stable@FreeBSD.ORG Sat Jun 6 14:49:56 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8571106564A for ; Sat, 6 Jun 2009 14:49:56 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-bw0-f214.google.com (mail-bw0-f214.google.com [209.85.218.214]) by mx1.freebsd.org (Postfix) with ESMTP id 408638FC08 for ; Sat, 6 Jun 2009 14:49:56 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by bwz10 with SMTP id 10so23252bwz.43 for ; Sat, 06 Jun 2009 07:49:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=PM5aTt3S8y1xeyo8MdfcJE3M11QjLLhcr7KG6SrOk0M=; b=VrF/QMLozaLci5eEaTHoJqze67S45c2Uw7DVhYJcECF624JfrXDKahkYypk4Y8WoY6 bnoB+uUOHpKQ1YknjO+CYSH70berQi0kz/yfxGTABAlz4vPhuACkEVmXc9pVVFiKv36G UnOkF3hgKuIN92buvtOVVymoRBlLPnEp2STxg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=RTrUXfZcEcnWdevFdv+BHS5s1Tzi15cjOkJTeqx8Sp1GeL3iiX0OPYVo4XQtgdxg+j mXsuxBpjaHRqVOt+iQWUsSlJOKiKkHm3kV1dFnoS3AWwZWV6PoGPdWiHNQCn8LU8SnQ/ ahqLIsG42F0A4hv35rDVxahdm8MQ88y9z05f0= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.223.103.207 with SMTP id l15mr2676080fao.2.1244299794051; Sat, 06 Jun 2009 07:49:54 -0700 (PDT) In-Reply-To: <86d49h300c.fsf@ra333.heimat.gr.jp> References: <86d49h300c.fsf@ra333.heimat.gr.jp> Date: Sat, 6 Jun 2009 16:49:54 +0200 X-Google-Sender-Auth: 173aead4ae9f08c6 Message-ID: <3bbf2fe10906060749xbbc2f2fy4c09f67711a63b@mail.gmail.com> From: Attilio Rao To: NAKAJI Hiroyuki Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: Big problem still remains with 7.2-STABLE locking up X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Jun 2009 14:49:57 -0000 2009/6/6 NAKAJI Hiroyuki : > Hi, > > I noticed, some months ago, frequent lockups on my RELENG_6 server with > ECS PM800-M2, Celeron 2.6GHz (UP), 2GB ram, ATA HDDs and 3Com NIC(xl0), > and then I gave up this old server. > > Last month, I replaced this 'unstable' server to the new one with > 7.2-RELEASE which worked very well until I setup it as 'a server'. The > problem began just after it started 'the services'. > > My story is very similar to Pete's. > http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/047487.htm= l > > I followed some instructions in the list thread. But unfortunately, the > big problem still remains. 7.2-STABLE server locks up frequently. > > Help! :-( > > The server is NEC Express5800 S70/SD. > > o CPU: Intel(R) Celeron(R) CPU 440 @ 2.00GHz (2280.25-MHz K8-class CPU) > o 6GB RAM > o ACPI APIC Table: > o 80GB and 250GB SATA HDDs > o http://www.heimat.gr.jp/~nakaji/localhost/dmesg.boot > > The kernel configuration is: > > include GENERIC > ident =C2=A0 HEIMAT > options MSGBUF_SIZE=3D81920 > makeoptions =C2=A0 =C2=A0 DEBUG=3D-g > options KDB > options DDB > options BREAK_TO_DEBUGGER > options QUOTA Were you unmounting any of the QUOTA'ed filesystems? I'm aware of a possible deadlock between quota and unmount path which is very difficult to trigger though. Anyways, the only one way we have to debug this is getting some help by the user. 1) Drop the option WITNESS_SPIKSPIN (as we would like to debug spinlocks too) and LOCK_PROFILING (in order to create higher contention and kill some barriers) 2) Once you get the deadlock break in the DDB debugger 3) Once you are in DDB informations which could be very useful are: db> show allpcpu db> show alllocks db> show lockedvnods db> ps db> allthreads Note that this is a lot of printout so you won't be able of collecting all these informations if not with a serial connection. 4) Dump the content so that we can further look at locks structure states once we identify something useful (ideally, keeping the machine up in DDB for that would be very useful, but often not viable) Let me know. Attilio --=20 Peace can only be achieved by understanding - A. Einstein