From owner-freebsd-stable@FreeBSD.ORG Tue Jul 29 20:41:52 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E9C8337B401; Tue, 29 Jul 2003 20:41:52 -0700 (PDT) Received: from mobile.hub.org (u134n133.eastlink.ca [24.224.134.133]) by mx1.FreeBSD.org (Postfix) with ESMTP id 01B0043F85; Tue, 29 Jul 2003 20:41:51 -0700 (PDT) (envelope-from scrappy@hub.org) Received: by mobile.hub.org (Postfix, from userid 1001) id C005B213; Wed, 30 Jul 2003 00:41:49 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by mobile.hub.org (Postfix) with ESMTP id AC5C41C8; Wed, 30 Jul 2003 00:41:49 -0300 (ADT) Date: Wed, 30 Jul 2003 00:41:49 -0300 (ADT) From: The Hermit Hacker To: Don Bowman In-Reply-To: Message-ID: <20030730004133.X17191@hub.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: "'freebsd-stable@freebsd.org'" cc: 'Robert Watson' Subject: RE: kernel deadlock X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2003 03:41:53 -0000 Suggestion: upgrade to 4.8-STABLE ... there hav On Tue, 29 Jul 2003, Don Bowman wrote: > From: Don Bowman [mailto:don@sandvine.com] > > > > From: Robert Watson [mailto:rwatson@freebsd.org] > > > On Tue, 29 Jul 2003, Dave Dolson wrote: > > > > > > > To follow up, I've discovered that the system has > > exhausted its "FFS > > > > node" malloc type. > > ... > > > > > > Some problems with this have turned up in -CURRENT on large-memory > > > machines where some of the scaling factors have been off. In > > > > We currently have kern.maxvnodes=70354 set (automatically > > scaled). This > > is a 1GB box. > > > > I will try re-running the test with less. > > > > when it hits kern.maxvnodes, what will it do? > > After applying the fixes from RELENG_4 for kern/52425, > I can still easily reproduce this hang without low memory. > Further debugging shows that vnlru process is waiting on > vlrup. This line is shown below. ie vnlru_nowhere is being > incremented ever 3 seconds. > > static void > vnlru_proc(void) > { > ... > s = splbio(); > for (;;) { > ... > if (done == 0) { > vnlru_nowhere++; > tsleep(vnlruproc, PPAUSE, "vlrup", hz * 3); > } > } > splx(s); > > syncher is in vlruwk wait from getnewvnode(). > > lots of other processes waiting on ffsvgt. > > this implies that vlrureclaim() was unable to free anything. > > i have maxvnode = 35k. as soon as i hit this value, my system locked > up [bash on serial shell non-responsive, serial driver echos chars, > can drop into ddb]. Processes which don't use filesystem seem to continue > to run ok. > > A couple of procs are waiting on inode: env, cron. These never come > out of waiting for it. > > suggestions? > > db> ps > pid proc addr uid ppid pgrp flag stat wmesg wchan cmd > 649 dc35a8a0 e0a32000 0 641 641 004104 3 ffsvgt c03698a8 atrun > 648 dc35a3c0 e0e36000 0 647 648 000014 3 vlruwk c0364c90 cron > 647 dc35b740 e03d4000 0 135 135 000004 3 ppwait dc35b740 cron > 646 dc35b0c0 e03ee000 0 635 101 004004 3 inode c368ee00 env > 645 dc35ad80 e03f1000 0 212 644 004006 3 ffsvgt c03698a8 grep > 644 dc35aa40 e0400000 0 212 644 004006 3 ffsvgt c03698a8 sysctl > 641 dc35a080 e0e4c000 0 640 641 004084 3 wait dc35a080 sh > 640 dc35a220 e0e39000 0 135 135 000084 3 piperd e037c5c0 cron > 635 dc35a560 e0e32000 0 101 101 004084 3 piperd e037cd40 sh > 456 dc35abe0 e03fc000 0 133 456 4004004 3 ffsvgt c03698a8 tclsh83 > 212 dc35bdc0 e0392000 0 199 212 004086 3 wait dc35bdc0 bash > 199 dc35c440 e036e000 0 1 199 004186 3 wait dc35c440 login > 187 dc35c2a0 e0376000 0 1 7 000086 3 select c037c460 snmpd > 169 dc35af20 e03e7000 0 1 169 000084 3 nanslp c0364970 > siocontrol > 163 dc35b260 e03e2000 0 1 163 000084 3 nanslp c0364970 wddt > 143 dc35b400 e03dd000 25 1 143 2000184 3 pause e03dd260 > sendmail > 140 dc35b5a0 e03d9000 0 1 140 000184 3 select c037c460 sendmail > 137 dc35b8e0 e03d0000 0 1 137 000184 3 select c037c460 sshd > 135 dc35ba80 e03c2000 0 1 135 000004 3 inode c35f4400 cron > 133 dc35bc20 e0397000 0 1 133 000084 3 select c037c460 inetd > 124 dc35bf60 e0382000 0 1 124 000084 3 select c037c460 syslogd > 101 dc35c100 e037e000 0 1 101 000084 3 wait dc35c100 dhclient > 6 dc35c5e0 defd1000 0 0 0 000204 3 vlrup dc35c5e0 vnlru > 5 dc35c780 defce000 0 0 0 000204 3 syncer c037c388 syncer > 4 dc35c920 defcb000 0 0 0 000204 3 psleep c0364b3c > bufdaemon > 3 dc35cac0 defc8000 0 0 0 000204 3 psleep c0373280 vmdaemon > 2 dc35cc60 defc5000 0 0 0 000204 3 psleep c0352118 > pagedaemon > 1 dc35ce00 dc361000 0 0 1 004284 3 wait dc35ce00 init > 0 c037b760 c040e000 0 0 0 000204 3 sched c037b760 swapper > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org