Date: Wed, 24 May 2006 23:48:53 -0400 From: "Howard Leadmon" <howard@leadmon.net> To: "'Howard Leadmon'" <howard@leadmon.net>, "'Rong-en Fan'" <grafan@gmail.com> Cc: 'Konstantin Belousov' <kostikbel@gmail.com>, freebsd-stable@freebsd.org, 'Kris Kennaway' <kris@obsecurity.org> Subject: RE: Trouble with NFSd under 6.1-Stable, any ideas? Message-ID: <003001c67fae$27a88370$071872cf@Leadmon.local> In-Reply-To: <001401c67f56$b02975e0$071872cf@Leadmon.local>
next in thread | previous in thread | raw e-mail | index | archive | help
I need to follow up to the below, as I am not sure why the below test with the vfs_lookup.c didn't pan out the first time, but with my new found knowledge on cvs I was determined to regress the system till I found the smoking gun so to speak, which I have done. First let me say that instead of running RELENG_6_1 like Rong-en is, I am running the RELENG_6 tree that I know updates more often, but seems to work well for me. OK, so as I said above I started to regress the system a couple days at a time, till suddenly NFS stared working again, so I knew at that point it was a change that was made. So then I started to narrow the time range, till I got to the point that it broke. Sure enough under the RELENG_6 branch, this time was as follows: *default tag=RELENG_6 date=2006.04.30.03.57.00 (Works OK) *default tag=RELENG_6 date=2006.04.30.03.58.00 (Broken) So what's changed at that delta, under the one that works vfs_lookup.c is: Edit src/sys/kern/vfs_lookup.c Add delta 1.80.2.6 2006.03.31.07.39.24 kris Under the one that fails the vfs_lookup.c is: Edit src/sys/kern/vfs_lookup.c Add delta 1.80.2.7 2006.04.30.03.57.46 kris So I stand corrected on my last post, the issue is in fact in this module, as just taking that module back to 1.80.2.6 fixes the problem with my server. I even took multiple NFS clients and gave them a heavy workload, and CPU still remained reasonable, and very responsive. As soon as I rev to the new version, NFS breaks badly and even a single client doing something like a du of a directory structure results in sluggishness and extreme CPU usage. I am not a coder, so not sure why this module was changed, but unless there is some good reason why the changes were needed I would suspect it needs to be rolled back, or something fixed. So Rong-en Fan, I think you were dead on with your analysis that the issue is in fact inside the vfs_lookup.c module. I hope this helps... --- Howard Leadmon - howard@leadmon.net http://www.leadmon.net > -----Original Message----- > From: owner-freebsd-stable@freebsd.org > [mailto:owner-freebsd-stable@freebsd.org] On Behalf Of Howard Leadmon > Sent: Wednesday, May 24, 2006 1:23 PM > To: 'Rong-en Fan' > Cc: 'Konstantin Belousov'; freebsd-stable@freebsd.org > Subject: RE: Trouble with NFSd under 6.1-Stable, any ideas? > > > Hello Rong-en, > > > As an update, I did the below, and I still had the issue with > either version > of vfs_lookup.c compiled in and running. > > On the bright side, I didn't realize you could step through > the cvs by date, guess I just never paid attention. So I > just stepped back to 'tag=RELENG_6 date=2006.04.20.00.00.00' > on my server, rebuilt and violla nfs is now running > perfect. > > So backing out something has fixed my problem, now to figure > out just what it > was. As I don't know what has caused this, I have done > complete buildworlds > to make sure everything updates which takes a few hours. I > am going to > start moving the cvs date forward till I get the problem > back, once I nail this down a bit more, I'll let you know > what I come up with. > > > > --- > Howard Leadmon > http://www.leadmon.net > > > > > -----Original Message----- > > From: Rong-en Fan [mailto:grafan@gmail.com] > > Sent: Tuesday, May 23, 2006 3:09 PM > > To: Howard Leadmon > > Cc: freebsd-stable@freebsd.org > > Subject: Re: Trouble with NFSd under 6.1-Stable, any ideas? > > > > On 5/23/06, Howard Leadmon <howard@leadmon.net> wrote: > > > > > > Hello Rong-en, > > > > > > Thanks for the info on getting the debugger configured, > > and on the serial > > > console. I will have to try and play with the serial > > console thing more, I > > > just tried putting in the flags and the damn thing hung, I > > had to boot > > > from CD and take the stuff back out. > > > > > > One thing you mention below that concerns me is that you > > have version 1.90 of > > > the vfs_lookup.c file. I just did a less on > > /usr/src/sys/kern/vfs_lookup.c > > > and I see the following: > > > > > > FreeBSD: src/sys/kern/vfs_lookup.c,v 1.80.2.7 2006/04/30 > > 03:57:46 kris > > > Exp > > > > > > > > > I even did a cvsup (I use cvsup2.FreeBSD.org) to make sure > > I had the > > > current stuff before rebuilding the kernel just now, and > > still I see the same thing. > > > Is something fishy going on here, or did you by chance > make a typo?? > > > > Sorry for the confusion. rev 1.90 is the number for -HEAD. > To back out > > this MFC'ed change for RELENG_6_1, please cvsup to > > RELENG_6_1 date=2006.04.30.03.57.00. Then you should see it is > > > > 1.80.2.6 2006/03/31 07:39:24 kris > > > > To verify the effect of this revision. Please run RELENG_6_1 with > > 2006.04.30.03.57.00 and 2006.04.30.04.00.00. > > > > Regards, > > Rong-En Fan > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?003001c67fae$27a88370$071872cf>