Date: Wed, 12 Nov 2003 19:16:04 -0500 (EST) From: Robert Watson <rwatson@freebsd.org> To: freebsd-current@freebsd.org Subject: Re: Still getting NFS client locking up Message-ID: <Pine.NEB.3.96L.1031112190842.96006C-100000@fledge.watson.org> In-Reply-To: <200311121738.13783.x@Vex.Net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 12 Nov 2003, Tim Middleton wrote: > On November 11, 2003 11:36 pm, Janet Sullivan wrote: > > So far I only have problems in a mixed -STABLE/-CURRENT environment. > > When the client & server are both -CURRENT I haven't had any problems. > > I just installed another -STABLE box to see if keeping them both -STABLE > helps. I haven't really tested the NFS yet as I didn't want to risk > locking the box up in the middle of a buildworld. If we can demonstrate the problem with both systems as -STABLE, that rules out a lot of things, and might also raise some questions about the hardware. > So i just mounted the NFS drive on the new test box and left it.... > Within an hour the NFS server box doing the build world was locked up > solid. I can't say if it was NFS mount related or not; nfsd wasn't > really doing anything. Doesn't seem like it would have been. Beginning > to wonder if it is some strange hardware problem on this box; which > coincidentally only shows up when there's an nfs mount! But that doesn't > explain why my normally rock solid desktop system tanked when being > tested as an NFS client to that STABLE box. Hmmm... One of the problems that can occur in -STABLE is a cascading failure when one file system is jammed up (i.e., an NFS mount from another system). Processes hang holding locks in NFS because the NFS session is stalled; other processes try to aquire the hold locks while holding additional locks, and before you know it a lot of very useful locks are held and can't be released due to an inability to free up locks at the cause. Many aspects of this problem are believed to be resolved in -CURRENT, but it's a touch cookies to crack without redoing VFS locking. If you have a spare system, it might be really interesting to install -STABLE on it, replicate data from your file server, point the client at that, and see if the problem still occurs there with the same load. You might also try swapping network cards: perhaps we're looking at a network device driver problem where loss of key packets, or packets over a certain size, is causing an unrecoverable failure. > Back to testing. I'm doing heavy disk I/O tests without any NFS mounts > now. If they go okay, back to the NFS mounting and testing... > > It seems to me there is something desperately wrong with NFS is mixing > -CURRENT and -STABLE NFS server/clients causes either side (in my case both > sides) to lock up solid. I mean, problems are problems... but solid lockups > with no crash messages or anything is ... nasty. Clearly there's a substantial problem, but it sounds like we're still having a lot of trouble identifying the circumstances that trigger the problem, and attempting to narrow things down. One of the problem with distributed system debugging is that it's often hard to track the problem down to a particular source when you catch it partway through a cascading failure. For example, it could well be that a server problem is triggering client symptoms, or it could be that a serious client problem might consume resources on the server such that other clients couldn't operate. Under these circumstances, it can be very difficult to track it down to a particular cause (a missing unlock on the server, for example). > > Are the folks seeing hangs getting any kind of console error messages? > > I see nothing. My server is completely locks up. Nothing responds. The > drive light (the times i've noticed) is frozen "on". On my desktop box > the mouse is dead as well. I can't help but wonder if the server isn't suffering an under-reported hardare failure. It might be interesting to see how quickly the problem vanishes when exchanging various elements. > > I don't see anything - performance just tanks to the point of being > > unusable. > > When testing with my desktop box as client, i noticed just before or > just when the NFS locked up the mouse and keyboard response would be > very erratic ... slow and jerky. This might suggest a high RPC load, deep queues in processing, or key locks held for extended periods of time. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1031112190842.96006C-100000>