From owner-freebsd-stable@FreeBSD.ORG Wed Nov 26 14:13:10 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3EB1D16A4CE for ; Wed, 26 Nov 2003 14:13:10 -0800 (PST) Received: from unix18.sihope.com (unix18.sihope.com [207.195.195.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 10C5643FD7 for ; Wed, 26 Nov 2003 14:13:09 -0800 (PST) (envelope-from adam@whee.org) Received: from adammxp (pc6.sihope.com [207.195.195.65]) by unix18.sihope.com (8.12.9p2/8.12.9) with SMTP id hAQMD8lU037976 for ; Wed, 26 Nov 2003 16:13:08 -0600 (CST) (envelope-from adam@whee.org) Message-ID: <00a401c3b46a$78cd7650$41c3c3cf@office.sihope.com> From: "Adam Maloney" To: References: <004f01c3b427$eaea8c70$41c3c3cf@office.sihope.com> <008601c3b455$379b72a0$41c3c3cf@office.sihope.com> Date: Wed, 26 Nov 2003 16:13:08 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Subject: Re: NFS/Sendmail issue since upgrading to 4.9-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2003 22:13:10 -0000 Sorry for the noise on the list about this. We think we've identified the issue. We made the change to make the inbound queues local, and the problem occured again almost immediately. That ruled out NFS, and left us with only one other realistic option. This custom application reads a directory list and for each qf file, attempts a LOCK_EX. For the past 2 years this has appeared to work fine with Sendmail - if Sendmail is still working on the qf file, we aren't able to get an exclusive lock, so we go on to the next file, etc. (Maybe Claus can confirm some of this behavior?) It appears that Sendmail changed from 8.12.9 (FreeBSD 4.8R) to 8.12.10 (FreeBSD 4.9-STA). What we believe is happening is that the qf file is being created, and our application is getting the exclusive lock before Sendmail. This makes Sendmail spit out the "resource temporarily unavailable" error, since it can't lock the file, and it leaves our application with an empty qf file that we are trying to process. It will take several more hours of testing before we're relatively confident that this is the issue. Hopefully I'll be posting one last time to confirm that everything is fixed. ----- Original Message ----- From: "Adam Maloney" To: Sent: Wednesday, November 26, 2003 1:40 PM Subject: Re: NFS/Sendmail issue since upgrading to 4.9-STABLE > Just an update for anyone paying attention to this issue. > > We are going to temporarily move the inbound Sendmail queue to be local > rather than NFS. We'll run like this through the holiday weekend and see if > the problems we're having continue or not. If they stop happening, then we > can be pretty confident that some change in the NFS client code is causing > this (the problems started when we upgraded the 1st client, before making > any changes to the server). > > We've also noticed another problem that we believe to be NFS related as well > with these same machines. It started happening at exactly the same time > (though less frequently). This 2nd issue is in a custom application that we > are running. At this time I don't have a lot of detail, but we are fairly > confident that it is also NFS related. Making the inbound Sendmail queues > local will also isolate this to be NFS or not. > > I will report back on Monday with our results. > > I appreciate everyone's input. > > ----- Original Message ----- > From: "Adam Maloney" > To: > Sent: Wednesday, November 26, 2003 8:16 AM > Subject: NFS/Sendmail issue since upgrading to 4.9-STABLE > > > > Hi all, > > > > I have 3 servers accepting mail, with sendmail in Queue-only mode dropping > > mail into an NFS queue (mounted from a 4th machine.) All 4 machines > > involved are running FreeBSD. > > > > I upgraded the 3 NFS clients from 4.8 to 4.9-STABLE on Monday, and started > > seeing this sendmail message: > > > > Nov 26 06:51:29 unix20 sm-mta[48069]: (real-message-id): SYSERR(root): > > queueup: cannot create queue file ./qf(real-message-id), euid=0: Resource > > temporarily unavailable > > > > The df(message-id) file is created and appears to be complete, however the > > qf file is 0 bytes. > > > > This happens sporadically, a few times a day on each machine (together > doing > > about 150,000 messages/day). > > > > I upgraded the NFS server from 4.8 to 4.9 yesterday, thinking that might > > help, but the problem is still there. > > > > Any ideas how to troubleshoot this one? > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >