From owner-freebsd-questions@FreeBSD.ORG Fri Jul 2 13:07:47 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5C56106566C for ; Fri, 2 Jul 2010 13:07:47 +0000 (UTC) (envelope-from jalmberg@identry.com) Received: from smtp-gw30.mailanyone.net (smtp-gw30.mailanyone.net [208.70.128.56]) by mx1.freebsd.org (Postfix) with ESMTP id 6AE908FC0A for ; Fri, 2 Jul 2010 13:07:47 +0000 (UTC) Received: from mailanyone.net by smtp-gw30.mailanyone.net with esmtpa (MailAnyone extSMTP jalmberg@identry.com) id 1OUfy5-0004w9-Ek; Fri, 02 Jul 2010 08:07:46 -0500 Message-ID: <4C2DE4A0.1000209@identry.com> Date: Fri, 02 Jul 2010 09:07:44 -0400 From: John Almberg User-Agent: Postbox 1.1.5 (Macintosh/20100613) MIME-Version: 1.0 To: Bas Smeelen References: <4C2DC4FC.7070004@identry.com> <4C2DCD58.3070103@identry.com> <4C2DCE9B.4090306@ose.nl> <4C2DD130.5070508@identry.com> <4C2DD599.5030101@infracaninophile.co.uk> <4C2DD70F.8040004@identry.com> <4C2DD89F.4010302@ose.nl> In-Reply-To: <4C2DD89F.4010302@ose.nl> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-questions@freebsd.org Subject: Re: ssh_exchange_identification: Connection closed by remote host X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 13:07:47 -0000 >>> If you can't log in -- even on the console -- then rebooting is really >>> your only option. Ctrl-Alt-Del should bring the system down cleanly if >>> you haven't disabled that functionality. Otherwise, just toggle the >>> power. >>> >>> The symptoms you're seeing could well be due to filesystem problems or >>> to some filesystem filling up (/tmp is a prime suspect) or due to >>> running out of memory+swap. Some sort of memory leak sounds pretty >>> likely actually. >>> >>> Probably best to bring the system up in single user mode and run fsck on >>> all the filesystems manually -- that will show if you've got h/w >>> problems with drives and possibly with disk controllers or cabling too. >>> Then check for overfull filesystems. You may not find any -- >>> rebooting >>> can clear a number of conditions where disk space is not released back >>> to the OS properly after use. You may or may not find any clues as to >>> what went wrong in the system logs. In the absence of any other clues, >>> the only option is to monitor the server closely and wait for something >>> similar to happen again. Hopefully if there is a next time, you'll be >>> able to catch it and fix the underlying problem before it takes the >>> machine out a second time. >>> >>> >>> >> Yes, I can't log in. I get a login prompt, but no password prompt. I'm >> going to try ctrl-alt-del and see what happens. >> >> Crossing fingers... >> > Sorry I missed that you can't login. > Good luck > So, ctrl-alt-del did the trick. I was able to log in and actually, the whole box came up and everything seems to be working. I thought for sure I'd find that my /var directory was full up, but it's only at 77% (that's the weak spot on this box... I wish I'd made the /var partition bigger.) The message log is full of these messages: 38054 Jul 2 08:13:02 qu kernel: swap_pager_getswapspace(16): failed If I run swapinfo, I get this: [master@qu:log]> swapinfo Device 1K-blocks Used Avail Capacity /dev/ar0s1b 2055952 208 2055744 0% I looked back in the log file to see if there were any clues when the problem began and found this: Jul 2 03:19:25 qu kernel: swap_pager: out of swap space Jul 2 03:19:26 qu kernel: swap_pager_getswapspace(7): failed Jul 2 03:19:26 qu kernel: pid 93543 (mysqld), uid 88, was killed: out of swap space Jul 2 03:19:26 qu kernel: pid 85077 (ruby18), uid 1023, was killed: out of swap space Jul 2 03:19:25 qu root: Check for bad ssh behavior Jul 2 03:20:05 qu root: Check for bad ssh behavior Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(15): failed Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(14): failed Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed Jul 2 03:20:49 qu last message repeated 2 times It pretty much goes on forever from there. Is there any other place I can look for clues as to why I ran out of swap space? This machine is basically a webserver, running apache/mysql and ruby on rails. It's been running for over a year with no problems. No new software introduced on the box, recently. -- John