Date: Fri, 02 Jul 2010 09:07:44 -0400 From: John Almberg <jalmberg@identry.com> To: Bas Smeelen <b.smeelen@ose.nl> Cc: freebsd-questions@freebsd.org Subject: Re: ssh_exchange_identification: Connection closed by remote host Message-ID: <4C2DE4A0.1000209@identry.com> In-Reply-To: <4C2DD89F.4010302@ose.nl> References: <4C2DC4FC.7070004@identry.com> <AANLkTinNCKfwgHf-ha0cys9cdTSk-rcYznWO3zcK2r0H@mail.gmail.com> <4C2DCD58.3070103@identry.com> <4C2DCE9B.4090306@ose.nl> <4C2DD130.5070508@identry.com> <4C2DD599.5030101@infracaninophile.co.uk> <4C2DD70F.8040004@identry.com> <4C2DD89F.4010302@ose.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
>>> If you can't log in -- even on the console -- then rebooting is really >>> your only option. Ctrl-Alt-Del should bring the system down cleanly if >>> you haven't disabled that functionality. Otherwise, just toggle the >>> power. >>> >>> The symptoms you're seeing could well be due to filesystem problems or >>> to some filesystem filling up (/tmp is a prime suspect) or due to >>> running out of memory+swap. Some sort of memory leak sounds pretty >>> likely actually. >>> >>> Probably best to bring the system up in single user mode and run fsck on >>> all the filesystems manually -- that will show if you've got h/w >>> problems with drives and possibly with disk controllers or cabling too. >>> Then check for overfull filesystems. You may not find any -- >>> rebooting >>> can clear a number of conditions where disk space is not released back >>> to the OS properly after use. You may or may not find any clues as to >>> what went wrong in the system logs. In the absence of any other clues, >>> the only option is to monitor the server closely and wait for something >>> similar to happen again. Hopefully if there is a next time, you'll be >>> able to catch it and fix the underlying problem before it takes the >>> machine out a second time. >>> >>> >>> >> Yes, I can't log in. I get a login prompt, but no password prompt. I'm >> going to try ctrl-alt-del and see what happens. >> >> Crossing fingers... >> > Sorry I missed that you can't login. > Good luck > So, ctrl-alt-del did the trick. I was able to log in and actually, the whole box came up and everything seems to be working. I thought for sure I'd find that my /var directory was full up, but it's only at 77% (that's the weak spot on this box... I wish I'd made the /var partition bigger.) The message log is full of these messages: 38054 Jul 2 08:13:02 qu kernel: swap_pager_getswapspace(16): failed If I run swapinfo, I get this: [master@qu:log]> swapinfo Device 1K-blocks Used Avail Capacity /dev/ar0s1b 2055952 208 2055744 0% I looked back in the log file to see if there were any clues when the problem began and found this: Jul 2 03:19:25 qu kernel: swap_pager: out of swap space Jul 2 03:19:26 qu kernel: swap_pager_getswapspace(7): failed Jul 2 03:19:26 qu kernel: pid 93543 (mysqld), uid 88, was killed: out of swap space Jul 2 03:19:26 qu kernel: pid 85077 (ruby18), uid 1023, was killed: out of swap space Jul 2 03:19:25 qu root: Check for bad ssh behavior Jul 2 03:20:05 qu root: Check for bad ssh behavior Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(15): failed Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(14): failed Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed Jul 2 03:20:49 qu last message repeated 2 times It pretty much goes on forever from there. Is there any other place I can look for clues as to why I ran out of swap space? This machine is basically a webserver, running apache/mysql and ruby on rails. It's been running for over a year with no problems. No new software introduced on the box, recently. -- John
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C2DE4A0.1000209>