Date: Thu, 23 Sep 2010 14:42:26 -0400 From: jhell <jhell@DataIX.net> To: Michael BlackHeart <amdmiek@gmail.com> Cc: freebsd-stable@freebsd.org, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: FreeBSD 8.1 Stable Unreasanoble Rebooting Message-ID: <4C9B9F92.5030102@DataIX.net> In-Reply-To: <4C9B9EAB.4050704@DataIX.net> References: <AANLkTimGZOwf5USXD4ANLBq0EBR1LvBakKAH88pmTLo%2B@mail.gmail.com> <20100916164930.GA31869@icarus.home.lan> <AANLkTinZRdrtdehrzqSRyom8hAo=gBLvtnwuSSgCC6B9@mail.gmail.com> <4C9B9EAB.4050704@DataIX.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 09/23/2010 14:38, jhell wrote:
> On 09/16/2010 15:33, Michael BlackHeart wrote:
>> 2010/9/16 Jeremy Chadwick <freebsd@jdc.parodius.com>:
>>> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote:
>>>> Today I've got a pretty strange event. It looks like a reboot but
>>>> unreasonable as far as I see. Before server's uptime was over month,
>>>> it's sometimes have to reboot for kernel updates or somethings like
>>>> that. I've digen all logs and didn't find a reason, so here they all.
>>>>
>>>> auth.log
>>>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating.
>>>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442.
>>>>
>>>> cron - nothing
>>>> debug.log - nothing
>>>> dmesg - nothing
>>>>
>>>> messages
>>>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create
>>>> socket: Protocol not supported (fdlimit.c:651)
>>>> Sep 16 13:45:31 diablo last message repeated 5 times
>>>> Sep 16 13:47:23 diablo last message repeated 13 times
>>>> Sep 16 13:57:40 diablo last message repeated 51 times
>>>> Sep 16 13:59:48 diablo last message repeated 12 times
>>>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953
>>>> Sep 16 14:00:18 diablo named[1575]: exiting
>>>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15
>>>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel
>>>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project.
>>>> {...}
>>>
>>> This sure looks like a legitimate reboot to me (e.g. shutdown -r now);
>>> note how your system daemons (named, syslogd) are being shut down with
>>> SIGTERM. You can check with "last" (shutdown/reboot vs. crash).
>>>
>>> <paranoid>
>>> I would highly recommend taking this machine offline and reinstalling
>>> the OS, in addition to newfs'ing all existing filesystems (restore from
>>> last known good backup). buildworld/installworld and
>>> buildkernel/installkernel may not be enough depending on what the
>>> individual did. It's likely the machine could be compromised in some
>>> way, especially if there's any service on it which is public-facing,
>>> regardless of authentication mechanisms you've deployed in front of it.
>>> </paranoid>
>>>
>>> --
>>> | Jeremy Chadwick jdc@parodius.com |
>>> | Parodius Networking http://www.parodius.com/ |
>>> | UNIX Systems Administrator Mountain View, CA, USA |
>>> | Making life hard for others since 1977. PGP: 4BD6C0CB |
>>>
>>>
>
>> That looks reasonable
>> last says:
>> reboot ~ th 16 sen 14:04
>> reboot ~ th 16 sen 14:03
>> shutdown ~ th 16 sen 13:59
>
>> and it's pretty good syncs with logs but there's no anybody access to
>> physical console 'cos it's not even plugged in. That's for the first.
>> Next, I've got, I believe, pretty strong passwords, and also root
>> can't log in directly, but wheel user also is in operators so he also
>> can reboot or shutdown, but there's no any attempts or successful
>> logins. All potentialy dangerous services run under their own
>> unprerileged users, and so on. Crontabs also doesn't contain scripts,
>> I prefer periodic system, and there's no anyway anything that cause
>> reboot.
>> Thing that worries me it that there were multiple reboots and shutdown
>> that goes up by itself without anyone pressing a button. And in
>> messages log there's fsck segment that indicates to unnormal shutdown
>> or reboot. It looks like it started to shutting down but was in some
>> case interrupted and after powering up it few times reboots itself.
>> But commonly FreeBSD doesn't reboot by it's own will.
>> The same hardware worked over a half a year under 8.0 stables without
>> such a problem. I just would like to understand from where this
>> problem comes up.
>> This machine doesn't contain any critical info so I'll wait for a bit.
>> Also I'd like to notice that recently I've tuned hdd's spindown exept
>> system hdd by atacontrol port, powerd and CPU frequency lowering in
>> idle, maybe something of this could cause this problem? And where
>> could I check this out?
>
> You might just want to go through your /etc/rc.d/*, crontabs,
> /etc/periodic/* and /etc/rc* to check for any commands that call
> shutdown(8) or reboot(8).
>
> Not really malicious but a rogue user that was once a staffer can plant
> a shutdown/reboot command in any one of the above matching files and
> have it run by at(1). This really sounds like the case here.
>
> 1). Check the above files. egrep -nr "(shutdown|reboot)" on those.
> 2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to
> unintelligible binaries that have been setuid=0(u+s) owner=root.
> 3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to
> another command or created a script containing shutdown(8) to another
> command that matches another system command or has replaced one.
>
> Thats just a few things to go on for now. Others may have more to add to it.
>
>
> Regards & Good luck,
>
One more thing I would like to add to this is audit(8). This can prove
to be an invaluable tool to find rogue binaries on the system and track
down exactly to where one is being executed. Since your looking at a
specific time-frame that this is happening it would be real easy for you
to track this back.
Regards again,
--
jhell,v
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C9B9F92.5030102>