Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 26 Jun 2006 10:42:32 -0400
From:      Mike Tancsa <mike@sentex.net>
To:        "Marc G. Fournier" <scrappy@hub.org>, freebsd-stable@freebsd.org
Subject:   Re: force panic of remote server ... possible?
Message-ID:  <6.2.3.4.0.20060626102033.02f2fcd8@64.7.153.2>
In-Reply-To: <20060626085321.T1114@ganymede.hub.org>
References:  <20060626085321.T1114@ganymede.hub.org>

next in thread | previous in thread | raw e-mail | index | archive | help
At 07:55 AM 26/06/2006, Marc G. Fournier wrote:

>For the server that I'm fighting with right now, where Dmitry 
>pointed out that it looks like a deadlock issue ... I have 
>dumpdev/savecore enabled, is there some way of forcing it to panic 
>when I know I actually have the deadlock, so that it will dump a core?
>
>DDB is a difficult option, since a keyboard isn't always attached to 
>the server when it boots ...

These are ugly quick hacks, but it might work for you... If the 
network still continues to function. you might be able to hack up a 
quick script to force a panic.  Hackup some kld (e.g. ichwd) with 
something like

# diff -u /usr/src/sys/dev/ichwd/ichwd.c.orig /usr/src/sys/dev/ichwd/ichwd.c
--- /usr/src/sys/dev/ichwd/ichwd.c.orig Mon Jun 26 09:50:33 2006
+++ /usr/src/sys/dev/ichwd/ichwd.c      Mon Jun 26 09:51:04 2006
@@ -225,6 +225,7 @@
         device_t ich = NULL;
         device_t dev;

+       panic("I played panicky idiot no 3 on the Poseidon Adventure");
         /* look for an ICH LPC interface bridge */
         for (id = ichwd_devices; id->desc != NULL; ++id)
                 if ((ich = pci_find_device(id->vendor, id->device)) != NULL)


Then run a script something like the one below.  Set target to be an 
ip that you control and is always up.  When you think your box has 
deadlocked, add a firewall rule on the target machine to block ICMP 
echos from the problem machine.  You might need to fiddle with 
max_tries to make it more aggressive.  If the target machine is on 
the local LAN you can make it a nice low value like 2 or 3.  Ideally, 
you would want to make a kld that would instead do the test for you, 
or you could perhaps hack up the software watchdog to call a panic 
for you. Dont know if that works or not as I have only used hardware watchdogs.


#!/bin/sh
timeout=5
no_resp_sleep=10
max_tries=25
normal_sleep=300
con_cnt=0
target=1.1.1.1


while true; do
     strings /boot/kernel/ichwd.ko > /dev/null   # try and make sure 
these binaries are cached
     strings /sbin/kldload > /dev/null # try and make sure these 
binaries are cached
     if /sbin/ping -c1 -t$timeout $target > /dev/null 2>&1; then
         no_resp=0
     else
         no_resp=$(($no_resp + 1))
     fi
     if [ $no_resp -gt $max_tries ]; then
             /sbin/kldload ichwd
     fi
     if [ $no_resp -gt 0 ]; then
         sleep $no_resp_sleep
     else
         sleep $normal_sleep
         if [ $con_cnt -lt 25 ]; then
             con_cnt=$(($con_cnt + 1))
         fi
     fi
done &


         ---Mike 




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6.2.3.4.0.20060626102033.02f2fcd8>