Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 20 Apr 2010 17:51:49 +0400
From:      pluknet <pluknet@gmail.com>
To:        John Baldwin <jhb@freebsd.org>
Cc:        freebsd-stable@freebsd.org, c0re <nr1c0re@gmail.com>, net@freebsd.org
Subject:   Re: FreeBSD 7.3, reboot after panic: double fault
Message-ID:  <r2oa31046fc1004200651mc8161796x355afcee15bf9580@mail.gmail.com>
In-Reply-To: <201004200748.09566.jhb@freebsd.org>
References:  <n2g44d4913e1004192353o89dd3437ue9a01c8d5e6b6de5@mail.gmail.com> <201004200748.09566.jhb@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 20 April 2010 15:48, John Baldwin <jhb@freebsd.org> wrote:
> On Tuesday 20 April 2010 2:53:16 am c0re wrote:
>> Hello All!
>> I've upgraded freebsd from 7.0 to 7.3 and all was good until I tryed to
>> configure gre interface and use ipfw fwd.
>> I'm actually does not know what was the point of failure in my
>> configuration.
>>
>> [ some details snipped ]
>>
>> It worked about one week and then I made some configuration changes:
>> added gre interface and 2 aliases:
>>
>> # cat /etc/rc.conf |grep
>> ifconfig_xl0=3D"inet 192.168.0.10 =A0netmask 255.255.255.0"
>> ifconfig_xl0_alias0=3D"192.168.0.11 netmask 255.255.255.255"
>> ifconfig_xl0_alias1=3D"192.168.0.12 netmask 255.255.255.255"
>> cloned_interfaces=3D"gre0"
>> ifconfig_gre0=3D"inet 192.168.250.6 192.168.250.5 tunnel 192.168.0.12
>> 192.168.200.15 netmask 255.255.255.252 link1 up"
>>
>> and
>>
>> # cat /etc/rc.local
>> #!/bin/sh
>> ipfw add fwd 192.168.250.5 icmp from 192.168.0.11 to any out via xl0
>> ipfw add fwd 192.168.250.5 tcp from 192.168.0.11 443 to any out via xl0
>> ipfw add allow ip from any to any
>>
>> # ifconfig gre0
>> gre0: flags=3Db050<POINTOPOINT,RUNNING,LINK0,LINK1,MULTICAST> metric 0 m=
tu
>> 1476
>> =A0 =A0 =A0 =A0 tunnel inet 192.168.0.12 --> 192.168.200.15
>> =A0 =A0 =A0 =A0 inet 192.168.250.6 --> 192.168.250.5 netmask 0xfffffffc
>>
>> I shutted down gre interface to prevent requests via gre to buggy IP.
>>
>> The main idea of such configurations was: fwd all connections to https t=
o
>> 192.168.0.1 via gre interface.
>> And also I made apache configurations to make it listen on 192.168.0.11 =
too.
>>
>> And make some tests: ping 192.168.0.11 - was fine, goes via gre. Telnet =
to
>> 192.168.0.11 =A0443 was fine too. Then I tryed to make browser https
>> connection to 192.168.0.11. Apache showed me certificate warning and I
>> accepted, then in browser nothing happened, it was trying to open page. =
But
>> server got kernel panic at that moment.
>>
>> At first time I thought that it was some power failure, I tryed 2 more t=
imes
>> and got same behaviour.
>>
>> So https works without kernel panic via 192.168.0.10 address but kernel
>> panics when I try do https via 192.168.0.11 address that source-forwarde=
d
>> via gre.
>
> Looks like the TCP output path got stuck in an infinite recursion loop un=
til
> it exhausted the kernel stack:
>
>> # cd /usr/obj/usr/src/sys/MYKERNEL
>> # kgdb kernel.debug /var/crash/vmcore.2
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you=
 are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. =A0Type "show warranty" for det=
ails.
>> This GDB was configured as "i386-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>> Fatal double fault:
>> eip =3D 0xc08e3ba3
>> esp =3D 0xccf6dfc4
>> ebp =3D 0xccf6e274
>> cpuid =3D 0; apic id =3D 00
>> panic: double fault
>> cpuid =3D 0
>> Uptime: 7m14s
>> Physical memory: 235 MB
>> Dumping 35 MB: 20 4
>>
>> Reading symbols from /boot/kernel/acpi.ko...Reading symbols from
>> /boot/kernel/acpi.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/acpi.ko
>> Reading symbols from /boot/kernel/if_gre.ko...Reading symbols from
>> /boot/kernel/if_gre.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/if_gre.ko
>> Reading symbols from /boot/kernel/linux.ko...Reading symbols from
>> /boot/kernel/linux.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/linux.ko
>> #0 =A0doadump () at pcpu.h:196
>> 196 =A0 =A0 =A0 =A0 =A0 =A0 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (=
td));
>> (kgdb) bt
>> #0 =A0doadump () at pcpu.h:196
>> #1 =A00xc07f2857 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdow=
n.c:418
>> #2 =A00xc07f2b29 in panic (fmt=3DVariable "fmt" is not available.
>> ) at /usr/src/sys/kern/kern_shutdown.c:574
>> #3 =A00xc0a7ea2b in dblfault_handler () at /usr/src/sys/i386/i386/trap.c=
:983
>> #4 =A00xc08e3ba3 in ipfw_chk (args=3D0xccf6e28c) at
>> /usr/src/sys/netinet/ip_fw2.c:2465
>> #5 =A00xc08e6ce1 in ipfw_check_out (arg=3D0x0, m0=3D0xccf6e390, ifp=3D0x=
c25c5c00,
>> dir=3D2, inp=3D0xc28ba708) at /usr/src/sys/netinet/ip_fw_pfil.c:248
>> #6 =A00xc08a1968 in pfil_run_hooks (ph=3D0xc0c55240, mp=3D0xccf6e420,
>> ifp=3D0xc25c5c00, dir=3D2, inp=3D0xc28ba708) at /usr/src/sys/net/pfil.c:=
78
>> #7 =A00xc08eb6f2 in ip_output (m=3D0xc2710b00, opt=3D0x0, ro=3D0xccf6e3f=
4, flags=3D0,
>> imo=3D0x0, inp=3D0xc28ba708) at /usr/src/sys/netinet/ip_output.c:443
>> #8 =A00xc08f4016 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1134
>> #9 =A00xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_off=
load.h:269
>> #10 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #11 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #12 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #13 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #14 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #15 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #16 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #17 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #18 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #19 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #20 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #21 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #22 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #23 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #24 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #25 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #26 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #27 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #28 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #29 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #30 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #31 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #32 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #33 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #34 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #35 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #36 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #37 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #38 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #39 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #40 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #41 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #42 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #43 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #44 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #45 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #46 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #47 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #48 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #49 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> ---Type <return> to continue, or q <return> to quit---
>> #50 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #51 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #52 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #53 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo=
ad.h:269
>> #54 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at
>> /usr/src/sys/netinet/tcp_output.c:1195
>> #55 0xc08fdcf8 in tcp_usr_send (so=3D0xc2ac1820, flags=3D0, m=3D0xc270ed=
00,
>> nam=3D0x0, control=3D0x0, td=3D0xc28e2d80) at tcp_offload.h:269
>> #56 0xc0850405 in sosend_generic (so=3D0xc2ac1820, addr=3D0x0, uio=3D0xc=
28766c0,
>> top=3D0xc270ed00, control=3D0x0, flags=3D0, td=3D0xc28e2d80) at
>> /usr/src/sys/kern/uipc_socket.c:1243
>> #57 0xc084bf7f in sosend (so=3D0xc2ac1820, addr=3D0x0, uio=3D0xc28766c0,=
 top=3D0x0,
>> control=3D0x0, flags=3D0, td=3D0xc28e2d80) at /usr/src/sys/kern/uipc_soc=
ket.c:1285
>> #58 0xc0833c5b in soo_write (fp=3D0xc28e84c0, uio=3D0xc28766c0,
>> active_cred=3D0xc28e5900, flags=3D0, td=3D0xc28e2d80) at
>> /usr/src/sys/kern/sys_socket.c:103
>> #59 0xc082d2e7 in dofilewrite (td=3D0xc28e2d80, fd=3D24, fp=3D0xc28e84c0=
,
>> auio=3D0xc28766c0, offset=3D-1, flags=3D0) at file.h:257
>> #60 0xc082d5c8 in kern_writev (td=3D0xc28e2d80, fd=3D24, auio=3D0xc28766=
c0) at
>> /usr/src/sys/kern/sys_generic.c:402
>> #61 0xc082d816 in writev (td=3D0xc28e2d80, uap=3D0xccf6fcfc) at
>> /usr/src/sys/kern/sys_generic.c:388
>> #62 0xc0a7f2d5 in syscall (frame=3D0xccf6fd38) at
>> /usr/src/sys/i386/i386/trap.c:1101
>> #63 0xc0a636a0 in Xint0x80_syscall () at
>> /usr/src/sys/i386/i386/exception.s:262
>> #64 0x00000033 in ?? ()
>> Previous frame inner to this frame (corrupt stack?)
>> (kgdb)
>> (kgdb) quit
>
> tcp_output() calls tcp_mtudisc() if ip_output() returns EMSGSIZE:
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0case EMSGSIZE:
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/*
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * For some reason the int=
erface we used initially
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * to send segments change=
d to another or lowered
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * its MTU.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 *
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * tcp_mtudisc() will find=
 out the new MTU and as
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * its last action, initia=
te retransmission, so it
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * is important to not do =
so here.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 *
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * If TSO was active we ei=
ther got an interface
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * without TSO capabilits =
or TSO was turned off.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Disable it for this con=
nection as too and
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * immediatly retry with M=
SS sized segments generated
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * by this function.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (tso)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0tp->t_flag=
s &=3D ~TF_TSO;
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0tcp_mtudisc(tp->t_inpcb, 0=
);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return (0);
>
> But tcp_mtudisc() calls tcp_output():
>
> =A0 =A0 =A0 =A0tcpstat.tcps_mturesent++;
> =A0 =A0 =A0 =A0tp->t_rtttime =3D 0;
> =A0 =A0 =A0 =A0tp->snd_nxt =3D tp->snd_una;
> =A0 =A0 =A0 =A0tcp_free_sackholes(tp);
> =A0 =A0 =A0 =A0tp->snd_recover =3D tp->snd_max;
> =A0 =A0 =A0 =A0if (tp->t_flags & TF_SACK_PERMIT)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0EXIT_FASTRECOVERY(tp);
> =A0 =A0 =A0 =A0tcp_output_send(tp);
> =A0 =A0 =A0 =A0return (inp);
>
> I'm not sure why it's not able to figure out the MTU, perhaps folks on ne=
t@
> can help. =A0However, it would seem that for the tcp_output() case,
> tcp_mtudisc() should probably not call tcp_output_send(), but instead
> tcp_output() should just loop back up to the top after calling tcp_mtudis=
c()
> and retry.
>

I'm afraid to be wrong but it looks similar to another report for 8.0-STABL=
E
(may it be a cross-major version regression somewhere around tcp_mtudisc()?=
):

http://lists.freebsd.org/pipermail/freebsd-stable/2010-April/056063.html

--=20
wbr,
pluknet



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?r2oa31046fc1004200651mc8161796x355afcee15bf9580>