Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 20 Apr 2010 07:48:09 -0400
From:      John Baldwin <jhb@freebsd.org>
To:        freebsd-stable@freebsd.org
Cc:        c0re <nr1c0re@gmail.com>, net@freebsd.org
Subject:   Re: FreeBSD 7.3, reboot after panic: double fault
Message-ID:  <201004200748.09566.jhb@freebsd.org>
In-Reply-To: <n2g44d4913e1004192353o89dd3437ue9a01c8d5e6b6de5@mail.gmail.com>
References:  <n2g44d4913e1004192353o89dd3437ue9a01c8d5e6b6de5@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday 20 April 2010 2:53:16 am c0re wrote:
> Hello All!
> I've upgraded freebsd from 7.0 to 7.3 and all was good until I tryed to
> configure gre interface and use ipfw fwd.
> I'm actually does not know what was the point of failure in my
> configuration.
>
> [ some details snipped ]
> 
> It worked about one week and then I made some configuration changes:
> added gre interface and 2 aliases:
> 
> # cat /etc/rc.conf |grep
> ifconfig_xl0="inet 192.168.0.10  netmask 255.255.255.0"
> ifconfig_xl0_alias0="192.168.0.11 netmask 255.255.255.255"
> ifconfig_xl0_alias1="192.168.0.12 netmask 255.255.255.255"
> cloned_interfaces="gre0"
> ifconfig_gre0="inet 192.168.250.6 192.168.250.5 tunnel 192.168.0.12
> 192.168.200.15 netmask 255.255.255.252 link1 up"
> 
> and
> 
> # cat /etc/rc.local
> #!/bin/sh
> ipfw add fwd 192.168.250.5 icmp from 192.168.0.11 to any out via xl0
> ipfw add fwd 192.168.250.5 tcp from 192.168.0.11 443 to any out via xl0
> ipfw add allow ip from any to any
> 
> # ifconfig gre0
> gre0: flags=b050<POINTOPOINT,RUNNING,LINK0,LINK1,MULTICAST> metric 0 mtu
> 1476
>         tunnel inet 192.168.0.12 --> 192.168.200.15
>         inet 192.168.250.6 --> 192.168.250.5 netmask 0xfffffffc
> 
> I shutted down gre interface to prevent requests via gre to buggy IP.
> 
> The main idea of such configurations was: fwd all connections to https to
> 192.168.0.1 via gre interface.
> And also I made apache configurations to make it listen on 192.168.0.11 too.
> 
> And make some tests: ping 192.168.0.11 - was fine, goes via gre. Telnet to
> 192.168.0.11  443 was fine too. Then I tryed to make browser https
> connection to 192.168.0.11. Apache showed me certificate warning and I
> accepted, then in browser nothing happened, it was trying to open page. But
> server got kernel panic at that moment.
> 
> At first time I thought that it was some power failure, I tryed 2 more times
> and got same behaviour.
> 
> So https works without kernel panic via 192.168.0.10 address but kernel
> panics when I try do https via 192.168.0.11 address that source-forwarded
> via gre.

Looks like the TCP output path got stuck in an infinite recursion loop until 
it exhausted the kernel stack:

> # cd /usr/obj/usr/src/sys/MYKERNEL
> # kgdb kernel.debug /var/crash/vmcore.2
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> 
> Fatal double fault:
> eip = 0xc08e3ba3
> esp = 0xccf6dfc4
> ebp = 0xccf6e274
> cpuid = 0; apic id = 00
> panic: double fault
> cpuid = 0
> Uptime: 7m14s
> Physical memory: 235 MB
> Dumping 35 MB: 20 4
> 
> Reading symbols from /boot/kernel/acpi.ko...Reading symbols from
> /boot/kernel/acpi.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/acpi.ko
> Reading symbols from /boot/kernel/if_gre.ko...Reading symbols from
> /boot/kernel/if_gre.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/if_gre.ko
> Reading symbols from /boot/kernel/linux.ko...Reading symbols from
> /boot/kernel/linux.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/linux.ko
> #0  doadump () at pcpu.h:196
> 196             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
> (kgdb) bt
> #0  doadump () at pcpu.h:196
> #1  0xc07f2857 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
> #2  0xc07f2b29 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:574
> #3  0xc0a7ea2b in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:983
> #4  0xc08e3ba3 in ipfw_chk (args=0xccf6e28c) at
> /usr/src/sys/netinet/ip_fw2.c:2465
> #5  0xc08e6ce1 in ipfw_check_out (arg=0x0, m0=0xccf6e390, ifp=0xc25c5c00,
> dir=2, inp=0xc28ba708) at /usr/src/sys/netinet/ip_fw_pfil.c:248
> #6  0xc08a1968 in pfil_run_hooks (ph=0xc0c55240, mp=0xccf6e420,
> ifp=0xc25c5c00, dir=2, inp=0xc28ba708) at /usr/src/sys/net/pfil.c:78
> #7  0xc08eb6f2 in ip_output (m=0xc2710b00, opt=0x0, ro=0xccf6e3f4, flags=0,
> imo=0x0, inp=0xc28ba708) at /usr/src/sys/netinet/ip_output.c:443
> #8  0xc08f4016 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1134
> #9  0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #10 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #11 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #12 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #13 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #14 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #15 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #16 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #17 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #18 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #19 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #20 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #21 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #22 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #23 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #24 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #25 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #26 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #27 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #28 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #29 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #30 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #31 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #32 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #33 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #34 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #35 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #36 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #37 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #38 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #39 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #40 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #41 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #42 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #43 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #44 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #45 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #46 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #47 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #48 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #49 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> ---Type <return> to continue, or q <return> to quit---
> #50 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #51 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #52 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #53 0xc08f6d98 in tcp_mtudisc (inp=0xc28ba708, errno=0) at tcp_offload.h:269
> #54 0xc08f4105 in tcp_output (tp=0xc25b2570) at
> /usr/src/sys/netinet/tcp_output.c:1195
> #55 0xc08fdcf8 in tcp_usr_send (so=0xc2ac1820, flags=0, m=0xc270ed00,
> nam=0x0, control=0x0, td=0xc28e2d80) at tcp_offload.h:269
> #56 0xc0850405 in sosend_generic (so=0xc2ac1820, addr=0x0, uio=0xc28766c0,
> top=0xc270ed00, control=0x0, flags=0, td=0xc28e2d80) at
> /usr/src/sys/kern/uipc_socket.c:1243
> #57 0xc084bf7f in sosend (so=0xc2ac1820, addr=0x0, uio=0xc28766c0, top=0x0,
> control=0x0, flags=0, td=0xc28e2d80) at /usr/src/sys/kern/uipc_socket.c:1285
> #58 0xc0833c5b in soo_write (fp=0xc28e84c0, uio=0xc28766c0,
> active_cred=0xc28e5900, flags=0, td=0xc28e2d80) at
> /usr/src/sys/kern/sys_socket.c:103
> #59 0xc082d2e7 in dofilewrite (td=0xc28e2d80, fd=24, fp=0xc28e84c0,
> auio=0xc28766c0, offset=-1, flags=0) at file.h:257
> #60 0xc082d5c8 in kern_writev (td=0xc28e2d80, fd=24, auio=0xc28766c0) at
> /usr/src/sys/kern/sys_generic.c:402
> #61 0xc082d816 in writev (td=0xc28e2d80, uap=0xccf6fcfc) at
> /usr/src/sys/kern/sys_generic.c:388
> #62 0xc0a7f2d5 in syscall (frame=0xccf6fd38) at
> /usr/src/sys/i386/i386/trap.c:1101
> #63 0xc0a636a0 in Xint0x80_syscall () at
> /usr/src/sys/i386/i386/exception.s:262
> #64 0x00000033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)
> (kgdb) quit

tcp_output() calls tcp_mtudisc() if ip_output() returns EMSGSIZE:

		case EMSGSIZE:
			/*
			 * For some reason the interface we used initially
			 * to send segments changed to another or lowered
			 * its MTU.
			 *
			 * tcp_mtudisc() will find out the new MTU and as
			 * its last action, initiate retransmission, so it
			 * is important to not do so here.
			 *
			 * If TSO was active we either got an interface
			 * without TSO capabilits or TSO was turned off.
			 * Disable it for this connection as too and
			 * immediatly retry with MSS sized segments generated
			 * by this function.
			 */
			if (tso)
				tp->t_flags &= ~TF_TSO;
			tcp_mtudisc(tp->t_inpcb, 0);
			return (0);

But tcp_mtudisc() calls tcp_output():

	tcpstat.tcps_mturesent++;
	tp->t_rtttime = 0;
	tp->snd_nxt = tp->snd_una;
	tcp_free_sackholes(tp);
	tp->snd_recover = tp->snd_max;
	if (tp->t_flags & TF_SACK_PERMIT)
		EXIT_FASTRECOVERY(tp);
	tcp_output_send(tp);
	return (inp);

I'm not sure why it's not able to figure out the MTU, perhaps folks on net@ 
can help.  However, it would seem that for the tcp_output() case, 
tcp_mtudisc() should probably not call tcp_output_send(), but instead 
tcp_output() should just loop back up to the top after calling tcp_mtudisc() 
and retry.

-- 
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201004200748.09566.jhb>