From owner-freebsd-stable@FreeBSD.ORG Tue Apr 20 13:51:52 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C80F3106564A; Tue, 20 Apr 2010 13:51:52 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id 00A878FC22; Tue, 20 Apr 2010 13:51:51 +0000 (UTC) Received: by bwz8 with SMTP id 8so5567813bwz.3 for ; Tue, 20 Apr 2010 06:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=AnP+huBkOJHK0m9w+ovY662TEpMWEBmCZ+hrgxqnnHM=; b=eAB6TcoY8ommvWy6Y3wwNKyMSimT1vknnlIx09wvoEEB/zWiStVh6fA5BAIeik13fO 46eLRazGA99nQwggAKyZrDQxhguVk37cZRcktnoN4Wd0SgXPuuMOOzwniCnfRXfNjef1 /IW+6xTorKG+xNHT/5JEZ4oNWCQmNPs2OHqb8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=feoiSU5SajZbv0kzYYNji6I38RdDGthwtvFzeGYK6fAby41BDUztoghPRtC9jsH5nz c9T90p9W5wP48oaBK4zyUoZOoNIXxwOQdw0ZjtdwEMUfDDGi2H/In6IKx31/8jNjfD4P 36Skgb85gdI9vTTZO6MDv4SEI/SUO+XYjj1+s= MIME-Version: 1.0 Received: by 10.204.47.232 with HTTP; Tue, 20 Apr 2010 06:51:49 -0700 (PDT) In-Reply-To: <201004200748.09566.jhb@freebsd.org> References: <201004200748.09566.jhb@freebsd.org> Date: Tue, 20 Apr 2010 17:51:49 +0400 Received: by 10.204.21.18 with SMTP id h18mr2490059bkb.177.1271771510528; Tue, 20 Apr 2010 06:51:50 -0700 (PDT) Message-ID: From: pluknet To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, c0re , net@freebsd.org Subject: Re: FreeBSD 7.3, reboot after panic: double fault X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 13:51:52 -0000 On 20 April 2010 15:48, John Baldwin wrote: > On Tuesday 20 April 2010 2:53:16 am c0re wrote: >> Hello All! >> I've upgraded freebsd from 7.0 to 7.3 and all was good until I tryed to >> configure gre interface and use ipfw fwd. >> I'm actually does not know what was the point of failure in my >> configuration. >> >> [ some details snipped ] >> >> It worked about one week and then I made some configuration changes: >> added gre interface and 2 aliases: >> >> # cat /etc/rc.conf |grep >> ifconfig_xl0=3D"inet 192.168.0.10 =A0netmask 255.255.255.0" >> ifconfig_xl0_alias0=3D"192.168.0.11 netmask 255.255.255.255" >> ifconfig_xl0_alias1=3D"192.168.0.12 netmask 255.255.255.255" >> cloned_interfaces=3D"gre0" >> ifconfig_gre0=3D"inet 192.168.250.6 192.168.250.5 tunnel 192.168.0.12 >> 192.168.200.15 netmask 255.255.255.252 link1 up" >> >> and >> >> # cat /etc/rc.local >> #!/bin/sh >> ipfw add fwd 192.168.250.5 icmp from 192.168.0.11 to any out via xl0 >> ipfw add fwd 192.168.250.5 tcp from 192.168.0.11 443 to any out via xl0 >> ipfw add allow ip from any to any >> >> # ifconfig gre0 >> gre0: flags=3Db050 metric 0 m= tu >> 1476 >> =A0 =A0 =A0 =A0 tunnel inet 192.168.0.12 --> 192.168.200.15 >> =A0 =A0 =A0 =A0 inet 192.168.250.6 --> 192.168.250.5 netmask 0xfffffffc >> >> I shutted down gre interface to prevent requests via gre to buggy IP. >> >> The main idea of such configurations was: fwd all connections to https t= o >> 192.168.0.1 via gre interface. >> And also I made apache configurations to make it listen on 192.168.0.11 = too. >> >> And make some tests: ping 192.168.0.11 - was fine, goes via gre. Telnet = to >> 192.168.0.11 =A0443 was fine too. Then I tryed to make browser https >> connection to 192.168.0.11. Apache showed me certificate warning and I >> accepted, then in browser nothing happened, it was trying to open page. = But >> server got kernel panic at that moment. >> >> At first time I thought that it was some power failure, I tryed 2 more t= imes >> and got same behaviour. >> >> So https works without kernel panic via 192.168.0.10 address but kernel >> panics when I try do https via 192.168.0.11 address that source-forwarde= d >> via gre. > > Looks like the TCP output path got stuck in an infinite recursion loop un= til > it exhausted the kernel stack: > >> # cd /usr/obj/usr/src/sys/MYKERNEL >> # kgdb kernel.debug /var/crash/vmcore.2 >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you= are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. =A0Type "show warranty" for det= ails. >> This GDB was configured as "i386-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> Fatal double fault: >> eip =3D 0xc08e3ba3 >> esp =3D 0xccf6dfc4 >> ebp =3D 0xccf6e274 >> cpuid =3D 0; apic id =3D 00 >> panic: double fault >> cpuid =3D 0 >> Uptime: 7m14s >> Physical memory: 235 MB >> Dumping 35 MB: 20 4 >> >> Reading symbols from /boot/kernel/acpi.ko...Reading symbols from >> /boot/kernel/acpi.ko.symbols...done. >> done. >> Loaded symbols for /boot/kernel/acpi.ko >> Reading symbols from /boot/kernel/if_gre.ko...Reading symbols from >> /boot/kernel/if_gre.ko.symbols...done. >> done. >> Loaded symbols for /boot/kernel/if_gre.ko >> Reading symbols from /boot/kernel/linux.ko...Reading symbols from >> /boot/kernel/linux.ko.symbols...done. >> done. >> Loaded symbols for /boot/kernel/linux.ko >> #0 =A0doadump () at pcpu.h:196 >> 196 =A0 =A0 =A0 =A0 =A0 =A0 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (= td)); >> (kgdb) bt >> #0 =A0doadump () at pcpu.h:196 >> #1 =A00xc07f2857 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdow= n.c:418 >> #2 =A00xc07f2b29 in panic (fmt=3DVariable "fmt" is not available. >> ) at /usr/src/sys/kern/kern_shutdown.c:574 >> #3 =A00xc0a7ea2b in dblfault_handler () at /usr/src/sys/i386/i386/trap.c= :983 >> #4 =A00xc08e3ba3 in ipfw_chk (args=3D0xccf6e28c) at >> /usr/src/sys/netinet/ip_fw2.c:2465 >> #5 =A00xc08e6ce1 in ipfw_check_out (arg=3D0x0, m0=3D0xccf6e390, ifp=3D0x= c25c5c00, >> dir=3D2, inp=3D0xc28ba708) at /usr/src/sys/netinet/ip_fw_pfil.c:248 >> #6 =A00xc08a1968 in pfil_run_hooks (ph=3D0xc0c55240, mp=3D0xccf6e420, >> ifp=3D0xc25c5c00, dir=3D2, inp=3D0xc28ba708) at /usr/src/sys/net/pfil.c:= 78 >> #7 =A00xc08eb6f2 in ip_output (m=3D0xc2710b00, opt=3D0x0, ro=3D0xccf6e3f= 4, flags=3D0, >> imo=3D0x0, inp=3D0xc28ba708) at /usr/src/sys/netinet/ip_output.c:443 >> #8 =A00xc08f4016 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1134 >> #9 =A00xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_off= load.h:269 >> #10 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #11 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #12 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #13 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #14 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #15 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #16 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #17 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #18 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #19 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #20 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #21 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #22 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #23 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #24 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #25 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #26 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #27 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #28 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #29 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #30 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #31 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #32 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #33 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #34 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #35 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #36 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #37 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #38 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #39 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #40 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #41 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #42 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #43 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #44 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #45 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #46 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #47 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #48 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #49 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> ---Type to continue, or q to quit--- >> #50 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #51 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #52 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #53 0xc08f6d98 in tcp_mtudisc (inp=3D0xc28ba708, errno=3D0) at tcp_offlo= ad.h:269 >> #54 0xc08f4105 in tcp_output (tp=3D0xc25b2570) at >> /usr/src/sys/netinet/tcp_output.c:1195 >> #55 0xc08fdcf8 in tcp_usr_send (so=3D0xc2ac1820, flags=3D0, m=3D0xc270ed= 00, >> nam=3D0x0, control=3D0x0, td=3D0xc28e2d80) at tcp_offload.h:269 >> #56 0xc0850405 in sosend_generic (so=3D0xc2ac1820, addr=3D0x0, uio=3D0xc= 28766c0, >> top=3D0xc270ed00, control=3D0x0, flags=3D0, td=3D0xc28e2d80) at >> /usr/src/sys/kern/uipc_socket.c:1243 >> #57 0xc084bf7f in sosend (so=3D0xc2ac1820, addr=3D0x0, uio=3D0xc28766c0,= top=3D0x0, >> control=3D0x0, flags=3D0, td=3D0xc28e2d80) at /usr/src/sys/kern/uipc_soc= ket.c:1285 >> #58 0xc0833c5b in soo_write (fp=3D0xc28e84c0, uio=3D0xc28766c0, >> active_cred=3D0xc28e5900, flags=3D0, td=3D0xc28e2d80) at >> /usr/src/sys/kern/sys_socket.c:103 >> #59 0xc082d2e7 in dofilewrite (td=3D0xc28e2d80, fd=3D24, fp=3D0xc28e84c0= , >> auio=3D0xc28766c0, offset=3D-1, flags=3D0) at file.h:257 >> #60 0xc082d5c8 in kern_writev (td=3D0xc28e2d80, fd=3D24, auio=3D0xc28766= c0) at >> /usr/src/sys/kern/sys_generic.c:402 >> #61 0xc082d816 in writev (td=3D0xc28e2d80, uap=3D0xccf6fcfc) at >> /usr/src/sys/kern/sys_generic.c:388 >> #62 0xc0a7f2d5 in syscall (frame=3D0xccf6fd38) at >> /usr/src/sys/i386/i386/trap.c:1101 >> #63 0xc0a636a0 in Xint0x80_syscall () at >> /usr/src/sys/i386/i386/exception.s:262 >> #64 0x00000033 in ?? () >> Previous frame inner to this frame (corrupt stack?) >> (kgdb) >> (kgdb) quit > > tcp_output() calls tcp_mtudisc() if ip_output() returns EMSGSIZE: > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0case EMSGSIZE: > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * For some reason the int= erface we used initially > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * to send segments change= d to another or lowered > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * its MTU. > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * tcp_mtudisc() will find= out the new MTU and as > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * its last action, initia= te retransmission, so it > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * is important to not do = so here. > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * If TSO was active we ei= ther got an interface > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * without TSO capabilits = or TSO was turned off. > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Disable it for this con= nection as too and > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * immediatly retry with M= SS sized segments generated > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * by this function. > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (tso) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0tp->t_flag= s &=3D ~TF_TSO; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0tcp_mtudisc(tp->t_inpcb, 0= ); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return (0); > > But tcp_mtudisc() calls tcp_output(): > > =A0 =A0 =A0 =A0tcpstat.tcps_mturesent++; > =A0 =A0 =A0 =A0tp->t_rtttime =3D 0; > =A0 =A0 =A0 =A0tp->snd_nxt =3D tp->snd_una; > =A0 =A0 =A0 =A0tcp_free_sackholes(tp); > =A0 =A0 =A0 =A0tp->snd_recover =3D tp->snd_max; > =A0 =A0 =A0 =A0if (tp->t_flags & TF_SACK_PERMIT) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0EXIT_FASTRECOVERY(tp); > =A0 =A0 =A0 =A0tcp_output_send(tp); > =A0 =A0 =A0 =A0return (inp); > > I'm not sure why it's not able to figure out the MTU, perhaps folks on ne= t@ > can help. =A0However, it would seem that for the tcp_output() case, > tcp_mtudisc() should probably not call tcp_output_send(), but instead > tcp_output() should just loop back up to the top after calling tcp_mtudis= c() > and retry. > I'm afraid to be wrong but it looks similar to another report for 8.0-STABL= E (may it be a cross-major version regression somewhere around tcp_mtudisc()?= ): http://lists.freebsd.org/pipermail/freebsd-stable/2010-April/056063.html --=20 wbr, pluknet