From owner-freebsd-net@FreeBSD.ORG Wed Jan 1 13:11:01 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 46ED698E; Wed, 1 Jan 2014 13:11:01 +0000 (UTC) Received: from mail-lb0-x235.google.com (mail-lb0-x235.google.com [IPv6:2a00:1450:4010:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2AF291539; Wed, 1 Jan 2014 13:10:59 +0000 (UTC) Received: by mail-lb0-f181.google.com with SMTP id q8so6674998lbi.40 for ; Wed, 01 Jan 2014 05:10:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=1cnhelaWIueBGMomxFQZW3GDRmcNbfyV5SHGCSOZIJY=; b=KqzucI8LxJIUqDB84rN+NVX6TjSmXKvMIr7viYxvs0rQEJQsjpn03b+3TRWd4JPNAb UyuvCtp5RdHNzzJLT6SeGpW3yv6CeeUtPycDMGTq5PrqlsVDWd974CAOl2E7rj+NQa9g GVDMKlMZ4Hlernv1qPcZPYAEeffVhYamM5rCBVW6SEO/M+qd2ylrwuVpylZi4sWePKeo zIwwJ4K3/j5zVfJEWt70dq1fdI0Z1ZcpMEdJdnfA2wYVw1KrnIFMF12huJqd5MjAFyoH 5wspCTcA7AYmwI5NaPMwVy7gq2cOFJGPIRNGAjzSk0mCmIrKGqBUlsUYPYyHxLoU8gyb ESJQ== MIME-Version: 1.0 X-Received: by 10.112.17.39 with SMTP id l7mr701161lbd.51.1388581858051; Wed, 01 Jan 2014 05:10:58 -0800 (PST) Received: by 10.114.242.33 with HTTP; Wed, 1 Jan 2014 05:10:57 -0800 (PST) In-Reply-To: <201312221310.rBMDA0KH022980@freefall.freebsd.org> References: <201312221304.rBMD4q38060416@oldred.freebsd.org> <201312221310.rBMDA0KH022980@freefall.freebsd.org> Date: Wed, 1 Jan 2014 13:10:57 +0000 Message-ID: Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) From: Nikolay Denev To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jan 2014 13:11:01 -0000 On Sun, Dec 22, 2013 at 1:10 PM, wrote: > Thank you very much for your problem report. > It has the internal identification `misc/185092'. > The individual assigned to look at your > report is: freebsd-bugs. > > You can access the state of your problem report at any time > via this link: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092 > > >Category: misc > >Responsible: freebsd-bugs > >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) > >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 > I'm trying to understand exactly what is happening here, and examining a core dump with kgdb I'm getting some output that confuses me : (kgdb) bt #0 doadump (textdump=3D-1011569920) at pcpu.h:233 #1 0xc06069b2 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xc0606d0e in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xc06de639 in rtfree (rt=3D) at /usr/src/sys/net/route.c:464 #4 0xc06e188d in route_output (m=3D) at /usr/src/sys/net/rtsock.c:951 #5 0xc06de18f in raw_usend (so=3D, flags=3D0, m=3D, nam=3D0x0, control=3D, td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D, top=3D, control=3D0x0, flags=3D, td=3D) at /usr/src/sys/kern/uipc_socket.c:1271 #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, to= p=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000) at /usr/src/sys/kern/uipc_socket.c:1315 #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000) at /usr/src/sys/kern/sys_socket.c:103 #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818, auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303 #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D) at /usr/src/sys/kern/sys_generic.c:467 #11 0xc064c4bc in sys_write (td=3D, uap=3D) at /usr/src/sys/kern/sys_generic.c:382 #12 0xc08614d3 in syscall (frame=3D) at subr_syscall.c:134 #13 0xc084cca1 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270 #14 0x281975b7 in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) fr 3 #3 0xc06de639 in rtfree (rt=3D) at /usr/src/sys/net/route.c:464 464 panic("rtfree 2"); (kgdb) print *rt $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bit = =3D 0, rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D { rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x1030000
, rn_Dupedkey =3D 0x0}, rn_node =3D { rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}}, {rn_= mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -94 '?', rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0 ""= , rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880}, rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R =3D 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060, rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt_rm= x =3D {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0}, rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags = =3D 0, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}} rn_Key with value of =93shutdown_post_sync=94 ? It=92s visible also in the raw_usend() frame: (kgdb) fr 5 #5 0xc06de18f in raw_usend (so=3D, flags=3D0, m=3D, nam=3D0x0, control=3D, td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 238 return ((*so->so_proto->pr_output)(m, so)); (kgdb) print *m $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data =3D = 0x0, mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0}, M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first =3D = 0x4}, len =3D -1012745856, flowid =3D 3282388448, csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos =3D 1= 17 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o', l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_pos= t_sy", sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103, 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr =3D {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D { eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, 0}, thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454}, ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_bu= f =3D 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0, ext_arg1 =3D 0x0, ext_arg2 =3D 0x0}, MH_databuf =3D '\0' , "file", '\0' , "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\0= 00\000\000\004\000\000\000\000\000\000\00000Y?", '\0' , "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\0= 00\000\000\000\000???", '\0' }}, M_databuf =3D "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync", '\0' , "file", '\0' , "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\0= 00\000\000\004\000\000\000\000\000\000\00000Y?", '\0' , "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\0= 00\000\000\000\000???", '\0' }} This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfree patch, which I thought earlier might be the cause of the panics I'm seeing). The machine is Soekris Net5501-70 with this kernel config : cpu I586_CPU cpu I686_CPU ident MARS options CPU_GEODE options CPU_SOEKRIS options HZ=3D2000 options DEVICE_POLLING options BPF_JITTER makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options TCP_OFFLOAD # TCP offload options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_DIRHASH # Improve performance on big directories options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options CAPABILITY_MODE # Capsicum capability mode options CAPABILITIES # Capsicum capabilities options PROCDESC # Support for process descriptors options INCLUDE_CONFIG_FILE # Include this file in kernel # Debugging support. Always need this: options KDB # Enable kernel debugger support. options KDB_TRACE # Print a stack trace for a panic. options KDB_UNATTENDED options TEXTDUMP_PREFERRED options TEXTDUMP_VERBOSE device pci device ata # Legacy ATA/SATA controllers options ATA_STATIC_ID # Static device numbering # ATA/SCSI peripherals device scbus # SCSI bus (required for ATA/SCSI) device da # Direct Access (disks) device pass # Passthrough device (direct ATA/SCSI access) # Add suspend/resume support for the i8254. device pmtimer # Serial (COM) ports device uart # Generic UART driver device miibus # MII bus support device vr # VIA Rhine, Rhine II # Wireless NIC cards device wlan # 802.11 support options IEEE80211_DEBUG # enable debug msgs options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's options IEEE80211_SUPPORT_MESH # enable 802.11s draft support device wlan_wep # 802.11 WEP support device wlan_ccmp # 802.11 CCMP support device wlan_tkip # 802.11 TKIP support device wlan_amrr # AMRR transmit rate control algorithm device ath # Atheros NICs device ath_pci # Atheros pci/cardbus glue device ath_hal # pci/cardbus chip support options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later device ath_rate_sample # SampleRate tx rate control for ath # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device gre device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device if_bridge options VIMAGE options ROUTETABLES=3D8 options RADIX_MPATH options SW_WATCHDOG device crypto device cryptodev device glxsb options BOOTVERBOSE=3D1 #device pf #device pflog #device pfsync device carp device enc device lagg device epair #options ALTQ #options ALTQ_CBQ #options ALTQ_RED #options ALTQ_RIO #options ALTQ_HFSC #options ALTQ_PRIQ options IPFIREWALL options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_NAT options LIBALIAS options IPDIVERT options DUMMYNET device bpf # Berkeley packet filter # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device umass # Disks/Mass storage - Requires scbus and da Also src.conf and make.conf : root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf WITHOUT_ACCT=3Dyes WITHOUT_ACPI=3Dyes WITHOUT_AMD=3Dyes WITHOUT_APM=3Dyes WITHOUT_ASSERT_DEBUG=3Dyes WITHOUT_AT=3Dyes WITHOUT_ATF=3Dyes WITHOUT_ATM=3Dyes WITHOUT_AUDIT=3Dyes WITHOUT_BLUETOOTH=3Dyes WITHOUT_CALENDAR=3Dyes WITHOUT_CDDL=3Dyes WITHOUT_CTM=3Dyes WITHOUT_DICT=3Dyes WITHOUT_FLOPPY=3Dyes WITHOUT_GAMES=3Dyes WITHOUT_HTML=3Dyes WITHOUT_INFO=3Dyes WITHOUT_IPFILTER=3Dyes WITHOUT_IPX=3Dyes #WITHOUT_KERNEL_SYMBOLS=3Dyes WITHOUT_LEGACY_CONSOLE=3Dyes WITHOUT_LOCALES=3Dyes WITHOUT_LPR=3Dyes WITHOUT_MAIL=3Dyes WITHOUT_NDIS=3Dyes WITHOUT_QUOTAS=3Dyes WITHOUT_ROUTED=3Dyes WITHOUT_SENDMAIL=3Dyes WITH_SVN=3Dyes WITHOUT_ZFS=3Dyes root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf CFLAGS=3D-O2 COPTFLAGS=3D -O -pipe CPUTYPE=3Dgeode KERNCONF=3DMARS NO_MODULES=3Dyes BOOTWAIT=3D0 DOC_LANG=3Den_US.ISO8859-1 --Nikolay