Date: Wed, 1 Jan 2014 13:21:13 +0000 From: Nikolay Denev <ndenev@gmail.com> To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) Message-ID: <CA%2BP_MZEqn4iK%2BWC-ouVNFEhMrq2tpH91qgcT5zJyaUnX%2BkVn1g@mail.gmail.com> In-Reply-To: <CA%2BP_MZHrGB0OdddCmhpOptA-sBUHOERCdjSfUFh20pJOQsZ7Kw@mail.gmail.com> References: <201312221304.rBMD4q38060416@oldred.freebsd.org> <201312221310.rBMDA0KH022980@freefall.freebsd.org> <CA%2BP_MZHrGB0OdddCmhpOptA-sBUHOERCdjSfUFh20pJOQsZ7Kw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev@gmail.com> wrote: > On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit@freebsd.org> wrote= : > >> Thank you very much for your problem report. >> It has the internal identification `misc/185092'. >> The individual assigned to look at your >> report is: freebsd-bugs. >> >> You can access the state of your problem report at any time >> via this link: >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092 >> >> >Category: misc >> >Responsible: freebsd-bugs >> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) >> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 >> > > I'm trying to understand exactly what is happening here, and examining a > core dump with kgdb I'm getting some output that confuses me : > > (kgdb) bt > #0 doadump (textdump=3D-1011569920) at pcpu.h:233 > #1 0xc06069b2 in kern_reboot (howto=3D260) at > /usr/src/sys/kern/kern_shutdown.c:447 > #2 0xc0606d0e in panic (fmt=3D<value optimized out>) at > /usr/src/sys/kern/kern_shutdown.c:754 > #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at > /usr/src/sys/net/route.c:464 > #4 0xc06e188d in route_output (m=3D<value optimized out>) at > /usr/src/sys/net/rtsock.c:951 > #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<= value > optimized out>, nam=3D0x0, control=3D<value optimized out>, > td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 > #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value optimized > out>, top=3D<value optimized out>, control=3D0x0, > flags=3D<value optimized out>, td=3D<value optimized out>) at > /usr/src/sys/kern/uipc_socket.c:1271 > #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, > top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000) > at /usr/src/sys/kern/uipc_socket.c:1315 > #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, > active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000) > at /usr/src/sys/kern/sys_socket.c:103 > #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818, > auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303 > #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value opt= imized > out>) at /usr/src/sys/kern/sys_generic.c:467 > #11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<value > optimized out>) at /usr/src/sys/kern/sys_generic.c:382 > #12 0xc08614d3 in syscall (frame=3D<value optimized out>) at > subr_syscall.c:134 > #13 0xc084cca1 in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:270 > #14 0x281975b7 in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) fr 3 > #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at > /usr/src/sys/net/route.c:464 > 464 panic("rtfree 2"); > (kgdb) print *rt > $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bi= t =3D 0, > rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D { > rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x10300= 00 > <Address 0x1030000 out of bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D { > rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}}, > {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -9= 4 '?', > rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0 = "", > rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880}, > rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R = =3D > 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060, > rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt_= rmx =3D > {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0}, > rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags= =3D 0, > lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}} > > > > rn_Key with value of =93shutdown_post_sync=94 ? > > It=92s visible also in the raw_usend() frame: > > (kgdb) fr 5 > #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<= value > optimized out>, nam=3D0x0, control=3D<value optimized out>, > td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 > 238 return ((*so->so_proto->pr_output)(m, so)); > (kgdb) print *m > $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data = =3D 0x0, > mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0}, > M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first = =3D 0x4}, len =3D > -1012745856, flowid =3D 3282388448, > csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos =3D= 117 > 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o', > l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_p= ost_sy", > sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103, > 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr = =3D > {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D { > eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, 0= }, > thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454}, > ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_= buf =3D > 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0, > ext_arg1 =3D 0x0, ext_arg2 =3D 0x0}, > MH_databuf =3D '\0' <repeats 56 times>, "file", '\0' <repeats 20 > times>, > "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000= \000\000\000\004\000\000\000\000\000\000\00000Y?", > '\0' <repeats 12 times>, > "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??= \000\000\000\000\000???", > '\0' <repeats 23 times>}}, > M_databuf =3D > "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync", > '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>, > "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000= \000\000\000\004\000\000\000\000\000\000\00000Y?", > '\0' <repeats 12 times>, > "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??= \000\000\000\000\000???", > '\0' <repeats 23 times>}} > > > This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfre= e > patch, which I thought earlier might be the cause of the panics I'm > seeing). > > The machine is Soekris Net5501-70 with this kernel config : > > cpu I586_CPU > cpu I686_CPU > ident MARS > options CPU_GEODE > options CPU_SOEKRIS > > options HZ=3D2000 > options DEVICE_POLLING > options BPF_JITTER > > makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols > > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options INET6 # IPv6 communications protocols > options TCP_OFFLOAD # TCP offload > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_DIRHASH # Improve performance on big directories > options PROCFS # Process filesystem (requires PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 > options COMPAT_FREEBSD5 # Compatible with FreeBSD5 > options COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7 > options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions > options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being interspersed= . > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) > options CAPABILITY_MODE # Capsicum capability mode > options CAPABILITIES # Capsicum capabilities > options PROCDESC # Support for process descriptors > options INCLUDE_CONFIG_FILE # Include this file in kernel > > # Debugging support. Always need this: > options KDB # Enable kernel debugger support. > options KDB_TRACE # Print a stack trace for a panic. > options KDB_UNATTENDED > > options TEXTDUMP_PREFERRED > options TEXTDUMP_VERBOSE > > device pci > device ata # Legacy ATA/SATA controllers > options ATA_STATIC_ID # Static device numbering > > # ATA/SCSI peripherals > device scbus # SCSI bus (required for ATA/SCSI) > device da # Direct Access (disks) > device pass # Passthrough device (direct ATA/SCSI access) > > # Add suspend/resume support for the i8254. > device pmtimer > > # Serial (COM) ports > device uart # Generic UART driver > > device miibus # MII bus support > device vr # VIA Rhine, Rhine II > > # Wireless NIC cards > device wlan # 802.11 support > options IEEE80211_DEBUG # enable debug msgs > options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's > options IEEE80211_SUPPORT_MESH # enable 802.11s draft support > device wlan_wep # 802.11 WEP support > device wlan_ccmp # 802.11 CCMP support > device wlan_tkip # 802.11 TKIP support > device wlan_amrr # AMRR transmit rate control algorithm > device ath # Atheros NICs > device ath_pci # Atheros pci/cardbus glue > device ath_hal # pci/cardbus chip support > options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors > options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation > options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later > device ath_rate_sample # SampleRate tx rate control for ath > > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device vlan # 802.1Q VLAN support > device tun # Packet tunnel. > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device gre > device faith # IPv6-to-IPv4 relaying (translation) > device firmware # firmware assist module > device if_bridge > > options VIMAGE > options ROUTETABLES=3D8 > options RADIX_MPATH > > options SW_WATCHDOG > > device crypto > device cryptodev > device glxsb > > options BOOTVERBOSE=3D1 > > #device pf > #device pflog > #device pfsync > device carp > device enc > device lagg > device epair > > #options ALTQ > #options ALTQ_CBQ > #options ALTQ_RED > #options ALTQ_RIO > #options ALTQ_HFSC > #options ALTQ_PRIQ > > options IPFIREWALL > options IPFIREWALL_DEFAULT_TO_ACCEPT > options IPFIREWALL_NAT > options LIBALIAS > options IPDIVERT > options DUMMYNET > > device bpf # Berkeley packet filter > > # USB support > options USB_DEBUG # enable debug msgs > device uhci # UHCI PCI->USB interface > device ohci # OHCI PCI->USB interface > device ehci # EHCI PCI->USB interface (USB 2.0) > device usb # USB Bus (required) > device umass # Disks/Mass storage - Requires scbus and da > > > Also src.conf and make.conf : > > root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf > WITHOUT_ACCT=3Dyes > WITHOUT_ACPI=3Dyes > WITHOUT_AMD=3Dyes > WITHOUT_APM=3Dyes > WITHOUT_ASSERT_DEBUG=3Dyes > WITHOUT_AT=3Dyes > WITHOUT_ATF=3Dyes > WITHOUT_ATM=3Dyes > WITHOUT_AUDIT=3Dyes > WITHOUT_BLUETOOTH=3Dyes > WITHOUT_CALENDAR=3Dyes > WITHOUT_CDDL=3Dyes > WITHOUT_CTM=3Dyes > WITHOUT_DICT=3Dyes > WITHOUT_FLOPPY=3Dyes > WITHOUT_GAMES=3Dyes > WITHOUT_HTML=3Dyes > WITHOUT_INFO=3Dyes > WITHOUT_IPFILTER=3Dyes > WITHOUT_IPX=3Dyes > #WITHOUT_KERNEL_SYMBOLS=3Dyes > WITHOUT_LEGACY_CONSOLE=3Dyes > WITHOUT_LOCALES=3Dyes > WITHOUT_LPR=3Dyes > WITHOUT_MAIL=3Dyes > WITHOUT_NDIS=3Dyes > WITHOUT_QUOTAS=3Dyes > WITHOUT_ROUTED=3Dyes > WITHOUT_SENDMAIL=3Dyes > WITH_SVN=3Dyes > WITHOUT_ZFS=3Dyes > > root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf > CFLAGS=3D-O2 > COPTFLAGS=3D -O -pipe > CPUTYPE=3Dgeode > KERNCONF=3DMARS > NO_MODULES=3Dyes > BOOTWAIT=3D0 > DOC_LANG=3Den_US.ISO8859-1 > > > > --Nikolay > > Also, originally I thought that the panic is when a multi path route is being deleted, however again from the coredump it seems that the panic happens when openvpn deletes the host route it installs for the remote openvpn server pointed to the default gw (before openvpn installs the new default gw pointing to the vpn tunnel) : (kgdb) fr 12 (kgdb) x/12sb td->td_proc->p_args 0xc4269780: "\001" 0xc4269782: "" 0xc4269783: "" 0xc4269784: "B" 0xc4269786: "" 0xc4269787: "" 0xc4269788: "/sbin/route" 0xc4269794: "delete" 0xc426979b: "-net" 0xc42697a0: "78.90.222.xxx" 0xc42697ad: "10.255.255.0" 0xc42697ba: "255.255.255.255" (kgdb) I'm trying to reproduce this on a VirtualBox instance now, however so far no luck (no OpenVPN running, just adding and removing routes). --Nikolay
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BP_MZEqn4iK%2BWC-ouVNFEhMrq2tpH91qgcT5zJyaUnX%2BkVn1g>