Date: Thu, 2 Jan 2014 16:14:36 +0000 From: Nikolay Denev <ndenev@gmail.com> To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) Message-ID: <CA%2BP_MZGmZqeFjn73zXMN0_TrshAY6wRwx=75t8FbXHrj=eggig@mail.gmail.com> In-Reply-To: <CA%2BP_MZF4pMYqXV-BbrVfUjirfD7=qzr5Nz=k1NvOz6_wmT4GrQ@mail.gmail.com> References: <201312221304.rBMD4q38060416@oldred.freebsd.org> <201312221310.rBMDA0KH022980@freefall.freebsd.org> <CA%2BP_MZHrGB0OdddCmhpOptA-sBUHOERCdjSfUFh20pJOQsZ7Kw@mail.gmail.com> <CA%2BP_MZEqn4iK%2BWC-ouVNFEhMrq2tpH91qgcT5zJyaUnX%2BkVn1g@mail.gmail.com> <CA%2BP_MZF4pMYqXV-BbrVfUjirfD7=qzr5Nz=k1NvOz6_wmT4GrQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 1, 2014 at 5:39 PM, Nikolay Denev <ndenev@gmail.com> wrote: > Ok, killing openvpn with -9 leaves the routes around, and particularly > interesting are the following routes : > > 78.90.222.xx 10.255.255.0 UGHS 0 5841 epair0 = =3D> > 78.90.222.xx/32 10.255.255.0 UGS 0 0 epair0 > > Now, if I do : > > route delete 78.90.222.xx 10.255.255.0 > > The route, with the H flag is deleted. If I repeat the command the > second route is deleted as well, even if the second command specifies > a netmask no panic. > > However the first delete command specifies the /32 mask like this : > > route delete 78.90.222.xx 10.255.255.0 255.255.255.255 > > Then I get "rtfree 2" kernel panic immediately. > > This seems to be happening as I'm manually installing static routes in > the vnet jail for the VPN remote endpoints , however OpenVPN adds such > routes too however differently, which results in two routing entries. > > For example : > > route add $IP $GW > and > route add $IP $GW 255.255.255.255 > > add to different route entries, one is /32 network, the other is a host r= oute. > > > > --Nikolay > > On Wed, Jan 1, 2014 at 1:21 PM, Nikolay Denev <ndenev@gmail.com> wrote: >> On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev@gmail.com> wrote: >>> >>> On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit@freebsd.org> wro= te: >>>> >>>> Thank you very much for your problem report. >>>> It has the internal identification `misc/185092'. >>>> The individual assigned to look at your >>>> report is: freebsd-bugs. >>>> >>>> You can access the state of your problem report at any time >>>> via this link: >>>> >>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092 >>>> >>>> >Category: misc >>>> >Responsible: freebsd-bugs >>>> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) >>>> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 >>> >>> >>> I'm trying to understand exactly what is happening here, and examining = a >>> core dump with kgdb I'm getting some output that confuses me : >>> >>> (kgdb) bt >>> #0 doadump (textdump=3D-1011569920) at pcpu.h:233 >>> #1 0xc06069b2 in kern_reboot (howto=3D260) at >>> /usr/src/sys/kern/kern_shutdown.c:447 >>> #2 0xc0606d0e in panic (fmt=3D<value optimized out>) at >>> /usr/src/sys/kern/kern_shutdown.c:754 >>> #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at >>> /usr/src/sys/net/route.c:464 >>> #4 0xc06e188d in route_output (m=3D<value optimized out>) at >>> /usr/src/sys/net/rtsock.c:951 >>> #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m= =3D<value >>> optimized out>, nam=3D0x0, control=3D<value optimized out>, >>> td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 >>> #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value optimiz= ed >>> out>, top=3D<value optimized out>, control=3D0x0, >>> flags=3D<value optimized out>, td=3D<value optimized out>) at >>> /usr/src/sys/kern/uipc_socket.c:1271 >>> #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10= , >>> top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000) >>> at /usr/src/sys/kern/uipc_socket.c:1315 >>> #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, >>> active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000) >>> at /usr/src/sys/kern/sys_socket.c:103 >>> #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818= , >>> auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303 >>> #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value o= ptimized >>> out>) at /usr/src/sys/kern/sys_generic.c:467 >>> #11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<value >>> optimized out>) at /usr/src/sys/kern/sys_generic.c:382 >>> #12 0xc08614d3 in syscall (frame=3D<value optimized out>) at >>> subr_syscall.c:134 >>> #13 0xc084cca1 in Xint0x80_syscall () at >>> /usr/src/sys/i386/i386/exception.s:270 >>> #14 0x281975b7 in ?? () >>> Previous frame inner to this frame (corrupt stack?) >>> Current language: auto; currently minimal >>> (kgdb) fr 3 >>> #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at >>> /usr/src/sys/net/route.c:464 >>> 464 panic("rtfree 2"); >>> (kgdb) print *rt >>> $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_= bit =3D 0, >>> rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D { >>> rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x103= 0000 >>> <Address 0x1030000 out of bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D { >>> rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}}, >>> {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D = -94 '?', >>> rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e= 0 "", >>> rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880}, >>> rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R = =3D >>> 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060, >>> rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, r= t_rmx =3D >>> {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0}, >>> rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_fla= gs =3D 0, >>> lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}} >>> >>> >>> >>> rn_Key with value of =93shutdown_post_sync=94 ? >>> >>> It=92s visible also in the raw_usend() frame: >>> >>> (kgdb) fr 5 >>> #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m= =3D<value >>> optimized out>, nam=3D0x0, control=3D<value optimized out>, >>> td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 >>> 238 return ((*so->so_proto->pr_output)(m, so)); >>> (kgdb) print *m >>> $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data = =3D 0x0, >>> mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0= }, >>> M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first = =3D 0x4}, len =3D >>> -1012745856, flowid =3D 3282388448, >>> csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos = =3D 117 >>> 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o', >>> l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "= _post_sy", >>> sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103, >>> 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr = =3D >>> {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D { >>> eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0,= 0}, >>> thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454}, >>> ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ex= t_buf =3D >>> 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0, >>> ext_arg1 =3D 0x0, ext_arg2 =3D 0x0}, >>> MH_databuf =3D '\0' <repeats 56 times>, "file", '\0' <repeats 2= 0 >>> times>, >>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\0= 00\000\000\000\004\000\000\000\000\000\000\00000Y?", >>> '\0' <repeats 12 times>, >>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203= ??\000\000\000\000\000???", >>> '\0' <repeats 23 times>}}, >>> M_databuf =3D >>> "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync= ", >>> '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>, >>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\0= 00\000\000\000\004\000\000\000\000\000\000\00000Y?", >>> '\0' <repeats 12 times>, >>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203= ??\000\000\000\000\000???", >>> '\0' <repeats 23 times>}} >>> >>> >>> This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtf= ree >>> patch, which I thought earlier might be the cause of the panics I'm >>> seeing). >>> >>> The machine is Soekris Net5501-70 with this kernel config : >>> >>> cpu I586_CPU >>> cpu I686_CPU >>> ident MARS >>> options CPU_GEODE >>> options CPU_SOEKRIS >>> >>> options HZ=3D2000 >>> options DEVICE_POLLING >>> options BPF_JITTER >>> >>> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols >>> >>> options SCHED_ULE # ULE scheduler >>> options PREEMPTION # Enable kernel thread preemption >>> options INET # InterNETworking >>> options INET6 # IPv6 communications protocols >>> options TCP_OFFLOAD # TCP offload >>> options FFS # Berkeley Fast Filesystem >>> options SOFTUPDATES # Enable FFS soft updates support >>> options UFS_DIRHASH # Improve performance on big directories >>> options PROCFS # Process filesystem (requires PSEUDOFS) >>> options PSEUDOFS # Pseudo-filesystem framework >>> options GEOM_PART_GPT # GUID Partition Tables. >>> options GEOM_LABEL # Provides labelization >>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >>> options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI >>> options KTRACE # ktrace(1) support >>> options STACK # stack(9) support >>> options SYSVSHM # SYSV-style shared memory >>> options SYSVMSG # SYSV-style message queues >>> options SYSVSEM # SYSV-style semaphores >>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensio= ns >>> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being interspers= ed. >>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >>> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) >>> options CAPABILITY_MODE # Capsicum capability mode >>> options CAPABILITIES # Capsicum capabilities >>> options PROCDESC # Support for process descriptors >>> options INCLUDE_CONFIG_FILE # Include this file in kernel >>> >>> # Debugging support. Always need this: >>> options KDB # Enable kernel debugger support. >>> options KDB_TRACE # Print a stack trace for a panic. >>> options KDB_UNATTENDED >>> >>> options TEXTDUMP_PREFERRED >>> options TEXTDUMP_VERBOSE >>> >>> device pci >>> device ata # Legacy ATA/SATA controllers >>> options ATA_STATIC_ID # Static device numbering >>> >>> # ATA/SCSI peripherals >>> device scbus # SCSI bus (required for ATA/SCSI) >>> device da # Direct Access (disks) >>> device pass # Passthrough device (direct ATA/SCSI access) >>> >>> # Add suspend/resume support for the i8254. >>> device pmtimer >>> >>> # Serial (COM) ports >>> device uart # Generic UART driver >>> >>> device miibus # MII bus support >>> device vr # VIA Rhine, Rhine II >>> >>> # Wireless NIC cards >>> device wlan # 802.11 support >>> options IEEE80211_DEBUG # enable debug msgs >>> options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's >>> options IEEE80211_SUPPORT_MESH # enable 802.11s draft support >>> device wlan_wep # 802.11 WEP support >>> device wlan_ccmp # 802.11 CCMP support >>> device wlan_tkip # 802.11 TKIP support >>> device wlan_amrr # AMRR transmit rate control algorithm >>> device ath # Atheros NICs >>> device ath_pci # Atheros pci/cardbus glue >>> device ath_hal # pci/cardbus chip support >>> options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors >>> options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation >>> options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later >>> device ath_rate_sample # SampleRate tx rate control for ath >>> >>> # Pseudo devices. >>> device loop # Network loopback >>> device random # Entropy device >>> device ether # Ethernet support >>> device vlan # 802.1Q VLAN support >>> device tun # Packet tunnel. >>> device md # Memory "disks" >>> device gif # IPv6 and IPv4 tunneling >>> device gre >>> device faith # IPv6-to-IPv4 relaying (translation) >>> device firmware # firmware assist module >>> device if_bridge >>> >>> options VIMAGE >>> options ROUTETABLES=3D8 >>> options RADIX_MPATH >>> >>> options SW_WATCHDOG >>> >>> device crypto >>> device cryptodev >>> device glxsb >>> >>> options BOOTVERBOSE=3D1 >>> >>> #device pf >>> #device pflog >>> #device pfsync >>> device carp >>> device enc >>> device lagg >>> device epair >>> >>> #options ALTQ >>> #options ALTQ_CBQ >>> #options ALTQ_RED >>> #options ALTQ_RIO >>> #options ALTQ_HFSC >>> #options ALTQ_PRIQ >>> >>> options IPFIREWALL >>> options IPFIREWALL_DEFAULT_TO_ACCEPT >>> options IPFIREWALL_NAT >>> options LIBALIAS >>> options IPDIVERT >>> options DUMMYNET >>> >>> device bpf # Berkeley packet filter >>> >>> # USB support >>> options USB_DEBUG # enable debug msgs >>> device uhci # UHCI PCI->USB interface >>> device ohci # OHCI PCI->USB interface >>> device ehci # EHCI PCI->USB interface (USB 2.0) >>> device usb # USB Bus (required) >>> device umass # Disks/Mass storage - Requires scbus and da >>> >>> >>> Also src.conf and make.conf : >>> >>> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf >>> WITHOUT_ACCT=3Dyes >>> WITHOUT_ACPI=3Dyes >>> WITHOUT_AMD=3Dyes >>> WITHOUT_APM=3Dyes >>> WITHOUT_ASSERT_DEBUG=3Dyes >>> WITHOUT_AT=3Dyes >>> WITHOUT_ATF=3Dyes >>> WITHOUT_ATM=3Dyes >>> WITHOUT_AUDIT=3Dyes >>> WITHOUT_BLUETOOTH=3Dyes >>> WITHOUT_CALENDAR=3Dyes >>> WITHOUT_CDDL=3Dyes >>> WITHOUT_CTM=3Dyes >>> WITHOUT_DICT=3Dyes >>> WITHOUT_FLOPPY=3Dyes >>> WITHOUT_GAMES=3Dyes >>> WITHOUT_HTML=3Dyes >>> WITHOUT_INFO=3Dyes >>> WITHOUT_IPFILTER=3Dyes >>> WITHOUT_IPX=3Dyes >>> #WITHOUT_KERNEL_SYMBOLS=3Dyes >>> WITHOUT_LEGACY_CONSOLE=3Dyes >>> WITHOUT_LOCALES=3Dyes >>> WITHOUT_LPR=3Dyes >>> WITHOUT_MAIL=3Dyes >>> WITHOUT_NDIS=3Dyes >>> WITHOUT_QUOTAS=3Dyes >>> WITHOUT_ROUTED=3Dyes >>> WITHOUT_SENDMAIL=3Dyes >>> WITH_SVN=3Dyes >>> WITHOUT_ZFS=3Dyes >>> >>> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf >>> CFLAGS=3D-O2 >>> COPTFLAGS=3D -O -pipe >>> CPUTYPE=3Dgeode >>> KERNCONF=3DMARS >>> NO_MODULES=3Dyes >>> BOOTWAIT=3D0 >>> DOC_LANG=3Den_US.ISO8859-1 >>> >>> >>> >>> --Nikolay >>> >> >> Also, originally I thought that the panic is when a multi path route is >> being deleted, however again from the coredump it seems that the panic >> happens when openvpn deletes the host route it installs for the remote >> openvpn server pointed to the default gw (before openvpn installs the ne= w >> default gw pointing to the vpn tunnel) : >> >> (kgdb) fr 12 >> (kgdb) x/12sb td->td_proc->p_args >> 0xc4269780: "\001" >> 0xc4269782: "" >> 0xc4269783: "" >> 0xc4269784: "B" >> 0xc4269786: "" >> 0xc4269787: "" >> 0xc4269788: "/sbin/route" >> 0xc4269794: "delete" >> 0xc426979b: "-net" >> 0xc42697a0: "78.90.222.xxx" >> 0xc42697ad: "10.255.255.0" >> 0xc42697ba: "255.255.255.255" >> (kgdb) >> >> I'm trying to reproduce this on a VirtualBox instance now, however so fa= r no >> luck (no OpenVPN running, just adding and removing routes). >> >> >> --Nikolay Some more testing shows that it's pretty easy to panic the machine. One just have to try to delete twice a host route that has the same GW as the default gw and the machine panics. # ifconfig em0 10.0.0.155/24 up # route add default 10.0.0.1 add net default: gateway: 10.0.0.1 # route add 8.8.8.8 10.0.0.1 add host 8.8.8.8: gateway: 10.0.0.1 # route delete 8.8.8.8 10.0.0.1 delete host 8.8.8.8: gateway: 10.0.0.1 # route delete 8.8.8.8 10.0.0.1 panic: rtfree 2 --Nikolay
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BP_MZGmZqeFjn73zXMN0_TrshAY6wRwx=75t8FbXHrj=eggig>