Date: Wed, 1 Jan 2014 13:10:57 +0000 From: Nikolay Denev <ndenev@gmail.com> To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) Message-ID: <CA%2BP_MZHrGB0OdddCmhpOptA-sBUHOERCdjSfUFh20pJOQsZ7Kw@mail.gmail.com> In-Reply-To: <201312221310.rBMDA0KH022980@freefall.freebsd.org> References: <201312221304.rBMD4q38060416@oldred.freebsd.org> <201312221310.rBMDA0KH022980@freefall.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit@freebsd.org> wrote: > Thank you very much for your problem report. > It has the internal identification `misc/185092'. > The individual assigned to look at your > report is: freebsd-bugs. > > You can access the state of your problem report at any time > via this link: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092 > > >Category: misc > >Responsible: freebsd-bugs > >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) > >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 > I'm trying to understand exactly what is happening here, and examining a core dump with kgdb I'm getting some output that confuses me : (kgdb) bt #0 doadump (textdump=3D-1011569920) at pcpu.h:233 #1 0xc06069b2 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xc0606d0e in panic (fmt=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at /usr/src/sys/net/route.c:464 #4 0xc06e188d in route_output (m=3D<value optimized out>) at /usr/src/sys/net/rtsock.c:951 #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<va= lue optimized out>, nam=3D0x0, control=3D<value optimized out>, td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value optimized o= ut>, top=3D<value optimized out>, control=3D0x0, flags=3D<value optimized out>, td=3D<value optimized out>) at /usr/src/sys/kern/uipc_socket.c:1271 #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, to= p=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000) at /usr/src/sys/kern/uipc_socket.c:1315 #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000) at /usr/src/sys/kern/sys_socket.c:103 #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818, auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303 #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value optim= ized out>) at /usr/src/sys/kern/sys_generic.c:467 #11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<value optim= ized out>) at /usr/src/sys/kern/sys_generic.c:382 #12 0xc08614d3 in syscall (frame=3D<value optimized out>) at subr_syscall.c:134 #13 0xc084cca1 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270 #14 0x281975b7 in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) fr 3 #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at /usr/src/sys/net/route.c:464 464 panic("rtfree 2"); (kgdb) print *rt $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bit = =3D 0, rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D { rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x1030000 <Address 0x1030000 out of bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D { rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}}, {rn_= mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -94 '?', rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0 ""= , rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880}, rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R =3D 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060, rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt_rm= x =3D {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0}, rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags = =3D 0, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}} rn_Key with value of =93shutdown_post_sync=94 ? It=92s visible also in the raw_usend() frame: (kgdb) fr 5 #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<va= lue optimized out>, nam=3D0x0, control=3D<value optimized out>, td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 238 return ((*so->so_proto->pr_output)(m, so)); (kgdb) print *m $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data =3D = 0x0, mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0}, M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first =3D = 0x4}, len =3D -1012745856, flowid =3D 3282388448, csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos =3D 1= 17 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o', l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_pos= t_sy", sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103, 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr =3D {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D { eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, 0}, thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454}, ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_bu= f =3D 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0, ext_arg1 =3D 0x0, ext_arg2 =3D 0x0}, MH_databuf =3D '\0' <repeats 56 times>, "file", '\0' <repeats 20 times>, "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\0= 00\000\000\004\000\000\000\000\000\000\00000Y?", '\0' <repeats 12 times>, "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\0= 00\000\000\000\000???", '\0' <repeats 23 times>}}, M_databuf =3D "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync", '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>, "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\0= 00\000\000\004\000\000\000\000\000\000\00000Y?", '\0' <repeats 12 times>, "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\0= 00\000\000\000\000???", '\0' <repeats 23 times>}} This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfree patch, which I thought earlier might be the cause of the panics I'm seeing). The machine is Soekris Net5501-70 with this kernel config : cpu I586_CPU cpu I686_CPU ident MARS options CPU_GEODE options CPU_SOEKRIS options HZ=3D2000 options DEVICE_POLLING options BPF_JITTER makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options TCP_OFFLOAD # TCP offload options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_DIRHASH # Improve performance on big directories options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options CAPABILITY_MODE # Capsicum capability mode options CAPABILITIES # Capsicum capabilities options PROCDESC # Support for process descriptors options INCLUDE_CONFIG_FILE # Include this file in kernel # Debugging support. Always need this: options KDB # Enable kernel debugger support. options KDB_TRACE # Print a stack trace for a panic. options KDB_UNATTENDED options TEXTDUMP_PREFERRED options TEXTDUMP_VERBOSE device pci device ata # Legacy ATA/SATA controllers options ATA_STATIC_ID # Static device numbering # ATA/SCSI peripherals device scbus # SCSI bus (required for ATA/SCSI) device da # Direct Access (disks) device pass # Passthrough device (direct ATA/SCSI access) # Add suspend/resume support for the i8254. device pmtimer # Serial (COM) ports device uart # Generic UART driver device miibus # MII bus support device vr # VIA Rhine, Rhine II # Wireless NIC cards device wlan # 802.11 support options IEEE80211_DEBUG # enable debug msgs options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's options IEEE80211_SUPPORT_MESH # enable 802.11s draft support device wlan_wep # 802.11 WEP support device wlan_ccmp # 802.11 CCMP support device wlan_tkip # 802.11 TKIP support device wlan_amrr # AMRR transmit rate control algorithm device ath # Atheros NICs device ath_pci # Atheros pci/cardbus glue device ath_hal # pci/cardbus chip support options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later device ath_rate_sample # SampleRate tx rate control for ath # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device gre device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device if_bridge options VIMAGE options ROUTETABLES=3D8 options RADIX_MPATH options SW_WATCHDOG device crypto device cryptodev device glxsb options BOOTVERBOSE=3D1 #device pf #device pflog #device pfsync device carp device enc device lagg device epair #options ALTQ #options ALTQ_CBQ #options ALTQ_RED #options ALTQ_RIO #options ALTQ_HFSC #options ALTQ_PRIQ options IPFIREWALL options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_NAT options LIBALIAS options IPDIVERT options DUMMYNET device bpf # Berkeley packet filter # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device umass # Disks/Mass storage - Requires scbus and da Also src.conf and make.conf : root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf WITHOUT_ACCT=3Dyes WITHOUT_ACPI=3Dyes WITHOUT_AMD=3Dyes WITHOUT_APM=3Dyes WITHOUT_ASSERT_DEBUG=3Dyes WITHOUT_AT=3Dyes WITHOUT_ATF=3Dyes WITHOUT_ATM=3Dyes WITHOUT_AUDIT=3Dyes WITHOUT_BLUETOOTH=3Dyes WITHOUT_CALENDAR=3Dyes WITHOUT_CDDL=3Dyes WITHOUT_CTM=3Dyes WITHOUT_DICT=3Dyes WITHOUT_FLOPPY=3Dyes WITHOUT_GAMES=3Dyes WITHOUT_HTML=3Dyes WITHOUT_INFO=3Dyes WITHOUT_IPFILTER=3Dyes WITHOUT_IPX=3Dyes #WITHOUT_KERNEL_SYMBOLS=3Dyes WITHOUT_LEGACY_CONSOLE=3Dyes WITHOUT_LOCALES=3Dyes WITHOUT_LPR=3Dyes WITHOUT_MAIL=3Dyes WITHOUT_NDIS=3Dyes WITHOUT_QUOTAS=3Dyes WITHOUT_ROUTED=3Dyes WITHOUT_SENDMAIL=3Dyes WITH_SVN=3Dyes WITHOUT_ZFS=3Dyes root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf CFLAGS=3D-O2 COPTFLAGS=3D -O -pipe CPUTYPE=3Dgeode KERNCONF=3DMARS NO_MODULES=3Dyes BOOTWAIT=3D0 DOC_LANG=3Den_US.ISO8859-1 --Nikolay
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BP_MZHrGB0OdddCmhpOptA-sBUHOERCdjSfUFh20pJOQsZ7Kw>