Date: Fri, 14 Dec 2018 23:51:40 +0000 From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 234026] [panic] [dummynet] Repeatable panic in dummynet due to locking issues and use-after-free Message-ID: <bug-234026-7501@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234026 Bug ID: 234026 Summary: [panic] [dummynet] Repeatable panic in dummynet due to locking issues and use-after-free Product: Base System Version: 11.2-STABLE Hardware: Any OS: Any Status: New Keywords: crash Severity: Affects Some People Priority: --- Component: kern Assignee: net@FreeBSD.org Reporter: eugen@freebsd.org Hi! I run multiple routers using FreeBSD 11.2-STABLE/amd64 r336962, ipfw+dummyn= et and net/mpd5 daemon that dynamically creates/destroys ngXXX interfaces for multiple PPPoE clients. If an interface ngXXX is destroyed while dummynet pipe/queue keeps mbuf with m_pkthdr.rcvif pointing to freed struct ifnet, kernel panices when taskqueue runs dummynet_task/dummynet_send/netisr_dispatch_src/ip_input sequence and I have crashdump. kgdb session follows: Script started on Sat Dec 15 06:47:49 2018 Command: kgdb kernel.debug /home/nanobsd/pppoe/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: stack pointer =3D 0x28:0xfffffe01244bb920 frame pointer =3D 0x28:0xfffffe01244bb9a0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 0 (dummynet) trap number =3D 12 panic: page fault cpuid =3D 0 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff802fc89b =3D db_trace_self_wrapper+0x2b/frame 0xfffffe01244bb5d0 vpanic() at 0xffffffff804f0ac7 =3D vpanic+0x177/frame 0xfffffe01244bb630 panic() at 0xffffffff804f0943 =3D panic+0x43/frame 0xfffffe01244bb690 trap_fatal() at 0xffffffff8076f2af =3D trap_fatal+0x35f/frame 0xfffffe01244= bb6e0 trap_pfault() at 0xffffffff8076f309 =3D trap_pfault+0x49/frame 0xfffffe0124= 4bb740 trap() at 0xffffffff8076eae4 =3D trap+0x2d4/frame 0xfffffe01244bb850 calltrap() at 0xffffffff8074ff3c =3D calltrap+0x8/frame 0xfffffe01244bb850 --- trap 0xc, rip =3D 0xffffffff804ec893, rsp =3D 0xfffffe01244bb920, rbp = =3D 0xfffffe01244bb9a0 --- __rw_rlock_hard() at 0xffffffff804ec893 =3D __rw_rlock_hard+0xf3/frame 0xfffffe01244bb9a0 ip_input() at 0xffffffff806444ca =3D ip_input+0x53a/frame 0xfffffe01244bba30 netisr_dispatch_src() at 0xffffffff8060ebe8 =3D netisr_dispatch_src+0xa8/fr= ame 0xfffffe01244bba80 dummynet_send() at 0xffffffff806723dd =3D dummynet_send+0x10d/frame 0xfffffe01244bbab0 dummynet_task() at 0xffffffff80671e1c =3D dummynet_task+0x2ec/frame 0xfffffe01244bbb20 taskqueue_run_locked() at 0xffffffff80548a54 =3D taskqueue_run_locked+0x154= /frame 0xfffffe01244bbb80 taskqueue_thread_loop() at 0xffffffff80549bb8 =3D taskqueue_thread_loop+0x98/frame 0xfffffe01244bbbb0 fork_exit() at 0xffffffff804ba803 =3D fork_exit+0x83/frame 0xfffffe01244bbb= f0 fork_trampoline() at 0xffffffff80750eee =3D fork_trampoline+0xe/frame 0xfffffe01244bbbf0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- Uptime: 57d17h28m40s Dumping 467 out of 4073 MB:..4%..11%..21%..31%..42%..52%..62%..72%..83%..93% Reading symbols from /boot/modules/tmpfs.ko...done. Loaded symbols for /boot/modules/tmpfs.ko #0 doadump (textdump=3D1) at pcpu.h:230 230 __asm("movq %%gs:%1,%0" : "=3Dr" (td) (kgdb) bt #0 doadump (textdump=3D1) at pcpu.h:230 #1 0xffffffff804f06c0 in kern_reboot (howto=3D260) at /home/src/sys/kern/kern_shutdown.c:383 #2 0xffffffff804f0b01 in vpanic (fmt=3D<value optimized out>, ap=3D<value optimized out>) at /home/src/sys/kern/kern_shutdown.c:776 #3 0xffffffff804f0943 in panic (fmt=3D<value optimized out>) at /home/src/sys/kern/kern_shutdown.c:707 #4 0xffffffff8076f2af in trap_fatal (frame=3D0xfffffe01244bb860, eva=3D274877908504) at /home/src/sys/amd64/amd64/trap.c:877 #5 0xffffffff8076f309 in trap_pfault (frame=3D0xfffffe01244bb860, usermode= =3D0) at pcpu.h:230 #6 0xffffffff8076eae4 in trap (frame=3D0xfffffe01244bb860) at /home/src/sys/amd64/amd64/trap.c:415 #7 0xffffffff8074ff3c in calltrap () at /home/src/sys/amd64/amd64/exception.S:231 #8 0xffffffff804ec893 in __rw_rlock_hard (rw=3D0xfffff80092e78190, td=3D0xfffff80001d02620, v=3D<value optimized out>) at /home/src/sys/kern/kern_rwlock.c:493 #9 0xffffffff806444ca in ip_input (m=3D<value optimized out>) at /home/src/sys/netinet/ip_input.c:795 #10 0xffffffff8060ebe8 in netisr_dispatch_src (proto=3D1, source=3D<value o= ptimized out>, m=3D<value optimized out>) at /home/src/sys/net/netisr.c:1120 #11 0xffffffff806723dd in dummynet_send (m=3D0x0) at /home/src/sys/netpfil/ipfw/ip_dn_io.c:774 #12 0xffffffff80671e1c in dummynet_task (context=3D<value optimized out>, pending=3D<value optimized out>) at /home/src/sys/netpfil/ipfw/ip_dn_io= .c:729 #13 0xffffffff80548a54 in taskqueue_run_locked (queue=3D0xfffff80006085e00) at /home/src/sys/kern/subr_taskqueue.c:463 #14 0xffffffff80549bb8 in taskqueue_thread_loop (arg=3D<value optimized out= >) at /home/src/sys/kern/subr_taskqueue.c:755 #15 0xffffffff804ba803 in fork_exit (callout=3D0xffffffff80549b20 <taskqueue_thread_loop>, arg=3D0xffffffff80c82c38, frame=3D0xfffffe01244bbc00) at /home/src/sys/kern/kern_fork.c:1072 #16 0xffffffff80750eee in fork_trampoline () at /home/src/sys/amd64/amd64/exception.S:972 ---Type <return> to continue, or q <return> to quit--- #17 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) frame 9 #9 0xffffffff806444ca in ip_input (m=3D<value optimized out>) at /home/src/sys/netinet/ip_input.c:795 795 IF_ADDR_RLOCK(ifp); (kgdb) l 790 * interface. Reception of forwarded directed broadcasts w= ould 791 * be handled via ip_forward() and ether_output() with the loopback 792 * into the stack for SIMPLEX interfaces handled by ether_output(). 793 */ 794 if (ifp !=3D NULL && ifp->if_flags & IFF_BROADCAST) { 795 IF_ADDR_RLOCK(ifp); 796 TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { 797 if (ifa->ifa_addr->sa_family !=3D AF_INET) 798 continue; 799 ia =3D ifatoia(ifa); (kgdb) p *ifp $1 =3D {if_link =3D {tqe_next =3D 0x4000000004, tqe_prev =3D 0x4000000006},= if_clones =3D { le_next =3D 0x4000000007, le_prev =3D 0x4000000009}, if_groups =3D {tqh= _first =3D 0x400000000a, tqh_last =3D 0x4000000011}, if_alloctype =3D 250 '=D0=97', if_softc =3D= 0x4000000104, if_llsoftc =3D 0x40000004d0, if_l2com =3D 0x40000004d4, if_dname =3D 0x4000000184 <Address 0x4000000184 out of bounds>, if_dunit = =3D 218, if_index =3D 64, if_index_reserved =3D 0, if_xname =3D 0xfffff80092e78060 "\220\001", if_description =3D 0x400000035e <Address 0x400000035e out of bounds>, if_= flags =3D 1050, if_drv_flags =3D 64, if_capabilities =3D 454, if_capenable =3D 64, if_lin= kmib =3D 0x4000000386, if_linkmiblen =3D 274877907462, if_refcount =3D 682, if_type =3D 64 '@', = if_addrlen =3D 0 '\0', if_hdrlen =3D 0 '\0', if_link_state =3D 0 '\0', if_mtu =3D 522, if_metric= =3D 64, if_baudrate =3D 274877907476, if_hwassist =3D 274877907488, if_epoch =3D 274877907500, if_lastchange =3D {tv_sec =3D 274877908294, tv_usec =3D 274877907730}, if= _snd =3D { ifq_head =3D 0x40000002e0, ifq_tail =3D 0x4000000334, ifq_len =3D 824, = ifq_maxlen =3D 64, ifq_mtx =3D { lock_object =3D {lo_name =3D 0x40000003c6 <Address 0x40000003c6 out of bounds>, lo_flags =3D 1298, lo_data =3D 64, lo_witness =3D 0x4000000332}, mt= x_lock =3D 274877907950}, ifq_drv_head =3D 0x40000002ae, ifq_drv_tail =3D 0x40000000fc, ifq_drv_l= en =3D 858, ifq_drv_maxlen =3D 64, altq_type =3D 870, altq_flags =3D 64, altq_disc = =3D 0x400000036a, altq_ifp =3D 0x4000000124, altq_enqueue =3D 0x4000000318, altq_dequeue = =3D 0x400000030a, altq_request =3D 0x400000036c, altq_clfier =3D 0x4000000188, altq_class= ify =3D 0x400000058d, altq_tbr =3D 0x400000058f, altq_cdnr =3D 0x4000000376}, if_linktask =3D= {ta_link =3D { stqe_next =3D 0x4000000262}, ta_pending =3D 460, ta_priority =3D 0, t= a_func =3D 0x4000000264, ta_context =3D 0x40000001b6}, if_addr_lock =3D {lock_object =3D { lo_name =3D 0x40000001b8 <Address 0x40000001b8 out of bounds>, lo_fla= gs =3D 1072, lo_data =3D 64, lo_witness =3D 0x400000026a}, rw_lock =3D 274877907356}, if_addrhead = =3D { tqh_first =3D 0x4000000382, tqh_last =3D 0x4000000196}, if_multiaddrs = =3D { tqh_first =3D 0x4000000120, tqh_last =3D 0x4000000218}, if_amcount =3D = 294, if_addr =3D 0x40000001be, if_broadcastaddr =3D 0x4000000064 <Address 0x4000000064 out of bounds>, if_afdata_lock =3D { ---Type <return> to continue, or q <return> to quit--- lock_object =3D {lo_name =3D 0x4000000192 <Address 0x4000000192 out of = bounds>, lo_flags =3D 810, lo_data =3D 64, lo_witness =3D 0x40000002de}, rw_lock =3D 27487790768= 4}, if_afdata =3D 0xfffff80092e78208, if_afdata_initialized =3D 441, if_fib = =3D 64, if_vnet =3D 0x40000000db, if_home_vnet =3D 0x4000000411, if_vlantrunk =3D 0x40000001bf, if_bpf =3D 0x40000001c1, if_pcount =3D 1051, if_bridge =3D 0x40000001c7, = if_lagg =3D 0x40000003ef, if_pf_kif =3D 0x4000000207, if_carp =3D 0x400000020b, if_label =3D 0x4000= 0002ab, if_netmap =3D 0x4000000215, if_output =3D 0x4000000219, if_input =3D 0x40= 000002af, if_start =3D 0x4000000221, if_ioctl =3D 0x400000022d, if_init =3D 0x40000= 002e1, if_resolvemulti =3D 0x40000002e5, if_qflush =3D 0x4000000305, if_transmit= =3D 0x4000000263, if_reassign =3D 0x4000000265, if_get_counter =3D 0x400000030b, if_request= encap =3D 0x400000026b, if_counters =3D 0xfffff80092e78410, if_hw_tsomax =3D 999, if_hw_tsomaxseg= count =3D 64, if_hw_tsomaxsegsize =3D 735, if_pspare =3D 0xfffff80092e78480, if_hw_addr= =3D 0x4000000039, if_pcp =3D 101 'e', if_bspare =3D 0xfffff80092e784a1 "", if_ispare =3D 0xfffff80092e784a4} (kgdb) frame 11 #11 0xffffffff806723dd in dummynet_send (m=3D0x0) at /home/src/sys/netpfil/ipfw/ip_dn_io.c:774 774 netisr_dispatch(NETISR_IP, m); (kgdb) p m $2 =3D (struct mbuf *) 0x0 (kgdb) l 769 case DIR_OUT: 770 ip_output(m, NULL, NULL, IP_FORWARDING, NUL= L, NULL); 771 break ; 772 773 case DIR_IN : 774 netisr_dispatch(NETISR_IP, m); 775 break; 776 777 #ifdef INET6 778 case DIR_IN | PROTO_IPV6: (kgdb) quit --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-234026-7501>