Date: Sat, 25 Aug 2018 12:12:03 -0700 From: Matthew Macy <mmacy@freebsd.org> To: Kristof Provost <kp@freebsd.org> Cc: Shawn Webb <shawn.webb@hardenedbsd.org>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: ifnet use after free Message-ID: <CAPrugNobGiWLuxu4C=T8VP2jrXiW06LtB-NG=_iNDg%2BDNSgUKw@mail.gmail.com> In-Reply-To: <3B8F3112-6DD3-4461-9256-ECB044FA6D44@FreeBSD.org> References: <20180824221955.7hkftov25otk6bjc@mutt-hbsd> <CAPrugNqiX5udzOchu=yBAEEqnkK-LAZZhTW4poen13Gguc1Xng@mail.gmail.com> <7A724399-B264-41A9-B85F-A49D3B0B4730@FreeBSD.org> <2287F455-EDFD-4065-BB83-33DDDF74D027@FreeBSD.org> <CAPrugNrT3qcy%2BE5LUVYMBKze8Cc=454ergJbbQs7GuzfFNOt3Q@mail.gmail.com> <3B8F3112-6DD3-4461-9256-ECB044FA6D44@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Aug 25, 2018 at 12:10 PM Kristof Provost <kp@freebsd.org> wrote: > You may be right about that. With memguard (on ifnet) it implicates pf: > > pfi_cleanup_vnet() at pfi_cleanup_vnet+0xa4/frame 0xfffffe084f775320 > vnet_pf_uninit() at vnet_pf_uninit+0x85f/frame 0xfffffe084f7757c0 > vnet_destroy() at vnet_destroy+0x12c/frame 0xfffffe084f7757f0 > prison_deref() at prison_deref+0x29d/frame 0xfffffe084f775830 > sys_jail_remove() at sys_jail_remove+0x28a/frame 0xfffffe084f775880 > amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe084f7759b0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe084f7759= b0 > --- syscall (508, FreeBSD ELF64, sys_jail_remove), rip =3D 0x8003130da, r= sp > =3D 0x7fffffffe848, rbp =3D 0x7fffffffe8d0 --- > > I=E2=80=99ll investigate that. Sorry for the noise. > Thanks for the pointer to memguard. Very useful. > > And thank you for updating me. -M > Kristof > > On 25 Aug 2018, at 19:44, Matthew Macy wrote: > > I'll take a look. But it's likely to not be the OP's issue. For future > reference memguard on the memory type in question is extremely useful in > catching use after free. > > -M > > On Sat, Aug 25, 2018 at 05:51 Kristof Provost <kp@freebsd.org> wrote: > >> On 25 Aug 2018, at 0:47, Kristof Provost wrote: >> >> On 25 Aug 2018, at 0:26, Matthew Macy wrote: >> >> On Fri, Aug 24, 2018 at 15:25 Shawn Webb <shawn.webb@hardenedbsd.org> >> wrote: >> >> Hey All, >> >> Somewhere in the last month or so, a use after free was introduced. I >> don't have the time right now to bisect the commits and figure out >> which commit introduced the breakage. Attached is the core.txt (which >> seems nonsensical because the dump is reporting on a different >> thread). If the core.txt gets scrubbed, I've posted it here: >> https://gist.github.com/796ea88cec19a1fd2a85f4913482286a >> >> Do you have any guidance on how to reproduce? The hardenedbsd rev isn=E2= =80=99t >> useful - the svn commit that it=E2=80=99s based against is what is neede= d. >> >> For what it=E2=80=99s worth, it=E2=80=99s not a hardenedbsd thing. I=E2= =80=99ve been chasing the >> same one (same offset, same allocation size, same most recent user). >> Something gets set to zero/NULL. 8 bytes on amd64, so presumably a point= er. >> >> I currently only trigger it on a development branch, but I=E2=80=99ll se= e if I >> can clean that up into something I can share tomorrow. >> >> In my test scenario it happens after shutdown of a vnet jail with a few >> interfaces in it (including a pfsync interface which will disappear with >> the jail), and new jails are started. It=E2=80=99s pretty reliable. >> >> At a guess something=E2=80=99s wrong with the delayed cleanup of ifnets = and vnet >> shutdown. >> >> I see this: >> >> Memory modified after free 0xfffff800623ab000(2040) val=3D0 @ 0xfffff800= 623ab398 >> panic: Most recently used by ifnet >> >> cpuid =3D 7 >> time =3D 1535199812 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008c= 8e13c0 >> vpanic() at vpanic+0x1a3/frame 0xfffffe008c8e1420 >> panic() at panic+0x43/frame 0xfffffe008c8e1480 >> mtrash_ctor() at mtrash_ctor+0x81/frame 0xfffffe008c8e14a0 >> uma_zalloc_arg() at uma_zalloc_arg+0x72c/frame 0xfffffe008c8e1510 >> malloc() at malloc+0x9a/frame 0xfffffe008c8e1560 >> if_alloc() at if_alloc+0x23/frame 0xfffffe008c8e1590 >> epair_clone_create() at epair_clone_create+0x239/frame 0xfffffe008c8e161= 0 >> if_clone_createif() at if_clone_createif+0x4a/frame 0xfffffe008c8e1660 >> ifioctl() at ifioctl+0x852/frame 0xfffffe008c8e1750 >> kern_ioctl() at kern_ioctl+0x2ba/frame 0xfffffe008c8e17b0 >> sys_ioctl() at sys_ioctl+0x15e/frame 0xfffffe008c8e1880 >> amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe008c8e19b0 >> fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe008c8e1= 9b0 >> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x80047b74a, rsp =3D= 0x7fffffffe208, rbp =3D 0x7fffffffe250 --- >> KDB: enter: panic >> [ thread pid 1426 tid 100466 ] >> Stopped at kdb_enter+0x3b: movq $0,kdb_why >> db> >> >> It does require a couple of bug fixes in pfsync to trigger. You can get >> them from the pfsync_vnet branch in >> https://github.com/kprovost/freebsd/tree/pfsync_vnet >> >> After that: >> kldload pfsync >> pkg install scapy >> cd /usr/tests/sys/netpfil/pf >> kyua test >> >> It should panic reliably. >> >> Regards, >> Kristof >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPrugNobGiWLuxu4C=T8VP2jrXiW06LtB-NG=_iNDg%2BDNSgUKw>