Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 25 Aug 2018 21:10:45 +0200
From:      "Kristof Provost" <kp@FreeBSD.org>
To:        "Matthew Macy" <mmacy@freebsd.org>
Cc:        "Shawn Webb" <shawn.webb@hardenedbsd.org>, freebsd-current@freebsd.org
Subject:   Re: ifnet use after free
Message-ID:  <3B8F3112-6DD3-4461-9256-ECB044FA6D44@FreeBSD.org>
In-Reply-To: <CAPrugNrT3qcy%2BE5LUVYMBKze8Cc=454ergJbbQs7GuzfFNOt3Q@mail.gmail.com>
References:  <20180824221955.7hkftov25otk6bjc@mutt-hbsd> <CAPrugNqiX5udzOchu=yBAEEqnkK-LAZZhTW4poen13Gguc1Xng@mail.gmail.com> <7A724399-B264-41A9-B85F-A49D3B0B4730@FreeBSD.org> <2287F455-EDFD-4065-BB83-33DDDF74D027@FreeBSD.org> <CAPrugNrT3qcy%2BE5LUVYMBKze8Cc=454ergJbbQs7GuzfFNOt3Q@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
You may be right about that. With memguard (on ifnet) it implicates pf:

pfi_cleanup_vnet() at pfi_cleanup_vnet+0xa4/frame 0xfffffe084f775320
vnet_pf_uninit() at vnet_pf_uninit+0x85f/frame 0xfffffe084f7757c0
vnet_destroy() at vnet_destroy+0x12c/frame 0xfffffe084f7757f0
prison_deref() at prison_deref+0x29d/frame 0xfffffe084f775830
sys_jail_remove() at sys_jail_remove+0x28a/frame 0xfffffe084f775880
amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe084f7759b0
fast_syscall_common() at fast_syscall_common+0x101/frame 
0xfffffe084f7759b0
--- syscall (508, FreeBSD ELF64, sys_jail_remove), rip = 0x8003130da, 
rsp = 0x7fffffffe848, rbp = 0x7fffffffe8d0 ---

I’ll investigate that. Sorry for the noise.
Thanks for the pointer to memguard. Very useful.

Kristof

On 25 Aug 2018, at 19:44, Matthew Macy wrote:

> I'll take a look. But it's likely to not be the OP's issue. For future
> reference memguard on the memory type in question is extremely useful 
> in
> catching use after free.
>
> -M
>
> On Sat, Aug 25, 2018 at 05:51 Kristof Provost <kp@freebsd.org> wrote:
>
>> On 25 Aug 2018, at 0:47, Kristof Provost wrote:
>>
>> On 25 Aug 2018, at 0:26, Matthew Macy wrote:
>>
>> On Fri, Aug 24, 2018 at 15:25 Shawn Webb <shawn.webb@hardenedbsd.org>
>> wrote:
>>
>> Hey All,
>>
>> Somewhere in the last month or so, a use after free was introduced. I
>> don't have the time right now to bisect the commits and figure out
>> which commit introduced the breakage. Attached is the core.txt (which
>> seems nonsensical because the dump is reporting on a different
>> thread). If the core.txt gets scrubbed, I've posted it here:
>> https://gist.github.com/796ea88cec19a1fd2a85f4913482286a
>>
>> Do you have any guidance on how to reproduce? The hardenedbsd rev 
>> isn’t
>> useful - the svn commit that it’s based against is what is needed.
>>
>> For what it’s worth, it’s not a hardenedbsd thing. I’ve been 
>> chasing the
>> same one (same offset, same allocation size, same most recent user).
>> Something gets set to zero/NULL. 8 bytes on amd64, so presumably a 
>> pointer.
>>
>> I currently only trigger it on a development branch, but I’ll see 
>> if I can
>> clean that up into something I can share tomorrow.
>>
>> In my test scenario it happens after shutdown of a vnet jail with a 
>> few
>> interfaces in it (including a pfsync interface which will disappear 
>> with
>> the jail), and new jails are started. It’s pretty reliable.
>>
>> At a guess something’s wrong with the delayed cleanup of ifnets and 
>> vnet
>> shutdown.
>>
>> I see this:
>>
>> Memory modified after free 0xfffff800623ab000(2040) val=0 @ 
>> 0xfffff800623ab398
>> panic: Most recently used by ifnet
>>
>> cpuid = 7
>> time = 1535199812
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
>> 0xfffffe008c8e13c0
>> vpanic() at vpanic+0x1a3/frame 0xfffffe008c8e1420
>> panic() at panic+0x43/frame 0xfffffe008c8e1480
>> mtrash_ctor() at mtrash_ctor+0x81/frame 0xfffffe008c8e14a0
>> uma_zalloc_arg() at uma_zalloc_arg+0x72c/frame 0xfffffe008c8e1510
>> malloc() at malloc+0x9a/frame 0xfffffe008c8e1560
>> if_alloc() at if_alloc+0x23/frame 0xfffffe008c8e1590
>> epair_clone_create() at epair_clone_create+0x239/frame 
>> 0xfffffe008c8e1610
>> if_clone_createif() at if_clone_createif+0x4a/frame 
>> 0xfffffe008c8e1660
>> ifioctl() at ifioctl+0x852/frame 0xfffffe008c8e1750
>> kern_ioctl() at kern_ioctl+0x2ba/frame 0xfffffe008c8e17b0
>> sys_ioctl() at sys_ioctl+0x15e/frame 0xfffffe008c8e1880
>> amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe008c8e19b0
>> fast_syscall_common() at fast_syscall_common+0x101/frame 
>> 0xfffffe008c8e19b0
>> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80047b74a, rsp = 
>> 0x7fffffffe208, rbp = 0x7fffffffe250 ---
>> KDB: enter: panic
>> [ thread pid 1426 tid 100466 ]
>> Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
>> db>
>>
>> It does require a couple of bug fixes in pfsync to trigger. You can 
>> get
>> them from the pfsync_vnet branch in
>> https://github.com/kprovost/freebsd/tree/pfsync_vnet
>>
>> After that:
>> kldload pfsync
>> pkg install scapy
>> cd /usr/tests/sys/netpfil/pf
>> kyua test
>>
>> It should panic reliably.
>>
>> Regards,
>> Kristof
>>





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B8F3112-6DD3-4461-9256-ECB044FA6D44>