Date: Thu, 2 Apr 2015 23:47:20 +0200 From: Mateusz Guzik <mjguzik@gmail.com> To: Yue Chen <ycyc321@gmail.com> Cc: PaX Team <pageexec@freemail.hu>, Benjamin Kaduk <bjk@freebsd.org>, HardenedBSD Core <core@hardenedbsd.org>, Oliver Pinter <oliver.pinter@hardenedbsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: How to traverse kernel threads? Message-ID: <20150402214720.GE549@dft-labs.eu> In-Reply-To: <CAKtBrB5WB0eCQqwwUXCKM8RVGaqeiatyKisyQZist4Zq4tJPKQ@mail.gmail.com> References: <CAKtBrB5KNqt6UJ1R_BQpPfTvQZdUzGvZZtT7Uz5qd4VrrfgEdw@mail.gmail.com> <20150321232622.GF14650@dft-labs.eu> <alpine.GSO.1.10.1503221644440.22210@multics.mit.edu> <CAPQ4ffuszSi%2B_SopJdCkoFr4OoY9=BZVbO6oo_s0sKrn8Rgjrw@mail.gmail.com> <CAKtBrB6ZF2FVExmDd%2Bt8yFpN0H7xHwaieWgvryR535Vc2cNBjw@mail.gmail.com> <20150327194920.GB18158@dft-labs.eu> <CAKtBrB6qeNTyemG-ws7X_OKcB2t-6muxJrOZ1Eyr0CRjjso5mQ@mail.gmail.com> <CAPQ4fft%2BmNJObMSw8SFvsweZsB-UKNUV2yghg4jB%2BdvSg9ho2w@mail.gmail.com> <20150330173358.GB9095@dft-labs.eu> <CAKtBrB5WB0eCQqwwUXCKM8RVGaqeiatyKisyQZist4Zq4tJPKQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Mar 30, 2015 at 10:59:16PM -0400, Yue Chen wrote: > > This would be a performance hit. > > The cost is to traverse the .text area and some other data in memory. It's > linear and only happens after a given time interval, like several minutes, > and does not put overhead on the normal execution. So the performance > overhead won't be high. > This was a remark about a runtime cost of executing kernel code with with an extra jump in every function. > > The only somewhat workable solution I see would require leaving function > entry points at constant addresses and only insert jumps to new locations. > > Yes, we do this. For the PIC direct jump/call/"instructions including > %RIP", we just need to update them. > I have trouble parsing what you wrote here. What I wrote here suggests taking existing foo_func, placing a copy at a new location and putting a jump to that address as the very first instruction of foo_func. (and zeroing/whatever the rest of old foo_func). Quite terrible. Also quite irrelevant to how you arrive at given func. Also see below. > > Also runtime relocation of everything definitely has a lot of unknown unknowns, > so all in all I would say this is a non-starter. > > FreeBSD kernel itself only has 1500+ indirect jumps. There are two types: > 1. jump table. Like switch-case. 2. Tail call. We have not found other > types like manual assembly code, or context restoration like that in > longjmp in libc. Maybe there are some other but we have not found. > Unknown unknowns have the problem of not being known. You have to be quite skilled in the area to come up with a working solution and I highly doubt either of us is such a person. > > Leave a lot of data at constant addresses, defeating the point to some > extent. > > One could consider a different approach where kernel data is randomly shuffled > around with some granularity and > > relevant symbol relocated prior to booting. > > Why do you think the data address exposure could be a serious problem? It > is non-executable and the code is randomized. Although for data itself, it > is really hard to protect from tampering and may cause other problems. > I see I wrote 'data', where I should have written '.text' or 'code', although from the context it should have been clear this refers to function entry points left behind. if you can get away with one function call, you don't need any infoleaks - you know the address of the function. That's one of the problems with leaving func entry points at known addresses. > Why the randomized data leads to kernel panic? I think if the data goes > somewhere else, the instruction addressing would follow, or page fault may > happen. If randomizing code, the exploit attempt may hit an illegal > instruction and the kernel panics. > What? Anyway I still don't understand how are you trying to achieve randomisation in the kernel. Previous mail suggests you just want to move funcs around and update all callsites and structs which store addresses to them. As outlined with 'struct meh' example (which can be found below) I do not believe this can work on real-world systems without a major surgery simply because there is no way to tell what looking like a pointer should be changed and what should not. I'm happy to be proved wrong. Kernel already leaks tons of addresses to userspace in documented ways, and I'm sure there are tons of situations where it does so as a result of a bug. I believe an alternative implementation I proposed which randomizes *all* kernel pages prior to booting it would be doable, but given aforementioned leaks not worth it. (well, striclty speaking you would want to move functions around with respect to each other, but keep the kernel physically contiguous so that you can still use superpages) Your previous mail in this thread suggests you are just starting any kind of work on kernel-level, which makes this very hard task even harder. Exploitation mitigation is a complicated subject on its own, I recommend reading what people were doing it have to say. Here is a piece about KASLR: https://forums.grsecurity.net/viewtopic.php?f=7&t=3367 In conclusion I strongly recommend dropping this project for the time being and focusing on something easier. > > return-oriented > > > > programming (ROP). It is basically a stronger form of ASLR. > > > > After each randomization procedure, the function return addresses > > saved in > > > > the stack are the ``old'' addresses before randomization, so we need to > > > > update them to the new addresses. > > > > That's why we need to get all the stack ranges to find those addresses. > > > > > > > > Also, in kernel, we believe that not only the return addresses in > > stacks > > > > need to be updated, there may exist other ``old'' saved instruction > > (PC) > > > > addresses in memory. Like in exception handling (maybe, do not know), > > > > debugging-purpose code and restartable atomic sequences (RAS) > > > > implementation. That's why I asked how to traverse all the kernel > > pages and > > > > get their virtual addresses here: > > > > > > https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html > > > > > > > > Now we found that it seems needed to traverse the ``pv_entry'' > > structure for > > > > x86_64 MMU. > > > > > > > > Another problem is that we do not know if FreeBSD has any form of > > special > > > > encodings or offset form for 64-bit instruction addresses (e.g., saved > > %RIP) > > > > on X86_64, instead of hard-coded addresses. For example, using a 32-bit > > > > offset instead of the 64-bit full address; and doing what glibc does > > for the > > > > setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the saved > > > > register values). > > > > > > > > Any suggestion or correction are highly appreciated. > > > > > > > > Best, > > > > Yue > > > > > > (Added HardenedBSD core and PaXTeam to CC.) > > > > > > Until you can not fixed all of the infoleaks from kernel (try sysctl > > > -a | grep 0x or similar command) the KASLR and other kernel address > > > space randomization techniques are easily bypass-able... > > > > > > > I do not believe this can be implemented reliably without some serious > > tinkering around the kernel. Surely I'm not a live patching expert (not > > any other kind of expert), but hear my out. > > > > It seems proposed approach is to move the kernel around and then updated > > all relevant pointers in all pages. > > > > I do not believe this can be done reliably without corrupting data, > > unless a major surgery is performed on the kernel. > > > > Consider: > > struct meh { > > func_t *m_func; > > size_t m_len; > > data *m_buf; > > }; > > > > Here we have 'm_func' which needs to be updated, but we don't know if > > that's the only pointer so we have to scan the entire struct. But m_buf > > can have data which looks like kernel pointers (i.e. matches some kernel > > funcs) and how will you know not to modify it? What if this is some sort > > of a hack and in fact you *should* modify it? > > > > CTF or some other solutions like that don't help if you just traverse > > all allocated buffers since you have no idea what the object you found > > really is. > > > > So, assuming this has to be done in runtime, the only somewhat workable > > solution I see would require leaving function entry points at contant > > addresses and only insert jumps to new locations. > > > > This would be a performance hit and would leave a lot of data at > > constant addresses, defeating the point to some extent. > > > > Also runtime relocation of everything definitely has a lot of unknown > > unknowns, so all in all I would say this is a non-starter. > > > > One could consider a different approach where kernel data is randomly > > shuffled around with some granularity and relevant symbol relocated > > prior to booting. > > > > This should provide unique enough layout, which paired with big > > likelyhood of a kernel panic on first bad exploit attempt may be an > > acceptable solution. > > > > But then the kernel may still have enough info leaks for this to not > > matter, so I would not be so eager to implement it. > > > > That said, what prompted this entire effort? Is there an operating > > system which got this to work or what? > > > > > > > > > > > > > > > > > > On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik <mjguzik@gmail.com> > > wrote: > > > >> > > > >> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: > > > >> > When using the following code on kernel module loading: > > > >> > > > > >> > > > ------------------------------------------------------------------------------------------ > > > >> > struct thread *td = kdb_thr_first(); > > > >> > td = kdb_thr_next(td); > > > >> > > > > >> > > > ------------------------------------------------------------------------------------------ > > > >> > The kernel panics. > > > >> > > > > >> > > > >> Panics how? > > > >> > > > >> Also you can easily see these functions don't lock anything, so it > > would > > > >> be assumed you took appropriate locks. > > > >> > > > >> Except it seems there routines are supposed to be only used when > > > >> execution is 'frozen' (e.g. when escaped to the debugger). > > > >> > > > >> > > > > >> > And when printing all threads in proc0 (all kernel threads?): > > > >> > > > > >> > > > ------------------------------------------------------------------------------------------ > > > >> > struct proc *p = pfind(0); > > > >> > FOREACH_THREAD_IN_PROC(p, td) { > > > >> > uprintf("td: %x\n", td); > > > >> > } > > > >> > > > > >> > > > >> proc0 is an exported symbol, no need to pfind. > > > >> > > > >> > td = curthread; > > > >> > uprintf("cur td: %x\n", td); > > > >> > > > > >> > > > ------------------------------------------------------------------------------------------ > > > >> > The ``curthread'' (from this kernel module running the above code) > > is > > > >> > not > > > >> > in the 0 process group. > > > >> > > > > >> > > > >> There is no 'curthread from kernel module'. > > > >> > > > >> My guess is you do this work from module initializator, and in that > > case > > > >> curthread is the thread which loads the module, and such a thread is > > > >> definitely not linked into proc0. > > > >> > > > >> Still nobody knows what you are trying to do. > > > >> > > > >> -- > > > >> Mateusz Guzik <mjguzik gmail.com> > > > > > > > > > > > > -- > > Mateusz Guzik <mjguzik gmail.com> > > -- Mateusz Guzik <mjguzik gmail.com>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150402214720.GE549>