Date: Sun, 05 Apr 2015 20:24:35 +0800 From: Julian Elischer <julian@freebsd.org> To: Yue Chen <ycyc321@gmail.com>, Mateusz Guzik <mjguzik@gmail.com>, Oliver Pinter <oliver.pinter@hardenedbsd.org>, Benjamin Kaduk <bjk@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, HardenedBSD Core <core@hardenedbsd.org>, PaX Team <pageexec@freemail.hu> Subject: Re: How to traverse kernel threads? Message-ID: <55212983.20201@freebsd.org> In-Reply-To: <CAKtBrB645am=hmuX2=w-NeDxY%2B7vEynZYpgVYxDZw5tsyrnZBA@mail.gmail.com> References: <CAKtBrB5KNqt6UJ1R_BQpPfTvQZdUzGvZZtT7Uz5qd4VrrfgEdw@mail.gmail.com> <20150321232622.GF14650@dft-labs.eu> <alpine.GSO.1.10.1503221644440.22210@multics.mit.edu> <CAPQ4ffuszSi%2B_SopJdCkoFr4OoY9=BZVbO6oo_s0sKrn8Rgjrw@mail.gmail.com> <CAKtBrB6ZF2FVExmDd%2Bt8yFpN0H7xHwaieWgvryR535Vc2cNBjw@mail.gmail.com> <20150327194920.GB18158@dft-labs.eu> <CAKtBrB6qeNTyemG-ws7X_OKcB2t-6muxJrOZ1Eyr0CRjjso5mQ@mail.gmail.com> <CAPQ4fft%2BmNJObMSw8SFvsweZsB-UKNUV2yghg4jB%2BdvSg9ho2w@mail.gmail.com> <20150330173358.GB9095@dft-labs.eu> <CAKtBrB5WB0eCQqwwUXCKM8RVGaqeiatyKisyQZist4Zq4tJPKQ@mail.gmail.com> <20150402214720.GE549@dft-labs.eu> <CAKtBrB645am=hmuX2=w-NeDxY%2B7vEynZYpgVYxDZw5tsyrnZBA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 4/5/15 6:09 AM, Yue Chen wrote: > Thank you so much for the suggestions and kind help. Please do not take negative comments too seriously. Remember that an experiment with a negative result is as important as an experiment with a positive result. And if you do in fact succeed, in the face of much negative comment, it would be an important step. Even if you do not succeed, a paper carefully outlining all the issued you faced and your designed answers will make very interesting reading. > > On Thu, Apr 2, 2015 at 5:47 PM, Mateusz Guzik <mjguzik@gmail.com> wrote: > >> On Mon, Mar 30, 2015 at 10:59:16PM -0400, Yue Chen wrote: >>>> This would be a performance hit. >>> The cost is to traverse the .text area and some other data in memory. >> It's >>> linear and only happens after a given time interval, like several >> minutes, >>> and does not put overhead on the normal execution. So the performance >>> overhead won't be high. >>> >> This was a remark about a runtime cost of executing kernel code with >> with an extra jump in every function. >> >>>> The only somewhat workable solution I see would require leaving >> function >>> entry points at constant addresses and only insert jumps to new >> locations. >>> Yes, we do this. For the PIC direct jump/call/"instructions including >>> %RIP", we just need to update them. >>> >> I have trouble parsing what you wrote here. >> >> What I wrote here suggests taking existing foo_func, placing a copy at a >> new location and putting a jump to that address as the very first >> instruction of foo_func. (and zeroing/whatever the rest of old >> foo_func). Quite terrible. >> >> Also quite irrelevant to how you arrive at given func. >> >> Also see below. >> >>>> Also runtime relocation of everything definitely has a lot of unknown >> unknowns, >>> so all in all I would say this is a non-starter. >>> >>> FreeBSD kernel itself only has 1500+ indirect jumps. There are two types: >>> 1. jump table. Like switch-case. 2. Tail call. We have not found other >>> types like manual assembly code, or context restoration like that in >>> longjmp in libc. Maybe there are some other but we have not found. >>> >> Unknown unknowns have the problem of not being known. You have to be >> quite skilled in the area to come up with a working solution and I >> highly doubt either of us is such a person. >> >>>> Leave a lot of data at constant addresses, defeating the point to some >>> extent. >>>> One could consider a different approach where kernel data is randomly >> shuffled >>> around with some granularity and >>>> relevant symbol relocated prior to booting. >>> Why do you think the data address exposure could be a serious problem? It >>> is non-executable and the code is randomized. Although for data itself, >> it >>> is really hard to protect from tampering and may cause other problems. >>> >> I see I wrote 'data', where I should have written '.text' or 'code', >> although from the context it should have been clear this refers to >> function entry points left behind. >> >> if you can get away with one function call, you don't need any infoleaks >> - you know the address of the function. That's one of the problems with >> leaving func entry points at known addresses. >> >>> Why the randomized data leads to kernel panic? I think if the data goes >>> somewhere else, the instruction addressing would follow, or page fault >> may >>> happen. If randomizing code, the exploit attempt may hit an illegal >>> instruction and the kernel panics. >>> >> What? >> >> Anyway I still don't understand how are you trying to achieve >> randomisation in the kernel. >> >> Previous mail suggests you just want to move funcs around and update all >> callsites and structs which store addresses to them. >> >> As outlined with 'struct meh' example (which can be found below) I do >> not believe this can work on real-world systems without a major surgery >> simply because there is no way to tell what looking like a pointer >> should be changed and what should not. I'm happy to be proved wrong. >> >> Kernel already leaks tons of addresses to userspace in documented ways, >> and I'm sure there are tons of situations where it does so as a result >> of a bug. >> >> I believe an alternative implementation I proposed which randomizes >> *all* kernel pages prior to booting it would be doable, but given >> aforementioned leaks not worth it. >> >> (well, striclty speaking you would want to move functions around with >> respect to each other, but keep the kernel physically contiguous so >> that you can still use superpages) >> >> Your previous mail in this thread suggests you are just starting any >> kind of work on kernel-level, which makes this very hard task even >> harder. >> >> Exploitation mitigation is a complicated subject on its own, I recommend >> reading what people were doing it have to say. Here is a piece about >> KASLR: https://forums.grsecurity.net/viewtopic.php?f=7&t=3367 >> >> In conclusion I strongly recommend dropping this project for the time >> being and focusing on something easier. >> >>>> return-oriented >>>>>> programming (ROP). It is basically a stronger form of ASLR. >>>>>> After each randomization procedure, the function return addresses >>>> saved in >>>>>> the stack are the ``old'' addresses before randomization, so we >> need to >>>>>> update them to the new addresses. >>>>>> That's why we need to get all the stack ranges to find those >> addresses. >>>>>> Also, in kernel, we believe that not only the return addresses in >>>> stacks >>>>>> need to be updated, there may exist other ``old'' saved instruction >>>> (PC) >>>>>> addresses in memory. Like in exception handling (maybe, do not >> know), >>>>>> debugging-purpose code and restartable atomic sequences (RAS) >>>>>> implementation. That's why I asked how to traverse all the kernel >>>> pages and >>>>>> get their virtual addresses here: >>>>>> >> https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html >>>>>> Now we found that it seems needed to traverse the ``pv_entry'' >>>> structure for >>>>>> x86_64 MMU. >>>>>> >>>>>> Another problem is that we do not know if FreeBSD has any form of >>>> special >>>>>> encodings or offset form for 64-bit instruction addresses (e.g., >> saved >>>> %RIP) >>>>>> on X86_64, instead of hard-coded addresses. For example, using a >> 32-bit >>>>>> offset instead of the 64-bit full address; and doing what glibc >> does >>>> for the >>>>>> setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the >> saved >>>>>> register values). >>>>>> >>>>>> Any suggestion or correction are highly appreciated. >>>>>> >>>>>> Best, >>>>>> Yue >>>>> (Added HardenedBSD core and PaXTeam to CC.) >>>>> >>>>> Until you can not fixed all of the infoleaks from kernel (try sysctl >>>>> -a | grep 0x or similar command) the KASLR and other kernel address >>>>> space randomization techniques are easily bypass-able... >>>>> >>>> I do not believe this can be implemented reliably without some serious >>>> tinkering around the kernel. Surely I'm not a live patching expert (not >>>> any other kind of expert), but hear my out. >>>> >>>> It seems proposed approach is to move the kernel around and then >> updated >>>> all relevant pointers in all pages. >>>> >>>> I do not believe this can be done reliably without corrupting data, >>>> unless a major surgery is performed on the kernel. >>>> >>>> Consider: >>>> struct meh { >>>> func_t *m_func; >>>> size_t m_len; >>>> data *m_buf; >>>> }; >>>> >>>> Here we have 'm_func' which needs to be updated, but we don't know if >>>> that's the only pointer so we have to scan the entire struct. But m_buf >>>> can have data which looks like kernel pointers (i.e. matches some >> kernel >>>> funcs) and how will you know not to modify it? What if this is some >> sort >>>> of a hack and in fact you *should* modify it? >>>> >>>> CTF or some other solutions like that don't help if you just traverse >>>> all allocated buffers since you have no idea what the object you found >>>> really is. >>>> >>>> So, assuming this has to be done in runtime, the only somewhat workable >>>> solution I see would require leaving function entry points at contant >>>> addresses and only insert jumps to new locations. >>>> >>>> This would be a performance hit and would leave a lot of data at >>>> constant addresses, defeating the point to some extent. >>>> >>>> Also runtime relocation of everything definitely has a lot of unknown >>>> unknowns, so all in all I would say this is a non-starter. >>>> >>>> One could consider a different approach where kernel data is randomly >>>> shuffled around with some granularity and relevant symbol relocated >>>> prior to booting. >>>> >>>> This should provide unique enough layout, which paired with big >>>> likelyhood of a kernel panic on first bad exploit attempt may be an >>>> acceptable solution. >>>> >>>> But then the kernel may still have enough info leaks for this to not >>>> matter, so I would not be so eager to implement it. >>>> >>>> That said, what prompted this entire effort? Is there an operating >>>> system which got this to work or what? >>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik <mjguzik@gmail.com> >>>> wrote: >>>>>>> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: >>>>>>>> When using the following code on kernel module loading: >>>>>>>> >>>>>>>> >> ------------------------------------------------------------------------------------------ >>>>>>>> struct thread *td = kdb_thr_first(); >>>>>>>> td = kdb_thr_next(td); >>>>>>>> >>>>>>>> >> ------------------------------------------------------------------------------------------ >>>>>>>> The kernel panics. >>>>>>>> >>>>>>> Panics how? >>>>>>> >>>>>>> Also you can easily see these functions don't lock anything, so it >>>> would >>>>>>> be assumed you took appropriate locks. >>>>>>> >>>>>>> Except it seems there routines are supposed to be only used when >>>>>>> execution is 'frozen' (e.g. when escaped to the debugger). >>>>>>> >>>>>>>> And when printing all threads in proc0 (all kernel threads?): >>>>>>>> >>>>>>>> >> ------------------------------------------------------------------------------------------ >>>>>>>> struct proc *p = pfind(0); >>>>>>>> FOREACH_THREAD_IN_PROC(p, td) { >>>>>>>> uprintf("td: %x\n", td); >>>>>>>> } >>>>>>>> >>>>>>> proc0 is an exported symbol, no need to pfind. >>>>>>> >>>>>>>> td = curthread; >>>>>>>> uprintf("cur td: %x\n", td); >>>>>>>> >>>>>>>> >> ------------------------------------------------------------------------------------------ >>>>>>>> The ``curthread'' (from this kernel module running the above >> code) >>>> is >>>>>>>> not >>>>>>>> in the 0 process group. >>>>>>>> >>>>>>> There is no 'curthread from kernel module'. >>>>>>> >>>>>>> My guess is you do this work from module initializator, and in >> that >>>> case >>>>>>> curthread is the thread which loads the module, and such a thread >> is >>>>>>> definitely not linked into proc0. >>>>>>> >>>>>>> Still nobody knows what you are trying to do. >>>>>>> >>>>>>> -- >>>>>>> Mateusz Guzik <mjguzik gmail.com> >>>>>> >>>> -- >>>> Mateusz Guzik <mjguzik gmail.com> >>>> >> -- >> Mateusz Guzik <mjguzik gmail.com> >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55212983.20201>