From owner-freebsd-dtrace@FreeBSD.ORG Mon Feb 24 04:14:58 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7EE323D4 for ; Mon, 24 Feb 2014 04:14:58 +0000 (UTC) Received: from mail-ie0-x232.google.com (mail-ie0-x232.google.com [IPv6:2607:f8b0:4001:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4B64E1C75 for ; Mon, 24 Feb 2014 04:14:58 +0000 (UTC) Received: by mail-ie0-f178.google.com with SMTP id ar20so3118900iec.9 for ; Sun, 23 Feb 2014 20:14:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=oBJ9GslXJjBhDUkGWhA1IVj/eNb0adPnRKwWkuoEfyM=; b=zkEZtwBuyaKe9fNMdGCaeiIXMlj2UkeRNjJuRMAFQW1fEgAlH2NWVZfuuvl2vantam B05PICoF+pQMB2ebrcZdMB0wq6qgfxJ6Kc5EUYuC7L2EPRMi4kyLRJpfcV0RTyD9+Sul Hitt8JZHiBR1AwlK/Gk+9y1vixLZsW3LPYmrnImWOWO4NEc5EXfGqmKr7nmGpdLt+Ple kbS9s9EN6G0E0+nDI0YnHKfislEP2Hbe0twxi6RjvhBLJ+F+mByBb4rgOYoZATbl/tfH jegHTx6SO8qPjKTSF5d93m3tUt7OXmRtFxQXV1zP6jYSJXOVohSOZdrChlZiFc6ak5i1 hq2Q== X-Received: by 10.43.156.18 with SMTP id lk18mr12665050icc.77.1393215297307; Sun, 23 Feb 2014 20:14:57 -0800 (PST) Received: from raichu (198-84-185-216.cpe.teksavvy.com. [198.84.185.216]) by mx.google.com with ESMTPSA id iq9sm22621488igb.7.2014.02.23.20.14.56 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 23 Feb 2014 20:14:56 -0800 (PST) Sender: Mark Johnston Date: Sun, 23 Feb 2014 23:14:54 -0500 From: Mark Johnston To: freebsd-dtrace@freebsd.org Subject: [patch] fasttrap process scratch space Message-ID: <20140224041454.GB2720@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Feb 2014 04:14:58 -0000 Hello, For those not familiar with MD parts of fasttrap, one of the things it has to do is ensure that any userland instruction that it replaces with a breakpoint gets executed in the traced process' context. For several common classes of instructions, fasttrap will emulate the instruction in the breakpoint handler; when it can't do that, it copies the instruction out to some scratch space in the process' address space and sets the PC of the interrupted thread to the address of that instruction, which is followed by a jump to the instruction following the breakpoint. There's a helpful block comment titled "Generic Instruction Tracing" around line 1585 of the x86 fasttrap_isa.c which describes the details of this. This functionality currently doesn't work on FreeBSD, mainly because we don't necessarily have any (per-thread) scratch space available for use in the process' address space. In illumos/Solaris, a small (< 64 byte) block is reserved in each thread's TLS for use by DTrace. It turns out that doing the same thing on FreeBSD is quite easy: http://people.freebsd.org/~markj/patches/fasttrap_scratch_hacky.diff Specifically, we need to ensure that TLS (allocated by the runtime linker) is executable and that we properly extract the offset to the scratch space from the FS segment register. I think this is somewhat hacky though, as it creates a dependency on libthr and rtld internals. A second approach is to have fasttrap dynamically allocate scratch space within the process' address space using vm_map_insert(9). My understanding is that Apple's DTrace implementation does this, and I've implemented this approach for FreeBSD here (which was done without referencing Apple code): http://people.freebsd.org/~markj/patches/fasttrap-scratch-space/fasttrap-scratch-space-1.diff The idea is to map pages of executable memory into the user process as needed, and carve them into scratch space chunks for use by individual threads. If a thread in fasttrap_pid_probe() needs scratch space, it calls a new function, fasttrap_scraddr(). If the thread already has scratch space allocated to it, it's used. Otherwise, if any free scratch space chunks are available in an already-mapped page, one of them is allocated to the thread and used. Otherwise, a new page is mapped using vm_map_insert(9). Threads hold onto their scratch space until they exit. That is, scratch space is never unmapped from the process, even if the controlling dtrace(1) process detaches. I added a handler for thread_dtor event which re-adds any scratch space held by the thread to the free list for that process. Per-process scratch space state is held in the fasttrap process handle (fasttrap_proc_t), since that turns out to be much easier than keeping it in the struct proc. Does anyone have any thoughts or comments on the approach or the patch? Any review or testing would be very much appreciated. For testing purposes, it's helpful to know that tracing memcpy() on amd64 will result in use of this scratch space code, as it starts with a "mov %rdi,%rax" on my machine at least. My main test case has been to run something like # dtrace -n 'pid$target:libc.so.7::entry {@[probefunc] = count()}' -p $(pgrep firefox) Attempting to trace all functions still results in firefox dying with SIGTRAP, but we're getting there. :) Thanks, -- -Mark