From owner-freebsd-arch@freebsd.org Sun Nov 22 02:44:34 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6374A3598F for ; Sun, 22 Nov 2015 02:44:34 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-vk0-x231.google.com (mail-vk0-x231.google.com [IPv6:2607:f8b0:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 612431999 for ; Sun, 22 Nov 2015 02:44:34 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by vkfr145 with SMTP id r145so19407668vkf.0 for ; Sat, 21 Nov 2015 18:44:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=Xgcntk3AjxLgbT0bGRs3UAw06gU/I78V11PZwUyAwac=; b=tjdwMeDq2COXkZrrq5PVeIYzfWbKV9dWuXeCkZsKqMT0FDPNm7UDwH+XIs+o6ATkeM moHeL3JGhaTstHK0a3OPJRMXNT7BZ3GF9dx2sOlyof7pgc5JqJDWHBivVlCg0CLSBn6V Wf0Gyhfr3WiJPAINcUTFtJfjzgcF7TGSEPXx7LuGGoOBDpx1RVgMMmpYBCQ+h0ek1ol5 yX0T62DYezJ09eWeEbnEH95rP2D+T4O+DufiXBC+G3EgDFvStiKOfpkEWd1VA6FhVqaH ghsa+6mymfMRrCbqMSuGBV6CkUlugO+4wZkmtOBAemCPlLjmW3BCtSF/vsyCa5Px7+kT 7SdA== X-Received: by 10.31.47.88 with SMTP id v85mr9438362vkv.118.1448160273473; Sat, 21 Nov 2015 18:44:33 -0800 (PST) Received: from wkstn-mjohnston.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by smtp.gmail.com with ESMTPSA id v145sm5514650vkv.6.2015.11.21.18.44.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 21 Nov 2015 18:44:33 -0800 (PST) Sender: Mark Johnston Date: Sat, 21 Nov 2015 18:45:42 -0800 From: Mark Johnston To: freebsd-arch@FreeBSD.org Subject: zero-cost SDT probes Message-ID: <20151122024542.GA44664@wkstn-mjohnston.west.isilon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2015 02:44:34 -0000 Hi, For the past while I've been experimenting with various ways to implement "zero-cost" SDT DTrace probes. Basically, at the moment an SDT probe site expands to this: if (func_ptr != NULL) func_ptr(); When the probe is enabled, func_ptr is set to dtrace_probe(); otherwise it's NULL. With zero-cost probes, the SDT_PROBE macros expand to func(); When the kernel is running, each probe site has been overwritten with NOPs. When a probe is enabled, one of the NOPs is overwritten with a breakpoint, and the handler uses the PC to figure out which probe fired. This approach has the benefit of incurring less overhead when the probe is not enabled; it's more complicated to implement though, which is why this hasn't already been done. I have a working implementation of this for amd64 and i386[1]. Before adding support for the other arches, I'd like to get some idea as to whether the approach described below is sound and acceptable. The main difficulty is in figuring out where the probe sites actually are once the kernel is running. In my patch, a probe site is a call to an externally-defined function which is defined in an automatically-generated C file. At link time, we first perform a partial link of all the kernel's object files. Then, a script uses the relocations against the still-undefined probe functions to generate 1) stub functions for the probes, so that the kernel can actually be linked, and 2) a linker set containing the offsets of each probe site relative to the beginning of the text section. The result is linked with the partially-linked kernel to generate the final kernel file. During boot, we iterate over the linker set, using the offsets plus the address of btext to overwrite probe sites with NOPs. SDT probes in kernel modules are handled differently (and more simply): the kernel linker just has special handling for relocations against symbols named __dtrace_sdt_*; this is how illumos/Solaris implements all of this. My uncertainty revolves around the use of relocations in the partially-linked kernel to determine the address of probe sites in the running kernel. With the GNU ld in base, this happens to work because the final link doesn't modify the text section. Is this something I can rely upon? Will this assumption be false with the advent of lld and LTO? Are there other, cleaner ways to implement what I described above? Thanks, -Mark [1] https://people.freebsd.org/~markj/patches/sdt-zerocost/