From owner-freebsd-performance@freebsd.org Fri Jan 13 21:21:49 2017 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 185DFCAE6B4 for ; Fri, 13 Jan 2017 21:21:49 +0000 (UTC) (envelope-from Alexander.Alexeev@dell.com) Received: from esa1.dell-outbound.iphmx.com (esa1.dell-outbound.iphmx.com [68.232.153.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.dell-outbound.iphmx.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E194215B0 for ; Fri, 13 Jan 2017 21:21:48 +0000 (UTC) (envelope-from Alexander.Alexeev@dell.com) DomainKey-Signature: s=smtpout; d=dell.com; c=simple; q=dns; h=Received:From:Received:Received:X-DKIM:DKIM-Signature: X-DKIM:Received:Received:Received:To:Subject:Thread-Topic: Thread-Index:Date:Message-ID:Accept-Language: Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator: x-originating-ip:Content-Type:MIME-Version: X-Sentrion-Hostname:X-RSA-Classifications; b=eT1LJVppM9DsXnIO4Hchq7N1PWE8wKY+nj/KPJ/3h44jeMBSZfTIq+hm Q6a+yMDoj8DbEqLKNxkCSJoCskZbw2p7BvKkNVS3JLMacdKaEpkAAFtjk p9xhTsN7cW01QDYYWwcwcUuYkWDa0c33MWl6PwyFfNhKMMCOH9i8P04il w=; DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dell.com; i=@dell.com; q=dns/txt; s=smtpout; t=1484341937; x=1515877937; h=from:to:subject:date:message-id:mime-version; bh=s1qmxT/1NiLKS5Denix2ZrUqA/Yibp3APYSCk9svAmU=; b=GQPTegNvjLk0ToiB9pZDQHzZjGTAhyEov+H4CIUUpX6EyoKssQoERy9o gnaQnG1sjUrzjeBAyHpl0RgllaeuKlQ8SKcqniG4v/gRLPlsvkp8XWLFa 13iRR0r+ROPxco4qUIRKJ6mZKENer8G4nGdgyBeRB2FfNbQFyif6H8tdk Q=; Received: from esa1.dell-outbound2.iphmx.com ([68.232.153.201]) by esa1.dell-outbound.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Jan 2017 15:12:12 -0600 From: "Alexeev, Alexander" Received: from mailuogwhop.emc.com ([168.159.213.141]) by esa1.dell-outbound2.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Jan 2017 03:20:11 +0600 Received: from maildlpprd01.lss.emc.com (maildlpprd01.lss.emc.com [10.253.24.33]) by mailuogwprd05.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id v0DLLbxI008443 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 13 Jan 2017 16:21:38 -0500 X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd05.lss.emc.com v0DLLbxI008443 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=rsa.com; s=jan2013; t=1484342498; bh=VYBlD5LIhEZQvdpbPcOY33qainE=; h=From:To:Subject:Date:Message-ID:Content-Type:MIME-Version; b=vgFZmHBJEXyXdSK4DDfDuEyI8IpccGWU/BtB8bcbZKdYe32IJ5XW2GjlyPi4mZCcv iqrB2bcWasofMB50oEOX3cigilZOhUFqNuluA8MapaHYWrIgqTauYxNVYc5x2CdwZG eY4vBy40DwPklxsruNlnIEipmx1/l+09PHXHmzm0= X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd05.lss.emc.com v0DLLbxI008443 Received: from mailusrhubprd01.lss.emc.com (mailusrhubprd01.lss.emc.com [10.253.24.19]) by maildlpprd01.lss.emc.com (RSA Interceptor) for ; Fri, 13 Jan 2017 16:20:50 -0500 Received: from MXHUB303.corp.emc.com (MXHUB303.corp.emc.com [10.146.3.29]) by mailusrhubprd01.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id v0DLLMcw020145 (version=TLSv1.2 cipher=AES128-SHA256 bits=128 verify=FAIL) for ; Fri, 13 Jan 2017 16:21:23 -0500 Received: from MX303CL02.corp.emc.com ([fe80::fcb7:fb6c:7ef7:f98f]) by MXHUB303.corp.emc.com ([10.146.3.29]) with mapi id 14.03.0266.001; Fri, 13 Jan 2017 16:21:22 -0500 To: "freebsd-performance@freebsd.org" Subject: PMC profiling with (kernel+user) call stacks Thread-Topic: PMC profiling with (kernel+user) call stacks Thread-Index: AdJt4vEM5Bm2IQToRpmPerQFXcCn4Q== Date: Fri, 13 Jan 2017 21:21:21 +0000 Message-ID: <210C625981C6714DB138F1B28AC66DCD21A4416D@MX303CL02.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.7.181.204] MIME-Version: 1.0 X-Sentrion-Hostname: mailusrhubprd01.lss.emc.com X-RSA-Classifications: public X-Mailman-Approved-At: Fri, 13 Jan 2017 21:32:11 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 21:21:49 -0000 Hello I am trying to adopt PMC for performance works and add some kind of post pr= ocessing to customized pmcstat(8) output. In order to customize output I use internal plugin interface, and it isn't = problem. What I found is that interface, or any other type of output from pmctsat pr= ofiling samples doesn't have kernel and user spaces stacks combined, only o= ne or another. For instance pmcstat -R profile -k /boot/kernel.amd64 > dump_profile.txt Stack traces (just short example) either kernel or user space callchain 0 0x2000616 2 6 s ... 0xffffffff80b49baf ... 0xffffffff80b4b2c2 ... 0xffffffff80b553b3 ... 0xffffffff8071bb2a ... 0xffffffff8071cb58 ... 0xffffffff8066d055 callchain 10059 0x3000616 3 6 u ... 0x801b39304 ... 0x811e46b09 ... 0x811f0c821 ... 0x811e99551 ... 0x801b4e9d5 ... 0x801b55510 callchain 10059 0x2000616 2 5 u ... 0x801b593d1 ... 0x801b4cb4a ... 0x801b53cd1 ... 0x802771cb5 ... 0x7fffda7d3000 callchain 10059 0x3000616 3 23 u ... 0x802cf9fa1 ... 0x801dc0018 ... 0x80fc6392b ... 0x80fc5eabd ... 0x80fc5e44d ... 0x80fcc3f5a ... 0x80fc5e327 ... 0x80fc5d578 ... 0x80fc5ce8c ... 0x80fcc3f5a If kernel execution initiated by user space code than trace started from [k= ernel:amd64_syscall+0x...] So the questions are - Is it possible to have callstacks combined in pmcstat out of the= box, so it would be seen what user space code cause kernel execution? - If not possible out of the box, what kind of post processing or = toolchain modification is required? I am running pmcstat of FreeBSD 10 Thanks, Alexander