From owner-freebsd-dtrace@FreeBSD.ORG  Wed Aug 21 06:45:02 2013
Return-Path: <owner-freebsd-dtrace@FreeBSD.ORG>
Delivered-To: freebsd-dtrace@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTP id 9853F589
 for <freebsd-dtrace@freebsd.org>; Wed, 21 Aug 2013 06:45:02 +0000 (UTC)
 (envelope-from brendan.gregg@joyent.com)
Received: from mail-pa0-f53.google.com (mail-pa0-f53.google.com
 [209.85.220.53])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 720392B23
 for <freebsd-dtrace@freebsd.org>; Wed, 21 Aug 2013 06:45:02 +0000 (UTC)
Received: by mail-pa0-f53.google.com with SMTP id lb1so473180pab.26
 for <freebsd-dtrace@freebsd.org>; Tue, 20 Aug 2013 23:44:56 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=google.com; s=20120113;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc:content-type;
 bh=TJ2/FyNp0IQDh+qOE7SYETPkg2/rAGZ0z69g7yZy4ts=;
 b=pR52z0a03bmZvDVD57Rrgbax36P6Gu/tPFkS/Mi+BmHsZZKx2X8PKCeu3v3mjJp/Um
 lP+Xze8AmIyFROYUaUZgc2WJn0SejWncCNNrBQb35frpQlDAH1DnmNshZA+YvNcHBg2z
 SRjsp6/GQYnF+GCr/t0hvbW9xTlIlwx4ZYr1J+IXFBoDufqodEJp5BO5mWOJ/V00PEDU
 XOMG8CeKQNHfB0JKctY6La2QpEuTRWLX4Fsh2RSLRhyz1xMbFTQ1prLAWrV4hIX9tTkZ
 5l3HhaAP8UKQkqpnPqE4QVA9S95O18gHesSQwGWEw4/tQTkTO1+uIaCGNJ0IptkLHaLs
 fxvg==
X-Gm-Message-State: ALoCoQkh9XSsVLlXvPRF0HoStWhWou7xSIPjR19RL3dJ28TEmgKi+2cgrB8mRNafgMW/A3Vrk2KD
MIME-Version: 1.0
X-Received: by 10.67.24.7 with SMTP id ie7mr7810330pad.112.1377067496223; Tue,
 20 Aug 2013 23:44:56 -0700 (PDT)
Received: by 10.70.55.34 with HTTP; Tue, 20 Aug 2013 23:44:56 -0700 (PDT)
In-Reply-To: <20130821045926.GA17196@raichu>
References: <20130821045926.GA17196@raichu>
Date: Tue, 20 Aug 2013 23:44:56 -0700
Message-ID: <CA+XzFFhw9beGi8=7heCY07dExvno=oMBYgzUrpht2LYXSbutqg@mail.gmail.com>
Subject: Re: DTrace network providers
From: Brendan Gregg <brendan.gregg@joyent.com>
To: Mark Johnston <markj@freebsd.org>
Content-Type: text/plain; charset=UTF-8
X-Content-Filtered-By: Mailman/MimeDel 2.1.14
Cc: freebsd-net@freebsd.org, freebsd-dtrace@freebsd.org
X-BeenThere: freebsd-dtrace@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "A discussion list for developers working on DTrace in FreeBSD."
 <freebsd-dtrace.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-dtrace>, 
 <mailto:freebsd-dtrace-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-dtrace>
List-Post: <mailto:freebsd-dtrace@freebsd.org>
List-Help: <mailto:freebsd-dtrace-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-dtrace>,
 <mailto:freebsd-dtrace-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Aug 2013 06:45:02 -0000

G'Day,

On Tue, Aug 20, 2013 at 10:00 PM, Mark Johnston <markj@freebsd.org> wrote:

> Hello!
>
> I've ported the ip, tcp and udp DTrace providers to FreeBSD, following
> the Solaris documentation here:
>
> https://wikis.oracle.com/display/DTrace/ip+Provider
> https://wikis.oracle.com/display/DTrace/tcp+Provider
> https://wikis.oracle.com/display/DTrace/udp+Provider
>
>
Fantastic!


> My implementation of these providers makes use of dynamic translators,
> for which FreeBSD support was added in r254468; this patch won't compile
> with earlier revisions. The use of dynamic translators means that
> existing DTrace scripts which use these providers will just work when run
> on FreeBSD - no modifications needed. In particular, all of the examples
> in the links above will work properly on FreeBSD with my diff.
>
> I've collected a bunch of example scripts for these providers and placed
> them here:
>
> http://people.freebsd.org/~markj/dtrace/network-providers/
>
>
They look familiar. :-) Which is a good sign for this port, as that's
what's supposed to happen: the same scripts run on any OS.

To run one you just need to execute "dtrace -s <script>".
>

Or chmod 755 the ones with interpreter lines.


> In general these providers make it trivial to monitor or count packets
> and bytes on a per-host/port/protocol/interface basis. One can also do
> neat things like watch TCP connection state transitions in real time
> (tcpstate.d) and measure connection latency. All of the probes
> correspond to logical events in their respective protocol
> implementations; all of the providers have send and receive probes, and
> the tcp provider has a few more.
>
> I didn't have to make any major changes to add support for these
> providers, but I've made a few small tweaks:
>
> 1. Add a little helper function for TCP state changes instead of
>    sprinkling a bunch of SDT_PROBE calls throughout the code.
>

Makes sense.


> 2. Set the IPv6 payload size in TCP and UDP before calling ip6_output().
>    This is done for the send probes so that the ipinfo_t argument
>    can be used to get the payload size. It's not quite correct since it
>    doesn't include the length of IPv6 option headers (they aren't known
>    yet), but I think that's ok for the purposes of these providers.
>

Excluding the IP header length is correct for ipinfo_t->ip_plength, which
is just supposed to be the payload length.

I see the patch makes an assumption about IPv4 header size being 20 bytes,
instead of picking it from the header (IHL), like:

        ip_plength = ntohs(I->ipha_length) -
            ((I->ipha_version_and_hdr_length & 0xf) << 2);

3. Set the IPv4 header version in udp_output(). This is already done for
>    IPv6.
>
> I'm hoping that none of this (or the rest of the diff) is controversial,
> but I wanted to make sure, so any review would be really appreciated.
>
> The patch is here:
>
>
> http://people.freebsd.org/~markj/patches/network-providers/network-providers-1.diff
>
> It depends on r254468. To use it, just recompile the kernel (assuming
> that KDTRACE_HOOKS is enabled) and copy
> $SRC/cddl/lib/libdtrace/{ip,tcp,udp}.d (added by the patch) to
> /usr/lib/dtrace. Then the example scripts I linked above should just
> work.
>

Great work Mark!

Here's some (from many) one-liners that this should make possible (more in
the DTrace book):

Who is connecting to what:
dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr,
args[3]->tcps_lport] = count(); }'

Who isn't connecting to what:
dtrace -n 'tcp:::accept-refused { @[args[2]->ip_daddr, args[4]->tcp_sport]
= count(); }'

What am I connecting to?
dtrace -n 'tcp:::connect-established { @[args[3]->tcps_raddr ,
args[3]->tcps_rport] = count(); }'

IP payload bytes for TCP send, size distribution by destination address:
dtrace -n 'tcp:::send { @[args[2]->ip_daddr] =
quantize(args[2]->ip_plength); }'

Many of these are very low overhead ways to understand network behavior -
lower than sniffing every packet. Plus you can pull out kernel state that
is never seen on the wire.

Brendan

-- 
Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan